Clustering - fix bug, document and write tests

[joannacknight]

There may still be an issue in the pre-processing code with the sub-clustering. The EmbeddingGemma search config said total_clusters: 2047, but there were only 2041 in the output file.

[joannacknight]

The basic clustering works fine so long as we don't hit the sub-clustering routine. I had made a mistake with the max_size of a cluster in the EmbeddingGemma config and it was set to 4,000 instead of 50,000. Changing the config and re-running and the output looks fine - it doesn't do any sub-clustering when max_size = 50000.

The sub-clustering works, so long as the resulting clusters are less than max_size. The issue seems to be in the recursion, when we have to sub-cluster a sub-cluster.

I am debugging and adding docstrings and tests for the clustering.py module.

Have re-named the ticket.