Account for vertical layers in domain decomposition

[sbrus89]

This is just something I was thinking about and was wondering if anyone in the group had thoughts...

It seems to me that there will be a load imbalance caused by our domain decomposition not accounting for the fact that open-ocean cells with more vertical layers have more work associated with them than cells in shallow regions with fewer vertical layers. This issue will probably get worse as we begin putting more resolution in shallow coastal regions. It could be mitigated by using weights in the domain decomposition such that shallow subdomains contain a larger number of elements relative to open ocean subdomains to balance the load.

@mark-petersen, @xylar, @philipwjones and others: Feel free to close this if any one of you doesn't think it makes sense to look into in the future.

[xylar]

@sbrus89, I have a vague memory that this was discussed earlier in the project and possibly also tried. I'm not sure what the results suggested and why we didn't go that route. I'm keen to see what @mark-petersen and @philipwjones remember of that.

[philipwjones]

@sbrus89 and @xylar My recollection (and it could be faulty) was that we did try this, since it should be an obvious win, and we could see gains in some configurations, but the gains were limited by several other things. First, at the limits of strong scaling where the production runs typically occur, the Metis partitioning starts to lead to irregular domains with worse connectivity so the load-balanced gains were offset by increased communication. Suspect this could be fixed with different partition algorithms but we didn't pursue. Second, at the time, the framework was highly inefficient (and still is less performant than it should be) so we need to fix some of the framework issues first. Third, I think the vertical levels were not always a good predictor of load - there are some aspects (like the barotropic mode) that are independent so tradeoffs need to be made. Finally, I think there was some idea that we'd be moving to a true ALE scheme and better use of the vertical coordinate in which we'd alway have a lot of active layers and we would allocate them differently rather than varying the number of levels. Maybe that will still happen. I think a weighted partitioning is still worth pursuing as we revise the framework. And I think it can be a runtime option once I can integrate my online partitioning. We could even have a capability that could run a time step as part of initialization and compute the real cost per column in production and use that for the weighting. But it hasn't been a real high priority until we fix some other stuff.

[mark-petersen]

@sbrus89 we had a summer student try this two years ago, Divya Jaganathan. Here is her poster:
https://drive.google.com/file/d/1nTcbrnE9sBGE9I5oKnFGRnyyiZm44jQB/view?usp=sharing
The take-home is that it made very little difference:

Here X=0 (red) is equal MPAS partitions of 2D cells, as is currently done, X=1 (black) is equal MPI partitions of 3D cells as you suggest, and X=0.5 (blue) is in between. You can get a little speed-up at X=0.5, but only a few percent in these tests. It's because of the cost of the barotropic solver, and also that you need to bring part of the vertical column into memory anyway.

[philipwjones]

I can't remember - did Divya add a barrier before the barotropic solver timer? Often the wait time from load imbalance shows up in the barotropic timer. And the barotropic is where you'd see a negative impact from more irregular subdomains. As I said above, it could be a nice feature to play with. But also agree that it's probably not the highest priority at the moment and not a clear win in any case.