Move ocean vector reconstruction to analysis member

[mark-petersen]

Currently, mpas_reconstruct is called each timestep, for numerous variables, and always interpolates to (x,y,z) and (zonal,meridional). This is wasteful for both compute time and memory usage. We can put these in two new analysis members:

• vector_reconstruction_lat_lon
• vector_reconstruction_xyz

and call them every 6 hours or daily, and only use lat_lon for global cases.

[mark-petersen]

The steps involved are:

1. Create a new branch from ocean/develop.
2. Add two new analysis members (AMs) using names above, starting from core_ocean/analysis_members/mpas_ocn_TEMPLATE.F and follow detailed instructions in that file.
3. All occurrences of call mpas_reconstruct should be removed from these directories: src/core_ocean/shared and src/core_ocean/mode_forward - see below.
4. The framework operator mpas_reconstruct is hard-wired to interpolate to both xyz and lat/lon. Take those two apart, and put the xyz portion in the AM vector_reconstruction_xyz and the lat/lon part in vector_reconstruction_lat_lon. All the output variables, like transportVelocityX etc, should now be in the Registry for each of the two new analysis members.
5. GM variables should only reconstruct if config_use_standardGM=.true.
6. Push your branch to your fork and make a PR.
7. Test as you go with the nightly regression suite. See instructions here. If AMs are called every time step, all fields, including vector reconstruction, should be bit-for-bit.
8. For global case (EC60to30) turn the two AMs off, and on but every 6 hours, and record speed-up and change in memory usage.
9. In E3SM, we will set the AM vector_reconstruction_xyz always off, and AM vector_reconstruction_lat_lon always on, but with a 6-hour frequency.

> cd src/core_ocean
> git grep -in  mpas_reconstruct
mode_forward/mpas_ocn_time_integration_rk4.F:745:         call mpas_reconstruct(meshPool,  normalVelocityNew, &
mode_forward/mpas_ocn_time_integration_rk4.F:750:         call mpas_reconstruct(meshPool, gradSSH,          &
mode_forward/mpas_ocn_time_integration_split.F:1930:         call mpas_reconstruct(meshPool, normalVelocityNew,  &
mode_forward/mpas_ocn_time_integration_split.F:1935:         call mpas_reconstruct(meshPool, gradSSH,          &
shared/mpas_ocn_diagnostics.F:1901:      call mpas_reconstruct(meshPool, normalTransportVelocity,          &
shared/mpas_ocn_diagnostics.F:1909:      call mpas_reconstruct(meshPool, normalGMBolusVelocity,          &
shared/mpas_ocn_diagnostics.F:1917:      call mpas_reconstruct(meshPool, relativeSlopeTopOfEdge,          &
shared/mpas_ocn_diagnostics.F:1925:      call mpas_reconstruct(meshPool, gmStreamFuncTopOfEdge,          &
shared/mpas_ocn_init_routines.F:694:      call mpas_reconstruct(meshPool, normalVelocity,        &

[xylar]

Wow, @mark-petersen, you're on a tear!

This seems like a good project, indeed.

[hrmoncada]

@mark-petersen, I will let you know if i have questions

[hrmoncada]

@mark-petersen ,I don't get the same using git grep -in mpas_reconstruct, I got more. Could you please let me know which version of the MPAS-Model/ocean are you using.

gr-fe1:Move_ocean_vector_reconstruction_412$ cd src/
gr-fe1:src$ cd core_ocean/
gr-fe1:core_ocean$ git grep -in mpas_reconstruct
analysis_members/mpas_ocn_eliassen_palm.F:990:         call mpas_reconstruct(meshPool, montgPotNormalGradOnEdge, &
analysis_members/mpas_ocn_eliassen_palm.F:1151:         call mpas_reconstruct(meshPool, ErtelPVNormalGradOnEdge, &
analysis_members/mpas_ocn_eliassen_palm.F:2643:      call mpas_reconstruct(meshPool, velNormalGradOnEdge, &
analysis_members/mpas_ocn_eliassen_palm.F:2651:      call mpas_reconstruct(meshPool, velNormalGradOnEdge, &
analysis_members/mpas_ocn_lagrangian_particle_tracking_interpolations.F:135:    call mpas_reconstruct(meshPool, u, uc
analysis_members/mpas_ocn_time_filters.F:362:           call mpas_reconstruct(meshPool, normalVelocityLowPass,  &
analysis_members/mpas_ocn_time_filters.F:367:           call mpas_reconstruct(meshPool, normalVelocityHighPass,  &
mode_forward/mpas_ocn_time_integration_rk4.F:745:         call mpas_reconstruct(meshPool,  normalVelocityNew, &
mode_forward/mpas_ocn_time_integration_rk4.F:750:         call mpas_reconstruct(meshPool, gradSSH,          &
mode_forward/mpas_ocn_time_integration_split.F:1930:         call mpas_reconstruct(meshPool, normalVelocityNew,  &
mode_forward/mpas_ocn_time_integration_split.F:1935:         call mpas_reconstruct(meshPool, gradSSH,          &
shared/mpas_ocn_diagnostics.F:1901:      call mpas_reconstruct(meshPool, normalTransportVelocity,          &
shared/mpas_ocn_diagnostics.F:1909:      call mpas_reconstruct(meshPool, normalGMBolusVelocity,          &
shared/mpas_ocn_diagnostics.F:1917:      call mpas_reconstruct(meshPool, relativeSlopeTopOfEdge,          &
shared/mpas_ocn_diagnostics.F:1925:      call mpas_reconstruct(meshPool, gmStreamFuncTopOfEdge,          &
shared/mpas_ocn_init_routines.F:694:      call mpas_reconstruct(meshPool, normalVelocity,        &

[mark-petersen]

@hrmoncada you are right, that is the full output on ocean/develop. A did not copy the whole thing before. You are on the right branch.

You can ignore everything in the directory analysis_members, because those are called infrequently. We are interested in removing those called every timestep, which are in mode_forward and shared.