You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
this can overload the CPU since every rank needs to run export on the CPU
May cause significant slow downs for long export time
To spin this a little further:
Everything until compile/cudagraph is very CPU heavy and broadly speaking the model is on the meta device and every rank performs the same set of computation. We may want to consider moving as much as we can into the pre-processing stage...
The text was updated successfully, but these errors were encountered:
Currently, we run export on every rank
To spin this a little further:
Everything until compile/cudagraph is very CPU heavy and broadly speaking the model is on the meta device and every rank performs the same set of computation. We may want to consider moving as much as we can into the pre-processing stage...
The text was updated successfully, but these errors were encountered: