Elastic resets

[t0278611]

Using model ema (at least for small-ish models) seems to drastically improve the validation results. I wonder why the ema weights are never used in training (a bit like the lookahead optimizer). It should be fairly forward to implement "elastic resets", where the online weights periodically get overwritten by the ema weights. I did not find the feature in the train args, if it exist please do point it out