Open
Description
Possible scenarios:
- Compare plain PiPPy with plain Megatron
- May be hard to implement
Compare HF MegatronBERT with PiPPy-wrapped HF BERTChoose a task (ForMaskedLM, ForCausalLM, ForNextSentencePrediction, ForPreTraining, ForSequenceClassification, ForMultipleChoice, ForTokenClassification, ForQuestionAnswering)May depend on Mirror HF examples with PiPPy #301Investigate if it's a fair comparison