Update torchft.md #1596
Update torchft.md #1596
Conversation
H-Huang
requested review from
tianyu-l,
fegin,
wwwjn and
wconstab
as code owners
meta-cla
bot
added
the
CLA Signed
There was a problem hiding this comment.
(adding a small comment if you don't mind as I have been using diloco with torchtitan for a bit)
could be usefull to add a clarification that the global_batch_size is split across all the diloco island. It's clear for HSDP but for diloco there is a distinction between global batch size (all diloco island) and the inner global batch size (inside each diloco island doing normal sharding)
H-Huang
force-pushed
the
torchft_readme
branch
from
c87fd44 to
d65112f
Compare
|
@samsja thanks :), added |
There was a problem hiding this comment.
what's the relationship with https://github.com/pytorch/torchtitan/blob/main/docs/torchft.md
Could we consolidate to just keep one of them?
|
@tianyu-l, oops I didn't realize. Vishal had asked if there were any docs and i didn't realize there was an existing one, so i just added something new. Let me see if there is anything to consolidate. If this is redundant, then ill just close this PR |
H-Huang
force-pushed
the
torchft_readme
branch
from
26113b0 to
d86e252
Compare
H-Huang
force-pushed
the
torchft_readme
branch
from
d86e252 to
014a6d3
Compare
Add some basic documentation on how to use Titan with TorchFt for DiLoCo. LMK if anything needs clarification @vishal9-team
Add some basic documentation on how to use Titan with TorchFt for DiLoCo.
LMK if anything needs clarification @vishal9-team