[grpo] Optimize vLLM weight synchronization

[hjh0119]

Optimize weight synchronization between the training model and the inference engine (vLLM):

LoRA

1. Synchronize/load only the trained adapter weights.(both colocate / server mode)
2. In server mode, transmit flattened adapter weights to reduce communication overhead of model parameters.

FULL

1. Remove the original per-tensor synchronization logic and adopt a bucketing strategy to reduce redundant communication requests and overhead, especially for MoE models (which have more tensors than dense models).

[hjh0119]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces an optimization for LoRA training with vLLM by enabling in-memory weight synchronization using flattened tensors. This avoids disk I/O and should improve training speed. The changes involve adding new arguments, new protocol definitions, and new methods in the rollout engine and GRPO trainer. A key part of the implementation is monkey-patching vLLM to support loading LoRA adapters from tensors. The overall approach is sound, but there are a few areas that need attention, such as ensuring deterministic adapter selection, cleaning up commented-out code, and addressing TODO comments.

[hjh0119]

Qwen2.5-VL-7B-Instruct, server mode, tp=2, dp=2 → 10× speed-up

[hjh0119]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant optimization for LoRA training with vLLM by synchronizing only the adapter weights instead of the full model. This is achieved by patching vLLM to load adapters from in-memory tensors and flattening the weights for more efficient communication. The changes are well-implemented across the trainer, rollout logic, and communication protocols. The documentation has also been updated to reflect these new features. My review includes a few suggestions to improve code quality and fix a minor typo in the documentation.