NVIDIAAnalysisDevelopersNovember 6, 2026

NVIDIA details Reshard collective to speed RL weight refit

Read original source →developer.nvidia.com

NCCL M2N Reshard, an open-source reference implementation in the NVIDIA/nccl-extensions GitHub repo, replaces gather-and-broadcast refit, which must move a payload on the order of the model's total weight size each cycle. Refit is complicated by training and inference using different parallel mappings across data, context, tensor, expert, and pipeline parallelism.

1 source

More stories today

Open the live feed