WeChat AI Team Details WeLM Models Scaling to 617B Parameters

The models, WeLM-HD4-80B and WeLM-HD4-617B, use Hidden Decoding, which expands each token into multiple internal computation streams without growing the Transformer backbone. The 617B model activates 23B parameters (3B for the 80B) and beat autoregressive baselines across nine benchmarks, at 4.4x training cost (5.1x for 80B).