AMD Releases 16B Instella-MoE Model With 2.8B Active Parameters as Open Benchmark Tops 76.7
Updated
Updated · MarkTechPost · Aug 1
AMD Releases 16B Instella-MoE Model With 2.8B Active Parameters as Open Benchmark Tops 76.7
3 articles · Updated · MarkTechPost · Aug 1
Summary
AMD unveiled Instella-MoE-16B-A3B, a decoder-only open language model trained from scratch on Instinct MI300X and MI325X GPUs, with 16 billion total parameters but only 2.8 billion activated per token.
AMD is publishing weights from every training stage, data mixtures, configs and SGLang inference code, though the model weights carry a research-only ResearchRAIL license while the training codebase is MIT licensed.
Two architecture choices underpin the release: Gated Multi-head Latent Attention and FarSkip-Collective, which AMD said delivered a 12.7% pre-training speedup and cut time to first token by as much as 39.2% in expert-parallel serving.
On performance, the base checkpoint scored a 76.7 average—ahead of fully open peers such as Moonlight-16B-A3B at 76.2—while the post-trained Think version reached 73.22 and lifted IFEval to 83.70.
The release targets AI labs, universities and enterprise R&D teams with data-center GPUs, offering a reproducible end-to-end MoE recipe and 64K-context experiments rather than a ready-made commercial endpoint.
Why did AMD open-source a massive AI training stack while strictly locking its highly optimized model weights out of commercial use?
Could AMD’s radical KV cache compression and expert-routing breakthroughs finally make complex, long-context AI serving affordable for smaller research teams?