Prime Intellect has added an overlapping backward-pass kernel to DeepSeek’s Mega MoE system, making it usable for training mixture-of-experts models. The company says the kernels were up to 3.11× faster than a default PyTorch implementation, while end-to-end training in Prime-RL was up to 1.34× faster than its baseline.
The end-to-end tests used H200 GPUs and Qwen3 models across four- and 16-node setups. Support depends on GPU type: the kernels cover bfloat16, with some FP8 formats limited to Hopper or Blackwell GPUs.
Why it matters
For teams training MoE models on supported Hopper or Blackwell GPUs, the kernels could reduce training time, though the reported speedups come from Prime Intellect’s own tests and may not apply to other setups.
