# Fireworks finds numeric mismatches can break RL training

> A Fireworks experiment shows that misaligned calculations between trainer and rollout engine cause reward collapse, and MoE models add extra alignment challenges.

Oossa · 2026-10-01 · https://oossa.com/en/fireworks-finds-numeric-mismatches-can-break-rl-training

Fireworks AI reports that when the trainer and rollout engine compute token probabilities differently, a reinforcement‑learning run on its GLM‑5.2 model collapsed reward after 20 steps. Aligning the calculations kept the KL divergence at zero and the reward stable for the whole 25‑step run. The team also found that in a Qwen‑3.5‑MoE model, small differences in how expert outputs are summed created a large k3 mismatch, showing that MoE routing needs both selection and precision alignment.

## The facts

- GLM‑5.2 experiment: reward fell from ~0.9 to <0.2 at step 20 without numeric alignment, stayed stable with alignment over 25 steps.
- Qwen‑3.5‑MoE test: k3 divergence of 0.296 (≈300× the 0.001 threshold) traced to expert‑output aggregation precision.

## Why it matters

Aligning trainer and rollout calculations lets ML teams avoid costly failed RL runs and scale reinforcement learning more reliably.

## Sources & references

1. [9/30/2026Reinforcement learning: Why alignment of numerics and MoE routing matter](https://fireworks.ai/blog/reinforcement-learning-why-alignment-of-numerics-and-MoE-routing-matter) – Fireworks AI

Last updated: 2026-10-01