Fireworks AI reports that when the trainer and rollout engine compute token probabilities differently, a reinforcement‑learning run on its GLM‑5.2 model collapsed reward after 20 steps. Aligning the calculations kept the KL divergence at zero and the reward stable for the whole 25‑step run. The team also found that in a Qwen‑3.5‑MoE model, small differences in how expert outputs are summed created a large k3 mismatch, showing that MoE routing needs both selection and precision alignment.
Why it matters
Aligning trainer and rollout calculations lets ML teams avoid costly failed RL runs and scale reinforcement learning more reliably.
