Soft‑prompt tuning helps LLM benchmarks focus on knowledge, not format

Aleph Alpha released a method that tweaks only a handful of learnable vectors—called soft prompts—to teach a frozen language model the exact answer format a benchmark expects. The tuning takes about 80 steps, roughly 1.5 minutes for a 7 billion‑parameter model, and improves format‑following accuracy to near‑100 percent without changing the model’s knowledge.

Because the model’s knowledge score stays flat while formatting improves, researchers can compare base models fairly and predict how they will rank after full post‑training, saving weeks of compute.