The paper NormTune proposes a simple add‑on for parameter‑efficient fine‑tuning (PEFT). It looks at how much each layer’s weights change and boosts layers that change too little. Adding NormTune to existing PEFT methods like LoRA, Bone and MoRA raised average accuracy by up to 24 % on hard reasoning tests such as AIME and MATH. With a larger rank (256), the NormTune‑enhanced methods matched full fine‑tuning performance, sometimes beating it by a few percent.
Why it matters
It lets researchers and developers achieve near‑full‑model accuracy on reasoning tasks with far less GPU time and memory.
