Researchers describe a way to speed up long-context text generation in hybrid large language models without retraining them or changing their weights. Their method has each full-attention layer focus on recent tokens, a pinned prefix and a limited set of relevant tokens selected from the wider context.
In tests across six models with 4 billion to 397 billion parameters, the researchers report matching baseline accuracy on 21 datasets. For Qwen3.5-4B, decoding 256,000 tokens was up to 2.22 times faster; these are the paper’s reported results, not an independent evaluation.
Why it matters
For teams running long-context hybrid models, the reported speedups could reduce decoding time without retraining, though the paper does not establish how the method performs in their own systems.
