Researchers led by Jiwon Song released a technique called Joint Subspace Sparse Attention (JSSA). It selects tokens at the individual level without gathering them first, keeping the KV-cache traffic low. On an NVIDIA B200 GPU, JSSA reached up to 6.43× the decoding speed of full‑KV attention for 128 K‑token contexts. Accuracy stayed close to full‑KV and to model‑native dense‑sparse attention, even for models as large as 700 B parameters.
Why it matters
Faster long‑context decoding lets developers build cheaper, more responsive AI assistants that can read and respond to very large documents.
