Yuxuan Liu and co‑authors released a paper on Oct 19 2026 describing First Passage Attention (FPA). FPA lowers a key’s logit based on the deepest dip of a learned field, allowing it to erase middle tokens while preserving earlier ones. The authors show that a constant‑size FPA head can select any subset of keys, something softmax heads need a larger dimension for. In a 110 M‑parameter language model, FPA matches the Forgetting Transformer’s perplexity and uniquely retrieves passkeys perfectly after eight times the usual training length, using only about 25 % of the key‑value cache.
Why it matters
It lets large language models keep important distant information while efficiently forgetting irrelevant middle parts, improving long‑context performance.
