# Researchers introduce First Passage Attention for selective forgetting

> A new attention method lets models drop intermediate context while keeping older tokens, scaling to a 110 M‑parameter model.

Oossa · 2026-10-19 · https://oossa.com/en/researchers-introduce-first-passage-attention-for-selective-forgetting

Yuxuan Liu and co‑authors released a paper on Oct 19 2026 describing First Passage Attention (FPA). FPA lowers a key’s logit based on the deepest dip of a learned field, allowing it to erase middle tokens while preserving earlier ones. The authors show that a constant‑size FPA head can select any subset of keys, something softmax heads need a larger dimension for. In a 110 M‑parameter language model, FPA matches the Forgetting Transformer’s perplexity and uniquely retrieves passkeys perfectly after eight times the usual training length, using only about 25 % of the key‑value cache.

## The facts

- First Passage Attention proposed in a paper released Oct 19 2026
- Scaled to a 110 M‑parameter language model and achieved perfect passkey retrieval at 8× training length

## Why it matters

It lets large language models keep important distant information while efficiently forgetting irrelevant middle parts, improving long‑context performance.

## Sources & references

1. [First Passage Attention](https://arxiv.org/abs/2610.22714) – arXiv, 2026-10-19

Last updated: 2026-10-19