Tsinghua researchers have made AKV, a cache-management system for LLM agents, available as open-source code on GitHub. Instead of shortening or summarizing an agent’s conversation when it grows long, AKV lets the agent drop selected stored information and reposition what remains. The project is built around SGLang, software for running language models, and includes both a component for choosing cache operations and a serving system that carries them out.
The distinction matters because an agent’s history can include many observations, tool calls and tool results. Keeping all of that in the model’s active context takes storage and processing. But editing earlier messages can make the system recalculate later context, a step called re-prefilling. The AKV paper says that can be costly, while removing too much history can hurt task accuracy. AKV instead keeps the conversation history intact and changes which parts of its stored key-value cache the model attends to.
How does AKV work?
AKV gives agents two cache operations. “Drop” excludes selected information from future attention, while “reposition” compacts the positions of retained information. The project’s example keeps the latest tool result, drops an earlier result after the newer one arrives, and then repositions the remaining cache. The original messages stay in the request; the AKV server applies the cache instructions. According to the paper, this avoids re-prefilling retained context.
The system also has to account for how its cache was changed. The paper notes that the same sequence of conversation tokens can lead to different cache states if the agent used different operations. That means a server cannot safely reuse cached information based only on matching the conversation prefix. AKV combines an agent-side policy with an SGLang-based server that manages cache reuse, removal and restoration. The project says the server uses a stateless interface compatible with existing serving APIs, while handling cache state internally.
What do the tests show?
In the authors’ experiments, a rolling-drop policy for tool responses raised average benchmark scores by 1.7 points and cut average cost by 27.7% compared with threshold-triggered summarization. The paper also reports up to 1.2 times the output throughput and a 1.5-times speedup in 95th-percentile request latency. A separate method for improving cache policies recursively reduced cost by 7.1% compared with policies designed by experts. These are results reported by the paper’s authors; the supplied materials do not report independent replication.
The GitHub instructions describe installing AKV with Rust, Cargo and Python 3.12. Their quick-start example serves GPT-OSS-120B through AKV and sends requests using an OpenAI-compatible chat-completions format. The project also describes integrating its policy into an existing agent harness by processing requests before sending them to the server. The reported results cover four agent benchmarks, but the supplied materials do not name those benchmarks or give enough detail to judge how the savings would translate to other models or workloads.
Why it matters
For teams building agents that repeatedly call tools, AKV offers a way to discard old tool results without rewriting the conversation history. The authors report lower costs in their tests, but whether a particular team would see similar savings depends on its model and workload.
