# AWS cuts first‑token latency for multi‑turn AI agents with bi‑directional KV transfer

> Bi‑directional key‑value cache sharing on EC2 P5en GPUs lowers TTFT up to 3.3× for long conversations.

Oossa · 2026-10-19 · https://oossa.com/en/aws-cuts-first-token-latency-for-multi-turn-ai-agents-with-bi-directional-kv-tra

AWS added bi‑directional KV cache transfer to its vLLM serving stack. The change lets a decode node send previously computed KV blocks back to the prefill node for the next turn, avoiding repeated work. In tests on a single‑prefill‑single‑decode (1P1D) setup using EC2 P5en instances with EFA, time‑to‑first‑token dropped up to 1.6× for Qwen3‑32B and up to 3.3× for GLM‑5‑FP8. The feature works on both Trainium and GPU clusters and is configurable via TTL and recompute thresholds.

## The facts

- Tested on Oct 19 2026 – "Mon Oct 19 2026 18:03:23 GMT+0200"
- TTFT reduced up to 1.6× for Qwen3‑32B and up to 3.3× for GLM‑5‑FP8 on EC2 P5en instances

## Why it matters

Developers of conversational assistants can keep response times low even as chat histories grow, saving accelerator cost.

## Sources & references

1. [Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM](https://aws.amazon.com/blogs/machine-learning/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv-transfer-in-vllm/) – AWS, 2026-10-19

Last updated: 2026-10-19