AWS cuts first‑token latency for multi‑turn AI agents with bi‑directional KV transfer

AWS added bi‑directional KV cache transfer to its vLLM serving stack. The change lets a decode node send previously computed KV blocks back to the prefill node for the next turn, avoiding repeated work. In tests on a single‑prefill‑single‑decode (1P1D) setup using EC2 P5en instances with EFA, time‑to‑first‑token dropped up to 1.6× for Qwen3‑32B and up to 3.3× for GLM‑5‑FP8. The feature works on both Trainium and GPU clusters and is configurable via TTL and recompute thresholds.