# Push‑based KV transfer cuts first‑token latency in vLLM

> AWS shows the new push mode can make first token appear up to three times faster and reduce per‑token time by 30% for the Qwen3‑32B model.

Oossa · 2026-10-19 · https://oossa.com/en/push-based-kv-transfer-cuts-first-token-latency-in-vllm

The vLLM engine now offers a push‑based key‑value (KV) cache transfer. It moves the data‑move setup to the prefill side, so the decode instance can start generating tokens sooner. In a benchmark on a 1‑prefill‑1‑decode (1P1D) setup using EC2 P5en instances with AWS Elastic Fabric Adapter, push mode cut time‑to‑first‑token by up to 3× and lowered time‑per‑output‑token by about 30% compared with the default pull mode.

## The facts

- Up to 3× lower TTFT and 30 % lower TPOT on Qwen3‑32B model
- Tested on EC2 P5en GPU instances with EFA on 2026‑10‑19

## Why it matters

Developers running high‑throughput LLM services can see noticeably faster responses, especially when handling many long prompts at once.

## Sources & references

1. [Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM](https://aws.amazon.com/blogs/machine-learning/push-based-kv-transfer-to-improve-ttft-and-tpot-for-disaggregated-inference-in-vllm/) – AWS, 2026-10-19

Last updated: 2026-10-19