The vLLM engine now offers a push‑based key‑value (KV) cache transfer. It moves the data‑move setup to the prefill side, so the decode instance can start generating tokens sooner. In a benchmark on a 1‑prefill‑1‑decode (1P1D) setup using EC2 P5en instances with AWS Elastic Fabric Adapter, push mode cut time‑to‑first‑token by up to 3× and lowered time‑per‑output‑token by about 30% compared with the default pull mode.
Why it matters
Developers running high‑throughput LLM services can see noticeably faster responses, especially when handling many long prompts at once.
