# AWS adds shared KV caching for self-hosted AI models

> Amazon ElastiCache for Valkey can share model context across GPU servers. AWS says it can cut inference costs by up to 40% and first-token wait by up to 80%.

Oossa · 2026-11-19 · https://oossa.com/en/aws-adds-shared-kv-caching-for-self-hosted-ai-models

AWS has added distributed key-value (KV) caching to Amazon ElastiCache for Valkey, letting self-hosted AI models share cached context across GPU servers. AWS says the service can reduce inference costs by up to 40% and time to first token—the wait before a model begins replying—by up to 80%. Those are AWS’s maximum reported reductions, not guaranteed results for every workload.

The shared cache stores reusable model state so another server can pick it up instead of calculating it again. That can help when a chat or agent sends the same long context across multiple turns, or when requests move between servers. AWS says cached state can persist for days and be retrieved in milliseconds.

## How it works

The cache runs on cache.i8ge.48xlarge nodes in the same AWS Availability Zone as the GPU fleet, using RDMA over Elastic Fabric Adapter to transfer data. It requires ElastiCache for Valkey 9.1 or later. Customers can connect it to existing vLLM, SGLang, or TensorRT-LLM deployments through the LMCache connector, with a configuration change.

AWS says it works with open-weight, fine-tuned, and self-trained models supported by the inference engine and LMCache. Its documentation names model families including Qwen, Kimi, GPT-OSS, DeepSeek, Gemma, and GLM. The service is available in 13 regions: three in North America, three in Europe, and seven in Asia Pacific.

## Which workloads may benefit?

AWS says the cache is best suited to long-running or multi-turn sessions, repeated context, and traffic spread across many GPU servers. Examples include coding agents, shared system prompts, and long-context or retrieval-augmented generation. It may offer little benefit for short prompts that are not reused, tasks such as translation or moderation, workloads dominated by output generation, and diffusion models.

Before deploying, teams can use the open-source KVTrace tool at no charge to simulate cache hit rates, estimated savings, and a break-even price using recorded vLLM or SGLang traffic. AWS charges by the hour for each cache node until the cluster is deleted; it says data transfer between GPU hosts and the cache in the same Availability Zone is free. The actual node price depends on AWS’s current pricing.

## The facts

- Distributed KV caching requires ElastiCache for Valkey 9.1 or later and cache.i8ge.48xlarge nodes.
- AWS lists availability in 13 regions across North America, Europe, and Asia Pacific.
- The LMCache connector supports integration with vLLM, SGLang, and TensorRT-LLM.
- AWS says KVTrace can estimate cache hit rates, savings, and break-even price before deployment.

## Why it matters

For teams running multi-turn chat or coding agents on their own GPU servers, shared caching could reduce repeated computation and help existing hardware handle more requests. AWS’s savings figures are maximum claims, so teams need to test their own traffic and compare the estimated savings with the hourly cache-node cost.

## Sources & references

1. [Amazon ElastiCache now supports distributed KV caching for self-hosted LLM inference](https://aws.amazon.com/about-aws/whats-new/2026/11/announcing-distributed-kv-caching-for-self-hosted-llms-with-amazon-elasticache/) – AWS, 2026-11-18
2. [Amazon Web Services (AWS)](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/distributed-kv-caching.html?utm_source=openai) – Amazon Web Services (AWS)

Last updated: 2026-11-19
