Google announced new inference engines that run the open‑source vLLM and SGLang projects on TPUs using a native PyTorch backend called TorchTPU. The engines were co‑developed with Inferact (vLLM) and RadixArk (SGLang) and already serve production workloads at companies like Meta. They include TPU‑specific optimizations such as custom Pallas kernels and KV‑cache management, aiming to make large‑scale, low‑latency serving easier for PyTorch users.
Why it matters
PyTorch developers can now deploy fast, cost‑efficient inference for large language models on Google’s TPU hardware without switching to JAX.
