# llama.cpp adds int8 CUDA kernel for few‑row matrix ops

> The open‑source LLM runtime now supports a new int8 kernel on Ampere‑class NVIDIA GPUs, speeding up speculative decoding.

Oossa · 2026-11-13 · https://oossa.com/en/llama-cpp-adds-int8-cuda-kernel-for-few-row-matrix-ops

The ggml‑org/llama.cpp project released version b12342 with a new CUDA int8 MMA kernel for few‑row matrix‑multiply‑quantize (MMQ) operations. It works with Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS quantizations on NVIDIA GPUs from the Ampere generation onward. The change is aimed at speeding up speculative decoding verification passes. The update also includes several test and guard fixes.

## The facts

- New int8 MMA kernel supports eight quantization formats on Ampere‑class NVIDIA GPUs.
- Release date: 2026-11-13

## Why it matters

Developers can run LLM inference faster on compatible GPUs, reducing latency for interactive AI applications.

## Sources & references

1. [ggml-org/llama.cpp b12342](https://github.com/ggml-org/llama.cpp/releases/tag/b12342) – llama.cpp, 2026-11-13

Last updated: 2026-11-13