The ggml‑org/llama.cpp project released version b12342 with a new CUDA int8 MMA kernel for few‑row matrix‑multiply‑quantize (MMQ) operations. It works with Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS quantizations on NVIDIA GPUs from the Ampere generation onward. The change is aimed at speeding up speculative decoding verification passes. The update also includes several test and guard fixes.
Why it matters
Developers can run LLM inference faster on compatible GPUs, reducing latency for interactive AI applications.
