Oossa

llama.cpp adds fast path for Zamba2 GPUs, boosting token speed

The ggml-cuda update adds a d_state=64 path, raising Zamba2‑7B‑Instruct generation from 11 to 15.1 tokens per second.

NoteBy Published by Oossa: 1 min read

The llama.cpp project released a GPU kernel tweak that adds a fast path for a state size of 64. Zamba2 and similar chips use this size, and before the change every scan step fell back to the CPU, limiting generation to about 6‑7 tokens per second. The new template raises throughput by roughly 35%, moving Zamba2‑7B‑Instruct from 11 to 15.1 tokens per second on the same hardware. Some scan calls (about 28%) still run on the CPU, so full parity isn’t reached yet.

Why it matters

Developers using Zamba2 GPUs can generate text noticeably faster without extra hardware.

Was this article useful?
Share

Read next

Oossa · Newsletter

The week in AI, explained

Every Monday: the stories worth knowing, in plain language. Free, no spam.