# Frozen 16 MB Vocabulary Beats Learned Embeddings in Small Language Models

> A new study shows a fixed 16 MB codebook can replace large learned embedding tables, keeping accuracy while cutting size and speeding inference.

Oossa · 2026-10-19 · https://oossa.com/en/frozen-16-mb-vocabulary-beats-learned-embeddings-in-small-language-models

Researchers led by Saikiran Bathula released a pre‑print on 15 Oct 2026 describing a tiny, frozen vocabulary for small language models. They swapped the usual learned embedding table in a Gemma‑style 270 M model with a 512‑bit codebook that is only 16 MB and has no trainable parameters. The model trained with this table matches or exceeds the original on benchmarks like LAMBADA and PIQA, while using far less memory and running 5–6× faster on a laptop CPU.

## The facts

- Paper submitted 15 Oct 2026 ("Thu, 15 Oct 2026 06:34:18 UTC")
- Frozen vocabulary size: 16 MB versus 168 MB learned table

## Why it matters

Developers can deploy tiny, fast language models on edge devices without sacrificing performance on common tasks.

## Sources & references

1. [Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models](https://arxiv.org/abs/2610.22428) – arXiv, 2026-10-19

Last updated: 2026-10-19