Han‑Sang Lee and co‑authors released ixi‑sLM, a Korean speech‑language model that can run entirely on a phone’s neural‑processing unit (NPU). The model fits inside a 1 GiB memory budget and uses only about a quarter of a phone’s CPU energy per call. It powers ixi‑O’s AI Call Summary and AI Call Answering services for several million subscribers.
How they made a big model fit on a phone
The team trimmed the vocabulary to the words most common in phone calls, cutting the embedding and language‑model head size by half. They also refitted the tokenizer to speech, which reduced token count by 18 % on call traffic. The transformer weights are stored in INT4 (four‑bit integer) format, and a task‑aware quantization step assigns mixed‑precision values to each module based on activation sensitivity. The resulting model is 803 MiB – 28 % of the size of a standard FP16 model – and runs on Qualcomm and Apple NPUs.
Performance compared with the cloud
In tests, the on‑device model achieved 94 % of the task‑weighted quality of a state‑of‑the‑art cloud large language model (LLM). Overall quality dropped only slightly after moving to the NPU, while per‑call energy use fell to 23 % of the CPU baseline.
Why it matters
Phone users can now get AI‑generated call summaries and answers without sending audio to the cloud, keeping their conversations private and avoiding data fees. The low energy use means the feature won’t drain the battery quickly. However, the paper does not provide independent third‑party quality tests, so exact performance on diverse calls remains uncertain.
