A new paper released on October 8, 2026 examines how a home‑automation system called Wactorz talks to language models. The system makes several kinds of calls – intent routing, action classification, device grounding, planning pipelines, and generating Python code. The authors tested nine models ranging from 0.8 billion parameters to a large hosted model. They used the system’s real production prompts on two Home Assistant installations, covering 280 real cases and 2 520 scored calls.
What the numbers show
The results reveal that bigger models are not always better. For four of the five call sites, the best local model performed statistically the same as both the small hosted model and a frontier model. Only the code‑generation site showed a measurable gap, with the best local model scoring slightly worse (p = 0.039 vs. the small host and p = 0.002 vs. the frontier host). Overall accuracy with per‑call routing to the optimal local model was 91.8 %, compared with 95.4 % when every call used the most capable model, but without any extra compute cost. In a live user test, handling only the two generative sites on a host matched full hosting (39 correct out of 43) while using just 28 % of the spending.
Safety quirks of tiny models
The benchmark also uncovered a safety issue at the actuation site. One 4 B model (Gemma4 E2B) mistakenly tried to control devices it didn’t own in 87.2 % of requests, while another model refused every request. These extreme behaviors illustrate that small models can resolve the accuracy‑refusal trade‑off in unpredictable ways.
Why it matters
For homeowners using open‑source automation, the findings mean they can run small, locally stored models for most tasks without paying for cloud hosting, keeping data private and costs low. However, they should be cautious with actuation commands, as some tiny models may behave unpredictably and either over‑act or refuse entirely.
