A research team has introduced WorldSketch, a robotics system that asks a vision-language model (VLM) to build a simple model of its surroundings instead of directly choosing robot movements. The model identifies task-relevant objects, how they can be touched or moved, and the motions that may be possible. A geometric planner then uses that model to plan contact and the robot’s whole-body movement.
The researchers report zero-shot success rates of 91% on RoboCasa, 97% on LIBERO-PRO and 76% on R2R-CE navigation. On RoboCasa, WorldSketch scored 58 percentage points above the next-best zero-shot agent baseline while using one-tenth as many VLM input tokens. On LIBERO-PRO, it beat CaP-X by 82 percentage points. For navigation, it matched step-by-step VLM agents’ success rate while using six times fewer input tokens. These are results reported by the paper’s authors, not independent evaluations.
How does the system revise its plan?
WorldSketch does not treat its first sketch as final. The VLM can query perception tools to check what it has understood, and the robot’s execution feedback can reveal when a prediction was wrong. The system can then update uncertain parts of its model and try again. The goal is to give the planner enough information to act without requiring the VLM to reason directly about precise geometry and contact.
The approach addresses two common design choices described by the researchers: using a VLM to select from a fixed library of skills, or asking it to generate actions and control programs. A fixed library can limit what the robot can do; direct action generation requires the VLM to handle detailed physical reasoning. WorldSketch instead puts the VLM in charge of constructing a task-specific model, with the planner handling movement.
The paper is a 40-page arXiv preprint, posted on November 3, 2026. Its abstract reports results across three named benchmarks, but does not say whether the evaluation used physical robots or establish how the system would perform outside those tests. The results therefore show promise within the reported evaluations, not proof of reliable everyday household use.
Why it matters
For people interested in home robots, WorldSketch offers a different way to connect visual understanding with movement: the VLM builds a model, and a planner uses it to act. The reported benchmark results are encouraging, but the paper’s abstract does not establish whether the system has been tested on physical robots or how reliably it would handle ordinary homes.
