Oossa

Unsloth beta lets AI models use your desktop apps

Unsloth’s v0.1.918-beta adds screen, mouse and keyboard control, app connections for creative tools, and new MiniMax-H3-Image options.

By Published by Oossa: 2 min read

Oscar Ivan Esquivel Arteaga · Unsplash

Unsloth’s v0.1.918-beta update adds a computer-use feature that lets an AI model view a user’s screen and control the mouse and keyboard. The feature works on macOS, Windows and Linux with supported local vision models, as well as connected providers. It is off by default and must be turned on in Settings.

Before a model takes its first action in a chat, Unsloth asks for permission. It also asks before switching to a new app and before risky steps such as using a terminal. The company says blocked apps will not be clicked or typed into, regardless of permission settings. Users can press Stop or move the pointer to a screen corner to take back control.

More apps can connect to chat

The update also adds Unreal Engine, Unity, Godot, Figma and Roblox to the app menu alongside Blender. These connections use MCP, a way for chat models to interact with other software. Unsloth says a model can use them to build scenes and games; its demos include a Pong game in Godot and a village square in Unreal Engine. Figma is read-only: a model can inspect frames, styles and assets but cannot edit them.

Users have to select an app, grant permission and test the connection before enabling it. Unsloth notes that these tools run code inside the connected app, and says only the app’s owner can use them. The release also adds nine providers to a searchable connections gallery, including Amazon Bedrock, Azure OpenAI, NVIDIA NIM and Perplexity.

Image models and chat controls

MiniMax-H3-Image can now run in GGUF format on CPUs, Macs and AMD hardware, and with FP8 or INT8 weights on NVIDIA GPUs. FP8 requires an NVIDIA GPU with compute capability SM 8.9 or newer. Users can also fine-tune the image model with LoRA, a method for adapting a model with a smaller set of added weights.

Unsloth says a 2048-pixel image decodes in 0.28 seconds rather than 2.82 seconds on an A100 80GB. That is a company-reported result, not an independent comparison. Other changes include a model picker in the chat box, project file browsing and automatic summaries for long chats on many connections. The update also supports running Inkling-2-Small as Dynamic GGUFs.

Why it matters

For people who run local AI models, computer use could let them ask a model to carry out tasks in desktop software rather than just give instructions. But it requires explicit setup and permissions, and the release does not establish how reliably it will handle everyday tasks beyond Unsloth’s own demos and tests.

Was this article useful?
Share

Read next

Oossa · Newsletter

The week in AI, explained

Every Monday: the stories worth knowing, in plain language. Free, no spam.