UI-TARS Desktop: ByteDance's AI Agent That Actually Sees and Controls Your Computer
The gap between AI reasoning and AI execution has been the defining frustration of the agent era. Models can plan complex workflows, write sophisticated code, and reason through multi-step problems. But when it comes to actually clicking a button, filling a form, or navigating a desktop application — they're blind. They live in text. They need APIs. And most software doesn't have them.
ByteDance's UI-TARS Desktop closes that gap. At 31,000+ GitHub stars and over 3,000 forks, it's one of the most significant open-source agent projects to emerge from a major tech company. It doesn't just think about tasks — it sees your screen, moves your mouse, and types your keystrokes. And it outperforms GPT-4o and Claude while doing it.
───
What UI-TARS Desktop Actually Is
UI-TARS Desktop is not one thing. It's a multimodal AI agent stack shipping two projects under one umbrella:
Agent TARS — a CLI and Web UI for general-purpose agent work. It brings GUI Agent and Vision capabilities into your terminal, browser, and product. It's built on MCP (Model Context Protocol) and connects to real-world tools: calendars, databases, email, APIs. Think of it as the "headless" side.
UI-TARS Desktop — a native desktop application that literally sees your screen and controls your computer. It takes screenshots, understands what it's looking at through vision-language models, and acts through mouse and keyboard input. Think of it as the "embodied" side.
Together they form something closer to how humans actually work than any agent that's come before. We don't just reason in language — we look, we point, we click. UI-TARS gives agents the same interface.
───
The Core Innovation: Visual Grounding
What makes UI-TARS different from Claude Code, OpenClaw, Hermes Agent, or any terminal-only agent is simple: it sees pixels, not just text.
The system uses the UI-TARS-1.5 vision-language model (available in 7B and 72B parameter versions), trained on approximately 50 billion tokens of screenshot data. This isn't a model that reads HTML and guesses where buttons might be. It parses screenshots — understanding element types, spatial relationships, bounding boxes, visual descriptions, and layout structure.
When you tell it to "install the autoDocstring extension in VS Code," it doesn't run a command. It:
1. Looks at your screen and recognizes VS Code isn't open
2. Clicks the VS Code icon
3. Waits for the window to fully load (it sees the loading state)
4. Identifies the Extensions tab in the sidebar by its visual position
5. Clicks it — and if the click misses, it notices the UI didn't change and tries again
6. Types "autoDocstring" into the search field
7. Watches for the install button to appear
8. Clicks Install and waits for confirmation
Every step is reasoned through visually. When something goes wrong, it doesn't crash — it notices the screen didn't change as expected and self-corrects. This is fundamentally different from API-based automation that breaks on the first unexpected dialog box.
───
Benchmark Dominance
The research paper behind UI-TARS, published by ByteDance and Tsinghua University, reports state-of-the-art performance across 10+ GUI benchmarks. The numbers are striking:
copy
| Benchmark | UI-TARS 72B | GPT-4o | Claude 3.5 | Gemini 1.5 Pro |
| -------------- | ----------- | ------ | ---------- | -------------- |
| VisualWebBench | 82.8% | 78.5% | 78.2% | — |
| WebSRC | 93.6%* | — | — | — |
| ScreenQA-short | 88.6% | — | — | — |
*7B model result on WebSRC
In OSWorld — which tests open-ended computer tasks — and AndroidWorld — 116 programmatic tasks across 20 mobile apps — UI-TARS consistently leads. The researchers note that Claude Computer Use "performs strongly in web-based tasks but significantly struggles with mobile scenarios," while UI-TARS "exhibits excellent performance in both website and mobile domains."
This cross-domain capability is significant. Most computer-use agents are web-specialized. UI-TARS works across desktop apps, mobile interfaces, and web applications — same model, same approach.
───
How It Works Under the Hood
UI-TARS's training pipeline is what makes the visual understanding possible:
Screenshot-based training data with parsed metadata — element descriptions, types, bounding boxes, visual descriptions, element functions, and text content. The model learns not just what a button looks like, but what it does.
State transition captioning — the model identifies and describes differences between two consecutive screenshots. This lets it recognize whether a click actually did something, a page loaded, or an error appeared.
Set-of-Mark (SoM) prompting — overlays distinct marks (letters, numbers) on specific screen regions. This gives the model a coordinate system for precise pointing: "click on the element marked 'B'" instead of "click somewhere in the top right."
Dual-system reasoning — the model performs both System 1 (fast, intuitive, automatic) and System 2 (slow, deliberate, multi-step) thinking. It plans, reflects, recognizes milestones, and corrects errors. When a click misses, it doesn't retry blindly — it reasons about why the click might have missed and adjusts.
Error correction training — researchers identified mistakes in training data, labeled corrective actions, and simulated recovery steps. The model learned not just to perform tasks, but to recover when things go wrong.
Short-term and long-term memory — handles immediate task context while retaining historical interactions to improve future decisions. Over time, the agent gets better at navigating interfaces it's seen before.
───
What the User Experience Looks Like
Open the desktop app, type a task in natural language, and watch it work:
• A thinking panel on the left shows the agent's step-by-step reasoning — what it sees, what it plans to do, why it's doing it
• The action window on the right shows your actual desktop as the agent controls it
• Every mouse movement, click, and keystroke is visible in real time
• The agent explains its reasoning before acting, so you can intervene before it does something wrong
This "explain-then-act" pattern builds trust in a way that opaque automation never can. You're not hoping the script worked — you're watching it work.
───
Agent TARS: The Terminal Side
For developers who prefer CLI workflows, Agent TARS provides the same capabilities in a terminal package:
Bash
npm install @agent-tars/cli@latest -g
agent-tars --provider anthropic --model claude-sonnet-4-6 --apiKey your-key
It supports multiple model providers (Volcengine/Doubao, Anthropic, OpenAI, and others), runs headful with a Web UI or headless as a server, and integrates with any MCP-compatible tool. The v0.3.0 release added streaming multi-tool support, runtime timing statistics, and an Event Stream Viewer for debugging agent data flow.
The hybrid browser agent is particularly clever: it can use visual grounding (looking at pixels), DOM analysis (reading HTML), or both — switching strategies based on what works better for the current page.
For isolated execution, it supports the AIO Agent Sandbox, letting the agent run in a containerized environment without risking your actual machine.
───
Remote Operation: The Killer Feature
Version 0.2.0 introduced something that changes the game: Remote Computer Operator and Remote Browser Operator — completely free.
No configuration. Click a button. The agent controls any remote computer or browser. This turns UI-TARS from a personal automation tool into something usable for remote support, distributed testing, cloud-based workflows, and multi-machine orchestration.
The remote browser operator is particularly practical for web scraping and testing: run agents against websites from cloud machines with different IPs, screen resolutions, and browser configurations — all through the same interface.
The remote computer operator opens possibilities for server administration, legacy system interaction, and cross-platform workflows where you need AI on a machine you're not sitting in front of.
───
The Rough Edges
UI-TARS is massive and mature by open-source standards, but it's not without friction:
386 open issues tells you this is actively developed and actively used. The issue count comes from real-world usage, not neglect. But expect to hit edge cases.
Model dependency is real. The best results come from ByteDance's own UI-TARS-1.5 and Seed-1.5-VL/1.6 models. Running with third-party models like Claude or GPT-4o works but doesn't leverage the full visual training pipeline. The 72B model requires serious hardware to run locally.
Visual automation is slower than API automation. If you're running the same task 100 times a day, write a script. UI-TARS shines for complex, infrequent, multi-application workflows — not repetitive operations where milliseconds matter.
Safety concerns are non-trivial. An agent that can move your mouse and type on your keyboard is an agent that can delete files, send messages, or make purchases. The thinking-panel transparency helps, but careful supervision and sandboxing are essential for high-stakes operations.
The ByteDance factor. For users concerned about Chinese tech company involvement, the Apache 2.0 license and fully open-source code provide transparency. Everything runs locally. Nothing phones home to ByteDance servers unless you explicitly use their hosted model APIs.
───
How It Compares
copy
| Aspect | UI-TARS | Claude Computer Use | holaOS | OpenClaw/Hermes |
| ----------------- | -------------------------------------- | -------------------------- | ----------------------- | --------------------- |
| Approach | Vision model sees screen + controls OS | API-based computer control | Shared visual workspace | Terminal + text tools |
| OS Control | Native mouse/keyboard | API-mediated | Via agent harness | Shell only |
| Mobile support | Yes (AndroidWorld-tested) | No (struggles) | No | No |
| Remote operation | Built-in, free | Limited | Via VNC | Not applicable |
| Model flexibility | Best with own models | Claude-only | Multi-model | Multi-model |
| License | Apache 2.0 | Proprietary | Modified Apache 2.0 | MIT |
| Stars | 31K+ | N/A (API) | 4.5K+ | 370K+ (OC) |
UI-TARS occupies a unique position: it's the only project that combines visual screen understanding, native OS control, remote operation, and open-source licensing in one package. It's not better than terminal agents for everything — but for tasks that require actual GUI interaction, nothing else in open source comes close.
───
Who This Is For
If your agent work lives entirely in terminals, code editors, and APIs, Agent TARS's CLI might be all you need. The visual desktop component adds overhead you don't require.
But if you've ever needed an agent to:
• Navigate a legacy enterprise application with no API
• Book flights across multiple travel sites comparing prices
• Change system settings across different operating systems
• Test your application visually across different screen sizes
• Automate workflows that span three different desktop applications
• Run agents on remote machines without setting up complex infrastructure
...then UI-TARS Desktop solves a problem no terminal agent can touch.
It's also positioned as a research platform. The batch trajectory generation, Atropos RL environments, and trajectory compression tools are explicitly designed for training the next generation of tool-calling models. If you're working on agent evaluation or training, the benchmark infrastructure alone is worth the install.
───
Links:
• GitHub: github.com/bytedance/UI-TARS-desktop — 31K+ stars
• Website: agent-tars.com
• Paper: UI-TARS: Pioneering Automated GUI Interaction with Native Agents
• Quick Start: npx @agent-tars/cli@latest or clone the desktop repo
• Discord: discord.gg/HnKcSBgTVx