Reads the tree, not the pixels
Waldo walks the accessibility tree, the same one screen readers use. Every control arrives with its name, role and real position, so any model picks the right thing instead of guessing at pixels.
Open source · for every AI agent · starting with the Mac
Give your AI a hand.
Waldo is the pair of hands any AI can borrow. He sees every window, presses every button and types into every field. When money, messages, passwords or deletes come up, he stops and asks you.
A
B
1
Month
Revenue
2
July
$15,920
3
August
$16,140
4
September
$16,150
5
6
Total
$48,210
7
Waldo wants to send an email as youSending as you is one of the four hard stops.
Todana@studio.co
SubjectQ3 numbers
“Hi Dana, Q3 came in at $48,210.”
Only a click from your own mouse counts.
refusedStopped.
Waldo let go of everything. Every tool refuses until you say go.
The real flow, drawn by hand. Any model, same hands. Push your pointer into the demo’s top-left corner to stop him.
Bring any model
- Claude
- GPT
- Gemini
- DeepSeek
- Grok
- Qwen
- Llama
- Mistral
- your own
Waldo speaks MCP, the open standard agents use to pick up tools. If your agent speaks it too, it can borrow his hands.
How it works
Good with his hands.
Three habits that make Waldo safe to lend to any AI.
ExportWaldo
Dear team, the plan foryou
Works behind you
Presses go straight to the control. The app stays put and your pointer stays yours. Keep working. He’ll work around you.
Dark mode
looked again…changed ✓
Checks his own work
After every act, Waldo looks again. If an app ignored him, he tries another way. Your agent hears what really happened, never a hopeful guess.
Asks first
Four things Waldo
never does alone.
The hard stops live in Waldo’s code, not in a prompt. No model, however clever, can talk its way past them.
Money
Payments, transfers, purchases, trades.
He doesn’t carry a wallet.
Sending as you
Email, messages, posts, form submissions.
Not even a thumbs-up.
Passwords & keys
Logins, keychains, API keys.
He looks away.
Deleting things
Deletes, erases, resets, sign-outs.
Measure twice. Ask once.
Only you can say yes. No agent can approve itself. The ask panel takes no keys and refuses any click that isn’t from your own mouse. Software clicks bounce off, and so do accessibility presses. Both are tested. (He has tried. See above.)
One switch
The panic corner.
Shove your pointer into the top-left corner of the screen. Every tool, for every agent, stops until you say go. Or type waldo stop.
Numbers
Fast hands.
Straight answers.
Each act takes Waldo a fraction of a second, then he checks it really happened. Whatever model you bring, its thinking is the slow part.
Efficientfractions of a second
0.35 sRead every control in a window241 controls in one Chrome window
0.43 sTake a screenshot for the modelone display, scaled to fit
0.15 sBring an app to the frontmedian, and confirmed
Effectivedoes it, then proves it
12/12Right app brought to the frontevery run
12/12Knew which app was in frontevery run
6/6Typed into stubborn fieldsones that ignore focus or already hold text
3.2 s inside Waldo for a real 38-step job in a web portal~90 ms to confirm a click changed something
Why Waldo
Every AI deserves hands. Every human deserves the final say.
Models are getting brilliant, yet most still can’t touch the apps you use every day. Waldo is the missing pair of hands: open source, model-agnostic, and wired so a person always holds the yes. We’re starting with the Mac.
Open sourceAny modelYou hold the yesRuns locally
Questions
Asked and answered.
What people ask before they lend Waldo to their AI. More answers in the guides.
What is Waldo?
Waldo is the pair of hands any AI can borrow. He is a free, open source MCP server for macOS. He sees every window, presses every button, and types into every field. When money, messages, passwords, or deletes come up, he stops and asks you.
How do I let Claude Code use my Mac?
Install Waldo and register him with Claude Code. Run git clone https://github.com/justusewaldo/waldo, then cd waldo, make install, waldo grant, and make register. Start a new Claude Code session and ask in plain words.
Does Waldo work with Codex, Gemini CLI, Cursor, and other agents?
Yes. Waldo speaks MCP, the open standard agents use to pick up tools. Codex, Gemini CLI, Cursor and friends all take the same command: waldo mcp.
Is it safe to let an AI agent use my Mac?
Waldo is wired so a person always holds the yes. Four hard stops live in his code, not in a prompt: money, sending as you, passwords and keys, and deleting things. Only a click from your own mouse can approve one.
Can the AI approve its own actions?
No. The ask panel takes no keys and refuses any click that did not come from your own mouse. Software clicks bounce off, and so do accessibility presses. Both are tested.
How do I stop an AI agent at once?
Push your pointer into the top-left corner of the screen. Every tool, for every agent, stops until you say go. Or run waldo stop.
Does Waldo read the screen with screenshots?
He reads the accessibility tree, the same one screen readers use. Every control arrives with its name, role, and real position, so any model picks the right thing instead of guessing at pixels. He takes a screenshot when the model asks to look.
Does Waldo send my screen to the cloud?
Waldo runs on your Mac and sends nothing to any server. What he reads goes to your agent, and your agent sends it to the model you chose, under that agent's own rules.
What does Waldo cost, and what do I need?
Waldo is free and open source under Apache 2.0. You need a Mac, Rust, and the Xcode command line tools, plus an agent that speaks MCP. He has run on Apple silicon with macOS 26.5 and 27.
Was Waldo called OS Control?
Yes. OS Control became Waldo in version 0.3.0. The tools, the gates, and the checks work as before. The command is now waldo.
Install
Bring Waldo home.
You need macOS, Rust and the Xcode command line tools. Build him once, then lend him to any agent.
Then ask your agent, in plain words:
Open System Settings and tell me which apps can record the screen.
$ git clone https://github.com/justusewaldo/waldo $ cd waldo $ make install # Waldo, the ask panel and the bar $ waldo grant # Accessibility + Screen Recording $ make register # hands the tools to Claude Code
$ git clone https://github.com/justusewaldo/waldo $ cd waldo && make install && waldo grant # then add Waldo to your agent’s MCP settings: { "mcpServers": { "waldo": { "command": "waldo", "args": ["mcp"] } } }
Codex, Gemini CLI, Cursor and friends all take the same command: waldo mcp. macOS asks permission for whichever app starts it.