cua-driver

Orchestrates macOS app automation via cua-driver. Whenever a user asks to drive a native macOS app, follow the loop in this skill rather than calling tools ad-hoc — the snapshot-before-action invariant is not optional and silently breaks if you skip it.

DeepChat MCP mode

When DeepChat auto-pins this skill because the Computer Use MCP server is enabled, use the available Computer Use tools in the current tool list directly. Treat examples such as click({...}) or get_window_state({...}) as calls to the matching MCP tool with the same JSON payload. Shell CLI examples are diagnostic references for shell-based agents.

App-name resolution belongs here. When matching a user request to an app, compare the request with each list_apps name and bundle_id. Consider the user's language, system language, English product names, English brand names, romanized or pinyin variants, and common abbreviations. Treat bundle_id as the strongest identity signal. When the requested app name is ambiguous, localized, translated, abbreviated, or written in another script, call list_apps, resolve the most credible bundle id, then call launch_app with that bundle id.

Sparse visible UI fallback

Many media and Electron apps expose a shallow AX tree while still showing enough pixels to act. Spotify-style snapshots with only AXWindow, title-bar buttons, and a menu bar are still actionable when the screenshot shows a visible, unambiguous target.

For a user-requested non-destructive action, continue within the same turn through the safest visible options:

1. Re-snapshot once if the first tree looks sparse.
2. Use zoom on the relevant screenshot region when the screenshot is wide, dense, or the target is small.
3. Prefer in-window controls visible in the screenshot: cards, primary buttons, bottom player controls, search fields, sidebars.
4. Use pixel click({pid, window_id, x, y}) or click({pid, window_id, x, y, from_zoom: true}) for visible targets missing from AX.
5. Re-snapshot after each action and look for evidence: selected state, text changes, playback progress, button icon changes, new panels, highlighted rows, or changed window content.
6. If the first safe click has no observable effect, try the next visible, semantically related target before asking the user.

Ask for user confirmation only when the visible candidates are ambiguous, the action is destructive, the target is off-screen, or the next step requires foregrounding the app. When the AX tree is weak and the screenshot presents clear controls, continue with the visual fallback steps above.

Menu-bar fallback is valid when the target app is already frontmost and the menu item names match the user's requested action. Inspect menus such as File, Edit, View, Playback, Window, and Help for semantic commands, then dispatch menu actions and re-snapshot. For backgrounded apps, prefer in-window AX or pixel actions.

The no-foreground contract — read this first

The user's frontmost app MUST NOT change. This is the whole reason cua-driver exists. Users pay for the right to keep typing in their editor while an agent drives another app in the background. Violate this rule and every other nice property the driver gives you (no cursor warp, no Space switch, no window raise) stops mattering — you just shipped the Accessibility Inspector with extra steps.

Before running any shell command, ask: "does this raise, activate, foreground, or make-key any app?" If yes, don't run it. Every one of the commands below activates the target on macOS and is therefore forbidden unless the user explicitly asked for frontmost state:

• Every form of the open CLI — open -a <App>, open -b <bundle-id>, open <file>, open <path-to-App.app>, open <url> — always activates. macOS routes all forms through LaunchServices, which unhides and foregrounds the target regardless of whether you passed an app name, a bundle id, a document, a URL, or the bundle path itself. The activation happens even when the only intent was "start the process." Never use open for any app launch. This includes launching a just-built .app from a local build dir (e.g. open build/Build/Products/Debug/MyApp.app) — resolve the CFBundleIdentifier from Info.plist and use launch_app with that id. See "The narrow carve-out" below for why launch_app is safe even when the app internally calls NSApp.activate.

• osascript -e 'tell application "X" to activate' — activates by design. Same for ... to open <file>, ... to launch, and anything with activate in the tell block.

• osascript -e 'tell application "System Events" to ... frontmost' in a mutating form (setting frontmost rather than reading it).

• AppleScript files that invoke activate, launch, or open against the target app.

• cliclick (moves the user's real cursor to the target coords before clicking — a focus-steal-equivalent even if the app's window state is unchanged).

• CGEventPost with cghidEventTap targeting a coordinate over a different app's window (warps the cursor, possibly activates on hit).

• AppleScriptTask, NSAppleScript, Process wrapping osascript that contains any of the above.

• NSRunningApplication.activate(options:) called from your own helper binary — same class.

• Dock clicks and any open invocation (see the first bullet — every form of open goes through LaunchServices which activates, full stop).

• Keyboard shortcuts that semantically mean "focus here" — most notably Chrome / Safari / Arc's ⌘L (focus omnibox) and Finder's ⌘⇧G (Go to Folder). These aren't pure key events — the receiving app interprets "user wants to type here" as activation intent and raises its window to be key. Even when delivered to a backgrounded pid via hotkey, the downstream app pulls focus. For omnibox navigation specifically, the correct path is launch_app({bundle_id: "com.google.Chrome", urls: ["https://…"]}) — no omnibox dance, no ⌘L, no focus-steal. Do NOT try set_value on the omnibox: Chrome's commit logic requires a "user-typed" signal that neither an AX value write nor CGEvent.postToPid keystrokes supply from a backgrounded pid — the URL lands in the field but Return fires as a no-op. See WEB_APPS.md → "Navigate to a URL" for the full pattern. The general principle: a shortcut that says "put my cursor inside this app" is a focus-steal; a shortcut that says "do this thing" (copy, save, quit) is fine.

• Tab-switching shortcuts in browsers (⌘1..⌘9, ⌘], ⌘[, ⌘⇧[, ⌘⇧]) are visibly disruptive even when delivered to a backgrounded pid. The app's key handler processes the shortcut, the window re-renders the new tab's content, the user sees their tabs flipping. There is no AX-only workaround: page content (HTML, form state, AXWebArea) populates only for the focused tab; inspecting a background tab requires activating it, which is the visible flip. Observed with Dia; the same mechanic applies to every Chromium-family browser (Chrome, Arc, Brave, Edge).

  Prefer the windows-over-tabs pattern: for each URL you need to drive backgrounded, use launch_app({bundle_id, urls: [url]}) — browsers open each URL in a new window. Each window has its own window_id, its own AX tree, and can be inspected / interacted with via element_index without activating or switching anything. Tabs are a UX grouping for humans; cua-driver workflows should default to windows. See WEB_APPS.md → "Tabs vs windows" for the full pattern.

  Tab-title enumeration (read-only) IS safe — walk a window's toolbar AX tree for AXTab / AXRadioButton children and read their AXTitles. Tab switching (activating one) is not.

Reading frontmost state is fine (osascript -e 'tell application "System Events" to get name of first application process whose frontmost is true'). Mutating it is not.

Corollary — the AXMenuBar rule. AXMenuBarItem + AXPick dispatches at the AX layer regardless of which app is frontmost, but macOS's on-screen menu bar always belongs to the frontmost app. If you drive a backgrounded app's menu bar, the AX call succeeds but the viewer sees the dispatch rendered over the frontmost app's menu bar — confusing in any observed session and routinely a silent no-op too, because action menu items go DISABLED when their owning app isn't the key window. So: only use menu-bar navigation when the target is already frontmost. For backgrounded targets, read state via in-window AX (window title, toolbar AXStaticText) and dispatch via in-window element_index or pixel clicks — both paths are frontmost-insensitive. Full rationale in "Navigating native menu bars" below.

"Open <app>" in user speech means launch, not activate. cua-driver launch_app is the one correct path for process startup — it's idempotent (no-op on a running app), returns the pid, and has an internal FocusRestoreGuard that catches NSApp.activate(ignoringOtherApps:) calls the target makes during application(_:open:) and clobbers the frontmost back to what it was before the launch. That guard is why launch_app with urls (e.g. {"bundle_id": "com.colliderli.iina", "urls": ["~/video.mp4"]}) is safe even for apps that normally foreground on media-load (Chrome, Electron, media players).

Defaults — always prefer cua-driver over shell shims

Default transport is the cua-driver CLI — Bash shelling out to cua-driver <tool-name> '<JSON-args>'. MCP tools (prefix mcp__cua-driver__*) only when the user explicitly asks for them. CLI wins because it picks up rebuilds instantly, failures are easier to diagnose, and there's no per-tool schema-load overhead.

Every reference to click(...), get_window_state(...) etc. in this skill means cua-driver click '{...}' — translate to MCP form only when MCP is requested.

Intent → tool mapping. If you find yourself reaching for the right column, something has gone wrong — re-read "The no-foreground contract" above:

Intent	Use	Don't use
Open / launch an app	launch_app({bundle_id}) or launch_app({bundle_id, urls:[...]})	open -a, osascript 'tell app … to launch/activate/open'
Find a pid	list_apps or launch_app's return	pgrep, ps, osascript frontmost
Enumerate an app's windows	list_windows({pid}) — or read the windows array launch_app already returns	osascript 'every window of app …'
Click / type / scroll / keys	click, type_text, scroll, press_key, hotkey	osascript, cliclick, raw CGEvent, open <url>
Drag / drag-and-drop / marquee select	drag({pid, from_x, from_y, to_x, to_y}) (pixel-only — macOS AX has no semantic drag)	cliclick dd:, osascript drag
Screenshot	screenshot or the PNG in get_window_state	screencapture
Quit an app	ask the user first, then hotkey({pid, keys:["cmd","q"]})	kill, killall, pkill
Hand a file/URL to an app	launch_app({bundle_id, urls:[<path>]})	open -a <App> <path>, open <url>

The narrow carve-out

The only legitimate use of osascript -e 'tell app X to activate' is when the user explicitly asked for frontmost state ("bring Chrome to the front", "make it frontmost", "I want to see X"). Reaching for it because a tool call returned something confusing is wrong — that's the skill's classic foot-in-the-door failure mode and it steals focus every time.

When a cua-driver call surprises you, diagnose cua-driver first:

• Tiny screenshot / empty tree_markdown? Check cua-driver get_config → capture_mode. Default "vision" omits the AX tree (PNG only), "ax" omits the PNG, "som" returns both. If a snapshot lacks a tree, capture_mode is almost certainly "vision" — either reason purely from the PNG or flip to "som" / "ax" via set_config.
• has_screenshot: false? The window capture failed (transient race against a close, or the window has no backing store yet). Re-snapshot; if persistent, pick a different window_id via list_windows.
• Invalid element_index / No cached AX state? You either skipped get_window_state this turn or passed a different window_id than the one the snapshot cached against. The cache is keyed on (pid, window_id) — indices don't carry across windows of the same app. Re-snapshot with the same window_id you're about to click in.
• Sparse Chromium AX tree? Retry get_window_state once — the tree populates on second call.

Only after those are ruled out, and only if the user's action genuinely needs frontmost state, fall through to the activate fallback. Always name the focus steal in your response ("I'll briefly bring Chrome to the front because …").

Self-check pattern

Before every Bash call whose command line touches any macOS app (launching, opening, clicking, typing, scripting, screenshotting), run the self-check:

1. Does this command foreground the target? If yes — stop and translate to the cua-driver equivalent from the mapping table.
2. Does this command move the user's real cursor? (cliclick, any CGEventPost at cghidEventTap over another app's window). If yes — stop; use click({pid, x, y}) which routes per-pid via SkyLight and never warps the cursor.
3. Does this command bypass cua-driver entirely? (osascript mutating GUI state, AppleScript files, external helpers.) If yes — stop; find the cua-driver tool that does the intent.

If all three are "no," the command is safe. If you can't answer, default to stop and ask rather than proceed. A single open -a run by accident kills the demo, the trust, and the user's in-flight editor state.

Prerequisites — check before starting

1. cua-driver is on $PATH (which cua-driver). If not, point the user at scripts/install-local.sh and stop.
2. Run cua-driver check_permissions (with the daemon up — see step 3). The default behavior also raises the system permission dialogs for any missing grants, so the user can grant on the spot. If either grant still reads false after that (user dismissed the dialog), tell them to open System Settings → Privacy & Security and grant Accessibility and Screen Recording to DeepChat Computer Use.app, then stop. Pass '{"prompt":false}' for a purely read-only status check that won't steal focus.
3. Start the daemon with open -n -g -a "DeepChat Computer Use" --args serve (the recommended form — goes through LaunchServices so TCC attributes the process to DeepChat Computer Use.app). cua-driver serve & also works; the CLI auto-relaunches through open -n -g -a "DeepChat Computer Use" when it detects a wrong-TCC context (any IDE-spawned shell: Claude Code, Cursor, VS Code, Conductor). Verify with cua-driver status.

Using cua-driver from the shell

Tool names are snake_case, management subcommands are kebab-case — no ambiguity. Tools invoked as cua-driver <tool-name> '<JSON-args>'. Management subcommands:

• open -n -g -a "DeepChat Computer Use" --args serve — start persistent daemon (required for element_index workflows; without it each CLI invocation spawns a fresh process and the per-pid element cache dies between calls). cua-driver serve & also works — the CLI auto-relaunches via open when the shell's TCC context is wrong. Pass --no-relaunch / CUA_DRIVER_NO_RELAUNCH=1 to opt out.
• cua-driver stop / status
• cua-driver list-tools, describe <tool>
• cua-driver recording start|stop|status — see RECORDING.md

Canonical multi-step workflow:

open -n -g -a "DeepChat Computer Use" --args serve
cua-driver launch_app '{"bundle_id":"com.apple.calculator"}'
# → {pid: 844, windows: [{window_id: 10725, ...}]}
cua-driver get_window_state '{"pid":844,"window_id":10725}'
cua-driver click '{"pid":844,"window_id":10725,"element_index":14}'
cua-driver stop

Agent cursor overlay

Visual cursor overlay for demos and screen recordings. Default: enabled. Toggle with cua-driver set_agent_cursor_enabled '{"enabled":true|false}'. A triangle pointer Bezier-glides to each click target, ring-ripples on landing, idle-hides after ~1.5s. Motion knobs: set_agent_cursor_motion takes any subset of start_handle, end_handle, arc_size, arc_flow, spring — tuneable at runtime, persisted to config.

Requires an AppKit runloop, which cua-driver serve / mcp bootstraps. One-shot CLI invocations skip the overlay entirely.

The core invariant — snapshot before AND after every action

Every action MUST be bracketed by get_window_state(pid, window_id):

• Before — the pre-action snapshot resolves the element_index you're about to use. Indices from previous turns are stale; the server replaces the element index map on every snapshot, keyed on (pid, window_id). Indices from turn N don't resolve in turn N+1, and indices from window A don't resolve against window B of the same app. Skip this and element-indexed actions fail with No cached AX state.
• After — the post-action snapshot verifies the action actually landed. Without it you can't tell a silent no-op from a real effect. The AX tree change (new value, new window, disappeared menu, disabled button, etc.) is your evidence that the action fired. If nothing changed, the action probably failed silently — say so, don't assume success.

This applies to pixel clicks too — re-snapshot after to confirm the click landed on the intended target.

Why window selection is the caller's job now

get_app_state used to pick a window for you via a max-area heuristic that returned the wrong surface on apps with large off-screen utility panels. Concrete reproducer: IINA's OpenSubtitles helper (600×432 off-screen) out-area'd the visible 320×240 player window, so get_app_state(pid) screenshot'd the invisible panel and clicks landed there silently. The new get_window_state(pid, window_id) makes the caller name the window explicitly — the driver validates that the window belongs to the pid and is on the current Space, then snapshots exactly what was asked for. Enumerate candidates via list_windows or read the windows array launch_app already returns.

Behavior matrix

Two orthogonal axes shape what the agent can do.

capture_mode → addressing mode

capture_mode	get_window_state returns	Use for actions
som (default)	tree + screenshot	element_index preferred; pixel fallback
ax	tree only (no PNG)	element_index only
vision	PNG only (no tree)	pixel only — see SCREENSHOT.md

vision was renamed from screenshot — the old name still decodes as a deprecated alias, so an on-disk "capture_mode": "screenshot" keeps working. Default is som so element_index clicks work the first time a user calls get_window_state; the other modes are opt-in when the caller specifically doesn't want one half of the work. Note the tool named screenshot is separate (raw PNG, no AX walk) and unrelated to the capture mode.

When a snapshot looks wrong (tiny screenshot / empty tree), check cua-driver get_config for capture_mode before anything else.

Pure-vision mode has its own caveats — Claude Code's vision pipeline downsamples dense text aggressively, so pixel grounding takes multiple correction cycles on text-heavy UIs. Read SCREENSHOT.md before driving anything in that mode; it documents the iterate/annotate/verify recipe plus the JPEG-over-PNG finding.

Window state → what works

state	get_window_state	click/set_value (AX)	press_key commit (Return/Space/Tab)	pixel click
frontmost	✅	✅	✅	✅
backgrounded / visible	✅	✅	✅	✅
minimized (Dock genie)	✅	✅ (no deminiaturize — AX actions fire on the minimized window in place)	❌ silent no-op / system beep — use set_value or click equivalent	❌ no on-screen bounds
hidden (hides=true / NSApp.hide)	✅	✅	depends	❌
on another Space	⚠️ AX tree often stripped to menu-bar-only on SwiftUI apps (System Settings) — AppKit apps usually fine. Response carries off_space: true + window_space_ids so you can detect it	✅	✅	❌ window not in current-Space list

Critical cell — minimized + keyboard commit. The keystroke reaches the app but AX focus doesn't propagate to renderer focus on a minimized window. Workarounds in order of preference: set_value to write the field's entire value directly, or AX-click a commit-equivalent button (Go, Submit, checkbox). Tell the user the window needs to un-minimize only as a last resort.

The canonical loop

launch_app(target)
  → pick window_id from the returned `windows` array
    (or call list_windows(pid) separately)
  → get_window_state(pid, window_id)
    → [act]  # every action also takes (pid, window_id)
  → get_window_state(pid, window_id) → verify

launch_app now returns a windows array alongside the pid, so the common case collapses to two calls (launch_app → get_window_state) without a separate list_windows hop.

1. Resolve target pid — always via launch_app

Always start with launch_app, whether or not the target is already running. It's idempotent (relaunching returns the existing pid with no side effects) and gives you the pid in one call — no list_apps hop.

• launch_app({bundle_id: "com.apple.finder"}) — preferred, unambiguous.
• launch_app({name: "Calculator"}) — when bundle_id isn't known.

launch_app is a hidden-launch primitive by design — that's the entire point of cua-driver: agents drive apps in the background while the user keeps typing in their real foreground app. The target's window is initialized (AX tree fully populated, clickable via element_index, the pid appears in list_apps) but not drawn on screen. The driver never activates or unhides apps on its own; that would violate the no-foreground contract the whole driver exists to protect.

If the user explicitly wants the window visible (usually for a demo or recording), they unhide it themselves — Dock click, Cmd-Tab, or Spotlight. Do not reach for open / osascript activate as a shortcut to make the window visible; those paths break the backgrounded invariant on every call, not just the call that "needed" the foreground. Say out loud what the user needs to do ("click the Todo app in your Dock to bring it forward") and let them do it.

Never shell out to any form of open (including open <path-to-App.app> for a just-built binary — resolve the bundle id from Info.plist and use launch_app with that), osascript 'tell app … to launch/open', or similar. Those paths activate the target, bypass the driver's focus-restore guard, and require a Bash permission prompt the agent loop shouldn't be burning on app launch. See "Prefer cua-driver tools over shell shims" above for the full intent → tool mapping.

list_apps is for app-level discovery (answering "what's installed / running / frontmost?") — not part of the core action loop. Skip it in the loop. For window-level questions — "does this app have a visible window?", "which Space is this window on?", "which of this pid's windows is the main one?" — call list_windows instead; the app record doesn't carry window state on purpose. In the common single-window case you can skip list_windows entirely and read the windows array that launch_app already returned.

2. Snapshot and act by element_index