simframe

Eyes, hands and memory for an agent driving the iOS Simulator or an Android emulator.

Website · npm

An agent driving a simulator is slow for three reasons, and only the first one is obvious:

1. Every look is a wait. simctl io screenshot costs ~130 ms of blocking latency, paid again on every glance — and paid twice whenever the agent captures mid-animation and has to look again.
2. Every step is a round trip. Tap, screenshot, reason, tap, screenshot. A twelve-step flow costs twelve model turns, and the model turns cost far more than the milliseconds.
3. Nothing is remembered. The same screen gets re-read and re-reasoned about every single time it appears.

And there is a fourth that is pure waste: an image is the most expensive way to ask what is on screen. A screenshot costs ~1,600 tokens when it is handled as a native image block and 15,000–25,000 when it is not, and it does not tell you what is tappable or where — you have to measure that by eye.

simframe attacks all four: a background loop keeps the newest frame warm, whole flows run in one call, screens the agent has seen before are answered from memory, and every answer is text with tap points in it. Nothing returns an image unless you ask for one.

It is also a study case, and the numbers are the point

Almost everything here was decided by a measurement rather than by an argument, and several of those measurements reversed a decision that had already shipped. They are written down in full, with what we expected beforehand, what it cost to find out, and the mistakes made getting there — because a benchmark that only records the winner teaches nothing.

docs/BENCHMARKS.md	the record: N, median, p95, machine, every round
docs/EXPERIMENTS.md	what we believed before measuring, and the fourteen times it was wrong
docs/DEFERRED.md	every known defect, open or closed, with the evidence
docs/ARTICLE.md	the argument the whole thing adds up to

The headline results, all free to reuse:

• A warm transition graph removes ~27% of an agent's model round trips — isolated with a three-pass field experiment, and corroborated to three points by an independent estimate. Operator knowledge is worth twice that, and no tool here carries it yet. (§14)
• A bigger local judge is not a better one. qwen3:14b scored lower than qwen3:8b — 82% against 91% — while being 79% larger and 62% slower. A one-line threshold on a number already computed beat all of them at 95% and zero latency.
• Every interactive element came from the accessibility tree — 72 of 72 — and 83% of them have no text at all. Screen recognition, though, did not need the tree once. The ladder splits there: semantics need a tree, identity needs pixels.
• Capture is ~31× faster than a screenshot for the same work, and the capture primitive itself about a thousand times faster.
• Three independent reporters converged on one thing: a refusal is cheap and a confident wrong answer is expensive. Every serious bug they found was a component reporting more certainty than it had.

What changed, measured

Same four-tab navigation flow, on a real production app:

	Before	With simframe
Look at the screen	~130–400 ms, blocking	~20 ms, already captured

| "Did anything change?" | a full image | ~2 ms, text only | | Finding a control | read tree (~570 ms) + reason | ~1 ms from memory | | A 4-step flow, verified | 4+ model round trips | 1 call, 3.6 s | | Same flow, 3rd run | no improvement — every run is the first | 3.7 s, 4/4 from memory, 4/4 verified | | A 10-step flow | 10 turns, 10 images (~16,000 tokens at best) | 1 turn, 0 images, ~1,650 characters | | Reading a screen | an image: ~1,600 tokens, no tap points | ~330 tokens of text, with tap points |

Every figure above is the cost inside a live process — the MCP server, or the daemon answering a socket — which is how an agent actually uses simframe. A one-shot simframe command from a shell pays about 200 ms of Node startup on top, and a frame sitting on an idle screen can be older than 20 ms because the capture loop throttles when nothing moves. ~20 ms is the read, not the process.

The four-tab tour, three times back to back from a cleared memory:

Pass	Wall clock	Steps verified	Controls from memory
1	10.2 s	0/4 — nothing is known yet	4/4
2	3.6 s	4/4	4/4
3	3.7 s	4/4	4/4

Every step is checked against what the same action did last time, and the run records its own preconditions — which input path, which daemon, whether the daemon was replaced mid-run — so a regression shows up in the measurement rather than hiding inside it. Earlier versions of this table quoted 7.4 s → 3.3 s with verification switched off; those numbers were measured while input was silently falling back to a slower path and the capture daemon was being replaced by every command, so they measured two bugs rather than the tool.

Install

npm install -g simframe
simframe doctor

Whichever of those two commands you run first builds a small Swift daemon from source — including doctor, which is why a cold doctor takes around 15 seconds and every later one takes two. It needs the Xcode command line tools, which you already have if you have a simulator. Without them simframe falls back to the original simctl loop and says so.

If more than one simulator is booted, name the one you mean — --device=<udid>, or export SIMFRAME_DEVICE=<udid> once per shell. simframe refuses to choose for you, because the first booted device is nobody's idea of "yours" and the command that would act on it is a tap.

doctor checks each capability separately and tells you what you have:

ok   xcrun              xcrun version 72.
ok   sips               available
ok   input driver      simframed: Indigo HID
ok   accessibility tree simframed: AXPTranslator, host-side
ok   on-device OCR      available
ok   booted simulator   iPhone 17 Pro (iOS 26.5)
ok   capture            frame #888 322x700 in 2ms (age 538ms)
ok   sensor mode        full — accessibility and OCR fused on every read (~164ms)
ok   local supervisor   none — not requested (SIMFRAME_SUPERVISOR is unset)
ok   local planner      none — not requested (SIMFRAME_PLANNER is unset)

The last three are experiments and none is their normal answer. See Local tiers, off by default.

Claude Code

claude mcp add --scope user simframe -- npx -y simframe mcp

--scope user makes it available in every session; without it the server is registered only for the directory you ran the command in.

Any other MCP client

{
  "mcpServers": {
    "simframe": { "command": "npx", "args": ["-y", "simframe", "mcp"] }
  }
}

Recovering without a round trip

The measured cost of driving an app is not perception — warm, an accessibility-only read is 85 ms and a fused read 142 ms. It is round trips: in one instrumented run, 75% of the wall time was the agent thinking and the call boundary, not simframe working. So the tools that matter most are the ones that let a batch survive a problem instead of handing it back.

Fallback selectors. {"tap": "Save", "or": ["Done", "Confirm"]} — tried locally in order, only an exhausted list reaching the model. Eligible after a selector that did not resolve and nothing else, because retrying from a screen you did not expect to be on is a second guess. A destructive-looking label is refused as a substitute even if you list it.

{"seek": "change username", "budget": 6} opens containers, checks, and comes back, depth first, inside a hard budget. It acts — opening a door changes state — and it refuses to open anything that commits, abandons or answers. It does not tap the target; it leaves you on the screen where the target resolves.

{"sweep": "all", "fill": {…}} reads a long screen a viewport at a time and fills each field while it is on screen. A form taller than the screen is only knowable in pieces — the tree publishes what is rendered — and one scroll gesture travels a non-deterministic distance, so finding a field and scrolling back to it does not work. Sweeping does: on a real web form it filled every field in one call. It detects both ends by measuring how far the content moved, ignoring fixed chrome, which is the only reliable signal available since nothing reports a scroll offset.

worked here before: puts the graph's own vocabulary in the map, most-used first, rather than reporting a count. When the remembered controls are not on the screen it says so instead, because that means two screens share one fingerprint — and confident advice on a misidentified screen is how a remembered label ends up pointing at a submit button.

Local tiers, off by default

Two on-device model experiments, both none unless asked for, both degrading to the existing matcher-then-model ladder, and CI runs with both off. They ship no weights: Apple's Foundation Models framework has nothing to download, which is the whole reason it clears this project's non-goal on shipping model weights.

flag	what it does
--sensor=ax-first	read the accessibility tree alone (~50 ms) and pay for OCR only when a resolve fails
--planner=apple	order the containers seek opens; it cannot choose an action
--supervisor=apple	when a step fails, answer wait, retry or stop — nothing else — before the failure reaches the model

All three are also per-call arguments on every MCP tool, because an MCP server's environment is fixed when it spawns and comparing two modes inside one session was otherwise impossible.

What is measured and what is not. The ranker: 5 of 6 top-1 on hand-written cases, median 564 ms warm, and on a real exploration it went to the right region in two steps where reading order wandered into version strings. ax-first made no measurable difference to how an agent drove a real app, with one small regression and one small win.

The supervisor, on 22 labelled failures from a seeded React Native app — and the number to judge it by is round trips, not accuracy:

handled locally, no round trip	9 of 22
escalated to Claude	13 of 22
a model round trip, measured in the field	10–16 s

So roughly 90–145 s saved on that population. Accuracy was 77 / 82 / 86% across three runs of the same questions — it is not deterministic, and every earlier single-run figure in this project carried that spread without reporting it. All of its errors were wait where stop was right, which is the cheap direction: a wrong wait costs a settle and a re-run, and stop already means "hand back to Claude with the unattempted steps" rather than "give up".

Still a bench, not the field. These are fixtures we designed, on a testbed we built, labelled by the person who then scored them. Numbers and conditions are in docs/BENCHMARKS.md; the judgements, including a phase cancelled by its own measurement, are in docs/DECISIONS.md.

And the measurements that changed our minds are in docs/EXPERIMENTS.md, with what we expected beforehand written down beside each — including the capacity comparison the supervisor design was assumed to make unnecessary. Asked the same 22 situations, three times each:

arm	accuracy	median	deterministic
always the commonest answer	55%	—	—
stillMs > 3000ms, no model at all	95%	0 ms	yes
Apple Foundation Models (~3B, on-device)	77 / 82 / 86%	~640 ms	no
qwen3:8b via Ollama (4-bit, 5.2 GB)	91%	919 ms	yes
qwen3:14b via Ollama (4-bit, 9.3 GB)	82%	1,489 ms	yes

Three results, and the third is the one that matters. The larger model scored lower than the smaller one. A free threshold on a number the daemon already computes beat all three, on a population it was never fitted to. And the accuracy ranking inverts the safety ranking: every arm errs in one direction only, Apple's errors are all wait where stop was right and both Qwen arms' are all stop where wait was right — and a wrong stop abandons a plan that would have worked, while a wrong wait costs a settle. A comparison reporting only the percentages would have recommended the wrong model.

The Ollama arm is an experiment, not a recommendation: off unless named, no weights shipped, no dependency added, and every arm reads the same briefing out of native/supervise.swift so no arm is answering a different question.

Two further results, both negative, both worth more than the table. A cascade — free rule first, model where it is unsure, Claude after that — was worse at every abstention band we tried (95% for the rule alone; 91%, 86%, 82% as more was handed to the model). And giving the model a fourth word, abstain, cost the Apple arm about a third of its accuracy (77/82/86% → 45/50/55%) while it never used the word once. One added paragraph, nothing else changed. So the fourth word ships off, and "add an abstain token" became "find a judge that will use one".

The supervisor's whole vocabulary is three words on purpose. It cannot invent a step, skip one, substitute a target or continue past an unexpected screen — not because a threshold forbids it but because those are not answers it can give. That constraint replaced an earlier version of the same idea that was given latitude over what to open and pressed a button labelled "YES, THIS FIXED MY PROBLEM" in a live app.

Capabilities are independent

Each layer works without the ones above it, and doctor tells you which you have. Observation needs nothing but Xcode.

Capability	Needs	Without it
Watch the screen, wait, recall	nothing extra	—
Read labels + coordinates from pixels	swiftc (Xcode CLT)	falls back to the accessibility tree alone
Tap, type, swipe	nothing extra	simframe observes but cannot touch
Accessibility tree	nothing extra	OCR alone still yields labels and coordinates

simframe doctor names which engine is carrying each capability, per device. Nothing beyond Xcode is required. Capture, input, text recognition and the accessibility tree all run in-process, in one daemon.

idb is still accepted as a fallback for input and for the tree, for a machine where the daemon cannot run — and SIMFRAME_AX_DRIVER=idb forces the tree back onto it, which is the escape hatch if an Xcode upgrade breaks the host-side path.

# optional fallback, not a requirement
brew tap facebook/fb && brew install idb-companion && pipx install fb-idb

Android

simframe's second backend drives an Android emulator with the same commands, the same screen map and the same memory as a simulator. Everything above the platform boundary — the frame store, settle, the structural fingerprint, the screen map, refs and the transition graph — runs on it unmodified, because the boundary hands it frames and nothing above it knows what a simulator is.

Capability	Android	How
Watch the screen, wait, recall	yes	frames at 41 ms through the emulator console, host-side — no adb in the capture path
Read labels + coordinates from pixels	yes	the same Vision OCR + CV, off the same PNG
Screen map, refs, screen memory, the graph	yes	unchanged above the boundary
Tap, type, swipe, keys	yes	the console's event mouse as a real down/move/up, event text for characters, input keyevent for keys
Clipboard, and paste into a field	yes	the emulator's gRPC setClipboard, over node:http2, no dependency, then KEYCODE_PASTE to deliver it
List/resolve devices, launch, terminate, open a URL, permissions	yes	adb, with the permission state read back off the device
Accessibility tree	not available (OCR + CV only)	uiautomator dump costs 2,012 ms a read, against 45 ms for the iOS tree. See docs/DEFERRED.md
A launch confirmed to have reached the front	not available (the launch is not checked)	iOS compares the pid simctl launch printed against the pid the device reports as frontmost, in 2–5 ms. Nothing here reports either; am start fronts synchronously, which is why it has not bitten — but that is not a check. See docs/DEFERRED.md

# an emulator is found the same way a simulator is
simframe devices          # ● Small_Phone_API_36  Android 16 (API 36)  emulator-5554
simframe ui --device=emulator-5554
simframe do --device=emulator-5554 flow.json

A host with a booted simulator and a booted emulator has no default, and simframe will not pick one for you: preferring iOS because it came first would tap a simulator while you were driving an emulator, and acting on the wrong device is worse than refusing. So a command with no device names both and stops. --device answers it per command; SIMFRAME_DEVICE answers it per shell:

export SIMFRAME_DEVICE=emulator-5554
simframe ui               # the emulator, without saying so every time

The tree is a deliberate omission, not an oversight. Making it fast needs a resident instrumentation APK on the device — the shape uiautomator2, Maestro and Appium all converged on — and that would be simframe's first runtime artifact installed onto your device. The perception ladder was built so a missing tier degrades rather than fails, and this is exactly that case: OCR and CV yield labels and coordinates on Android today, and a tap by label works without a tree at all. simframe doctor reports the tier as optional with that number, so the gap is visible rather than silent, and the criteria for revisiting it are in docs/DEFERRED.md under Phase 8b.

What the missing tree costs, measured rather than hand-waved. Screen identity is weaker on Android than on iOS, and specifically so. Tokens per screen, and where they come from:

Screen	Tokens	Regions	Roles	Chrome labels
launcher	1	nav-bar 1	text 1	0
Settings root	9	content 9	text 9	0
example.com in Chrome	6	content 3, nav-bar 3	text 6	0

Every token has role text, because without a tree nothing infers a button from a rectangle reliably enough to say so, and no screen here carries a chrome label at all. So on Android a screen is recognised by the geometry of its text, which is thinner and noisier than the iOS mix of roles, chrome labels and geometry. Flows still work; screen memory is doing more guessing, and that is the honest cost of the tier being absent.

It is also why the obvious fix for the iOS drift — dropping content-region text out of identity, which would be a strict improvement there — is not available: it would leave Settings' root with zero tokens, and zero tokens is no identity at all. See docs/BENCHMARKS.md.

That last column read 3 before this measurement changed it. Chrome's address bar is chrome by every structural test there is, so the screen's identity contained "== example.com" — a URL, meaning the same browser on a different page was a different screen and every learned route through it broke on navigation — plus ":" and "+", which are OCR reading punctuation off icons. A chrome label now has to be a name: two letters at minimum, and not an address.

The emulator's own gRPC surface was checked for anything tree-shaped and has nothing: 43 RPCs for sensors, input, screenshots and VM state, and no notion of a view. That question is settled, not open. The same surface is what carries the clipboard.

The tools

Read first, act in batches, and look at pixels only when the question is about pixels. Every tool description says so, because a tool surface that does not steer the model is a tool surface the model uses wrong.

Tool	What it does
sim_ui	Start here. The screen as a numbered text map: region, type, label, state, tap point, source. A tenth the cost of a screenshot and strictly more useful.
sim_do	The main tool. A whole flow in one call — tap, type, scroll, wait, assert — each step settling before the next and verified against what it did last time.
sim_state	The cheapest question there is: has anything changed since your last look, and which regions moved.
sim_goto	Walk to a screen simframe has been to before, planning the route through remembered transitions.
sim_flow_run	Replay a flow that verified end to end.
sim_find	Resolve an intent to one control, without acting on it.
sim_tap · sim_type_into · sim_scroll_to · sim_wait_for · sim_assert	Single actions, for when you genuinely only have one step. Each is one sim_do step underneath.
sim_launch · sim_open_url · sim_permission	Launch with arguments and environment; open a deep link; grant a privacy permission instead of tapping a system alert.
sim_wait	Waits for the screen to change and then settle.
sim_look	The only tool that returns an image, capped at 1024 px. For layout, colour, spacing — questions text cannot answer.
sim_recall · sim_strip	Look backwards: a text timeline of what happened, or recent frames tiled into one image.
sim_storage	What the app believes, as opposed to what it drew: its UserDefaults and, for React Native, its AsyncStorage. Reads the data container off disk, so it answers on a device that is not running.
sim_capture · sim_devices	Manage capture loops; list simulators.

What the screen looks like as text

iPhone 17 Pro · 402x874pt · screen a1b2c3d4 "Inbox" (known, 3 known exits)
last action: [2] tap — ok: matches the outcome seen 5x before
nav-bar:
  #1 button    24,64      Back
  #2 text      201,64     Inbox
content:
  #3 cell      201,140    Weekly digest
  #4 field     201,196    Search = weekly ~ weekly|
  #5 switch    201,252    Notifications = 1
tab-bar:
  #6 text      62,835     Inbox
  #7 text      201,835    Settings

Region first, because "Inbox" the title and "Inbox" the tab differ only by where they are. A tap point, because that is what an action needs. And a number, which is a selector: whatever this calls #3, the next call can tap as #3 without describing it. A ref is valid only while that screen is showing — used on a different screen it refuses rather than tapping whatever now sits there.

= something is what the control contains, from the accessibility tree, and ~ something is what OCR read off the pixels. Both are printed, and where they disagree that is the point: one is authoritative and the other is what is actually on screen, and a field mid-edit can legitimately differ. A row with no = is a control that reports no value, not an empty one.