mirror of
https://github.com/obra/superpowers.git
synced 2026-07-10 03:59:04 +08:00
77 lines
3.7 KiB
Markdown
77 lines
3.7 KiB
Markdown
# Driving a Desktop App (Computer Use)
|
|
|
|
Drive the live app through its accessibility tree, not screen-pixel guesses,
|
|
whenever an accessibility-driven tool is available. The worked example
|
|
throughout is macOS accessibility automation (an app-state dump plus
|
|
element-indexed click/type actions); the same dump-act-re-dump discipline
|
|
applies to any platform's accessibility layer.
|
|
|
|
## Dump, act, re-dump
|
|
|
|
Before touching anything, pull a full app-state dump — the accessibility
|
|
tree, not a screenshot. Read every element index and role off *that* dump;
|
|
never guess or reuse an index from a previous dump, since insertions and
|
|
removals renumber the tree.
|
|
|
|
```text
|
|
get_app_state {app}
|
|
click {app, element_index} # index/role read from the dump above
|
|
type_text {app, text}
|
|
get_app_state {app} # re-dump — did the field you predicted change?
|
|
```
|
|
|
|
Re-dump after every action, not just at the end. An action without a
|
|
following dump is a click you can't prove happened — you only have proof once
|
|
you've read the state back and it shows the change.
|
|
|
|
## Quote the observed state into the record
|
|
|
|
The evidence is the before → after value read from the dump, quoted directly
|
|
into the report or commit — not a description of the click. A counter that
|
|
should now read a higher page, a selection whose label changed after a
|
|
"next" action: put the literal *old value* and *new value* side by side so a
|
|
reader can re-run the same action and check for the same transition. "I
|
|
clicked the button" proves nothing; "field X read `A`, then `B`" is
|
|
falsifiable.
|
|
|
|
## Isolate before you drive
|
|
|
|
Copy the built app to a throwaway location under a distinct bundle
|
|
identifier and reset its permission grants before scripting it, so a driving
|
|
session can't corrupt the real app's session state or permissions. Build any
|
|
harness the driving needs outside the project's own repo — end-to-end
|
|
driving should never mutate the project under test.
|
|
|
|
## The escalation ladder
|
|
|
|
Accessibility automation on a real desktop is not always available cleanly.
|
|
Climb a ladder of approaches, and when a rung is blocked, record *why* before
|
|
trying the next one:
|
|
|
|
1. **Accessibility scripting** (on macOS, `osascript`/AppleScript) — the cheap
|
|
default. Blocked signature: a permission error before any command runs
|
|
(no Accessibility grant, e.g. `osascript` error `-1719`).
|
|
2. **UI-test harness** (on macOS, an XCUITest automation session) — the
|
|
"proper" way to drive the real app end to end. Blocked signature: the
|
|
harness process itself never establishes its automation session (an
|
|
unsigned test runner killed before it attaches) — that's the harness
|
|
failing to bootstrap, not a bug in the app under test.
|
|
3. **Raw input injection** (on macOS, a coordinate-clicking tool such as
|
|
`cliclick` plus `screencapture` after each action) — the fallback of last
|
|
resort when both of the above are blocked. Coarser than element-indexed
|
|
driving, so screenshot after every action and confirm the click landed on
|
|
the intended window before trusting the result.
|
|
|
|
Every rung you tried belongs in the report, including the ones that failed —
|
|
not only the one that worked. Diagnose each blocked rung enough to state the
|
|
failure cleanly (permission denied, session never attached, wrong window
|
|
frontmost) before moving on; a rung abandoned without a stated reason is
|
|
indistinguishable from one you never tried.
|
|
|
|
## A blocked ladder is a report, not an excuse
|
|
|
|
If every rung is blocked, that is the result: write down what you tried, what
|
|
each rung's failure looked like, and stop there. Never fall back to
|
|
describing what the UI "should" do, and never fabricate a dump or a
|
|
before/after value you didn't actually read back from the running app.
|