App Testing & QA

​

The basic pattern

Open my app "Acme Notes" on the test phone, go through the onboarding flow
as a new user, and report every screen you saw and anything that looked
broken. Ask me before granting any permissions.

• Name the app exactly as it appears on the device. Verify it is installed first with phone_get_state, which lists launchable apps with label and package name.
• “Ask me before granting any permissions” turns permission dialogs into needs_user_control handoffs instead of silent grants.
• Ask for a structured report - “for each screen: name, what you did, what happened, anything unexpected” - so results drop straight into your bug tracker.

​

Sizing test runs

• Raise per_call_timeout_s (default 300 s, max 900) for genuinely long walks.
• Split into several tasks in one session - one per flow (onboarding, settings, checkout). A task that hits terminal: "iteration_limit" is the signal to break it up further. Splitting also means a crash in one flow doesn’t cost you the report from the previous flows.

Task 1: "Open Acme Notes, complete onboarding as a new user, report each screen."
Task 2: "Now create a note titled 'QA test', add a photo, and verify it saves."
Task 3: "Now open settings, toggle dark mode, and report any visual glitches."

​

Watching the run

• Set include_steps: true on phone_task_monitor to pull the task’s recorded steps - screenshots and agent thinking - and since_step to fetch only new ones as the run progresses.
• Execution is visible on the device screen, so you can literally watch the test happen.
• Use phone_task_message to steer mid-run: “skip the newsletter dialog and continue.”

​

Repeatable runs

​

Related