test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) by thymikee · Pull Request #1793 · callstack/agent-device

thymikee · 2026-08-17T17:40:06Z

Summary

#1781 A1. The nightly Android job (Android Full Emulator Suite) has failed every scheduled run since the full tier landed, always on the same assertion:

AssertionError [ERR_ASSERTION]: step: request Android microphone permission for deny
command: agent-device click id="automation-request-microphone" ...
"message": "Selector did not match: id=\"automation-request-microphone\"",
scenario: full:lifecycle-system

Root cause

settings permission reset microphone maps to pm revoke (documented in commands.md), and Android kills the app process when a granted runtime permission is revoked. The accept round grants the permission; the reset that opens the deny round therefore terminates the fixture, and the click lands on the launcher.

Proven on the image the lane pins (android-36 google_apis_playstore, Pixel 7 profile):

--- reset (revoke) while NOT granted
pid: 4259                       # app survives
--- grant / revoke while GRANTED
pid after revoke:               # empty — process gone
topResumedActivity=ActivityRecord{... com.google.android.apps.nexuslauncher/.NexusLauncherActivity}

The nightly artifacts agree: in run 31990896404 the scroll bottom immediately before the failing click reports "summary": "Already at bottom", "passes": 0 (the launcher is not scrollable), while the same step in the accept round scrolled 1440px.

Why #1610 did not fix it

#1610 fixed a different defect in the same helper — a one-shot alert get racing the asynchronous prompt — by switching to alert wait 10000. That made the accept round reach its prompt, which moved the failure one step later, onto the revoke-kill. Before #1610 the job failed at {"alert":null,"message":"No alert visible"} (run 30974032723); from 2026-08-06 onwards it fails at the deny click (runs 31070190261, 31924041273, 31990896404).

Why there was more than one defect

full:lifecycle-system (added in #1484) and full:observability-artifacts have never executed end to end — #1484 explicitly deferred to the smoke tier for device validation, and the nightly has died inside the first of them every night since. Running them live surfaced the whole chain:

#	Defect	Symptom
1	reset revokes → app killed	`Selector did not match: id="automation-request-microphone"`
2	post-revoke readback used `home` + `open`	cold start lands on the tabs home, `wait "Automation lab"` times out
3	relaunched Automation lab starts above the controls	`get text id="automation-microphone-permission"` finds nothing (Android snapshots are on-screen only)
4	Automation lab is a root route without the tab bar	`click label="Form"` finds nothing
5	every `open` re-activates the test IME	`keyboard status` returns `visible:false`, `inputMethodPackage: …imehelper`
6	`scroll bottom` overshoots the IME diagnostic	`wait "Android fill input was captured…"` times out
7	`scroll down 0.7` overshoots the quick actions on pixel_7	`press id="home-open-catalog"` finds nothing
8	batch targets below the fold / not co-visible	`get text id="dismiss-notice"` finds nothing
9	event page (4) smaller than the events each page read appends	`events pagination exceeded 100 pages`

What this changes

live-lifecycle-scenario.ts: one openAutomationLab helper relaunches the deep-link route and reveals its controls; every permission reset and the post-revoke readback go through it. The IME section closes the session (the only thing that restores the previous IME), reopens the tabs root with --no-test-ime, and reveals the diagnostic with a bounded scroll.
live-observability-scenario.ts: reveal distances corrected for the pinned pixel_7 profile, the batch's is target changed to an element co-visible with dismiss-notice, and the event page size raised above the tail each page read appends.
examples/test-app/src/screens/CatalogScreen.tsx: stickyHeaderIndices pins the scroll-state
canary so Android can read it at any scroll offset.
examples/test-app/replays/gesture-lab-android.ad and
test/integration/replays/android/fixture/01-navigation-scroll.ad: one selector and the
multi-pointer start points re-derived from the live fixture on the pinned profile.

No production code is touched.

Validation

Live, on a local Pixel_7_CI AVD (API 36, google_apis_playstore, same profile/API as the lane), with the fixture APK and the packaged snapshot/IME helpers:

Red before: AGENT_DEVICE_ANDROID_E2E_SCENARIOS=full:lifecycle-system reproduced the nightly assertion verbatim.
Green after: the same filtered run passes (pass 1, full:lifecycle-system: 38241ms).
Full tier, unfiltered, in lane order: bootstrap → inventory → automation-system → form-input → keyboard-ime → capture-close → lifecycle-system → observability-artifacts all pass.

pnpm check:quick, pnpm format:check, and node --test test/integration/smoke-android-emulator-coverage.test.ts (10/10) pass.

Scope

Six files: live-lifecycle-scenario.ts, live-observability-scenario.ts, live-form-scenario.ts
(one constant exported), the 01-navigation-scroll.ad and gesture-lab-android.ad fixtures, and
CatalogScreen.tsx in the fixture app. No production code.

Full-tier evidence

Hosted, on the lane's own image — run 32107665052, a workflow_dispatch of Replay Nightly on this exact head. All nine scenarios pass with no retries:

Android emulator live run: 324515ms
  smoke:fixture-bootstrap 13044ms   smoke:inventory 2972ms
  smoke:automation-system 83799ms   smoke:form-input 28573ms
  smoke:keyboard-ime 21224ms        smoke:capture-close 8770ms
  full:lifecycle-system 71423ms     full:observability-artifacts 26642ms
  full:fixture-replays 66515ms
✔ live Android emulator fixture E2E (324516ms)

That job still ends red on a separate suite that runs after the E2E — pnpm gate replay-android, 4 passed (8), 4 failed, every failure carrying agent-device's own "a system surface … covers the app" diagnosis (Silent, Collapse, Android System). It starts on two Settings-app replays this PR does not touch, clears again three files later, and the gate has not executed since 2026-07-30 because the workflow's set -eu script never reached it while the E2E was failing. Detail in this comment; deliberately left to a follow-up rather than widening this PR into a third suite.

Local reproduction

The lane's own command passes end to end on a Pixel 7 / API 36 AVD (the profile and image
replays-manual.yml pins), against a CI-equivalent fixture APK — cached native plus head JS
through the same repack-android-apk.sh the workflow runs:

Android emulator live run: 153432ms
  smoke:fixture-bootstrap: 1735ms      smoke:inventory: 1140ms
  smoke:automation-system: 27585ms     smoke:form-input: 10205ms
  smoke:keyboard-ime: 6503ms           smoke:capture-close: 3673ms
  full:lifecycle-system: 33443ms       full:observability-artifacts: 21130ms
  full:fixture-replays: 46607ms
✔ live Android emulator fixture E2E   pass 1  fail 0

Getting there needed three more repairs, each a drift the lane had never executed far enough to
see:

01-navigation-scroll.ad clicked label="Catalog, 0 new notifications". fix: restore scheduled workflow health #1543 made the cart
badge conditional, so the live label is plain Catalog — which the iOS twin of this file
already used.
The catalog scroll canary sits inside the scrolling content, and Android snapshots carry
on-screen nodes only, so every state except the initial top was unobservable — wait "Catalog scroll: down|bottom|up" could not pass at any swipe coordinates. Rather than tune
the .ad around a canary that scrolls away, stickyHeaderIndices pins that one line, so all
four states are readable at any offset on both platforms.
gesture-lab-android.ad started its multi-pointer gestures at y=1040 — inside the target
when the file was last repaired (fix(test): repair Android replay fixtures against live device reality #1538), but 90px from its top edge after feat: add selector-targeted drag gestures #1567 moved the
card (targets now span y=949-1525), so the second pointer landed outside the view and the
gesture read as a no-op. Multi-pointer gestures now start at the target centre, and the
header comment records the geometry they depend on. Isolated on device: (220,1040) fails at
distance 180 and 500, (300,1237) passes at both — position, not distance.

One flake seen once in three full runs: record stop reported Android screenrecord ownership could not be confirmed for pid … in full:observability-artifacts. It passed on the scenario
re-run and on the full re-run; it is a step this PR does not touch, so it is noted rather than
chased.

CI note: the Coverage red was not this branch (input for #1781 A4)

Coverage failed three times on this PR with no failed test — Error: [vitest-pool]: Worker forks emitted error / Caused by: Error: Worker exited unexpectedly, last file logged before the ~36s gap always subprocess-stub scripts/fuzz/corpus-replay.test.ts, summary Test Files 942 passed (943). It passes on re-run (job 95465796922). What the evidence says:

Not the branch state. git diff --stat origin/main...HEAD is six files — four scenario/harness files under test/integration/android-emulator-e2e/, one .ad replay fixture, and the fixture app screen — with origin/main an ancestor of HEAD and no lockfile or config change. (At the time this note was written the diff was the three scenario files; the fixture repairs below were added after.) And no Vitest project includes that directory (vitest.config.ts: unit-core, subprocess-stub, provider-integration, interaction-contract, output-economy) — these scenarios run under node --test. The Coverage job executes the same 943 files with or without this diff.
Not reproducible locally. pnpm test:coverage:ci (the exact CI script) twice on this branch: run 1 exit 1 on an unrelated CPU-contention timeout (runner-client.test.ts > ensureXctestrunArtifact aborts only the disconnected request build, Test timed out in 5000ms) with zero Worker exited unexpectedly; run 2 fully green, Test Files 943 passed (943), exit 0.
The same signature is hitting other branches. Same afternoon, identical last-logged file and error: claude/agent-device-issue-1783-9dcc8e (job 95439415417, 16:58) and claude/agent-device-1774-67603d (jobs 95428461468 at 16:07 and 95421425175 at 15:14). 95428461468 has this PR's exact shape — no failed test, one file unaccounted for (940 passed (941)).

So it is (b): a subprocess-stub fork being killed under runner load. Two properties make it a job-level red rather than a blip: that project runs fileParallelism: false, maxWorkers: 1, isolate: true while its tests spawn real processes, and a killed fork surfaces as an unhandled error, which scripts/lib/contention-retry.ts deliberately refuses to retry ("No retry: the run failed for a reason a rerun cannot re-check"). A human re-run then passes, which is the worst shape: the policy that exists to keep contention honest converts a killed worker into a red that only manual intervention clears. Handing that to #1781 A4 (the subprocess-stub / contention-retry decision) rather than widening the retry policy from this PR.

How to verify

Trigger the workflow manually on this branch and watch the Android job:

gh workflow run replays-nightly.yml --ref test/1781-a1-replays-nightly-android

Or locally, against a booted pixel_7/API 36 AVD:

AGENT_DEVICE_ANDROID_E2E=1 AGENT_DEVICE_ANDROID_E2E_TIER=full \
AGENT_DEVICE_ANDROID_E2E_SCENARIOS=full:lifecycle-system,full:observability-artifacts \
AGENT_DEVICE_ANDROID_SERIAL=<serial> \
AGENT_DEVICE_FIXTURE_APP_PATH=examples/test-app/android/app/build/outputs/apk/release/app-release.apk \
AGENT_DEVICE_FIXTURE_APP_ID=com.callstack.agentdevicelab \
node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-android-emulator.test.ts

Draft because the lane cannot go green until the replay fixture above is decided.

github-actions · 2026-08-17T17:40:59Z

Size Report

Metric	Base	Current
JS raw	2.26 MB	2.26 MB
JS gzip	744.4 kB	744.4 kB
npm tarball	863.7 kB	863.7 kB
npm unpacked	3.01 MB	3.01 MB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	25.2 ms	25.6 ms	+0.4 ms
CLI --help	62.6 ms	63.2 ms	+0.5 ms

Top changed chunks: no changes in the largest emitted chunks.

#1781 A1) The nightly Android job has failed on `click id="automation-request-microphone"` since the full tier landed: `settings permission reset microphone` runs `pm revoke`, and revoking a *granted* runtime permission kills the app process, so the round after an accept clicked into an empty launcher surface. Reproduced on the pinned CI image (android-36 google_apis_playstore, Pixel 7): pm revoke leaves pid 4259 alive when the permission is denied and kills it when it is granted, with NexusLauncher resumed afterwards. Fixing that exposed the rest of a scenario that had never executed end to end: the post-revoke readback cold-started on the tabs home instead of Automation lab, the relaunched Automation lab needed its controls revealed, the Form tab does not exist on the Automation root route, that section needs the system IME back, and the IME diagnostic sits above the bottom of the form. The observability scenario then failed the same way (reveal distances tuned for a taller device) plus an event-timeline walk whose page was smaller than the events each page read appends. Validated live against a local Pixel_7_CI emulator (API 36, same profile as the lane): the full tier now runs bootstrap -> inventory -> automation-system -> form-input -> keyboard-ime -> capture-close -> lifecycle-system -> observability-artifacts and stops only in full:fixture-replays.

thymikee · 2026-08-17T18:13:50Z

Not ready yet.

The nested batch regression is still not independent: the get step resolves dismiss-notice, while the is step checks release-notice, its parent. A successful child lookup already establishes that parent is present, so this cannot independently prove the nested is operation. Please use an independently co-visible target (for example gesture-lab-card), after confirming the pinned Pixel 7 snapshot.

Before this can address #1781 A1, please also provide a successful exact-head Android Full Emulator Suite run (or equivalent uploaded full-tier artifact/evidence). full:fixture-replays remains red, so the linked A1 fix-or-disable sequence is incomplete until its fix or the separate disabling change is ready.

Coverage is red due to an unhandled Vitest worker-fork exit and needs a clean rerun. Bundle Size is a GitHub 503 while posting its report, so I consider that infrastructure rather than a source finding.

… canary (#1781 A1) Review follow-up. The nested batch regression now checks a sibling card instead of the notice that owns `dismiss-notice`: resolving a child already proves its parent is present, so the old target could not fail on its own. Confirmed on a Pixel 7 / API 36 emulator that `gesture-lab-card` and `dismiss-notice` are on screen together at the scenario's existing 0.3 reveal (both present at 0.2-0.4; the card is gone by 0.5). Getting a full-tier run to complete then required repairing what the lane had never executed: - `01-navigation-scroll.ad` clicked `label="Catalog, 0 new notifications"`. #1543 made the cart badge conditional, so the live label is `Catalog` — what the iOS twin already used. - The catalog scroll canary lives inside the scrolling content, and Android accessibility snapshots carry on-screen nodes only, so every state except the initial `top` was unobservable: `wait "Catalog scroll: down|bottom|up"` could never pass, whatever the swipe coordinates were. `stickyHeaderIndices` pins that one line, which makes all four states readable at any offset on both platforms rather than tuning the .ad around a canary that scrolls away. - `gesture-lab-android.ad` started its multi-pointer gestures at y=1040, inside the target when the file was last repaired but 90px from its top edge after #1567 moved the card (targets now span y=949-1525). The second pointer landed outside the view, which reads as "the gesture did nothing". Multi-pointer gestures now start at the target centre, and the header comment records the geometry they depend on. Evidence: the lane's own command (`AGENT_DEVICE_ANDROID_E2E_TIER=full` over smoke-android-emulator.test.ts) passes end to end on a Pixel 7 / API 36 AVD with a CI-equivalent fixture APK (cached native + head JS through the same repack the workflow runs): 9/9 scenarios, 153s.

thymikee · 2026-08-18T06:20:29Z

All three addressed; the branch now carries a green full-tier run.

Independent batch target. You were right that a child lookup already proves its parent, so release-notice could not fail on its own. The is step now checks gesture-lab-card, a sibling card. Confirmed on a Pixel 7 / API 36 AVD (my own Pixel_7_a1, not the one you have booted) that it is co-visible with dismiss-notice at the scenario's existing scroll down 0.3 — both present at 0.2/0.3/0.4, the card gone by 0.5 — so the reveal distance did not need to move.

Exact-head full-tier evidence. full:fixture-replays is fixed rather than left red, and the lane's own command now passes end to end:

Android emulator live run: 153432ms
  smoke:fixture-bootstrap 1735ms   smoke:inventory 1140ms
  smoke:automation-system 27585ms  smoke:form-input 10205ms
  smoke:keyboard-ime 6503ms        smoke:capture-close 3673ms
  full:lifecycle-system 33443ms    full:observability-artifacts 21130ms
  full:fixture-replays 46607ms
✔ live Android emulator fixture E2E   pass 1  fail 0

Run against a CI-equivalent fixture APK: the cached native build plus head JS through the same repack-android-apk.sh the workflow runs for Android artifacts, so the JS under test is this branch's examples/test-app source.

Three further drifts had to be repaired to get there, all of them invisible until the scenario ran that far:

01-navigation-scroll.ad clicked label="Catalog, 0 new notifications"; fix: restore scheduled workflow health #1543 made the badge conditional, so the live label is plain Catalog — what the iOS twin already used.
The catalog scroll canary lives inside the scrolling content, and Android snapshots carry on-screen nodes only, so every state except the initial top was unobservable — wait "Catalog scroll: down|bottom|up" could not pass at any swipe coordinates. stickyHeaderIndices pins that one line instead, so all four states are readable at any offset on both platforms.
gesture-lab-android.ad started its multi-pointer gestures at y=1040 — fine when fix(test): repair Android replay fixtures against live device reality #1538 repaired it, 90px from the target's top edge after feat: add selector-targeted drag gestures #1567 moved the card (targets now span y=949-1525), so the second pointer landed outside the view and the gesture read as a no-op. Isolated on device: (220,1040) fails at distance 180 and 500, (300,1237) passes at both, so it is position, not distance. Multi-pointer gestures now start at the target centre and the header comment records the geometry.

One honest caveat: in three full runs, record stop once reported Android screenrecord ownership could not be confirmed for pid … in full:observability-artifacts. It passed on both the scenario-only re-run and the next full run; it is a step this PR does not touch, so I noted it rather than chasing it.

Coverage. Green on the current head after a clean re-run. The PR body keeps the analysis: three failures with no failed test, Worker exited unexpectedly after subprocess-stub scripts/fuzz/corpus-replay.test.ts, the same signature on three other branches the same afternoon, no local repro in two pnpm test:coverage:ci runs — and contention-retry.ts classifying a killed fork as non-retryable is what turns it into a red that only a manual re-run clears. That belongs to #1781 A4 rather than this PR. Bundle Size was the 503 you identified and passes on re-run.

With full:fixture-replays fixed, the A1 sequence no longer depends on the disabling change; #1794 remains the fallback if you would rather park the lane than carry these fixture repairs.

thymikee · 2026-08-18T06:28:48Z

Re-reviewed exact head 8d36ad8. Clean code/readiness verdict: the batch assertion now targets an independently observable sibling, the full-tier production CLI/daemon/device route exercises it, and the documented Pixel 7/API 36 exact-head run completed all 9 scenarios. Exact-head deterministic checks and Android smoke are green; iOS Smoke is still pending and no hosted Replay Nightly dispatch exists for this branch, which remains residual evidence risk. Minor body cleanup: scope now spans 6 files, not the stated 3. Ready for human review.

thymikee · 2026-08-18T06:52:52Z

Hosted dispatch done — that retires the "no hosted Replay Nightly dispatch for this branch" residual: run 32107665052 on this exact head.

The full tier passed on the hosted emulator, all nine scenarios, no retries:

Android emulator live run: 324515ms
  smoke:fixture-bootstrap 13044ms   smoke:inventory 2972ms
  smoke:automation-system 83799ms   smoke:form-input 28573ms
  smoke:keyboard-ime 21224ms        smoke:capture-close 8770ms
  full:lifecycle-system 71423ms     full:observability-artifacts 26642ms
  full:fixture-replays 66515ms
✔ live Android emulator fixture E2E (324516ms)

That is the same result as the local Pixel 7 run, on the lane's own pixel_7 / API 36 image with the fixture app the workflow builds (artifact + JS repack), so the exact-head evidence no longer rests on my machine.

The job still ends red, and not on this PR's path. After the E2E the script runs a separate suite, pnpm gate replay-android (test/integration/replays/android), which finished 4 passed (8), 4 failed. All four failures carry agent-device's own system-surface diagnosis:

⨯ 05-app-lifecycle.ad   step 4 (find "text" "Search" "exists"): find did not match any element
⨯ 06-swipe-gestures.ad  step 3 (find "text" "Search" "exists"): find did not match any element
⨯ 02-selector-routes-covered-diagnosis.ad  step 2: wait timed out for selector: id="field-name".
     Current surface: Silent, Collapse, Android System, •
⨯ 01-navigation-scroll.ad step 2: wait timed out for text: Agent Device Tester
   hint: A system surface (notification shade, quick settings, or another system overlay) covers
         the app … (all four)

Why I read that as not attributable to this branch:

The surface it names is the volume panel plus notification shade (Silent, Collapse, Android System), not a selector mismatch.
It starts at 05-app-lifecycle.ad, after 03-scroll-discovery.ad passed, and clears again by 04-text-input-keyboard.ad / 01-settings.ad / 02-deep-navigation.ad — a device-state window, not a per-file content problem, and it survived all three retries inside that window.
The two files where it starts drive the system Settings app (open settings --relaunch, find text "Search"), which this PR does not touch. My edit to 01-navigation-scroll.ad is at step 3; it never reached step 3, failing at step 2's wait "Agent Device Tester" under the same overlay.
This gate has not executed since 2026-07-30: the workflow's script: runs under set -eu, so once the full-tier E2E started failing on 07-31 the gate was never reached. Its drift is newly visible because the E2E now completes — the same shape as everything else this PR uncovered, one suite further along.

I have left it alone rather than widen this PR into a third suite (system-app replays plus a system-surface recovery step are their own change, and the E2E evidence you asked for is complete without them). Happy to take it as a follow-up if you would rather have the whole job green in one go — say the word and I will pick it up in a separate PR.

github-actions · 2026-08-18T07:58:34Z

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-18 07:58 UTC

thymikee force-pushed the test/1781-a1-replays-nightly-android branch from def917d to 99d209e Compare August 17, 2026 18:03

thymikee marked this pull request as ready for review August 17, 2026 18:46

thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Aug 18, 2026

thymikee merged commit e3f3a24 into main Aug 18, 2026
29 of 31 checks passed

thymikee deleted the test/1781-a1-replays-nightly-android branch August 18, 2026 07:57

thymikee mentioned this pull request Aug 18, 2026

fix(scroll): keep saturated scroll gestures out of the status bar; gate Android replays from android/emulator (#1781 A1) #1820

Merged

Navigation Menu

Uh oh!

Uh oh!

test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) - #1793

test(android): restore full-tier lifecycle and observability scenarios (#1781 A1)#1793
thymikee merged 2 commits into
mainfrom
test/1781-a1-replays-nightly-android

thymikee commented Aug 17, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 17, 2026 •

edited

Loading

Uh oh!

thymikee commented Aug 17, 2026

Uh oh!

thymikee commented Aug 18, 2026

Uh oh!

thymikee commented Aug 18, 2026

Uh oh!

thymikee commented Aug 18, 2026

Uh oh!

Uh oh!

github-actions Bot commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

thymikee commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause

Why #1610 did not fix it

Why there was more than one defect

What this changes

Validation

Scope

Full-tier evidence

Local reproduction

CI note: the Coverage red was not this branch (input for #1781 A4)

How to verify

Uh oh!

github-actions Bot commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Size Report

Uh oh!

thymikee commented Aug 17, 2026

Uh oh!

thymikee commented Aug 18, 2026

Uh oh!

thymikee commented Aug 18, 2026

Uh oh!

thymikee commented Aug 18, 2026

Uh oh!

Uh oh!

github-actions Bot commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

thymikee commented Aug 17, 2026 •

edited

Loading

github-actions Bot commented Aug 17, 2026 •

edited

Loading