ci: move parked device replay suites to a dispatch-only workflow (#1781 A1) - #1794
ci: move parked device replay suites to a dispatch-only workflow (#1781 A1)#1794thymikee wants to merge 1 commit into
Conversation
Size Report
Startup median (7 runs, lower is better):
Top changed chunks: no changes in the largest emitted chunks. |
… A1)
Both full-tier device jobs have failed every scheduled run since 2026-07-24: the
Android suite inside full-tier scenarios that had never executed end to end, the
iOS suite on varying steps. They move to .github/workflows/replays-manual.yml,
which has no `schedule:`, so the schedule stops emitting a guaranteed failure while
the suites stay runnable on demand.
A job-level `if: github.event_name == 'workflow_dispatch'` would have looked the
same and lied: `workflowLanes()` decides `qualifying` per workflow FILE and never
reads job-level `if:`, so the manifest kept reporting replay-android, replay-ios,
and replay-ios-device as scheduled-lane owners — the silent-owner-loss failure the
manifest exists to catch. A separate file is what the file-level model already
reads correctly.
Those three checks now have no pull_request/schedule owner, so they are declared as
MANUAL_ONLY_OWNERS rather than folded into UNPROVABLE_OWNERS, whose claim ("it runs,
this loader cannot see it") is no longer true for replay-android. check:gate-manifest
drops from 48 to 46 wired checks and names the three on every run. Two tests pin it:
a dispatch-only lane is non-qualifying however many gates it declares, and every
manual-only declaration must name a registered check that no qualifying lane owns, so
a re-scheduled lane cannot keep a stale exemption.
a6e8c6c to
2198ff5
Compare
thymikee
changed the title
|
P1: MANUAL_ONLY_OWNERS is currently only a negative allowlist. The new test proves each entry is registered and has no qualifying owner, but it never proves a workflow_dispatch lane still declares and can run that check. Deleting a parked manual job, or its run-gate step, would therefore leave check:gate-manifest green and continue to print the check as manual-only: parked coverage has silently become deleted coverage. Please derive/attest each manual-only declaration from a dispatch-only lane and add a regression that deleting its declaration fails. Android needs an explicit attestation too: replay-android remains inside the opaque third-party action, so the existing loader cannot supply it by itself. The loss of scheduled replay coverage itself matches #1781 A1 (manual means off). The current Bundle Size and CodeQL failures are GitHub 503 infrastructure failures; the wording/count issue in the manifest output is non-blocking. |
|
CI status on this head: 25 pass, 2 fail. Both failures are |
1 participant
Summary
#1781A1 fallback.Replay Nightlyhas failed every scheduled run since 2026-07-24. Two jobs are responsible and neither has an owner:full:lifecycle-system, a scenario that had never executed end to end. Diagnosis and a validated fix for it (and forfull:observability-artifacts) are in test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) #1793; even with that, the last scenariofull:fixture-replaysstill fails on a replay fixture whose scroll canary is authored for a taller device than thepixel_7profile the lane pins.Both jobs move, byte-for-byte, into
.github/workflows/replays-manual.yml, which has noschedule:. The parser fuzz lane keeps its nightly schedule inreplays-nightly.ymland is untouched; the macOS job is already gone (#1787). Nothing about how the suites run changes — dispatch them with:Why a separate file rather than
if: github.event_name == 'workflow_dispatch'The first revision of this PR used a job-level
if:. It would have parked the jobs and lied about it:workflowLanes()(scripts/gate/workflows.ts) computesqualifying = 'pull_request' in on || 'schedule' in ononce per workflow file and never reads job-levelif:. Sopnpm check:gate-manifestkept printingwhile
replay-android,replay-ios, andreplay-ios-device— declared only in those two jobs — no longer ran on any schedule. That is exactly the failure the manifest exists to catch: "a check that silently loses its owner looks exactly like a green build" (docs/agents/testing.md). A separateworkflow_dispatch-only file is what the existing file-level model already reads correctly.What that exposed, and how it is declared
With the jobs moved, the audit reports the truth:
These are declared in
scripts/gate/declarations.tsas a newMANUAL_ONLY_OWNERSrecord rather than being folded intoUNPROVABLE_OWNERS, because the two facts are opposites:UNPROVABLE_OWNERSclaims "the suite runs; this loader cannot see it" — which stopped being true forreplay-androidthe moment its lane stopped running automatically, so its stale prose moved with it.UNPROVABLE_OWNERSis now empty, with a comment saying why.MANUAL_ONLY_OWNERSsays "nothing runs this until someone dispatches it", with the reason and the exit condition per entry.check:gate-manifestnow reports:Named, not counted — the ids read like a list of things nothing runs, where "3 manual-only" would read like a tally.
Two tests pin the new path (
pnpm check:gate-manifest:test, 30/30):a workflow_dispatch-only lane owns nothing, however many gates it declares— plants a dispatch-only workflow declaringreplay-iosand asserts the gate is read but the lane is non-qualifying.every manual-only declaration names a registered check no qualifying lane owns— the ratchet in the other direction: putting a lane back on a schedule without deleting its exemption fails the suite (verified by plantinglayeringin the record: the test goes red).docs/agents/testing.mdno longer calls the iOS full tier "the scheduled/manual nightly workflow"; it namesReplay Manual, states that the three checks run only on dispatch, and points at the declaration.Validation
pnpm check:toolingpasses (format, lint, typecheck, layering, depgraph, gate-manifest + its tests, production-exports, tmpdir-leaks, mcp-metadata, build, bundle-owner-files, package). The moved job bodies are byte-identical to the originals — only the job ids changed (nightly-android/nightly-ios→manual-android/manual-ios); job names, steps, pins, env, and artifact names are unchanged.Notes for whoever picks these up
full:lifecycle-system/step: wait for Automation lab(run31990896404) — the same scenario family as the Android bug in test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) #1793, where a permission reset kills the app and the relaunch does not land on the expected route. That class is worth checking first. The iOS scenario lives intest/integration/ios-simulator-e2e/live-full-scenarios.ts, separate from the Android files test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) #1793 touches.Exit condition
Move the two jobs back into
replays-nightly.ymland delete theirMANUAL_ONLY_OWNERSentries once a dispatch run is green. The audit test fails if the entries outlive the parking, so this cannot quietly become permanent.