[stable] Backport: Retry e2e tests once in CI, keeping retried tests visible (#3530) - #3555
Draft
alangenfeld wants to merge 1 commit into
Draft
[stable] Backport: Retry e2e tests once in CI, keeping retried tests visible (#3530)#3555alangenfeld wants to merge 1 commit into
alangenfeld wants to merge 1 commit into
Conversation
* [ci] Retry e2e tests once in CI, keeping retried tests visible Over the last 10 days ~93 Tests runs were manually re-run until green, some taking 6 attempts: the e2e suites drive real deployments, and a single test losing a timing race fails a whole 20+ minute matrix job. A CI-only vitest retry (retry: 1) absorbs those single-test races. beforeEach/afterEach hooks run per attempt, so suites with file-restore hooks (dev.test.ts) retry cleanly. Retries must not hide real races, so a retried-then-passed test stays visible everywhere a failure would have been: the github-reporter emits ::warning annotations and an e2e-flaky-*.json sidecar, every e2e job uploads it, and aggregate-e2e-results.js renders a 'Flaky E2E Tests (passed on retry)' section in both the per-job step summary and the PR comment, with per-app occurrence counts. Harnesses whose failures are themselves the signal pin retry: 0: event-log-race-repro (a pass runs the full configured budget) and benchmarks (a regression should not be papered over by a luckier second sample). Local runs keep retry at 0 so races reproduce while debugging. * Add changeset --------- (cherry picked from commit c041d3d) Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
🦋 Changeset detectedLatest commit: 423e7e2 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests🌍 Community Worlds (83 failed)turso (83 failed):
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Backport of #3530 to stable — its e2e matrix hits the same single-test timing races, where one lost race fails a 20+ minute job.
Test Plan
Existing e2e coverage runs in CI; the aggregation script's flaky section was checked against synthetic artifact directories.
Backport notes
Manual backport of c041d3d (#3530). The automated backport job (run) recommended backporting but failed at the push step: the workflow's
GITHUB_TOKENcannot push commits that modify.github/workflows/files (Resource not accessible by integration).Conflict resolutions, worth reviewing carefully:
vitest.config.ts— retry block merged onto stable's config (which keeps itsbenchmark.includesection and has noconfigDefaultsexclude).github/scripts/aggregate-e2e-results.js—renderFlakySectioncall placed before stable's "Details by Category" section (stable predates main's "E2E Test Summary" layout).github/workflows/tests.yml— took stable's version wholesale and re-applied thee2e-flaky-*.jsonupload paths by hand against stable's five upload blocks (stable's artifact names usecanary/stablesuffixes, notartifactSuffix)packages/core/e2e/benchmark.test.ts— hunk dropped; file does not exist on stable