Skip to content

Fix Design Decision Gate false failures caused by the LLM invocation cap - #52836

Draft
pelikhan with Copilot wants to merge 2 commits into
mainfrom
copilot/deep-report-investigate-merge-blocking-failure
Draft

Fix Design Decision Gate false failures caused by the LLM invocation cap#52836
pelikhan with Copilot wants to merge 2 commits into
mainfrom
copilot/deep-report-investigate-merge-blocking-failure

Conversation

Copilot AI commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Three independent monitors flagged the merge-blocking Design Decision Gate 🏗️ workflow failing at a high rate while still burning real token spend — a Turns=0-style crash signature that turned out to be something else entirely.

Root cause

max-turns doubles as the AWF API proxy's hard per-run LLM invocation cap. This gate's ADR-review task typically needs ~20-22 turns to complete, but max-turns was set to 20 — so it reliably hit 429 Maximum LLM invocations exceeded (20/20). Worse, the agent had often already written a valid safe-output (e.g. an add_comment with the completed ADR review) before hitting the cap, but every harness (claude/codex/copilot) treated the cap error as an unconditional hard failure and discarded that completed work.

Changes

  • Increased max-turns from 20 → 30 in design-decision-gate.md (recompiled lock file) to give headroom above the observed completion point.
  • claude_harness.cjs, codex_harness.cjs, copilot_harness.cjs: when the invocation cap is saturated but safe-outputs already contain a terminal/expected result, exit 0 instead of failing — mirroring the existing suppression pattern already used for "numerous permission-denied" and "partial execution" cases.
  • Added harness tests covering the new suppression behavior for all three engines.
// Before: cap saturation was always fatal, even with completed work
if (nonRetryableGuard.maxRunsExceeded) return { action: "stop" }; // exit 1

// After: suppress the false-red when safe-outputs already hold the result
if (nonRetryableGuard.maxRunsExceeded && hasExpectedSafeOutputs(safeOutputsPath)) {
  return { action: "stop", exitCode: 0 };
}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Investigate Design Decision Gate merge-blocking failure Fix Design Decision Gate false failures caused by the LLM invocation cap Aug 15, 2026
Copilot AI requested a review from pelikhan August 15, 2026 05:39
@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

Category: bug | Risk: high | Score: 82/100

  • Impact: 35/50 (fixes merge-blocking Design Decision Gate false failures burning real token spend across the org)
  • Urgency: 30/30 (flagged by three independent monitors — actively impacting CI reliability)
  • Quality: 17/20 (no reviews yet, CI status unknown, but clear root-cause narrative)

Recommended action: fast_track
High-urgency CI reliability fix — actively blocking merges and burning token spend. Batched with reliability-fixes cluster (#52757, #52837, #52854).

Generated by 🔧 PR Triage Agent · auto · 46.6 AIC · ⌖ 2.63 AIC · ⊞ 8.1K ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[deep-report] Investigate Design Decision Gate merge-blocking failure/cost hotspot (cross-verified by 3 monitors)

2 participants