diff --git a/CHANGELOG.md b/CHANGELOG.md index da03535b..8cc4bb16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added +- **ImputationDiD `aux_partition` variants now anchored against Stata `did_imputation + avgeffectsby()` (no library behavior change beyond a docstring wording fix).** The + coarser `aux_partition="cohort"` / `"horizon"` groupings — previously the + least-validated ImputationDiD surface (no R analogue; hand-calc only) — are now + measured against the authors' Stata `avgeffectsby(Ei)` / `avgeffectsby(K)` + (`K = t − Ei`), extending the PR #712 arm + (`benchmarks/stata/generate_imputation_loo_golden.do` → + `didimputation_loo_stata_golden.json`; `tests/test_imputation_loo_stata_parity.py` + gates 6-10, max observed deviation ~1.3e-8 against the `abs=1e-7` gate; default-arm + golden values byte-identical). Coverage: `horizon` at the overall + all 6 event-study + horizons, `cohort` at the overall, plus a deterministic **unbalanced subsample** + (`mod(unit,4)==0 & time>=6`, 1305 rows) re-running all three partitions. Three + measured facts are now pinned: on a balanced uniform-weight panel the `cohort` + partition is an arithmetic identity with the default (the unbalanced block is where + it genuinely diverges, ~23% larger SE); the `horizon` partition genuinely differs + per-horizon at h=0..3 (coinciding only at the single-cohort horizons h=4,5); and + "coarser ⇒ more conservative" is typical but not guaranteed — under + `leave_one_out=True` a coarser SE can be *smaller* (h=2 inversion; every measured + non-LOO comparison keeps the weak ordering). The `aux_partition` + docstring/REGISTRY/tutorial wording was corrected accordingly, and the old + `test_coarser_partition_more_conservative` (whose DGP made the ordering an exact + equality) was replaced by an identity pin + a genuine unbalanced-divergence test. + ## [3.9.1] - 2026-08-17 ### Added diff --git a/METHODOLOGY_REVIEW.md b/METHODOLOGY_REVIEW.md index 7ef3e7dd..a4baf8cc 100644 --- a/METHODOLOGY_REVIEW.md +++ b/METHODOLOGY_REVIEW.md @@ -557,7 +557,7 @@ and covariate-adjusted specifications.) - **Untreated-residual hardening.** `_compute_residuals_untreated` now preserves NaN for missing FE (symmetric with the treated path) instead of a silent `fillna(0.0)` that could mask a rank-condition logic error (provably inert on valid data). **Deviations from the reference / library extensions** (see `REGISTRY.md` `## ImputationDiD`): -- **Deviation from R:** `didimputation` computes the Equation 8 aggregator (`Σ v²τ̂/Σ v²`) at the cohort×event-time partition only (no partition control); at that partition it equals the unit-clustered Equation 8 = diff-diff's default `aux_partition="cohort_horizon"`. diff-diff additionally offers `aux_partition="cohort"`/`"horizon"` (validated by hand-calc, no R analogue). +- **Deviation from R:** `didimputation` computes the Equation 8 aggregator (`Σ v²τ̂/Σ v²`) at the cohort×event-time partition only (no partition control); at that partition it equals the unit-clustered Equation 8 = diff-diff's default `aux_partition="cohort_horizon"`. diff-diff additionally offers `aux_partition="cohort"`/`"horizon"` (validated by hand-calc and Stata `avgeffectsby()` parity, no R analogue). - Multiplier bootstrap on the Theorem-3 influence function (library extension, not in the paper). - Survey-design TSL variance on the influence function (library extension). - NaN inference for undefined statistics; Proposition-5 refuse-to-estimate (NaN + warning). diff --git a/TODO.md b/TODO.md index 9a0fa06c..93b76bcb 100644 --- a/TODO.md +++ b/TODO.md @@ -87,7 +87,6 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m | `worktree-rm` safety via a tested argv helper: the prose rewrite (ask-before-remove confirmation gate, detached-HEAD reachability/rescue, tip-identity force-delete guard) was reverted to the main version because editing the prose repeatedly reintroduced shell-injection (last: sourcing a state file built from a git-derived branch name). Restore those guards in a `worktree_rm.py` that takes the name via file ingress, invokes git through argv arrays, and has a metacharacter-branch/path injection regression test — the pattern that worked for `pr_prepare.py`/`premerge_scan.py`. | `.claude/commands/worktree-rm.md`, `.claude/scripts/` | skill-audit | Heavy | Medium | | `premerge_scan.py` should scan the staged blob (`git show :path`) for staged methodology files, not the working-tree copy — a stage-then-revert-working-copy edit currently reads the safe working version and misses the staged violation. Union staged-index findings with unstaged/untracked filesystem findings. | `.claude/scripts/premerge_scan.py` | skill-audit | Mid | Low | | Re-add committed-range methodology scanning to `/push-pr-update` §3b (clean tree, commits ahead) using `premerge_scan.py --range`, with the comparison ref passed as **data** (resolved into a quoted variable in one Bash call, never a raw ``). It was removed to avoid ref interpolation; the helper already implements and tests `--range`. | `.claude/commands/push-pr-update.md` | skill-audit | Quick | Low | -| Extend the ImputationDiD-LOO Stata anchor to the coarser `aux_partition ∈ {"cohort", "horizon"}` variants (Stata `did_imputation ..., leaveout avgeffectsby(Ei)` / `avgeffectsby(K)`, `K = t - Ei`). These differ from the default only at the *overall* aggregate (a no-op per-horizon) and are the least-validated surface — no R analogue, currently only hand-calc. Extends `benchmarks/stata/generate_imputation_loo_golden.do` + golden. | `benchmarks/stata/`, `tests/test_imputation_loo_stata_parity.py` | stata-arm | Mid | Low | | Codex reviewer isolation (repo-wide): `codex --sandbox read-only` blocks writes but does NOT confine READS to the worktree/repo (verified: a probe read `/etc/hosts`), and prompt guards are not a security boundary. Affects `/ai-review-local` (which already documents and accepts this surface) and plan-review-eval dual arms equally. Evaluate OS-level isolation for codex invocations (container / `sandbox-exec` profile / dedicated low-privilege account exposing only the worktree). | `.claude/scripts/openai_review.py`, `tools/plan-review-eval/` | plan-review-eval local review R6 | Heavy | Medium | | plan-review-eval dual arms: terminate the surviving peer subprocess when the first reviewer of a dual pair fails (today an early Claude failure still waits out the codex peer — up to its 3600s ceiling — before the run becomes INFRA_ERROR; needs process handles threaded through `_call_claude`/`call_codex` or a shared cancellation event). | `tools/plan-review-eval/plan_adapters/plan_reviewer.py` | plan-review-eval local review R3 | Mid | Low | | Reimplement the retired `/review-plan` `--updated` (delta re-review, `## Delta Assessment` section) and `--pr ` (PR-comment feedback-coverage) modes in the plan-review skill — descoped from the initial skill ship (the advertised-but-unimplemented flags were removed). `--updated` needs prior-review load + a delta diff; `--pr` needs a `gh` comment fetch + a new reviewer-prompt token. | `.claude/skills/plan-review/SKILL.md` | plan-review skill local review | Mid | Low | diff --git a/benchmarks/data/didimputation_loo_stata_golden.json b/benchmarks/data/didimputation_loo_stata_golden.json index e3396a14..6ef2d39c 100644 --- a/benchmarks/data/didimputation_loo_stata_golden.json +++ b/benchmarks/data/didimputation_loo_stata_golden.json @@ -3,9 +3,10 @@ "estimator": "ImputationDiD leave-one-out (BJS 2024 App. A.9) SE - Stata did_imputation leaveout", "generator": "benchmarks/stata/generate_imputation_loo_golden.do", "source_panel": "benchmarks/data/didimputation_test_panel.csv", - "point_anchor": "benchmarks/data/didimputation_golden.json (overall.att, event_study.att)", - "cmd": "did_imputation y unit time Ei, [horizons(0/5)] leaveout avgeffectsby(Ei t) cluster(unit)", - "avgeffectsby": "Ei t (== library aux_partition=cohort_horizon; pinned explicitly)", + "point_anchor": "benchmarks/data/didimputation_golden.json (overall.att, event_study.att) for the balanced arms; library-computed (informational) for the unbalanced block", + "cmd": "did_imputation y unit time Ei, [horizons(0/5)] [leaveout] avgeffectsby(Ei t | Ei | K) cluster(unit); unbalanced block re-runs all three partitions on the subsample", + "avgeffectsby": "default arm: Ei t (== library aux_partition=cohort_horizon; pinned explicitly)", + "avgeffectsby_variants": "Ei == cohort, K == horizon (K = t - Ei); each pinned explicitly", "se_convention": "A.9 leave-one-out finite-sample variance; unit-clustered; se_nonloo is the non-leaveout cluster SE (== R didimputation).", "ssc_versions": { "did_imputation": "Version: November 22, 2023", @@ -24,5 +25,26 @@ "3": {"att": 2.4998545630646829, "se": 0.029299586939814812, "se_nonloo": 0.029048460851672311}, "4": {"att": 2.9648169454740363, "se": 0.043460391002896281, "se_nonloo": 0.043064691747540094}, "5": {"att": 3.4673514576204787, "se": 0.043504637171120686, "se_nonloo": 0.043073537575459654} + }, + "variants": { + "cohort": {"overall": {"att": 2.0456679010334402, "se": 0.021041161683307705, "se_nonloo": 0.020879383567946323}}, + "horizon": { + "overall": {"att": 2.0456678935832633, "se": 0.021135663550101814, "se_nonloo": 0.021029986100480732}, + "event_study": { + "0": {"att": 0.99293657879682562, "se": 0.029291895495013806, "se_nonloo": 0.029142569815289478}, + "1": {"att": 1.51296485612299, "se": 0.025851152972887623, "se_nonloo": 0.025729536673645869}, + "2": {"att": 2.0064992950637248, "se": 0.03034366662276871, "se_nonloo": 0.030235883748585404}, + "3": {"att": 2.4998545649099237, "se": 0.029638800366029311, "se_nonloo": 0.029508891639513458}, + "4": {"att": 2.9648169169496308, "se": 0.043460391007636372, "se_nonloo": 0.043064691747399318}, + "5": {"att": 3.4673514290960732, "se": 0.043504637175855947, "se_nonloo": 0.043073537575276655} + } + } + }, + "unbalanced": { + "drop_rule": "mod(unit,4)==0 & time>=6", + "n_rows": 1305, + "cohort_horizon": {"att": 1.9647746285960803, "se": 0.023082924221374316, "se_nonloo": 0.022875332984551428}, + "cohort": {"att": 1.9647746285960803, "se": 0.028488231748503588, "se_nonloo": 0.028263480747048431}, + "horizon": {"att": 1.9647746382159736, "se": 0.023163427162940238, "se_nonloo": 0.023022816565355601} } } diff --git a/benchmarks/stata/README.md b/benchmarks/stata/README.md index 80f093dc..9c739a3f 100644 --- a/benchmarks/stata/README.md +++ b/benchmarks/stata/README.md @@ -146,16 +146,38 @@ regeneration is therefore scoped to a fixed Stata + fixed installed SSC versions ## What the generator does No clean-sample reconstruction: `did_imputation` consumes the raw committed R-arm panel -`didimputation_test_panel.csv` (180 units, cohorts {3,5} + never-treated); the only -mapping is `Ei = first_treat` (missing for never-treated). The partition is pinned -`avgeffectsby(Ei t)` **explicitly** (== library `aux_partition="cohort_horizon"`). This -is also `did_imputation`'s current default, but pinning it keeps the validation estimand -self-describing and robust to a future default change. The per-horizon command is: +`didimputation_test_panel.csv` (180 units, cohorts {3,5} + never-treated); the mappings +are `Ei = first_treat` (missing for never-treated) and `K = time - Ei` (the relative +time grouped on under `avgeffectsby(K)`). Every partition is pinned **explicitly**: +`avgeffectsby(Ei t)` (== library `aux_partition="cohort_horizon"`, the default arm), +`avgeffectsby(Ei)` (== `"cohort"`), `avgeffectsby(K)` (== `"horizon"`). `Ei t` is also +`did_imputation`'s current default, but pinning each keeps the validation estimand +self-describing and robust to a future default change. The default arm's per-horizon +command is: ```stata did_imputation y unit time Ei, horizons(0/5) leaveout avgeffectsby(Ei t) cluster(unit) ``` +## Coarser-partition variants + unbalanced subsample + +The `variants` block anchors the coarser `aux_partition` values on the same balanced +panel: `horizon` at the overall AND horizons 0..5 (its per-horizon SEs genuinely differ +from the default at h=0..3 and coincide at the single-cohort horizons h=4,5), `cohort` +at the overall only — because on a BALANCED panel the cohort partition is an +**arithmetic identity** with the default (only `v != 0` rows contribute per group; the +generator asserts the degeneracy at `reldif < 1e-6`, observed ~1e-12, and the Python +gate 7 pins it committed-vs-committed at 1e-9). The `unbalanced` block is where +`cohort` genuinely bites: a deterministic subsample (`drop if mod(unit,4)==0 & +time>=6`, N=1305 — reconstructed identically in pandas by the test; the row count alone +does not identify the subsample, the numeric parity gates do) re-runs all three +partitions at the overall, with the cohort LOO SE measuring ~23% above the default's. +The unbalanced point pin is **library-anchored** (no R anchor exists for the subsample; +informational — the Python test is authoritative). Under `leave_one_out=True` the +`horizon` partition's SE can be *smaller* than the default's (parity panel h=2), so +"coarser ⇒ more conservative" is typical, not guaranteed — recorded in the REGISTRY +ImputationDiD partition note. + Agreement is cross-implementation, not bit-identical (`did_imputation` goes through `reghdfe`, the library through its own sparse IF solver): the **SE** agrees to ~1e-9 and the **point** to ~2e-8; the parity test gates at `abs=1e-7`. Both the LOO and the non-LOO @@ -179,10 +201,20 @@ grep -E '^r\([0-9]+\);' generate_imputation_loo_golden.log # must print nothin ```json { - "meta": { "...": "...", "avgeffectsby": "Ei t (== cohort_horizon)", + "meta": { "...": "...", "avgeffectsby": "default arm: Ei t (== cohort_horizon)", + "avgeffectsby_variants": "Ei == cohort, K == horizon (K = t - Ei)", "ssc_versions": {"did_imputation": "...", "reghdfe": "...", "...": "..."} }, "overall": {"att": , "se": , "se_nonloo": , "N": }, - "event_study": { "0": {"att": , "se": , "se_nonloo": }, "...": "..." } + "event_study": { "0": {"att": , "se": , "se_nonloo": }, "...": "..." }, + "variants": { + "cohort": {"overall": {"att": , "se": , "se_nonloo": }}, + "horizon": {"overall": {"...": "..."}, "event_study": {"0": {"...": "..."}, "...": "..."}} + }, + "unbalanced": { + "drop_rule": "mod(unit,4)==0 & time>=6", "n_rows": 1305, + "cohort_horizon": {"att": , "se": , "se_nonloo": }, + "cohort": {"...": "..."}, "horizon": {"...": "..."} + } } ``` diff --git a/benchmarks/stata/generate_imputation_loo_golden.do b/benchmarks/stata/generate_imputation_loo_golden.do index bf186a36..495235c6 100644 --- a/benchmarks/stata/generate_imputation_loo_golden.do +++ b/benchmarks/stata/generate_imputation_loo_golden.do @@ -19,6 +19,10 @@ *! *! Outputs (checked into the repo) *! benchmarks/data/didimputation_loo_stata_golden.json +*! (default avgeffectsby(Ei t) arm: overall + horizons 0..5, LOO + non-LOO; +*! "variants" block: balanced-panel avgeffectsby(Ei) [== aux_partition="cohort", +*! overall] and avgeffectsby(K) [== "horizon", overall + horizons 0..5]; +*! "unbalanced" block: deterministic subsample, all three partitions, overall) *! *! Usage (run from the repo root, AFTER benchmarks/stata/requirements.do) *! /Applications/Stata/StataSE.app/Contents/MacOS/stata-se -b do \ @@ -31,15 +35,24 @@ *! sole owner is benchmarks/R/generate_didimputation_golden.R (180 units, cohorts *! {3,5} + never-treated, t=1..8). If that panel is ever regenerated, the R golden *! AND this Stata golden must both be regenerated. -*! - No clean-sample reconstruction: did_imputation consumes the raw panel; the only -*! mapping is Ei = first_treat (missing for never-treated). -*! - avgeffectsby(Ei t) is passed EXPLICITLY (== library aux_partition="cohort_horizon"). -*! did_imputation currently defaults to avgeffectsby(Ei t) too; pinning it explicitly -*! makes the validation estimand self-describing and robust to a future default change. -*! - The in-.do point gate (1e-6, informational) checks the Stata coef against the -*! R-anchored points from didimputation_golden.json and aborts early on a gross bug. -*! The AUTHORITATIVE parity gate is tests/test_imputation_loo_stata_parity.py, which -*! compares the recomputed library output against this committed Stata golden. +*! - No clean-sample reconstruction: did_imputation consumes the raw panel; the +*! mappings are Ei = first_treat (missing for never-treated) and K = time - Ei +*! (the relative time did_imputation groups on under avgeffectsby(K)). +*! - Every avgeffectsby() partition is passed EXPLICITLY: Ei t (== library +*! aux_partition="cohort_horizon", the default arm), Ei (== "cohort"), K (== +*! "horizon"). did_imputation currently defaults to avgeffectsby(Ei t); pinning each +*! explicitly keeps the validation estimand self-describing and robust to a future +*! default change. +*! - The unbalanced block re-runs all three partitions on a deterministic subsample +*! (drop if mod(unit,4)==0 & time>=6; N=1305) because on the BALANCED panel the +*! cohort partition is an arithmetic identity with the default (only v!=0 rows +*! contribute per group), so only an unbalanced sample exercises it distinctly. +*! - The in-.do point gates (1e-6, informational) check the Stata coef against the +*! R-anchored points from didimputation_golden.json for the balanced arms, and +*! against a library-computed point for the unbalanced block (no R anchor exists for +*! the subsample) - both abort early on a gross bug. The AUTHORITATIVE parity gate is +*! tests/test_imputation_loo_stata_parity.py, which compares the recomputed library +*! output against this committed Stata golden. version 19 clear all @@ -149,6 +162,7 @@ quietly count assert r(N) == 1440 // 180 units x 8 periods gen Ei = first_treat replace Ei = . if first_treat == 0 +gen K = time - Ei * ------------------------------------------------------------------------------ * Overall ATT: LOO + non-LOO. Coefficient is named `tau`. @@ -176,6 +190,83 @@ forvalues h = 0/5 { scalar se_nl_`h' = _se[tau`h'] } +* ------------------------------------------------------------------------------ +* Balanced `horizon` variant: avgeffectsby(K) == library aux_partition="horizon". +* Overall + horizons 0..5, LOO + non-LOO. Points are partition-invariant, so the +* R-anchored rpt_* pins apply unchanged. +* ------------------------------------------------------------------------------ +did_imputation y unit time Ei, leaveout avgeffectsby(K) cluster(unit) +scalar att_hz_all = _b[tau] +scalar se_hz_all = _se[tau] +assert reldif(att_hz_all, rpt_all) < 1e-6 + +did_imputation y unit time Ei, avgeffectsby(K) cluster(unit) +scalar se_hz_all_nl = _se[tau] + +did_imputation y unit time Ei, horizons(0/5) leaveout avgeffectsby(K) cluster(unit) +forvalues h = 0/5 { + scalar att_hz_`h' = _b[tau`h'] + scalar se_hz_`h' = _se[tau`h'] + assert reldif(att_hz_`h', rpt_`h') < 1e-6 +} +did_imputation y unit time Ei, horizons(0/5) avgeffectsby(K) cluster(unit) +forvalues h = 0/5 { + scalar se_hz_nl_`h' = _se[tau`h'] +} + +* ------------------------------------------------------------------------------ +* Balanced `cohort` variant: avgeffectsby(Ei) == library aux_partition="cohort". +* Overall only - on this BALANCED panel the cohort partition is an arithmetic +* identity with the default (only v!=0 rows contribute per group, and the +* uniform-weight overall makes the cohort mean equal the mean of cell means), so +* the assert below pins the degeneracy (observed ~1e-12; standard 1e-6 bound for +* robustness to SSC solver drift). The distinct-cohort measurement lives in the +* unbalanced block. +* ------------------------------------------------------------------------------ +did_imputation y unit time Ei, leaveout avgeffectsby(Ei) cluster(unit) +scalar att_co_all = _b[tau] +scalar se_co_all = _se[tau] +assert reldif(att_co_all, rpt_all) < 1e-6 +assert reldif(se_co_all, se_all) < 1e-6 + +did_imputation y unit time Ei, avgeffectsby(Ei) cluster(unit) +scalar se_co_all_nl = _se[tau] + +* ------------------------------------------------------------------------------ +* Unbalanced subsample: drop the tail periods of every 4th unit, then re-run all +* three partitions (overall only). This is where `cohort` genuinely diverges +* from the default (~23% larger SE). The point pin is LIBRARY-anchored (no R +* anchor exists for the subsample); informational - the Python test is +* authoritative. +* ------------------------------------------------------------------------------ +scalar rpt_ub = 1.9647746414090286 +preserve +drop if mod(unit, 4) == 0 & time >= 6 +quietly count +assert r(N) == 1305 + +did_imputation y unit time Ei, leaveout avgeffectsby(Ei t) cluster(unit) +scalar att_ub_ch = _b[tau] +scalar se_ub_ch = _se[tau] +assert reldif(att_ub_ch, rpt_ub) < 1e-6 +did_imputation y unit time Ei, avgeffectsby(Ei t) cluster(unit) +scalar se_ub_ch_nl = _se[tau] + +did_imputation y unit time Ei, leaveout avgeffectsby(Ei) cluster(unit) +scalar att_ub_co = _b[tau] +scalar se_ub_co = _se[tau] +assert reldif(att_ub_co, rpt_ub) < 1e-6 +did_imputation y unit time Ei, avgeffectsby(Ei) cluster(unit) +scalar se_ub_co_nl = _se[tau] + +did_imputation y unit time Ei, leaveout avgeffectsby(K) cluster(unit) +scalar att_ub_hz = _b[tau] +scalar se_ub_hz = _se[tau] +assert reldif(att_ub_hz, rpt_ub) < 1e-6 +did_imputation y unit time Ei, avgeffectsby(K) cluster(unit) +scalar se_ub_hz_nl = _se[tau] +restore + * ------------------------------------------------------------------------------ * Emit JSON by hand at %21.17g. Reuses the LPDiD conventions: every STRING field * carries a trailing comma (the compound-quote `"' delimiter swallows a trailing @@ -190,9 +281,10 @@ file write `fh' `" "meta": {"' _n file write `fh' `" "estimator": "ImputationDiD leave-one-out (BJS 2024 App. A.9) SE - Stata did_imputation leaveout","' _n file write `fh' `" "generator": "benchmarks/stata/generate_imputation_loo_golden.do","' _n file write `fh' `" "source_panel": "benchmarks/data/didimputation_test_panel.csv","' _n -file write `fh' `" "point_anchor": "benchmarks/data/didimputation_golden.json (overall.att, event_study.att)","' _n -file write `fh' `" "cmd": "did_imputation y unit time Ei, [horizons(0/5)] leaveout avgeffectsby(Ei t) cluster(unit)","' _n -file write `fh' `" "avgeffectsby": "Ei t (== library aux_partition=cohort_horizon; pinned explicitly)","' _n +file write `fh' `" "point_anchor": "benchmarks/data/didimputation_golden.json (overall.att, event_study.att) for the balanced arms; library-computed (informational) for the unbalanced block","' _n +file write `fh' `" "cmd": "did_imputation y unit time Ei, [horizons(0/5)] [leaveout] avgeffectsby(Ei t | Ei | K) cluster(unit); unbalanced block re-runs all three partitions on the subsample","' _n +file write `fh' `" "avgeffectsby": "default arm: Ei t (== library aux_partition=cohort_horizon; pinned explicitly)","' _n +file write `fh' `" "avgeffectsby_variants": "Ei == cohort, K == horizon (K = t - Ei); each pinned explicitly","' _n file write `fh' `" "se_convention": "A.9 leave-one-out finite-sample variance; unit-clustered; se_nonloo is the non-leaveout cluster SE (== R didimputation).","' _n file write `fh' `" "ssc_versions": {"' _n file write `fh' `" "did_imputation": "`v_didimp'","' _n @@ -215,7 +307,8 @@ _jnum N_all "%12.0f" local nn = r(s) file write `fh' `" "overall": {"att": `a', "se": `s', "se_nonloo": `snl', "N": `nn'},"' _n -* event_study block (comma-prefixed; no trailing comma, no blank first line) +* event_study block (comma-prefixed entries, no blank first line; the block close +* carries a trailing comma because the variants/unbalanced blocks follow) file write `fh' `" "event_study": {"' local sep "" forvalues h = 0/5 { @@ -228,8 +321,68 @@ forvalues h = 0/5 { file write `fh' "`sep'" _n `" "`h'": {"att": `a', "se": `s', "se_nonloo": `snl'}"' local sep "," } -file write `fh' _n " }" _n +file write `fh' _n " }," _n + +* variants block: balanced-panel avgeffectsby(Ei) / avgeffectsby(K) +file write `fh' `" "variants": {"' _n +_jnum att_co_all +local a = r(s) +_jnum se_co_all +local s = r(s) +_jnum se_co_all_nl +local snl = r(s) +file write `fh' `" "cohort": {"overall": {"att": `a', "se": `s', "se_nonloo": `snl'}},"' _n +_jnum att_hz_all +local a = r(s) +_jnum se_hz_all +local s = r(s) +_jnum se_hz_all_nl +local snl = r(s) +file write `fh' `" "horizon": {"' _n +file write `fh' `" "overall": {"att": `a', "se": `s', "se_nonloo": `snl'},"' _n +file write `fh' `" "event_study": {"' +local sep "" +forvalues h = 0/5 { + _jnum att_hz_`h' + local a = r(s) + _jnum se_hz_`h' + local s = r(s) + _jnum se_hz_nl_`h' + local snl = r(s) + file write `fh' "`sep'" _n `" "`h'": {"att": `a', "se": `s', "se_nonloo": `snl'}"' + local sep "," +} +file write `fh' _n " }" _n +file write `fh' " }" _n +file write `fh' " }," _n + +* unbalanced block: deterministic subsample, all three partitions, overall only +file write `fh' `" "unbalanced": {"' _n +file write `fh' `" "drop_rule": "mod(unit,4)==0 & time>=6","' _n +file write `fh' `" "n_rows": 1305,"' _n +_jnum att_ub_ch +local a = r(s) +_jnum se_ub_ch +local s = r(s) +_jnum se_ub_ch_nl +local snl = r(s) +file write `fh' `" "cohort_horizon": {"att": `a', "se": `s', "se_nonloo": `snl'},"' _n +_jnum att_ub_co +local a = r(s) +_jnum se_ub_co +local s = r(s) +_jnum se_ub_co_nl +local snl = r(s) +file write `fh' `" "cohort": {"att": `a', "se": `s', "se_nonloo": `snl'},"' _n +_jnum att_ub_hz +local a = r(s) +_jnum se_ub_hz +local s = r(s) +_jnum se_ub_hz_nl +local snl = r(s) +file write `fh' `" "horizon": {"att": `a', "se": `s', "se_nonloo": `snl'}"' _n +file write `fh' " }" _n file write `fh' "}" _n file close `fh' -display "Wrote benchmarks/data/didimputation_loo_stata_golden.json (overall + 6 horizons)" +display "Wrote benchmarks/data/didimputation_loo_stata_golden.json (default arm + variants + unbalanced)" diff --git a/diff_diff/imputation.py b/diff_diff/imputation.py index 8a05514b..56b441ff 100644 --- a/diff_diff/imputation.py +++ b/diff_diff/imputation.py @@ -114,9 +114,15 @@ class ImputationDiD(ImputationDiDBootstrapMixin, _ImputationAggregationMixin, Ba computed for abs(h) <= horizon_max. aux_partition : str, default="cohort_horizon" Controls the auxiliary model partition for Theorem 3 variance: - - "cohort_horizon": Groups by cohort x relative time (tightest SEs) - - "cohort": Groups by cohort only (more conservative) - - "horizon": Groups by relative time only (more conservative) + - "cohort_horizon": Groups by cohort x relative time (finest) + - "cohort": Groups by cohort only (coarser) + - "horizon": Groups by relative time only (coarser) + Coarser partitions pool more observations per auxiliary group and are + typically -- but not necessarily -- more conservative: on a balanced + panel with uniform weights "cohort" coincides exactly with the default, + and per-horizon "horizon" SEs can be smaller than the default's under + ``leave_one_out=True`` (measured against Stata ``did_imputation + avgeffectsby()``; see the REGISTRY ImputationDiD partition note). pretrends : bool, default=False If True, event study includes pre-treatment horizons for visual pre-trends assessment. Pre-period effects should be ~0 under diff --git a/docs/benchmarks.rst b/docs/benchmarks.rst index dd08bfc2..f1ed32a0 100644 --- a/docs/benchmarks.rst +++ b/docs/benchmarks.rst @@ -48,7 +48,9 @@ where no runnable R reference exists: - Dube, Girardi, Jorda & Taylor (2025); no runnable R analogue * - ``ImputationDiD`` (leave-one-out SE) - Stata ``did_imputation, leaveout`` - - Borusyak, Jaravel & Spiess (2024) Supp. App. A.9; no runnable R analogue + - Borusyak, Jaravel & Spiess (2024) Supp. App. A.9; no runnable R analogue. + Covers all three ``aux_partition`` values via ``avgeffectsby(Ei t | Ei | K)``, + plus an unbalanced subsample where the cohort partition genuinely diverges * - ``ETWFE`` / ``CallawaySantAnna`` (cross-check) - Stata ``jwdid`` / ``csdid`` - Wooldridge (2021) ETWFE and CS via the authors' Stata implementations diff --git a/docs/doc-deps.yaml b/docs/doc-deps.yaml index 683af42e..9c019c33 100644 --- a/docs/doc-deps.yaml +++ b/docs/doc-deps.yaml @@ -298,6 +298,8 @@ sources: type: methodology - path: benchmarks/stata/README.md type: methodology + - path: docs/benchmarks.rst + type: performance - path: docs/api/imputation.rst type: api_reference - path: docs/tutorials/11_imputation_did.ipynb diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 31d84fb7..f1713378 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -1890,7 +1890,7 @@ Auxiliary model residuals (Equation 8): - `epsilon_tilde_it = Y_it - alpha_hat_i - beta_hat_t [- X'delta_hat] - tau_tilde_g` (treated) - `epsilon_tilde_it = Y_it - alpha_hat_i - beta_hat_t [- X'delta_hat]` (untreated, i.e., Step 1 residuals) -The `aux_partition` parameter controls the partition: `"cohort_horizon"` (default, tightest SEs), `"cohort"` (coarser, more conservative), `"horizon"` (groups by relative time only). +The `aux_partition` parameter controls the partition: `"cohort_horizon"` (default, finest), `"cohort"` (coarser), `"horizon"` (groups by relative time only). Coarser partitions pool more observations per auxiliary group and are typically -- but not necessarily -- more conservative: on a balanced panel with uniform weights `"cohort"` coincides exactly with the default, and per-horizon `"horizon"` SEs can be smaller than the default's under `leave_one_out=True` (see the partition-parity facts in the Note (deviation from R) below). *Pre-trend test (Test 1, Equation 9):* ``` @@ -1959,8 +1959,8 @@ where `W_it(h) = 1[K_it = h]` are lead indicators, estimated on `Omega_0` only. - **Note:** Survey weights enter ImputationDiD via weighted iterative FE (Step 1), survey-weighted ATT aggregation (Step 3), and design-based variance via `compute_survey_if_variance()`. PSU clustering, stratification, and FPC are fully supported in the Theorem 3 variance path. When `resolved_survey` is present, the observation-level influence function (`v_it * epsilon_tilde_it`) is passed to `compute_survey_if_variance()` which applies the stratified PSU-level sandwich with FPC correction. Strata also enters survey df (n_PSU - n_strata) for t-distribution inference. Bootstrap + survey supported (Phase 6) via PSU-level multiplier weights. - **Bootstrap inference:** Uses multiplier bootstrap on the Theorem 3 influence function: `psi_i = sum_t v_it * epsilon_tilde_it`. Cluster-level psi sums are pre-computed for each aggregation target (overall, per-horizon, per-group), then perturbed with multiplier weights (Rademacher by default; configurable via `bootstrap_weights` parameter to use Mammen or Webb weights, matching CallawaySantAnna). This is a library extension (not in the paper) consistent with CallawaySantAnna/SunAbraham bootstrap patterns. - **Auxiliary residuals (Equation 8):** Implements the paper's *unit-clustered* Equation 8 aggregator, `tau_tilde_g = sum_i (sum_{t in G_g,i} v_it)(sum_{t in G_g,i} v_it * tau_hat_it) / sum_i (sum_{t in G_g,i} v_it)^2` (Borusyak-Jaravel-Spiess 2024, eq. 8, p. 3272; minimal-excess-variance derivation in Supplementary Appendix A.8): for each unit form the within-unit weight sum `a_{i,g}` and weighted-effect sum `b_{i,g}` over the unit's observations in group `g`, then combine across units. Groups partition `Omega_1` via `aux_partition` (default `"cohort_horizon"` = cohort × event-time; also `"cohort"` / `"horizon"`). Unimputable (NaN `tau_hat`) and off-target observations carry `v_it = 0` and are excluded from the aggregation — exact for finite `tau_hat` (a zero-weight row adds 0 to both `a` and `b`) and NaN-safe; a group with no contributing observations falls back to the unweighted group mean (a variance no-op, since `psi_g = sum_t v_it * eps_tilde_it = 0` there). -- **Note (deviation from R):** R `didimputation::did_imputation` computes the auxiliary aggregator as `sum(v_it^2 * tau_hat_it) / sum(v_it^2)` grouped by cohort × event-time only (no partition control is exposed). At that partition each unit contributes at most one observation per group, so the paper's unit-clustered Equation 8 reduces exactly to `sum(v^2 * tau)/sum(v^2)` — i.e. diff-diff matches R at the default `aux_partition="cohort_horizon"` (pinned in `tests/test_methodology_imputation.py::TestImputationDiDParityR`). diff-diff additionally offers the coarser `aux_partition="cohort"` / `"horizon"` groupings (where a unit may contribute several observations to a group), which have no R analogue and are validated by hand-calculation. Both implement the same paper Equation 8; only the available partition granularity differs. The earlier observation-level mean `sum(v*tau)/sum(v)` (pre-3.5.x) coincided with this only under uniform within-group weights; it was corrected to the exact unit-clustered form during the ImputationDiD methodology validation. -- **Note (leave-one-out variance refinement, Supp. App. A.9):** the opt-in `leave_one_out=True` applies the Borusyak-Jaravel-Spiess (2024) Supplementary Appendix A.9 finite-sample refinement. The non-LOO `tau_tilde_g` (eq. 8) is built from the fitted `tau_hat_it`, which contain the noise `epsilon_it`, so it partially overfits and the auxiliary residuals `epsilon_tilde_it` are too small, biasing the variance **downward**. LOO recomputes each unit's group aggregate excluding that unit, `tau_tilde_it^LO = sum_{j!=i} v_jg^2 T_jg / sum_{j!=i} v_jg^2`, implemented efficiently (A.9) by rescaling each treated residual `epsilon_tilde_it^LO = epsilon_tilde_it / (1 - v_ig^2 / sum_j v_jg^2)` — where `v_ig = sum_{t in G_g,i} v_it` and `sum_j v_jg^2` are already materialized in `_compute_auxiliary_residuals_treated` as `a_{i,g}` and `per_group['den']`. That rescale reproduces the direct-LOO per-unit cluster sum `psi_i = sum_t v_it * epsilon_tilde_it` **exactly** (`psi_i^rescale = a_{i,g}(T_ig - tau_tilde_g) D/(D - a^2) = psi_i^direct-LOO`; a machine-precision-verified identity — **paper-fidelity** to the source `tau_tilde_it^LO`, not merely internal consistency). At the **default unit clustering** LOO gives a larger, less-downward-biased SE (Prop. A8: unbiased for an upper bound on the true variance; an equal-weight K-unit group inflates residuals by exactly `K/(K-1)`). **Default `leave_one_out=False`** preserves R `didimputation` parity (which omits LOO). **Edge (App. A.9 fn. 51):** a group with a single positive-weight unit has an undefined LOO (rescale factor `1/0`); such groups keep the non-LOO residual and the fit emits one consolidated `UserWarning` (a coarser `aux_partition` reduces singletons); a genuinely unit-dominated `>=2`-unit group keeps its large finite factor (intended inflation). **Composition scope:** the rescale operates on `epsilon_tilde`, so it flows through the coarser-`cluster=` CR sum, the analytical survey PSU-TSL variance, and the multiplier bootstrap unchanged — but Prop. A8's upper-bound guarantee and the `LOO >= non-LOO` direction hold at the default unit clustering only (under a coarser `cluster=`, per-unit `psi` inflation can partially cancel), so those compositions are a documented library extension, not paper-derived. **Replicate-weight survey designs** (BRR / Fay / JK1 / JKn / SDR) raise `NotImplementedError` with `leave_one_out=True`: replicate variance is computed by per-replicate point-estimate refits, bypassing the conservative-IF residual path where the LOO rescale lives, so LOO would silently no-op (fail-closed, no-silent-failures). **Effective-singleton guard:** a group's singleton test counts units with *positive* squared weight (not raw rows), and the leave-one-out denominator is the exact sum of the *other* units' squared weights (not `D - v_ig^2`, which can cancel to `<= 0` in float64 for an extremely dominated `>=2`-unit group and would silently revert it to non-LOO). **Reference / validation:** the authors' Stata `did_imputation` ships the same option (`leaveout`), and the library LOO SE is now **measured against it** — the LOO SE matches to ~1e-9 (the point to ~2e-8; cross-implementation: `did_imputation` goes through `reghdfe`, the library through its own sparse IF solver; the parity test gates at `abs=1e-7`) across the overall ATT and all event-study horizons on the committed panel (`benchmarks/stata/generate_imputation_loo_golden.do` → `benchmarks/data/didimputation_loo_stata_golden.json`; `tests/test_imputation_loo_stata_parity.py`, which also three-way-confirms the non-LOO SE against R). The partition is pinned explicitly `avgeffectsby(Ei t)` (== `aux_partition="cohort_horizon"`), which is also `did_imputation`'s current default — pinned to keep the validation estimand self-describing and robust to a future default change. Stata is node-locked (not in CI, golden committed, exactly like the R arm), so the CI-runnable guards remain the exact psi-identity + hand-calc + MC coverage (`tests/test_methodology_imputation.py::TestB2024AppendixA9LeaveOneOut`). Source: arXiv:2108.12419v5 App. A.9 (the REStud Supplementary Material is canonical); the main-article review's A.9 GAP is now filled (see the `borusyak-jaravel-spiess-2024-review.md` provenance note). The stronger Prop. A8 variant that *also* leaves out for the `delta_hat` covariate estimation (exact-unbiased upper bound) is noted but not implemented — the Stata option is the `tau_tilde` residual rescale only. +- **Note (deviation from R):** R `didimputation::did_imputation` computes the auxiliary aggregator as `sum(v_it^2 * tau_hat_it) / sum(v_it^2)` grouped by cohort × event-time only (no partition control is exposed). At that partition each unit contributes at most one observation per group, so the paper's unit-clustered Equation 8 reduces exactly to `sum(v^2 * tau)/sum(v^2)` — i.e. diff-diff matches R at the default `aux_partition="cohort_horizon"` (pinned in `tests/test_methodology_imputation.py::TestImputationDiDParityR`). diff-diff additionally offers the coarser `aux_partition="cohort"` / `"horizon"` groupings (where a unit may contribute several observations to a group), which have no R analogue and are validated by hand-calculation **and Stata `did_imputation avgeffectsby()` parity** (`avgeffectsby(Ei)` == `"cohort"`, `avgeffectsby(K)` == `"horizon"` with `K = t - Ei`; overall + event-study for `horizon`, overall + an unbalanced subsample for `cohort`; `tests/test_imputation_loo_stata_parity.py` gates 6-10, max observed deviation ~1.3e-8 against the `abs=1e-7` gate). Two measured partition facts are pinned by that arm: **(i)** on a balanced panel with uniform weights the `"cohort"` partition is an *arithmetic identity* with the default (only `v_it != 0` rows contribute to a group's aggregate, so per-horizon targets collapse `cohort` groups to the `(Ei, h)` cell, and the uniform-weight overall makes the cohort mean equal the mean of cell means) -- an unbalanced sample is required for it to differ (measured ~23% larger SE on the parity subsample); **(ii)** "coarser => more conservative" is typical, not guaranteed: under `leave_one_out=True` the `"horizon"` partition's SE is measurably *smaller* than the default's at some surfaces (parity panel h=2: 0.0303437 < 0.0303711; every measured non-LOO comparison satisfies the weak `coarse >= fine` ordering). Both implement the same paper Equation 8; only the available partition granularity differs. The earlier observation-level mean `sum(v*tau)/sum(v)` (pre-3.5.x) coincided with this only under uniform within-group weights; it was corrected to the exact unit-clustered form during the ImputationDiD methodology validation. +- **Note (leave-one-out variance refinement, Supp. App. A.9):** the opt-in `leave_one_out=True` applies the Borusyak-Jaravel-Spiess (2024) Supplementary Appendix A.9 finite-sample refinement. The non-LOO `tau_tilde_g` (eq. 8) is built from the fitted `tau_hat_it`, which contain the noise `epsilon_it`, so it partially overfits and the auxiliary residuals `epsilon_tilde_it` are too small, biasing the variance **downward**. LOO recomputes each unit's group aggregate excluding that unit, `tau_tilde_it^LO = sum_{j!=i} v_jg^2 T_jg / sum_{j!=i} v_jg^2`, implemented efficiently (A.9) by rescaling each treated residual `epsilon_tilde_it^LO = epsilon_tilde_it / (1 - v_ig^2 / sum_j v_jg^2)` — where `v_ig = sum_{t in G_g,i} v_it` and `sum_j v_jg^2` are already materialized in `_compute_auxiliary_residuals_treated` as `a_{i,g}` and `per_group['den']`. That rescale reproduces the direct-LOO per-unit cluster sum `psi_i = sum_t v_it * epsilon_tilde_it` **exactly** (`psi_i^rescale = a_{i,g}(T_ig - tau_tilde_g) D/(D - a^2) = psi_i^direct-LOO`; a machine-precision-verified identity — **paper-fidelity** to the source `tau_tilde_it^LO`, not merely internal consistency). At the **default unit clustering** LOO gives a larger, less-downward-biased SE (Prop. A8: unbiased for an upper bound on the true variance; an equal-weight K-unit group inflates residuals by exactly `K/(K-1)`). **Default `leave_one_out=False`** preserves R `didimputation` parity (which omits LOO). **Edge (App. A.9 fn. 51):** a group with a single positive-weight unit has an undefined LOO (rescale factor `1/0`); such groups keep the non-LOO residual and the fit emits one consolidated `UserWarning` (a coarser `aux_partition` reduces singletons); a genuinely unit-dominated `>=2`-unit group keeps its large finite factor (intended inflation). **Composition scope:** the rescale operates on `epsilon_tilde`, so it flows through the coarser-`cluster=` CR sum, the analytical survey PSU-TSL variance, and the multiplier bootstrap unchanged — but Prop. A8's upper-bound guarantee and the `LOO >= non-LOO` direction hold at the default unit clustering only (under a coarser `cluster=`, per-unit `psi` inflation can partially cancel), so those compositions are a documented library extension, not paper-derived. **Replicate-weight survey designs** (BRR / Fay / JK1 / JKn / SDR) raise `NotImplementedError` with `leave_one_out=True`: replicate variance is computed by per-replicate point-estimate refits, bypassing the conservative-IF residual path where the LOO rescale lives, so LOO would silently no-op (fail-closed, no-silent-failures). **Effective-singleton guard:** a group's singleton test counts units with *positive* squared weight (not raw rows), and the leave-one-out denominator is the exact sum of the *other* units' squared weights (not `D - v_ig^2`, which can cancel to `<= 0` in float64 for an extremely dominated `>=2`-unit group and would silently revert it to non-LOO). **Reference / validation:** the authors' Stata `did_imputation` ships the same option (`leaveout`), and the library LOO SE is now **measured against it** — the LOO SE matches to ~1e-9 (the point to ~2e-8; cross-implementation: `did_imputation` goes through `reghdfe`, the library through its own sparse IF solver; the parity test gates at `abs=1e-7`) across the overall ATT and all event-study horizons on the committed panel (`benchmarks/stata/generate_imputation_loo_golden.do` → `benchmarks/data/didimputation_loo_stata_golden.json`; `tests/test_imputation_loo_stata_parity.py`, which also three-way-confirms the non-LOO SE against R). The anchor covers **all three `aux_partition` values**: the default `avgeffectsby(Ei t)` (== `"cohort_horizon"`) plus the coarser `avgeffectsby(Ei)` / `avgeffectsby(K)` variants (== `"cohort"` / `"horizon"`; balanced overall + event-study, and an unbalanced subsample where the cohort partition genuinely diverges — see the Note (deviation from R) above). Each partition is pinned explicitly (`avgeffectsby(Ei t)` is also `did_imputation`'s current default) to keep the validation estimand self-describing and robust to a future default change. Stata is node-locked (not in CI, golden committed, exactly like the R arm), so the CI-runnable guards remain the exact psi-identity + hand-calc + MC coverage (`tests/test_methodology_imputation.py::TestB2024AppendixA9LeaveOneOut`). Source: arXiv:2108.12419v5 App. A.9 (the REStud Supplementary Material is canonical); the main-article review's A.9 GAP is now filled (see the `borusyak-jaravel-spiess-2024-review.md` provenance note). The stronger Prop. A8 variant that *also* leaves out for the `delta_hat` covariate estimation (exact-unbiased upper bound) is noted but not implemented — the Stata option is the `tau_tilde` residual rescale only. - **Note:** The Step-1 iterative FE solver (`_iterative_fe`) routes through the shared bincount Gauss-Seidel helper `diff_diff.utils._iterative_fe_solve`, and the covariate/pre-trend within-transformation routes through the shared MAP engine `diff_diff.utils.demean_by_groups` (factorize-once + `np.bincount`, optional Rust kernel; group order `[time, unit]` preserving the historical time-then-unit sweep) — the same convergence contract, accumulation-order numerics (~1e-10 vs the pre-3.7 pandas loops, not bit-for-bit), and `max_iter=10_000` budget documented under "Absorbed Fixed Effects with Survey Weights". Both surfaces emit `UserWarning` via `diff_diff.utils.warn_if_not_converged` when `max_iter` exhausts without reaching `tol` (the demean warning now carries the shared-engine label naming the affected variables rather than the estimator name). Silent return of the current iterate was classified as a silent failure under the Phase 2 audit and replaced with an explicit signal to match the logistic/Poisson IRLS pattern in `linalg.py`. - **Note:** Zero-total-weight groups (e.g. whole PSUs zeroed by JK1/BRR replicate weights, which reach Step 1 unmasked — `keep_mask` only drops always-treated units): a unit/period whose observations ALL carry zero weight has no identifying contribution and surfaces as `NaN` FE (key retained for the rank-condition membership check; matches the SpilloverDiD `_iterative_fe_subset` REGISTRY contract — never a silent finite `0.0`), and the shared demean engine's inert-row guard leaves those rows un-demeaned instead of NaN-poisoning the column. Before 3.7 the pandas loops divided 0/0 there: the covariate replicate path NaN-poisoned `y_dm`/`X_dm`, failed EVERY replicate refit inside `solve_ols(check_finite=True)`, and returned NaN SEs after a non-convergence warning storm; a main fit with zero-weight rows + covariates raised the same opaque `ValueError`. Both now produce finite results. `TwoStageDiD._mask_nan_ytilde`'s "non-finite imputed outcomes" `UserWarning` is suppressed (via `warn_nan=False`) ONLY inside the replicate-refit closures, where NaN FE for zeroed PSUs is expected mechanics — the main-fit warning is unchanged. - **Note:** `vcov_type` is permanently narrow to `{"hc1"}` per the Theorem 3 IF-based variance decomposition. Analytical-sandwich families `{classical, hc2, hc2_bm}` are rejected at `__init__` — the per-unit influence function aggregation has no equivalent single design matrix on which hat-matrix leverage or Bell-McCaffrey Satterthwaite DOF can be defined. `cluster=` invokes per-cluster IF summation (Theorem 3 equation 7 conservative variance, `sigma_sq = (cluster_psi_sums**2).sum()` — plain CR1 with no Stata-style `(n-1)/(n-p)` finite-sample factor because the IF has no design-matrix `p` in the OLS sense); `cluster=None` (the default) routes the SAME Theorem 3 cluster-summed IF variance with `cluster_var = unit` (the unit column passed to `fit()`), so the summary renders `"CR1 cluster-robust at , G="` rather than the generic `"HC1"` label; `survey_design=` invokes TSL on the combined IF. Under bootstrap (`n_bootstrap > 0`) the analytical variance-family label is suppressed in `summary()` because `fit()` overwrites the reported SE/CI/p-value with bootstrap_results (mirrors the canonical `DiDResults` gate at `results.py:213-226`). `vcov_type='conley'` is deferred to the ImputationDiD Conley follow-up row in DEFERRED.md. diff --git a/docs/methodology/papers/borusyak-jaravel-spiess-2024-review.md b/docs/methodology/papers/borusyak-jaravel-spiess-2024-review.md index 4395e65b..85990152 100644 --- a/docs/methodology/papers/borusyak-jaravel-spiess-2024-review.md +++ b/docs/methodology/papers/borusyak-jaravel-spiess-2024-review.md @@ -189,4 +189,4 @@ No regularization, bandwidth, factor-count, or cross-validation tuning (the esti - No contradictions were found between the three extraction passes or against the direct page reads; all numbered equations rendered cleanly (no `[UNREADABLE EQUATION]`). - **Resolved in PR-B:** the auxiliary `τ̃_g` aggregator now implements the exact unit-clustered Eq. (8) (`_compute_auxiliary_residuals_treated` + docstring + REGISTRY label), validated by white-box hand-calc + R `didimputation` parity (`tests/test_methodology_imputation.py`). The earlier obs-level mean was a valid conservative simplification but not the variance-minimizing form. - **Done in PR-B:** the **R `didimputation` parity fixture** and **`tests/test_methodology_imputation.py`** are on file. The `v*_it` weights (Supp. Prop. A3, absent from this PDF) were validated *empirically* against the reference: the exact two-way-FE projection `-A₀(A₀'A₀)⁻¹A₁'w` now matches `didimputation` (observed ~1e-10; tests assert SE `abs=1e-7`) — the prior FE-only closed form `-(w_i/n0_i + w_t/n0_t − w/N₀)` was a balanced-panel approximation that biased the SE ~27% on staggered (unbalanced-Ω₀) designs and was corrected in PR-B. The leave-one-out variance refinement (Supp. App. A.9) is now **implemented** as the opt-in `leave_one_out` parameter (default False, preserving R `didimputation` parity) — see the A.9 provenance note below. -- **Supp. App. A.9 provenance (leave-one-out, added post-review):** A.9 was a GAP in this main-article review (the REStud Supplementary Material is not in the reviewed PDF). It was subsequently sourced from the arXiv preprint supplement (**arXiv:2108.12419v5, Appendix A.9 "Leave-Out Conservative Variance Estimation"**) to implement the opt-in `leave_one_out` parameter. The **REStud Supplementary Material is the canonical version**; per this doc's version-pin scope note, the efficient-rescale formula `ε̃^LO = ε̃/(1 − v_ig²/Σ_j v_jg²)`, its exact ψ-level equivalence to the direct leave-one-out `τ̃_it^LO`, Prop. A8 (unbiased for an upper bound), and the footnote-51 single-positive-weight-unit edge are recorded in `docs/methodology/REGISTRY.md` `## ImputationDiD` (NOT transcribed here). Validated by the exact ψ-identity + hand-calc + MC coverage (`tests/test_methodology_imputation.py::TestB2024AppendixA9LeaveOneOut`); the authors' Stata `did_imputation` ships the same option (`leaveout`), now a committed Stata parity anchor matching the library LOO SE to ~1e-9 at the overall ATT and all event-study horizons (`benchmarks/stata/generate_imputation_loo_golden.do`; `tests/test_imputation_loo_stata_parity.py`). +- **Supp. App. A.9 provenance (leave-one-out, added post-review):** A.9 was a GAP in this main-article review (the REStud Supplementary Material is not in the reviewed PDF). It was subsequently sourced from the arXiv preprint supplement (**arXiv:2108.12419v5, Appendix A.9 "Leave-Out Conservative Variance Estimation"**) to implement the opt-in `leave_one_out` parameter. The **REStud Supplementary Material is the canonical version**; per this doc's version-pin scope note, the efficient-rescale formula `ε̃^LO = ε̃/(1 − v_ig²/Σ_j v_jg²)`, its exact ψ-level equivalence to the direct leave-one-out `τ̃_it^LO`, Prop. A8 (unbiased for an upper bound), and the footnote-51 single-positive-weight-unit edge are recorded in `docs/methodology/REGISTRY.md` `## ImputationDiD` (NOT transcribed here). Validated by the exact ψ-identity + hand-calc + MC coverage (`tests/test_methodology_imputation.py::TestB2024AppendixA9LeaveOneOut`); the authors' Stata `did_imputation` ships the same option (`leaveout`), now a committed Stata parity anchor matching the library LOO SE to ~1e-9 at the overall ATT and all event-study horizons — and, via `avgeffectsby(Ei)`/`avgeffectsby(K)`, across all three `aux_partition` values including an unbalanced subsample (`benchmarks/stata/generate_imputation_loo_golden.do`; `tests/test_imputation_loo_stata_parity.py`). diff --git a/docs/tutorials/11_imputation_did.ipynb b/docs/tutorials/11_imputation_did.ipynb index abd336f5..caad2e95 100644 --- a/docs/tutorials/11_imputation_did.ipynb +++ b/docs/tutorials/11_imputation_did.ipynb @@ -206,7 +206,7 @@ "source": [ "### Auxiliary Model Partition\n", "\n", - "The `aux_partition` parameter controls the auxiliary model partition for the conservative variance estimator (Theorem 3). Finer partitions give tighter SEs but may overfit with few observations per group." + "The `aux_partition` parameter controls the auxiliary model partition for the conservative variance estimator (Theorem 3). Coarser partitions pool more observations per auxiliary group and are typically -- but not necessarily -- more conservative: on a balanced panel with uniform weights `\"cohort\"` coincides exactly with the default, and under `leave_one_out=True` a coarser partition's SE can even be smaller (see the REGISTRY ImputationDiD partition note)." ] }, { diff --git a/tests/test_imputation.py b/tests/test_imputation.py index dfbd1b8b..2dbe319b 100644 --- a/tests/test_imputation.py +++ b/tests/test_imputation.py @@ -827,36 +827,51 @@ def test_aux_partition_horizon(self): ) assert results.overall_se > 0 - def test_coarser_partition_more_conservative(self): - """Test that coarser partition gives more conservative (larger) SEs.""" + def test_cohort_partition_coincides_on_balanced_uniform_panel(self): + """On a BALANCED panel with uniform weights the cohort partition is an + arithmetic identity with the default cohort_horizon: only v != 0 rows + contribute to a group's Eq. 8 aggregate, and the uniform-weight overall + makes the cohort mean equal the mean of cell means. Coarser is therefore + "typically, not guaranteed" more conservative - see REGISTRY + ## ImputationDiD, Note (deviation from R). The identity is the strongest + pin on this DGP (the old ordering assertion was a one-sided band around + this exact equality); genuine divergence is asserted by the companion + test below on an unbalanced subsample. + """ data = generate_test_data(n_units=200, seed=42) - est_fine = ImputationDiD(aux_partition="cohort_horizon") - results_fine = est_fine.fit( - data, - outcome="outcome", - unit="unit", - time="time", - first_treat="first_treat", - ) + kwargs = dict(outcome="outcome", unit="unit", time="time", first_treat="first_treat") + results_fine = ImputationDiD(aux_partition="cohort_horizon").fit(data, **kwargs) + results_coarse = ImputationDiD(aux_partition="cohort").fit(data, **kwargs) - est_coarse = ImputationDiD(aux_partition="cohort") - results_coarse = est_coarse.fit( - data, - outcome="outcome", - unit="unit", - time="time", - first_treat="first_treat", + # rtol=0: numpy's default rtol=1e-7 would mask the identity pin. + np.testing.assert_allclose( + results_coarse.overall_se, results_fine.overall_se, rtol=0, atol=1e-12 ) - # Coarser partition pools more observations per group, so the - # conservative auxiliary model imposes equality over a larger set and the - # SE is >= the finer-partition SE. Tightened from the old 0.95 fudge: the - # unit-clustered Eq. 8 aggregator preserves the ordering exactly (a broken - # ordering would be a methodology finding, not a tolerance to relax). - # Meaningful coarse>fine divergence under non-uniform weights is covered - # in tests/test_methodology_imputation.py. - assert results_coarse.overall_se >= results_fine.overall_se - 1e-9 + def test_coarser_partition_diverges_on_unbalanced_panel(self): + """Companion to the identity test: on an unbalanced subsample the coarse + partitions genuinely diverge from the default (a unit contributes several + observations to a group with non-uniform effective weighting). Margins are + measurement-derived on this DGP: cohort/fine = 1.0159, horizon/fine = + 1.0051 (non-LOO; under leave_one_out=True the horizon ratio inverts to + 0.9986 - the "coarser => more conservative" heuristic is typical, not + guaranteed). No golden dependency: this is the divergence coverage that + runs even where benchmarks/data/ is absent. + """ + data = generate_test_data(n_units=200, seed=42) + # Drop the last 3 periods (t >= 7 on the t=0..9 panel) for every 4th unit; + # a 2-period drop does not clear the 1.01 margin (measured 1.0092). + sub = data.loc[(data["unit"] % 4 != 0) | (data["time"] < 7)] + assert len(sub) < len(data) + + kwargs = dict(outcome="outcome", unit="unit", time="time", first_treat="first_treat") + se_fine = ImputationDiD(aux_partition="cohort_horizon").fit(sub, **kwargs).overall_se + se_cohort = ImputationDiD(aux_partition="cohort").fit(sub, **kwargs).overall_se + se_horizon = ImputationDiD(aux_partition="horizon").fit(sub, **kwargs).overall_se + + assert se_cohort > se_fine * 1.01 + assert se_horizon > se_fine * 1.002 def test_invalid_aux_partition(self): """Test that invalid aux_partition raises ValueError.""" diff --git a/tests/test_imputation_loo_stata_parity.py b/tests/test_imputation_loo_stata_parity.py index 3ab16b0f..20abebb2 100644 --- a/tests/test_imputation_loo_stata_parity.py +++ b/tests/test_imputation_loo_stata_parity.py @@ -8,9 +8,10 @@ Second Stata parity arm, and the first SSC-dependent one. The golden is produced by `benchmarks/stata/generate_imputation_loo_golden.do`, which runs `did_imputation ..., -leaveout avgeffectsby(Ei t) cluster(unit)` on the committed R-arm panel +[leaveout] avgeffectsby(...) cluster(unit)` on the committed R-arm panel `didimputation_test_panel.csv` (no clean-sample reconstruction - did_imputation takes -the raw panel; the only mapping is Ei = first_treat, missing for never-treated). +the raw panel; the mappings are Ei = first_treat, missing for never-treated, and +K = time - Ei for the `avgeffectsby(K)` variant). The library uses its own sparse IF solver while did_imputation goes through reghdfe, so agreement is cross-implementation, not bit-identical: the SE agrees to ~1e-9 and the @@ -29,6 +30,34 @@ 5. **Warning-cleanliness** - the LOO fit emits no A.9-singleton `UserWarning`, proving the LOO rescale is genuinely exercised rather than silently falling back to non-LOO. +Coarser-`aux_partition` extension (Stata `avgeffectsby(Ei)` == `aux_partition="cohort"`, +`avgeffectsby(K)` == `"horizon"`, `K = t - Ei`), gates 6-10: +6. **Balanced `horizon` variant** - point + LOO SE + non-LOO SE at the overall AND every + horizon h=0..5 (max observed deviation ~1.1e-8; per-horizon SEs genuinely differ from + the default at h=0..3 and coincide at the single-cohort horizons h=4,5 - the TODO + row's "no-op per-horizon" claim was wrong). +7. **Balanced `cohort` variant** - (a) library vs Stata at the overall; (b) the + committed-vs-committed degeneracy identity: on this BALANCED panel the cohort + partition is an arithmetic identity with the default (only v != 0 rows contribute per + group), so Stata's `avgeffectsby(Ei)` overall matches the default-arm golden to 1e-9 + (observed ~1e-12; leg (a) is transitively implied by leg (b) + gates 1-3, kept as a + direct anchor). The distinct-cohort measurement is gate 9's unbalanced block. +8. **Library-side identity pins** (library-only assertions, no golden values; shares this + module's `_skip_if_missing` guard, so it skips in the isolated-install CI leg - the + no-benchmarks-dir divergence coverage lives in tests/test_imputation.py:: + TestImputationVariance::test_coarser_partition_diverges_on_unbalanced_panel): + `cohort` == default bit-tight + (rtol=0, atol=1e-12) at overall + every horizon, both variance modes; `horizon` + == default at h=4,5 only, differing at h=0..3. +9. **Unbalanced subsample** (drop_rule `mod(unit,4)==0 & time>=6`, 1305 rows, + reconstructed identically on both sides; the row count alone does NOT identify the + subsample - every residue class gives 1305 rows - so sample equivalence is pinned by + the numeric gates, with `drop_rule`/`n_rows` consistency asserts riding along): + point + LOO SE + non-LOO SE per partition, plus the mechanism gate that `cohort` + LOO SE differs from the default by >10% (measured ratio 1.234). +10. **Warning-cleanliness for every new LOO fit** - mirrors gate 5 across the variant + and unbalanced fixtures. + Guard per ``feedback_golden_file_pytest_skip``: CI isolated-install jobs copy ``tests/`` only, not ``benchmarks/data/``, so a missing fixture downgrades to pytest.skip rather than fail. Regenerate (after `benchmarks/stata/requirements.do`) with:: @@ -45,6 +74,7 @@ import pandas as pd import pytest +from numpy.testing import assert_allclose from diff_diff import ImputationDiD @@ -74,6 +104,13 @@ HORIZONS = [0, 1, 2, 3, 4, 5] +# Coarser-partition arm (gates 6-10): Stata avgeffectsby() <-> library aux_partition. +VARIANT_PARTITIONS = {"cohort": "Ei", "horizon": "K"} +# Deterministic unbalanced subsample rule; MUST match the golden's recorded drop_rule +# and the generator's `drop if` line (the numeric gates pin actual sample equivalence). +UNBALANCED_DROP_RULE = "mod(unit,4)==0 & time>=6" +UNBALANCED_N_ROWS = 1305 + # Cross-platform tolerance: the library IF variance vs did_imputation (reghdfe) agree # ~1e-9..1e-10 same-machine; 1e-7 matches the repo's imputation SE-parity convention # and absorbs cross-platform BLAS variation. rel=0 -> purely absolute bound. @@ -146,6 +183,88 @@ def by_h(res): } +def _require_block(golden: dict, key: str) -> dict: + """A PRESENT golden missing a new block is a stale/reverted regeneration - a real + inconsistency that must FAIL, not skip (the skip convention covers absent files).""" + assert key in golden, ( + f"golden {STATA_GOLDEN_PATH.name} lacks the '{key}' block - stale golden? " + f"Regenerate via benchmarks/stata/generate_imputation_loo_golden.do" + ) + return golden[key] + + +def _fit_partition(panel, aux_partition: str, leave_one_out: bool): + """One event-study fit; returns (results, [warning messages]). The event_study + aggregate populates the per-horizon surfaces gates 6/8 read. The DEPRECATED + fit-time ``aggregate=`` route is deliberate, matching this module's convention + (see the M-021/M-022 pytestmark comment above: parity tests pin the fit-time + surface shape-for-shape; the shim warning is filtered by message).""" + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = ImputationDiD(leave_one_out=leave_one_out, aux_partition=aux_partition).fit( + panel, + outcome="y", + unit="unit", + time="time", + first_treat="first_treat", + aggregate="event_study", + ) + assert res.event_study_effects is not None + return res, [str(w.message) for w in caught] + + +@pytest.fixture(scope="module") +def library_fit_variants() -> dict: + """Balanced-panel fits for gates 6-8: {cohort, horizon, cohort_horizon} x + {LOO, non-LOO} = 6 event-study fits (the default partition is refit as gate 8's + identity reference).""" + _skip_if_missing() + panel = pd.read_csv(PANEL_PATH) + out: dict = {"loo_warnings": []} + for part in ("cohort", "horizon", "cohort_horizon"): + for loo in (True, False): + res, msgs = _fit_partition(panel, part, loo) + if loo: + out["loo_warnings"] += msgs + es = res.event_study_effects + assert es is not None + mode = "loo" if loo else "nonloo" + out[f"{part}_{mode}"] = { + "att": float(res.overall_att), + "se": float(res.overall_se), + "es": {h: (float(es[h]["effect"]), float(es[h]["se"])) for h in HORIZONS}, + } + return out + + +@pytest.fixture(scope="module") +def library_fit_unbalanced() -> dict: + """Unbalanced-subsample fits for gates 9-10: 3 partitions x {LOO, non-LOO}, overall + only. The pandas reconstruction mirrors the generator's `drop if` rule exactly.""" + _skip_if_missing() + panel = pd.read_csv(PANEL_PATH) + # keep rows NOT matching UNBALANCED_DROP_RULE (De Morgan of mod(unit,4)==0 & time>=6) + sub = panel.loc[(panel["unit"] % 4 != 0) | (panel["time"] < 6)] + assert len(sub) == UNBALANCED_N_ROWS + out: dict = {"loo_warnings": []} + for part in ("cohort_horizon", "cohort", "horizon"): + for loo in (True, False): + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = ImputationDiD(leave_one_out=loo, aux_partition=part).fit( + sub, + outcome="y", + unit="unit", + time="time", + first_treat="first_treat", + ) + if loo: + out["loo_warnings"] += [str(w.message) for w in caught] + mode = "loo" if loo else "nonloo" + out[f"{part}_{mode}"] = (float(res.overall_att), float(res.overall_se)) + return out + + # ----- Gate 1: point (library vs Stata, same estimand) ----- @@ -230,3 +349,139 @@ def test_loo_fit_emits_no_singleton_warning(library_fit): if "leave-one-out" in m.lower() or "singleton" in m.lower() or "single" in m.lower() ] assert not singletons, f"unexpected LOO-singleton warning(s): {singletons}" + + +# ----- Gate 6: balanced `horizon` variant (avgeffectsby(K)), overall + every horizon ----- + + +def test_variant_horizon_overall_matches_stata(stata_golden, library_fit_variants): + blk = _require_block(stata_golden, "variants")["horizon"]["overall"] + loo = library_fit_variants["horizon_loo"] + nonloo = library_fit_variants["horizon_nonloo"] + assert loo["att"] == pytest.approx(blk["att"], abs=ATOL, rel=0) + assert loo["se"] == pytest.approx(blk["se"], abs=ATOL, rel=0) + assert nonloo["se"] == pytest.approx(blk["se_nonloo"], abs=ATOL, rel=0) + + +@pytest.mark.parametrize("h", HORIZONS) +def test_variant_horizon_event_study_matches_stata(h, stata_golden, library_fit_variants): + blk = _require_block(stata_golden, "variants")["horizon"]["event_study"][str(h)] + att, loo_se = library_fit_variants["horizon_loo"]["es"][h] + _, nonloo_se = library_fit_variants["horizon_nonloo"]["es"][h] + assert att == pytest.approx(blk["att"], abs=ATOL, rel=0) + assert loo_se == pytest.approx(blk["se"], abs=ATOL, rel=0) + assert nonloo_se == pytest.approx(blk["se_nonloo"], abs=ATOL, rel=0) + + +# ----- Gate 7: balanced `cohort` variant (avgeffectsby(Ei)) + degeneracy identity ----- + + +def test_variant_cohort_overall_matches_stata_and_degenerates(stata_golden, library_fit_variants): + """(a) library vs Stata avgeffectsby(Ei) at the overall; (b) committed-vs-committed: + on this BALANCED panel the cohort partition is an arithmetic identity with the + default, so Stata's cohort block matches the default-arm golden to 1e-9 (observed + ~1e-12 - pure reghdfe accumulation noise). Leg (a) is transitively implied by (b) + + gates 1-3; kept as a direct anchor. The distinct-cohort measurement is gate 9.""" + blk = _require_block(stata_golden, "variants")["cohort"]["overall"] + loo = library_fit_variants["cohort_loo"] + nonloo = library_fit_variants["cohort_nonloo"] + assert loo["att"] == pytest.approx(blk["att"], abs=ATOL, rel=0) + assert loo["se"] == pytest.approx(blk["se"], abs=ATOL, rel=0) + assert nonloo["se"] == pytest.approx(blk["se_nonloo"], abs=ATOL, rel=0) + # (b) degeneracy identity, committed vs committed. The SEs pin the identity at 1e-9 + # (observed ~1e-12); the att only at ATOL - did_imputation's POINT jitters ~4e-9 + # across runs (first-stage reghdfe noise), while the identity claim is about the + # variance construction (the library att is bit-identical across partitions). + default = stata_golden["overall"] + assert blk["att"] == pytest.approx(default["att"], abs=ATOL, rel=0) + assert blk["se"] == pytest.approx(default["se"], abs=1e-9, rel=0) + assert blk["se_nonloo"] == pytest.approx(default["se_nonloo"], abs=1e-9, rel=0) + + +# ----- Gate 8: library-side identity pins (protect the documented partition facts) ----- + + +def test_cohort_partition_is_identity_with_default_on_balanced_panel(library_fit_variants): + """`aux_partition="cohort"` == the default at every surface on the balanced panel + (bit-equal locally; rtol=0 so numpy's default rtol=1e-7 cannot mask the pin).""" + for mode in ("loo", "nonloo"): + co, ch = ( + library_fit_variants[f"cohort_{mode}"], + library_fit_variants[f"cohort_horizon_{mode}"], + ) + assert_allclose(co["att"], ch["att"], rtol=0, atol=1e-12) + assert_allclose(co["se"], ch["se"], rtol=0, atol=1e-12) + for h in HORIZONS: + assert_allclose(co["es"][h], ch["es"][h], rtol=0, atol=1e-12) + + +def test_horizon_partition_differs_except_single_cohort_horizons(library_fit_variants): + """`aux_partition="horizon"` per-horizon SEs differ from the default at h=0..3 and + coincide exactly at h=4,5 - the horizons only cohort 3 reaches, where the K group + equals the (Ei, h) cell (corrects the old "no-op per-horizon" claim).""" + for mode in ("loo", "nonloo"): + hz, ch = ( + library_fit_variants[f"horizon_{mode}"], + library_fit_variants[f"cohort_horizon_{mode}"], + ) + for h in (4, 5): + assert_allclose(hz["es"][h][1], ch["es"][h][1], rtol=0, atol=1e-12) + for h in (0, 1, 2, 3): + assert ( + abs(hz["es"][h][1] - ch["es"][h][1]) > 10 * ATOL + ), f"h={h} ({mode}): horizon-partition SE unexpectedly equals the default's" + + +# ----- Gate 9: unbalanced subsample - where the cohort partition genuinely bites ----- + + +def test_unbalanced_block_consistency(stata_golden): + """drop_rule/n_rows consistency (golden <-> test literals). The row count alone does + NOT identify the subsample (every mod-4 residue class gives 1305 rows); actual + sample equivalence is pinned by the numeric gates below.""" + blk = _require_block(stata_golden, "unbalanced") + assert blk["drop_rule"] == UNBALANCED_DROP_RULE + assert blk["n_rows"] == UNBALANCED_N_ROWS + + +def test_meta_records_variant_mapping(stata_golden): + """The golden's meta must record the avgeffectsby <-> aux_partition mapping the + variant gates rely on (drift detection for the generator's meta twin).""" + meta_line = stata_golden["meta"]["avgeffectsby_variants"] + for lib_partition, stata_var in VARIANT_PARTITIONS.items(): + # Assert the exact pairing ("Ei == cohort"), not mere co-occurrence - a + # swapped mapping must fail here, not only in the numeric gates. + assert f"{stata_var} == {lib_partition}" in meta_line + + +@pytest.mark.parametrize("partition", ["cohort_horizon", "cohort", "horizon"]) +def test_unbalanced_matches_stata(partition, stata_golden, library_fit_unbalanced): + blk = _require_block(stata_golden, "unbalanced")[partition] + att, loo_se = library_fit_unbalanced[f"{partition}_loo"] + _, nonloo_se = library_fit_unbalanced[f"{partition}_nonloo"] + assert att == pytest.approx(blk["att"], abs=ATOL, rel=0) + assert loo_se == pytest.approx(blk["se"], abs=ATOL, rel=0) + assert nonloo_se == pytest.approx(blk["se_nonloo"], abs=ATOL, rel=0) + + +def test_unbalanced_cohort_genuinely_diverges(stata_golden): + """Mechanism gate: on the unbalanced subsample the cohort partition must differ from + the default by >10% (measured ratio ~1.234) - proving the block exercises the + coarse-partition path rather than re-measuring the balanced degeneracy.""" + blk = _require_block(stata_golden, "unbalanced") + assert blk["cohort"]["se"] > 1.10 * blk["cohort_horizon"]["se"] + + +# ----- Gate 10: warning-cleanliness for every new LOO fit (mirrors gate 5) ----- + + +def test_variant_and_unbalanced_loo_fits_emit_no_singleton_warning( + library_fit_variants, library_fit_unbalanced +): + msgs = library_fit_variants["loo_warnings"] + library_fit_unbalanced["loo_warnings"] + singletons = [ + m + for m in msgs + if "leave-one-out" in m.lower() or "singleton" in m.lower() or "single" in m.lower() + ] + assert not singletons, f"unexpected LOO-singleton warning(s): {singletons}"