Skip to content

feat: executed MMM calibration tutorials (PyMC-Marketing + Meridian) + CI interop canary - #775

Merged
igerber merged 1 commit into
mainfrom
feature/mmm-tutorials
Aug 16, 2026
Merged

feat: executed MMM calibration tutorials (PyMC-Marketing + Meridian) + CI interop canary#775
igerber merged 1 commit into
mainfrom
feature/mmm-tutorials

Conversation

@igerber

@igerber igerber commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Summary

  • Two new tutorials run the diff-diff -> MMM hand-off end-to-end against the real frameworks, each fitting the MMM without and with the DiD calibration and asserting in-notebook that the ROI posterior moves to the simulated truth and tightens:
    • 29_mmm_calibration_pymc.ipynb - staggered geo spend-boost experiment (with a demand-chasing spend history, the classic bias motivating lift-test calibration, and an in-test budget freeze); CallawaySantAnna + aggregate('simple') + scale=G_treated exported via to_pymc_marketing_lift_test at the national-weekly grain and fed to pymc-marketing 1.0's MMM.add_lift_test_measurements. Committed run: ROI 3.52 (90% 1.78-5.36) uncalibrated -> 2.21 (1.91-2.50) calibrated, truth 2.0.
    • 30_mmm_calibration_meridian.ipynb - geo-holdout staggered launch (exactly roi_m's full-spend-vs-zero estimand, with a demand ramp + common national shocks); aggregate('total') -> to_meridian_roi_prior + meridian_calibration_mask, the generated to_code() snippet executed verbatim into google-meridian 1.8's national model. Committed run: ROI 3.29 (90% 3.02-3.57) default -> 2.54 (2.47-2.62) calibrated, truth 2.5.
  • Both notebooks pass strict sampler-health gates in the committed runs (zero divergences, zero max-tree-depth events, whole-model max R-hat under PyMC's 1.01 warning bar / Meridian's documented 1.2 convention, chain-aware ArviZ R-hat + bulk/tail ESS on the reported ROI), are self-contained and kernelspec-free, and are excluded from CI execution (the 06/10/26 precedent - the two frameworks cannot even share one environment: pymc-marketing 1.0 needs arviz>=1.2, meridian needs arviz<0.20).
  • CI interop canary: new mmm-interop.yml workflow (label-gated, SHA-pinned, paths-scoped to diff_diff/mmm.py + diff_diff/aggregation.py) installs floating framework ranges (weekly cron doubles as an upstream-drift canary, the balance-job precedent) and runs tests/test_mmm_interop_pymc.py / test_mmm_interop_meridian.py - schema/dims round-trips, functional value retention over all four lift fields via logp sensitivity, to_code() exec + mask/prior value retention through the public DataFrameInputDataBuilder, and a float32-aware Meridian defaults-drift pin. Import-canary steps prevent silently-skipped green runs; TestCiWorkflowLabelEventGuard/GUARD_COVERED_WORKFLOWS/rust-test paths updated in the same diff.
  • Drift guards in the normal suite (test_t29_*/test_t30_*_drift.py): DGP constant + mechanism-line sync, sampler-config needles, integration-cell + committed-output presence (money plot, per-field lift-row pins, linearity-scope and full-panel-fit prose pins), kernelspec-free check, and diff-diff-only recomputation of the printed DiD numbers; also wired as a step in execute-notebooks so notebook-only PRs run them.
  • Docs sweep: tutorials index cards/toctree (+ notebook counts 28 -> 31 on both surfaces), tutorials README entries with per-entry dependency requirements, doc-deps registration (mmm.py + aggregation.py blocks, incl. the practitioner guide), route-qualified Step-8 MMM hand-off in llms-practitioner.txt (content-pinned in tests/test_guides.py), api/mmm.rst seealso, CHANGELOG, TODO (PR-B row delivered; new optional scheduled-execution-lane row), dev-status environment note. Meridian provenance claims relabeled "execution-validated on 1.8.0" (behavioral 1.7.0 claims deliberately unchanged).
  • Adjacent fix: tutorial 28's 4a./4b./4c. headings renamed (their digit-leading HTML ids trip docs-tests' search-anchor guard on the next docs build - latent on main; markdown-only).

Methodology references (required if estimator / math changes)

  • Method name(s): No estimator/math changes - MMM exporter interop tutorials + tests only (diff_diff/mmm.py edits are comment/docstring version-provenance updates).
  • Paper / source link(s): PyMC-Marketing lift-test calibration (MMM.add_lift_test_measurements docs); Google Meridian "Set custom priors using past experiments" + configure-model (roi_calibration_period) + ROI/mROI parameterization docs; Callaway & Sant'Anna (2021) for the DiD side.
  • Any intentional deviations from the source (and why): None. The Meridian 1.7.0-pinned to_code templates and defaults are execution-validated unchanged on 1.8.0 (recorded at the provenance sites; behavioral 1.7.0 statements deliberately keep their version pin).

Validation

  • Tests added/updated: tests/test_mmm_interop_pymc.py, tests/test_mmm_interop_meridian.py (run in the new CI jobs; importorskip elsewhere), tests/test_t29_mmm_calibration_pymc_drift.py, tests/test_t30_mmm_calibration_meridian_drift.py (normal suite), tests/test_guides.py (Step-8 content pin), tests/test_openai_review.py (workflow-contract locks).
  • Backtest / simulation / notebook evidence (if applicable): both notebooks executed end-to-end in dedicated Python 3.12 venvs against pymc-marketing 1.0.0 and google-meridian 1.8.0 (committed outputs carry the sampler-health printouts and acceptance-assert results); interop suites pass against those installs; Sphinx -W build + docs-IA + doc-snippets + guide/naming guards green.

Security / privacy

  • Confirm no secrets/PII in this PR: Yes

…+ CI interop canary

Two new tutorials run the diff-diff -> MMM hand-off END-TO-END against the
real frameworks, both fitting the MMM without and with the DiD calibration
and asserting (in-notebook) that the ROI posterior moves to the simulated
truth and tightens:

- docs/tutorials/29_mmm_calibration_pymc.ipynb - staggered geo spend-boost
  experiment with a demand-chasing spend history (the classic bias that
  motivates lift-test calibration) and an in-test budget freeze; CS +
  aggregate('simple') + scale=G_treated exported via
  to_pymc_marketing_lift_test at the national-weekly grain (grain
  reconciliation taught explicitly, with the staggered-average ->
  single-transition construction scoped to its linear/homogeneous DGP and
  the correct nonlinear-case designs spelled out: simultaneous boost, or
  full-panel fit + aggregation restricted to full-rollout periods);
  pymc-marketing 1.0 MMM.add_lift_test_measurements; campaign-ROI posterior
  via the original-scale contribution accessor and a boost-removed
  counterfactual. Committed run: ROI 3.52 (90% 1.78-5.36) uncalibrated ->
  2.21 (1.91-2.50) calibrated, truth 2.0.
- docs/tutorials/30_mmm_calibration_meridian.ipynb - geo-holdout staggered
  LAUNCH (exactly roi_m's full-spend-vs-zero estimand) with a demand ramp
  and common national shocks; aggregate('total') -> to_meridian_roi_prior +
  meridian_calibration_mask, the generated to_code() snippet executed
  verbatim into google-meridian 1.8; a NATIONAL Meridian model (aggregation
  destroys the holdout contrast, which is precisely why the experiment
  prior carries information the model cannot re-derive). Committed run:
  ROI 3.29 (90% 3.02-3.57) default -> 2.54 (2.47-2.62) calibrated,
  truth 2.5.

Both notebooks are self-contained (no data files), kernelspec-free, use
scoped warnings suppression only (diff-diff's UserWarning loud-failure
channel stays visible), and pass STRICT sampler-health gates in the
committed runs: zero divergences, zero maximum-tree-depth events,
whole-model max Rhat under PyMC's 1.01 warning bar (29: 1.0069/1.0062,
chains=4, tune=3000, target_accept=0.999, max_treedepth=14) or Meridian's
documented 1.2 convergence convention (30: 1.0038/1.0175, n_chains=4),
plus chain-aware ArviZ Rhat/bulk+tail-ESS asserts on the reported ROI
quantity and posterior acceptance asserts with slack margins and
reader-side reproducibility notes. They execute locally in dedicated
Python 3.12 venvs (the frameworks cannot share one environment:
pymc-marketing 1.0 needs arviz>=1.2 while meridian needs arviz<0.20) and
are excluded from CI execution like 06/10/26; an optional scheduled
end-to-end execution lane is tracked in TODO.md.

CI protection (hybrid):
- NEW .github/workflows/mmm-interop.yml - two label-gated jobs (SHA-pinned
  actions, contents:read; paths-scoped to diff_diff/mmm.py +
  diff_diff/aggregation.py, the two files that own the exporter contract)
  installing FLOATING framework ranges (pymc-marketing>=1.0,<2 /
  google-meridian>=1.8,<2, the balance-job precedent, so the weekly cron is
  a genuine upstream-drift canary) with an import-canary step so
  importorskip cannot silently green an install failure.
- tests/test_mmm_interop_pymc.py - schema acceptance, wrong-schema pin
  (pytest.raises(KeyError, match="delta_y"), probed against the live API),
  an unconditional dims-carrying round-trip, and FUNCTIONAL value retention
  over ALL four lift fields (pymc-marketing 1.0 stores lift tests only as a
  model potential, so retention is asserted via logp sensitivity: identical
  frame -> identical logp; perturbed x/delta_x/delta_y/sigma -> changed
  logp).
- tests/test_mmm_interop_meridian.py - defaults-drift canary
  (_MERIDIAN_PARAM_DEFAULTS vs live PriorDistribution(), float32-aware),
  to_code() exec with mu/sigma + mask value retention, Meridian
  instantiation via the PUBLIC DataFrameInputDataBuilder (no dependence on
  Meridian's internal test fixtures).
- tests/test_t29_*/test_t30_*_drift.py - diff-diff-only guards in the
  NORMAL suite: DGP constant + mechanism-line sync (t26 pattern), sampler
  config needles, integration-cell + committed-output presence (incl. the
  money-plot image and per-field lift-row pins - the ROI ratio alone would
  let the treated-geo multiplier cancel), kernelspec-free check,
  linearity-scope + full-panel-fit prose pins, and recomputed DiD numbers
  vs printed outputs (loose bands, t27 convention). Run as a step in
  execute-notebooks so notebook-only PRs exercise them.
- tests/test_openai_review.py EXPECTED_JOBS + GUARD_COVERED_WORKFLOWS gain
  mmm-interop.yml; rust-test.yml paths gain it (workflow-contract lock).

Meridian version-claim sweep (spike validated 1.8.0): PROVENANCE claims
amended to "execution-validated on 1.8.0" (mmm.py defaults comment,
template-pin comment, to_code docstring; REGISTRY primary-sources +
defaults notes; references.rst; doc-deps note). BEHAVIORAL claims stay
1.7.0-pinned by design - they record the version whose source the behavior
was verified against (mmm.py roi_calibration_period docstrings + runtime
ValueError strings, REGISTRY mask-acceptance note, CHANGELOG history line,
test_mmm.py comment). The generated-snippet header strings ("template
pinned to Meridian 1.7.0") also stay unchanged - deliberate no-op: the pin
statement remains true and the committed notebook output carries them.

Docs: tutorials index (2 cards + labeled toctree entries in Business
Applications; notebook count 28 -> 31 on both count surfaces - the
directory already held 29 files), tutorials README entries with per-entry
dependency requirements (balance precedent; no backfill of the
pre-existing 05-14/16/28 catalog gaps - deliberate no-op), doc-deps
registration under BOTH diff_diff/mmm.py and diff_diff/aggregation.py plus
the practitioner guide, llms-practitioner.txt Step-8 route-qualified MMM
hand-off (guarded by a new Step-8 content pin in tests/test_guides.py),
api/mmm.rst seealso, CHANGELOG entry, TODO PR-B row delivered/removed +
new execution-lane row, dev-status.md environment note.
llms.txt/llms-full.txt tutorial catalogs untouched - deliberate no-op
(already selective; 27/28 absent too).

Adjacent fixes required to keep docs-tests green: tutorial 28's three
"### 4a./4b./4c." headings renamed to letter-leading forms - they produced
digit-leading HTML ids that trip the search-anchor guard in docs-tests.yml
on the next docs build (latent on main; markdown-only edit, no
re-execution; no cross-references or drift-test pins existed). Widget
metadata/outputs (sampler progress artifacts) and machine-local paths are
stripped from committed notebook outputs - nbsphinx -W fails on widget
state without ipywidgets in the docs env.
@github-actions

Copy link
Copy Markdown

Overall assessment

Looks good — no unmitigated P0/P1 findings.

Executive summary

  • No estimator, weighting, variance, inference, or default behavior changes.
  • PyMC lift-test scaling matches the Registry’s same-grain contract.
  • Meridian total-ROI and calibration-mask semantics match the cited methodology.
  • Earlier notebook-29 concerns about staggered nonlinear extrapolation are explicitly resolved and regression-guarded.
  • End-to-end MCMC CI execution remains absent but is properly tracked as P3.

Methodology

Severity: None
Impact: The tutorials correctly use aggregate("simple") × treated geos for the PyMC national-weekly row and aggregate("total") / total spend for Meridian roi_m. Mask scope and full-panel estimation guidance are also correct. See docs/methodology/REGISTRY.md:L6538-L6569 and tests/test_t29_mmm_calibration_pymc_drift.py:L210-L250.
Concrete fix: None.

Code Quality

Severity: None
Impact: Production changes are provenance comments/docstrings only. New tests clearly separate framework integration from numerical drift checks.
Concrete fix: None.

Performance

Severity: None
Impact: Heavy framework installations are isolated, path-scoped, and scheduled weekly rather than added to normal CI.
Concrete fix: None.

Maintainability

Severity: None
Impact: Workflow guards, schema checks, source needles, and rendered-value recomputation provide reasonable drift protection.
Concrete fix: None.

Tech Debt

Severity: P3 — informational, tracked
Impact: The committed MCMC results are not re-executed in CI, so sampler or posterior drift could remain unnoticed. This is explicitly tracked in TODO.md:L79.
Concrete fix: Optional future cron jobs executing each notebook in its exact-pin isolated environment. No approval action required.

Security

Severity: None
Impact: No secrets or PII found; workflow permissions are read-only and third-party actions are SHA-pinned.
Concrete fix: None.

Documentation/Tests

Severity: None
Impact: Documentation surfaces and notebook counts are synchronized. Real-framework canaries validate schemas and retained values, while normal-suite guards recompute the DiD/export side.
Concrete fix: None.

Reviewer note: static parsing succeeded, but tests could not be executed in this review environment because pytest and numerical dependencies are unavailable.

@igerber igerber added the ready-for-ci Triggers CI test workflows label Aug 16, 2026
@igerber
igerber merged commit d6a30d2 into main Aug 16, 2026
43 of 44 checks passed
@igerber
igerber deleted the feature/mmm-tutorials branch August 16, 2026 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-ci Triggers CI test workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant