From 3c2ffc7c3bf37c5bfbc5c4280a866d5b8e5890c9 Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 16 Aug 2026 16:28:45 -0400 Subject: [PATCH] feat: executed MMM calibration tutorials (PyMC-Marketing + Meridian) + CI interop canary Two new tutorials run the diff-diff -> MMM hand-off END-TO-END against the real frameworks, both fitting the MMM without and with the DiD calibration and asserting (in-notebook) that the ROI posterior moves to the simulated truth and tightens: - docs/tutorials/29_mmm_calibration_pymc.ipynb - staggered geo spend-boost experiment with a demand-chasing spend history (the classic bias that motivates lift-test calibration) and an in-test budget freeze; CS + aggregate('simple') + scale=G_treated exported via to_pymc_marketing_lift_test at the national-weekly grain (grain reconciliation taught explicitly, with the staggered-average -> single-transition construction scoped to its linear/homogeneous DGP and the correct nonlinear-case designs spelled out: simultaneous boost, or full-panel fit + aggregation restricted to full-rollout periods); pymc-marketing 1.0 MMM.add_lift_test_measurements; campaign-ROI posterior via the original-scale contribution accessor and a boost-removed counterfactual. Committed run: ROI 3.52 (90% 1.78-5.36) uncalibrated -> 2.21 (1.91-2.50) calibrated, truth 2.0. - docs/tutorials/30_mmm_calibration_meridian.ipynb - geo-holdout staggered LAUNCH (exactly roi_m's full-spend-vs-zero estimand) with a demand ramp and common national shocks; aggregate('total') -> to_meridian_roi_prior + meridian_calibration_mask, the generated to_code() snippet executed verbatim into google-meridian 1.8; a NATIONAL Meridian model (aggregation destroys the holdout contrast, which is precisely why the experiment prior carries information the model cannot re-derive). Committed run: ROI 3.29 (90% 3.02-3.57) default -> 2.54 (2.47-2.62) calibrated, truth 2.5. Both notebooks are self-contained (no data files), kernelspec-free, use scoped warnings suppression only (diff-diff's UserWarning loud-failure channel stays visible), and pass STRICT sampler-health gates in the committed runs: zero divergences, zero maximum-tree-depth events, whole-model max Rhat under PyMC's 1.01 warning bar (29: 1.0069/1.0062, chains=4, tune=3000, target_accept=0.999, max_treedepth=14) or Meridian's documented 1.2 convergence convention (30: 1.0038/1.0175, n_chains=4), plus chain-aware ArviZ Rhat/bulk+tail-ESS asserts on the reported ROI quantity and posterior acceptance asserts with slack margins and reader-side reproducibility notes. They execute locally in dedicated Python 3.12 venvs (the frameworks cannot share one environment: pymc-marketing 1.0 needs arviz>=1.2 while meridian needs arviz<0.20) and are excluded from CI execution like 06/10/26; an optional scheduled end-to-end execution lane is tracked in TODO.md. CI protection (hybrid): - NEW .github/workflows/mmm-interop.yml - two label-gated jobs (SHA-pinned actions, contents:read; paths-scoped to diff_diff/mmm.py + diff_diff/aggregation.py, the two files that own the exporter contract) installing FLOATING framework ranges (pymc-marketing>=1.0,<2 / google-meridian>=1.8,<2, the balance-job precedent, so the weekly cron is a genuine upstream-drift canary) with an import-canary step so importorskip cannot silently green an install failure. - tests/test_mmm_interop_pymc.py - schema acceptance, wrong-schema pin (pytest.raises(KeyError, match="delta_y"), probed against the live API), an unconditional dims-carrying round-trip, and FUNCTIONAL value retention over ALL four lift fields (pymc-marketing 1.0 stores lift tests only as a model potential, so retention is asserted via logp sensitivity: identical frame -> identical logp; perturbed x/delta_x/delta_y/sigma -> changed logp). - tests/test_mmm_interop_meridian.py - defaults-drift canary (_MERIDIAN_PARAM_DEFAULTS vs live PriorDistribution(), float32-aware), to_code() exec with mu/sigma + mask value retention, Meridian instantiation via the PUBLIC DataFrameInputDataBuilder (no dependence on Meridian's internal test fixtures). - tests/test_t29_*/test_t30_*_drift.py - diff-diff-only guards in the NORMAL suite: DGP constant + mechanism-line sync (t26 pattern), sampler config needles, integration-cell + committed-output presence (incl. the money-plot image and per-field lift-row pins - the ROI ratio alone would let the treated-geo multiplier cancel), kernelspec-free check, linearity-scope + full-panel-fit prose pins, and recomputed DiD numbers vs printed outputs (loose bands, t27 convention). Run as a step in execute-notebooks so notebook-only PRs exercise them. - tests/test_openai_review.py EXPECTED_JOBS + GUARD_COVERED_WORKFLOWS gain mmm-interop.yml; rust-test.yml paths gain it (workflow-contract lock). Meridian version-claim sweep (spike validated 1.8.0): PROVENANCE claims amended to "execution-validated on 1.8.0" (mmm.py defaults comment, template-pin comment, to_code docstring; REGISTRY primary-sources + defaults notes; references.rst; doc-deps note). BEHAVIORAL claims stay 1.7.0-pinned by design - they record the version whose source the behavior was verified against (mmm.py roi_calibration_period docstrings + runtime ValueError strings, REGISTRY mask-acceptance note, CHANGELOG history line, test_mmm.py comment). The generated-snippet header strings ("template pinned to Meridian 1.7.0") also stay unchanged - deliberate no-op: the pin statement remains true and the committed notebook output carries them. Docs: tutorials index (2 cards + labeled toctree entries in Business Applications; notebook count 28 -> 31 on both count surfaces - the directory already held 29 files), tutorials README entries with per-entry dependency requirements (balance precedent; no backfill of the pre-existing 05-14/16/28 catalog gaps - deliberate no-op), doc-deps registration under BOTH diff_diff/mmm.py and diff_diff/aggregation.py plus the practitioner guide, llms-practitioner.txt Step-8 route-qualified MMM hand-off (guarded by a new Step-8 content pin in tests/test_guides.py), api/mmm.rst seealso, CHANGELOG entry, TODO PR-B row delivered/removed + new execution-lane row, dev-status.md environment note. llms.txt/llms-full.txt tutorial catalogs untouched - deliberate no-op (already selective; 27/28 absent too). Adjacent fixes required to keep docs-tests green: tutorial 28's three "### 4a./4b./4c." headings renamed to letter-leading forms - they produced digit-leading HTML ids that trip the search-anchor guard in docs-tests.yml on the next docs build (latent on main; markdown-only edit, no re-execution; no cross-references or drift-test pins existed). Widget metadata/outputs (sampler progress artifacts) and machine-local paths are stripped from committed notebook outputs - nbsphinx -W fails on widget state without ipywidgets in the docs env. --- .github/workflows/mmm-interop.yml | 129 +++ .github/workflows/notebooks.yml | 23 + .github/workflows/rust-test.yml | 2 + CHANGELOG.md | 23 + TODO.md | 2 +- diff_diff/guides/llms-practitioner.txt | 19 + diff_diff/mmm.py | 11 +- docs/api/mmm.rst | 8 + docs/dev-status.md | 21 + docs/doc-deps.yaml | 20 +- docs/index.rst | 2 +- docs/methodology/REGISTRY.md | 4 +- docs/references.rst | 2 +- .../28_rdd_scholarship_illusion.ipynb | 6 +- docs/tutorials/29_mmm_calibration_pymc.ipynb | 878 ++++++++++++++++ .../30_mmm_calibration_meridian.ipynb | 970 ++++++++++++++++++ docs/tutorials/README.md | 14 + docs/tutorials/index.rst | 18 +- tests/test_guides.py | 19 + tests/test_mmm_interop_meridian.py | 148 +++ tests/test_mmm_interop_pymc.py | 174 ++++ tests/test_openai_review.py | 2 + tests/test_t29_mmm_calibration_pymc_drift.py | 250 +++++ ...test_t30_mmm_calibration_meridian_drift.py | 215 ++++ 24 files changed, 2947 insertions(+), 13 deletions(-) create mode 100644 .github/workflows/mmm-interop.yml create mode 100644 docs/tutorials/29_mmm_calibration_pymc.ipynb create mode 100644 docs/tutorials/30_mmm_calibration_meridian.ipynb create mode 100644 tests/test_mmm_interop_meridian.py create mode 100644 tests/test_mmm_interop_pymc.py create mode 100644 tests/test_t29_mmm_calibration_pymc_drift.py create mode 100644 tests/test_t30_mmm_calibration_meridian_drift.py diff --git a/.github/workflows/mmm-interop.yml b/.github/workflows/mmm-interop.yml new file mode 100644 index 00000000..aa20a30f --- /dev/null +++ b/.github/workflows/mmm-interop.yml @@ -0,0 +1,129 @@ +name: MMM Interop + +# Schema-drift canary for the diff_diff.mmm exporters against the REAL MMM +# frameworks (pymc-marketing, google-meridian). Lives in its own workflow file +# because `paths:` is workflow-level: inside notebooks.yml these heavy jobs +# would fire on every notebook-prose PR and pay full TensorFlow/PyMC installs. +# +# Version policy: CI installs FLOATING ranges (like the balance job's +# "balance>=0.21") so the weekly cron genuinely resolves new upstream releases +# and acts as an upstream-drift canary; the tutorial notebooks and their local +# execution venvs stay pinned to exact versions for reproducibility. +# +# Fragile edge: google-meridian pins an exact tfp-nightly build. If that wheel +# ever disappears from PyPI, first bump/resolve google-meridian to the newest +# release; if none resolves, flip this workflow to cron-only in a hotfix PR — +# never leave it red on labeled PRs. + +on: + push: + branches: [main] + paths: + # The exporters' behavior is owned by mmm.py plus the pinned + # AggregationResult container shape (aggregation.py); estimator-internal + # changes cannot alter the exporter contract without touching these, and + # the normal suite owns the container contract itself - so the heavy + # framework installs do not fan out to unrelated library changes. + - 'diff_diff/mmm.py' + - 'diff_diff/aggregation.py' + - 'pyproject.toml' + - 'tests/test_mmm_interop_pymc.py' + - 'tests/test_mmm_interop_meridian.py' + - '.github/workflows/mmm-interop.yml' + pull_request: + branches: [main] + types: [opened, synchronize, reopened, labeled, unlabeled] + paths: + # The exporters' behavior is owned by mmm.py plus the pinned + # AggregationResult container shape (aggregation.py); estimator-internal + # changes cannot alter the exporter contract without touching these, and + # the normal suite owns the container contract itself - so the heavy + # framework installs do not fan out to unrelated library changes. + - 'diff_diff/mmm.py' + - 'diff_diff/aggregation.py' + - 'pyproject.toml' + - 'tests/test_mmm_interop_pymc.py' + - 'tests/test_mmm_interop_meridian.py' + - '.github/workflows/mmm-interop.yml' + schedule: + # Weekly Sunday 7am UTC (offset from notebooks.yml) — upstream-drift canary: + # the floating installs resolve the latest framework releases. + - cron: '0 7 * * 0' + +permissions: + contents: read + +jobs: + mmm-pymc-interop: + name: PyMC-Marketing interop smoke + # Same ready-for-ci label gate as the notebooks.yml jobs (keep in sync). + if: >- + github.event_name != 'pull_request' + || (contains(github.event.pull_request.labels.*.name, 'ready-for-ci') + && (github.event.action != 'labeled' && github.event.action != 'unlabeled' + || github.event.label.name == 'ready-for-ci')) + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: '3.12' + + - name: Install dependencies + run: | + pip install numpy pandas scipy pytest "pymc-marketing>=1.0,<2" + # Add repo root to Python path so tests can import diff_diff + # (pip install -e . requires the Rust/maturin toolchain; .pth avoids that) + python -c "import site; print(site.getsitepackages()[0])" | xargs -I{} sh -c 'echo "$PWD" > {}/diff_diff_dev.pth' + + - name: Import canary + # Its own step so an install/transitive-import failure FAILS the job + # instead of being swallowed by pytest.importorskip into an all-skipped + # green run. + run: python -c "import pymc_marketing; print(pymc_marketing.__version__)" + + - name: Run interop smoke tests + env: + DIFF_DIFF_BACKEND: python + run: | + pytest tests/test_mmm_interop_pymc.py -v --tb=short + + mmm-meridian-interop: + name: Meridian interop smoke + # Same ready-for-ci label gate as the notebooks.yml jobs (keep in sync). + if: >- + github.event_name != 'pull_request' + || (contains(github.event.pull_request.labels.*.name, 'ready-for-ci') + && (github.event.action != 'labeled' && github.event.action != 'unlabeled' + || github.event.label.name == 'ready-for-ci')) + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: '3.12' + + - name: Install dependencies + run: | + pip install numpy pandas scipy pytest "google-meridian>=1.8,<2" + # Add repo root to Python path so tests can import diff_diff + # (pip install -e . requires the Rust/maturin toolchain; .pth avoids that) + python -c "import site; print(site.getsitepackages()[0])" | xargs -I{} sh -c 'echo "$PWD" > {}/diff_diff_dev.pth' + + - name: Import canary + # Its own step so an install/transitive-import failure FAILS the job + # instead of being swallowed by pytest.importorskip into an all-skipped + # green run. + run: python -c "import meridian; print(meridian.__version__)" + + - name: Run interop smoke tests + env: + DIFF_DIFF_BACKEND: python + run: | + pytest tests/test_mmm_interop_meridian.py -v --tb=short diff --git a/.github/workflows/notebooks.yml b/.github/workflows/notebooks.yml index 274805a7..d2e268db 100644 --- a/.github/workflows/notebooks.yml +++ b/.github/workflows/notebooks.yml @@ -10,6 +10,9 @@ on: - '.github/workflows/notebooks.yml' # the interop drift guard runs only in this workflow (balance installed) - 'tests/test_t26_composition_drift_calibration_drift.py' + # the MMM tutorial drift guards run as a step in execute-notebooks + - 'tests/test_t29_mmm_calibration_pymc_drift.py' + - 'tests/test_t30_mmm_calibration_meridian_drift.py' pull_request: branches: [main] types: [opened, synchronize, reopened, labeled, unlabeled] @@ -20,6 +23,9 @@ on: - '.github/workflows/notebooks.yml' # the interop drift guard runs only in this workflow (balance installed) - 'tests/test_t26_composition_drift_calibration_drift.py' + # the MMM tutorial drift guards run as a step in execute-notebooks + - 'tests/test_t29_mmm_calibration_pymc_drift.py' + - 'tests/test_t30_mmm_calibration_meridian_drift.py' schedule: # Weekly Sunday 6am UTC — smoke test that notebooks still execute cleanly - cron: '0 6 * * 0' @@ -63,6 +69,8 @@ jobs: --ignore=docs/tutorials/06_power_analysis.ipynb \ --ignore=docs/tutorials/10_trop.ipynb \ --ignore=docs/tutorials/26_composition_drift_calibration.ipynb \ + --ignore=docs/tutorials/29_mmm_calibration_pymc.ipynb \ + --ignore=docs/tutorials/30_mmm_calibration_meridian.ipynb \ -v \ --tb=short # Excluded notebooks (too slow for pure-Python CI without Rust backend): @@ -72,6 +80,21 @@ jobs: # 26_composition_drift_calibration — requires the balance package; # runs in the isolated interop-notebooks job below so this job's # minimal env keeps enforcing that tutorials add no dependencies + # 29/30_mmm_calibration_* — require pymc-marketing / google-meridian, + # which cannot even share ONE environment (arviz conflict) and run + # multi-minute MCMC; executed locally with committed outputs. Their + # drift guards run in the next step, and the mmm-interop.yml workflow + # smoke-tests the exporters against the real frameworks (no sampling). + + - name: Run MMM tutorial drift guards + # The MMM notebooks are excluded from execution above, so their committed + # surface is protected here: DGP/sampler/integration needles + diff-diff + # recomputation of the DiD-side numbers (no MMM frameworks needed). + env: + DIFF_DIFF_BACKEND: python + run: | + pytest tests/test_t29_mmm_calibration_pymc_drift.py \ + tests/test_t30_mmm_calibration_meridian_drift.py -v --tb=short - name: Upload failed notebook outputs if: failure() diff --git a/.github/workflows/rust-test.yml b/.github/workflows/rust-test.yml index 4883441b..5ca666a2 100644 --- a/.github/workflows/rust-test.yml +++ b/.github/workflows/rust-test.yml @@ -19,6 +19,7 @@ on: # this suite (enforced by TestRustTestWorkflowPathFilter). - '.github/workflows/docs-tests.yml' - '.github/workflows/notebooks.yml' + - '.github/workflows/mmm-interop.yml' - '.github/workflows/ci-gate.yml' - '.github/workflows/release-build-check.yml' # lint.yml carries the ruff/black pin-sync contract locked by @@ -51,6 +52,7 @@ on: # this suite (enforced by TestRustTestWorkflowPathFilter). - '.github/workflows/docs-tests.yml' - '.github/workflows/notebooks.yml' + - '.github/workflows/mmm-interop.yml' - '.github/workflows/ci-gate.yml' - '.github/workflows/release-build-check.yml' # lint.yml carries the ruff/black pin-sync contract locked by diff --git a/CHANGELOG.md b/CHANGELOG.md index 91bf6f6a..bad43cff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 REGISTRY's LWDiD section gains the maintainer implementation notes (RI convention, dual overall-ATT conventions, Sec 4.3 rejection, inference default, API conformance). +- **Executed MMM calibration tutorials + CI interop canary.** Two new tutorials + run the diff-diff -> MMM hand-off end-to-end against the real frameworks: + `29_mmm_calibration_pymc.ipynb` (a staggered geo spend-boost experiment + exported with `to_pymc_marketing_lift_test` and fed to pymc-marketing 1.0's + `MMM.add_lift_test_measurements`) and `30_mmm_calibration_meridian.ipynb` (a + geo-holdout launch exported with `to_meridian_roi_prior` + + `meridian_calibration_mask`, with the generated `to_code()` snippet executed + verbatim in google-meridian 1.8). Both fit the MMM without and with the + calibration and demonstrate the ROI posterior moving to the simulated truth + and tightening, with in-notebook sampler-health and acceptance asserts. The + heavy frameworks stay out of diff-diff's dependencies: the notebooks execute + locally (committed outputs; they cannot even share one environment - the two + frameworks pin conflicting `arviz` ranges) and are guarded by + diff-diff-only drift tests (`tests/test_t29_*`/`test_t30_*`), while a new + `mmm-interop.yml` workflow smoke-tests the exporters against floating + framework releases on exporter-touching PRs (paths-scoped to `mmm.py` + + `aggregation.py`, the two files that own the exporter contract) and a + weekly cron + (`tests/test_mmm_interop_pymc.py`, `tests/test_mmm_interop_meridian.py` - + schema, dims, value-retention, and Meridian defaults-drift pins; the + 1.7.0-pinned `to_code()` templates and `_MERIDIAN_PARAM_DEFAULTS` are + execution-validated on Meridian 1.8.0). The practitioner guide's Step 8 + gains the route-qualified MMM hand-off pointer. - **`results.aggregate('total')` - the estimator-owned total incremental outcome** on CallawaySantAnna, EfficientDiD, ImputationDiD, and TwoStageDiD, promoted into the library-wide aggregation vocabulary. The single diff --git a/TODO.md b/TODO.md index 2ef1d085..9a0fa06c 100644 --- a/TODO.md +++ b/TODO.md @@ -76,11 +76,11 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m | Issue | Location | Origin | Effort | Priority | |-------|----------|--------|--------|----------| +| Optional scheduled end-to-end execution gate for the MMM tutorials (29/30): a cron-only workflow (or extension of `mmm-interop.yml`) that executes both notebooks in isolated exact-pin environments, so a stale/invalid committed posterior cannot stay green indefinitely - today the hybrid posture (deliberate: notebooks execute locally with committed outputs; CI smoke-tests the exporters without sampling; drift tests pin source + committed-output needles) leaves the MCMC claims un-re-executed in CI | `.github/workflows/mmm-interop.yml`, `docs/tutorials/29_mmm_calibration_pymc.ipynb`, `docs/tutorials/30_mmm_calibration_meridian.ipynb` | mmm-interop | Mid | Low | | Committed `fixest::feols` event-study golden for TWFE `event_study=True` (within + pooled specs, unbalanced + covariate panels, matched CR1 cluster convention, per-period effects + vcov block) - the in-suite gates are shared-core cross-checks (TWFE-within == MPD-absorb, pooled == MPD bit-exact), so a defect common to the shared core would pass; the live-R harness (`benchmarks/R/benchmark_multiperiod.R`, `feols(y ~ treated * time_f \| unit)`) validated the within design in `docs/benchmarks.rst` but is not a committed regression test - follow the `fixest_did_twfe_golden.json` committed-golden pattern (pytest.skip when absent) | `tests/test_fixest_did_twfe_parity.py`, `benchmarks/R/` | 3(a) R2 | Mid | Medium | | Type-blind `n_bootstrap` acceptance in already-validated estimators - HAD bool (`isinstance(..., int)` passes `True`, runs as 1 replicate), dCDH bool+float (its bare `< 0` check passes both `True` and `2.5`), TROP float (`2.5` passes the `>= 2` floor), SyntheticDiD float under all three variance methods + bool/negative under jackknife (its floor check is skipped there) - align these local checks with the `utils.validate_n_bootstrap` type guard (M-081 kept them out of the sweep: it scoped to previously-UNvalidated estimators only) | `diff_diff/had.py`, `diff_diff/chaisemartin_dhaultfoeuille.py`, `diff_diff/trop.py`, `diff_diff/synthetic_did.py` | 2(d) PR-B | Quick | Low | | Evaluate adding the `BaseEstimator` param surface (get_params/set_params) to the exported classes that never had it - `PowerAnalysis`, `LinearRegression`, `BusinessReport`, `DiagnosticReport`, `TWFEWeightsResult` (a NEW public surface, deliberately out of the 2(c)-i pure-refactor scope; `LinearRegression` is the one `fit`-bearing class excluded from the contract suite's roster-completeness test). | `diff_diff/linalg.py`, `diff_diff/power.py` | mixin PR | Mid | Low | | Tighten the mypy suppressions that back the enforced-zero posture: burn down `prep_dgp`'s per-module `[index]` override (needs a None-vs-array restructure that preserves the seeded RNG stream), and evaluate re-enabling the globally disabled codes (`arg-type`, `return-value`, `var-annotated`, `assignment`) one at a time — `assignment` alone hid several real annotation drifts found during the 2026-07 triage. | `pyproject.toml` `[tool.mypy]`, `diff_diff/prep_dgp.py` | lint-CI | Mid | Low | -| MMM interop PR-B: calibration tutorial notebook (fit DiD/CS -> scope -> `to_pymc_marketing_lift_test` / `to_meridian_roi_prior`) + a `llms-practitioner.txt` Step 8 pointer to the exporters as the MMM hand-off. | `docs/tutorials/`, `diff_diff/guides/llms-practitioner.txt` | mmm-interop | Mid | Low | | Tracking-file contract guard test: reject NEW active deferred-work pointers at `TODO.md` (deferred rows live in `DEFERRED.md`; allowlist for historical/past-tense prose and actionable-row pointers) and assert rows cross-linking a `docs/v4-deprecations.yaml` `M-xxx` id don't restate ledger status. Origin: tracking-split local review R2. | `tests/`, `TODO.md`, `DEFERRED.md` | tracking-split | Quick | Low | | Real-data CI canary for dataset-backed replication tests: `test_methodology_lwdid.py`'s Prop 99 / Walmart goldens skip (visibly) when loaders fall back to synthetic; add a lane or canary asserting `df.attrs["source"] == "lwdid_ssc_ancillary"` in CI so network regressions cannot silently de-gate the replication tests. Follow-on from the loader-fallback repair (#723), which made provenance explicit but deliberately did not add a network-dependent CI lane. | `tests/test_methodology_lwdid.py`, `.github/workflows/` | LWDiD validation suite | Quick | Low | | df-provenance completion follow-up (PR C descoping): a Wooldridge public per-row df channel (a per-key dict that can also carry the hc2_bm BM contrast dofs the scalar `_df_analytic_fallback` cannot represent — the unified surface's Wooldridge df column stays NaN until then); ImputationDiD lead-horizon `event_study_df` (needs the `_compute_lead_coefficients`/`_aggregate_event_study` return-path change plus bootstrap-clearing semantics for post rows); and the `inference_df` (DiD/MPD/SA/StackedDiD) vs `df_inference` (CS/dCDH) adapter-naming unification. NOTE (M-024): StackedDiD's `df_survey=None` on analytical fits is DELIBERATE - the container resolver's `df_inference` fallback not matching `inference_df` is load-bearing (normal-theory FLCI; see the REGISTRY StackedDiD M-024 Note and the pin in `tests/test_event_study_consumers.py`) - the unification must preserve that outcome or renegotiate it explicitly. | `diff_diff/wooldridge_results.py`, `diff_diff/imputation.py`, `diff_diff/results_base.py` | #variance-PR-C | Mid | Low | diff --git a/diff_diff/guides/llms-practitioner.txt b/diff_diff/guides/llms-practitioner.txt index ebcd8e33..a35b69bf 100644 --- a/diff_diff/guides/llms-practitioner.txt +++ b/diff_diff/guides/llms-practitioner.txt @@ -591,6 +591,25 @@ guidance = practitioner_next_steps(results) # Returns context-aware suggestions for what to do next ``` +### Hand off to an MMM (marketing mix model) + +When the DiD measured a geo experiment on a marketing channel, export the +result as MMM calibration input via `diff_diff.mmm` - the route depends on +the grain the target MMM row represents: + +- The MMM row IS the campaign total (e.g. Meridian ROI priors): + `results.aggregate('total')` -> `to_meridian_roi_prior(aggregation_result=..., + spend=...)` - the total container arrives pre-scaled; `scale` is not accepted. +- The MMM row's population/period grain differs from the total's span (e.g. a + PyMC-Marketing weekly lift row for a national model): + `results.aggregate('simple')` or `aggregate('group')` plus an explicit + `scale=` -> `to_pymc_marketing_lift_test(aggregation_result=..., scale=..., + x=..., delta_x=...)` - every lift-row field must describe the SAME + observation (same channel, population, and period span). + +Executed end-to-end examples: tutorials 29 (PyMC-Marketing lift test) and +30 (Meridian ROI prior + calibration mask). + --- ## Common Pitfalls diff --git a/diff_diff/mmm.py b/diff_diff/mmm.py index d4739c9b..2e7f5ea0 100644 --- a/diff_diff/mmm.py +++ b/diff_diff/mmm.py @@ -135,12 +135,16 @@ # Meridian prior parameters this exporter can target, with each one's Meridian # default LogNormal(mu, sigma) per channel (verified against -# meridian/model/prior_distribution.py at 1.7.0): roi_m is the return on a +# meridian/model/prior_distribution.py at 1.7.0; execution-validated unchanged on +# 1.8.0, 2026-08, and continuously checked by tests/test_mmm_interop_meridian.py's +# defaults-drift canary): roi_m is the return on a # channel's full spend (zero-spend counterfactual), mroi_m the marginal return. # Channels without an experiment keep the default in the vector snippet. _MERIDIAN_PARAM_DEFAULTS = {"roi_m": (0.2, 0.9), "mroi_m": (0.0, 0.5)} -# The .to_code() templates are pinned against google-meridian 1.7.0 (2026-06); they +# The .to_code() templates are pinned against google-meridian 1.7.0 (2026-06) and +# execution-validated on 1.8.0 (2026-08: the generated snippets exec verbatim into a +# ModelSpec that Meridian 1.8.0 accepts); they # are convenience snippets, not a programmatic contract. Meridian's roi_m/mroi_m have # batch shape n_media_channels; a scalar LogNormal broadcasts to EVERY channel, so the # scalar template is gated behind an explicit single_channel opt-in. @@ -817,7 +821,8 @@ def to_code( roi_calibration_period: Optional[Union[str, np.ndarray]] = None, full_model_window: bool = False, ) -> str: - """Ready-to-paste Meridian snippet (channel- and time-scoped; 1.7.0 pinned). + """Ready-to-paste Meridian snippet (channel- and time-scoped; 1.7.0 pinned, + execution-validated on 1.8.0). Meridian's ``roi_m``/``mroi_m`` prior has batch shape ``n_media_channels`` and a scalar distribution broadcasts to EVERY media channel - a TV experiment's diff --git a/docs/api/mmm.rst b/docs/api/mmm.rst index 3d3063a5..f5bdbcb6 100644 --- a/docs/api/mmm.rst +++ b/docs/api/mmm.rst @@ -8,6 +8,14 @@ Assemble Marketing Mix Model (MMM) calibration inputs from experiment results. Overview -------- +.. seealso:: + + Tutorials 29 (:doc:`../tutorials/29_mmm_calibration_pymc`) and 30 + (:doc:`../tutorials/30_mmm_calibration_meridian`) run both hand-offs + end-to-end against the real frameworks - a lift-test calibration in + PyMC-Marketing and an ROI-prior calibration in Meridian - including the + with/without-calibration posterior comparison. + MMM practitioners calibrate their models against experimental evidence. The two dominant Python MMM frameworks consume that evidence in different shapes: diff --git a/docs/dev-status.md b/docs/dev-status.md index 337b8860..3a8bc55d 100644 --- a/docs/dev-status.md +++ b/docs/dev-status.md @@ -152,3 +152,24 @@ Linux. covariates); logistic-regression overflow in edge cases (separate from the BLAS bug). - **Long-term:** revert to the `@` operator when numpy ≥ 2.3 becomes the minimum supported version. + +## MMM tutorial environments (notebooks 29 / 30) + +The two executed MMM calibration tutorials need frameworks that CANNOT share one +environment: pymc-marketing 1.0 requires `arviz>=1.2,<2.0` while google-meridian +requires `arviz<0.20`. Each notebook therefore has its own Python 3.12 venv and +Jupyter kernel (TensorFlow does not support the 3.14 dev environment): + +- `~/.venvs/diffdiff-mmm-pymc` -> kernel `diffdiff-mmm-pymc` + (`pymc-marketing==1.0.0` + matplotlib/ipykernel/jupyter/nbconvert/pytest) +- `~/.venvs/diffdiff-mmm-meridian` -> kernel `diffdiff-mmm-meridian` + (`google-meridian==1.8.0` + the same tooling) + +Both use the CI jobs' `diff_diff_dev.pth` shim (the repo root written into +site-packages) instead of `pip install -e .` (the maturin build backend would +demand a Rust build). Committed notebooks stay kernelspec-free; every local +execution names the kernel explicitly (`--nbmake-kernel=...` / +`nbconvert --ExecutePreprocessor.kernel_name=...`) - never the default +`python3` kernel. Fragile edge: google-meridian pins an exact `tfp-nightly` +build; the response protocol lives in `.github/workflows/mmm-interop.yml`'s +header comment. diff --git a/docs/doc-deps.yaml b/docs/doc-deps.yaml index 7a054780..683af42e 100644 --- a/docs/doc-deps.yaml +++ b/docs/doc-deps.yaml @@ -1073,7 +1073,7 @@ sources: - path: docs/methodology/REGISTRY.md section: "MMM Calibration Export (interop)" type: methodology - note: "Lift-test scaling semantics, spend-weighted pooling, lognormal closed form (pinned to Meridian 1.7.0), sign/positivity policies." + note: "Lift-test scaling semantics, spend-weighted pooling, lognormal closed form (pinned to Meridian 1.7.0; execution-validated on 1.8.0), sign/positivity policies." - path: docs/api/mmm.rst type: api_reference - path: README.md @@ -1087,6 +1087,15 @@ sources: - path: diff_diff/guides/llms-full.txt section: "MMM Calibration Export" type: user_guide + - path: diff_diff/guides/llms-practitioner.txt + section: "Step 8 MMM hand-off" + type: user_guide + - path: docs/tutorials/29_mmm_calibration_pymc.ipynb + type: tutorial + note: "pymc-marketing interop: executed lift-test calibration; drift guard tests/test_t29_mmm_calibration_pymc_drift.py" + - path: docs/tutorials/30_mmm_calibration_meridian.ipynb + type: tutorial + note: "meridian interop: executed ROI-prior calibration + to_code snippet; drift guard tests/test_t30_mmm_calibration_meridian_drift.py" diff_diff/power.py: drift_risk: low @@ -1231,6 +1240,15 @@ sources: type: user_guide - path: docs/r_comparison.rst type: user_guide + - path: diff_diff/guides/llms-practitioner.txt + section: "Step 8 MMM hand-off (aggregate routes)" + type: user_guide + - path: docs/tutorials/29_mmm_calibration_pymc.ipynb + type: tutorial + note: "aggregate('simple')+scale lift-row route taught end-to-end" + - path: docs/tutorials/30_mmm_calibration_meridian.ipynb + type: tutorial + note: "aggregate('total') container route taught end-to-end" diff_diff/results_base.py: drift_risk: low diff --git a/docs/index.rst b/docs/index.rst index 303ac99e..7b61e46c 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -78,7 +78,7 @@ Explore the Documentation :link: tutorials/index :link-type: doc - 28 hands-on notebooks, from basic 2x2 DiD to survey-weighted and + 31 hands-on notebooks, from basic 2x2 DiD to survey-weighted and spillover-aware designs. .. grid-item-card:: User Guide diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 917dbf84..31d84fb7 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -6537,7 +6537,7 @@ estimator-focused: ## MMM Calibration Export (interop) -**Primary sources:** [PyMC-Marketing lift-test calibration](https://www.pymc-marketing.io/en/stable/notebooks/mmm/mmm_lift_test.html) (`MMM.add_lift_test_measurements` schema); [Google Meridian, "Set custom prior distributions using past experiments"](https://developers.google.com/meridian/docs/advanced-modeling/set-custom-priors-past-experiments) and the closed-form lognormal conversion in `meridian/model/prior_distribution.py` (`lognormal_dist_from_mean_std`, Meridian 1.7.0); the roi_m/mroi_m estimand definitions in [Meridian's ROI parameterization docs](https://developers.google.com/meridian/docs/advanced-modeling/roi-mroi-contribution-parameterizations); the `roi_calibration_period` mask shape/semantics contract in [Meridian's "Set the ROI calibration period"](https://developers.google.com/meridian/docs/user-guide/configure-model) (configure-model guide). +**Primary sources:** [PyMC-Marketing lift-test calibration](https://www.pymc-marketing.io/en/stable/notebooks/mmm/mmm_lift_test.html) (`MMM.add_lift_test_measurements` schema); [Google Meridian, "Set custom prior distributions using past experiments"](https://developers.google.com/meridian/docs/advanced-modeling/set-custom-priors-past-experiments) and the closed-form lognormal conversion in `meridian/model/prior_distribution.py` (`lognormal_dist_from_mean_std`, Meridian 1.7.0; execution-validated on 1.8.0, 2026-08); the roi_m/mroi_m estimand definitions in [Meridian's ROI parameterization docs](https://developers.google.com/meridian/docs/advanced-modeling/roi-mroi-contribution-parameterizations); the `roi_calibration_period` mask shape/semantics contract in [Meridian's "Set the ROI calibration period"](https://developers.google.com/meridian/docs/user-guide/configure-model) (configure-model guide). **Module:** `diff_diff/mmm.py` @@ -6559,7 +6559,7 @@ estimator-focused: *Meridian ROI prior (`to_meridian_roi_prior`):* - Per experiment `roi = incremental_outcome / spend`, `roi_sd = incremental_outcome_se / spend * se_widening`. Multiple experiments pooled spend-weighted - the spend-weighted average ROI the [Meridian FAQ](https://developers.google.com/meridian/docs/faqs) suggests (citing sec 3.4 of Google's MMM calibration whitepaper): `roi_mean = sum(w_i * roi_i)`, `w_i = spend_i / sum(spend)`. -- **Estimand (`parameter={roi_m,mroi_m}`, default roi_m):** `roi_m` is the return on the channel's full spend (a zero-spend/full-holdout estimand; `spend` = total channel spend); `mroi_m` is the marginal return of a spend change (`spend` = the spend change). Under saturation these differ, so the caller declares which their `incremental_outcome` measures; the returned prior and `.to_code()` target that parameter with its own Meridian default for non-experiment channels (roi_m `LogNormal(0.2, 0.9)`, mroi_m `LogNormal(0.0, 0.5)`, both verified at 1.7.0). Sign, aggregation, and population are the caller's responsibility - a go-dark contract's numerator is `Y_exposed - Y_zero`, which the caller supplies with the correct sign. +- **Estimand (`parameter={roi_m,mroi_m}`, default roi_m):** `roi_m` is the return on the channel's full spend (a zero-spend/full-holdout estimand; `spend` = total channel spend); `mroi_m` is the marginal return of a spend change (`spend` = the spend change). Under saturation these differ, so the caller declares which their `incremental_outcome` measures; the returned prior and `.to_code()` target that parameter with its own Meridian default for non-experiment channels (roi_m `LogNormal(0.2, 0.9)`, mroi_m `LogNormal(0.0, 0.5)`, both verified at 1.7.0, execution-validated unchanged on 1.8.0, and continuously checked by the `tests/test_mmm_interop_meridian.py` defaults-drift canary). Sign, aggregation, and population are the caller's responsibility - a go-dark contract's numerator is `Y_exposed - Y_zero`, which the caller supplies with the correct sign. - Lognormal conversion matches Meridian's `lognormal_dist_from_mean_std` exactly: `sigma = sqrt(log1p((s/m)^2))`, `mu = ln(m) - log1p((s/m)^2)/2` (`log1p` keeps a ~1e-8 relative SE from rounding `sigma` to 0). - A non-positive pooled ROI mean raises (lognormal positivity), pointing to pooling, wider priors, or Meridian's contribution/coefficient parameterizations. `spend`, `incremental_outcome_se`, and `se_widening` must be finite and positive; `incremental_outcome` finite. - **Note:** Pooled `roi_sd = sqrt(sum((w_i * sd_i)^2))` treats experiments as independent (no covariance term). Ignoring covariance can misstate the pooled sd - it is anti-conservative when the net weighted covariance is positive, which experiments sharing control units or overlapping windows typically induce - and the direction cannot be determined from marginal SEs alone; the docstring instructs users to widen via `se_widening` (a conservative heuristic, not an exact correction). diff --git a/docs/references.rst b/docs/references.rst index a3611ff4..f92547cb 100644 --- a/docs/references.rst +++ b/docs/references.rst @@ -371,7 +371,7 @@ MMM Calibration Interop - **Google.** "Set Custom Prior Distributions Using Past Experiments." *Google Meridian documentation*. https://developers.google.com/meridian/docs/advanced-modeling/set-custom-priors-past-experiments - Documents the experiment-to-ROI-prior calibration workflow and caveats; the lognormal ``(mu, sigma)`` closed form emitted by ``to_meridian_roi_prior`` matches Meridian's ``prior_distribution.lognormal_dist_from_mean_std`` (verified against Meridian 1.7.0 source). + Documents the experiment-to-ROI-prior calibration workflow and caveats; the lognormal ``(mu, sigma)`` closed form emitted by ``to_meridian_roi_prior`` matches Meridian's ``prior_distribution.lognormal_dist_from_mean_std`` (verified against Meridian 1.7.0 source; execution-validated on 1.8.0, 2026-08). - **Google.** "Set the ROI Calibration Period." *Google Meridian documentation (configure-model guide)*. https://developers.google.com/meridian/docs/user-guide/configure-model diff --git a/docs/tutorials/28_rdd_scholarship_illusion.ipynb b/docs/tutorials/28_rdd_scholarship_illusion.ipynb index f3dd143a..73a047db 100644 --- a/docs/tutorials/28_rdd_scholarship_illusion.ipynb +++ b/docs/tutorials/28_rdd_scholarship_illusion.ipynb @@ -471,7 +471,7 @@ "An RD estimate is only as credible as its design checks. Three standard ones, all\n", "runnable with the estimator itself.\n", "\n", - "### 4a. Covariate balance\n", + "### Diagnostic A: covariate balance\n", "\n", "Continuity has a testable implication: **predetermined covariates cannot jump at the\n", "cutoff** - the scholarship offer cannot change your parents' income or your high-school\n", @@ -527,7 +527,7 @@ "absence supports it, but no balance test can prove continuity of the (unobservable)\n", "potential-outcome functions.\n", "\n", - "### 4b. Placebo cutoffs\n", + "### Diagnostic B: placebo cutoffs\n", "\n", "At any threshold where nothing is assigned, the estimator should find nothing.\n", "Practical guides examine one or more artificial cutoffs across the support;\n", @@ -585,7 +585,7 @@ "cutoffs cannot rule out discontinuities everywhere else - but finding none where none\n", "should exist is exactly the supporting evidence this check exists to collect.\n", "\n", - "### 4c. Bandwidth sensitivity\n", + "### Diagnostic C: bandwidth sensitivity\n", "\n", "The bandwidth is the estimator's main tuning parameter, so referees will ask: does the\n", "answer depend on it? Overriding the selector with fixed `h` values maps out the\n", diff --git a/docs/tutorials/29_mmm_calibration_pymc.ipynb b/docs/tutorials/29_mmm_calibration_pymc.ipynb new file mode 100644 index 00000000..bfa2c12c --- /dev/null +++ b/docs/tutorials/29_mmm_calibration_pymc.ipynb @@ -0,0 +1,878 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "964fea06", + "metadata": {}, + "source": [ + "# MMM Calibration with PyMC-Marketing Lift Tests\n", + "\n", + "Marketing mix models fit on observational spend history inherit its biases: when\n", + "search budgets chase demand, the model credits search with sales that demand would\n", + "have produced anyway. A geo experiment breaks that circularity - and diff-diff's\n", + "MMM exporters carry the experiment's result into\n", + "[PyMC-Marketing](https://www.pymc-marketing.io/) as a *lift test* with one function\n", + "call.\n", + "\n", + "This tutorial runs the full pipeline **end-to-end with the real frameworks**:\n", + "\n", + "1. simulate a geo-week world where search spend chased demand historically, then a\n", + " staggered geo **spend-boost experiment** was run under a budget freeze;\n", + "2. measure the boost's incremental sales with `CallawaySantAnna` -\n", + " `results.aggregate('total')` for the campaign headline, and\n", + " `results.aggregate('simple')` for the per-geo-week ATT the lift row needs;\n", + "3. export the measurement with `to_pymc_marketing_lift_test` via the container\n", + " route - the only number you supply is the count of treated geos (the\n", + " grain-reconciliation `scale`);\n", + "4. fit the national PyMC-Marketing MMM **without and with** the lift test and watch\n", + " the search-ROI posterior move to the truth and tighten.\n", + "\n", + "**Requirements.** This notebook needs `pymc-marketing==1.0.0` (Python 3.12), which\n", + "is *not* a diff-diff dependency: `pip install \"pymc-marketing==1.0.0\" diff-diff`.\n", + "It cannot share an environment with the Meridian tutorial (notebook 30) - the two\n", + "frameworks pin incompatible `arviz` ranges - so each tutorial gets its own\n", + "environment/kernel. Runtime is a few minutes (two small MCMC fits).\n", + "\n", + "**Reproducibility.** MCMC draws depend on framework and transitive-dependency\n", + "versions and on the platform. The committed outputs were produced with\n", + "pymc-marketing 1.0.0; re-executing under other versions may shift posterior\n", + "numbers. The `assert` cells check the qualitative claims (sampler health, and\n", + "that calibration moves the posterior toward the simulated truth and tightens it)\n", + "with slack margins - if one trips on your machine, it means exactly that claim\n", + "did not reproduce in your environment." + ] + }, + { + "cell_type": "markdown", + "id": "1b76eaeb", + "metadata": {}, + "source": [ + "## A world where spend chased demand\n", + "\n", + "Fifteen geos, 104 weeks, two channels. The **search** channel is the experiment\n", + "channel and is deliberately simple: zero carryover (an immediate-response channel)\n", + "and a *linear* true response - every search dollar returns `BETA_SEARCH = 2.0`\n", + "dollars of sales, so the true incremental ROI is 2.0 by construction. The **tv**\n", + "channel keeps adstock and saturation for realism.\n", + "\n", + "The bias mechanism: for the first 52 weeks, search budgets *chased seasonal\n", + "demand* (`spend = base * (1 + 0.5 * season)` while the same season moves sales\n", + "directly). From week 52 budgets were **frozen** - standard geo-test hygiene, and\n", + "what makes the lift test's baseline spend `x` a single well-defined number - and\n", + "from week 60 a staggered boost of \\$50/week rolled out across nine treated geos\n", + "(cohorts at weeks 60, 68, 76), leaving six geos as controls." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "dba18972", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:57:05.565006Z", + "iopub.status.busy": "2026-08-16T19:57:05.564773Z", + "iopub.status.idle": "2026-08-16T19:57:06.439770Z", + "shell.execute_reply": "2026-08-16T19:57:06.439363Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
geoweeksalessearch_spendtv_spendfirst_treat
0003543.352739100.000000203.57869660
1013687.103795106.026834212.79826360
2023753.001771111.965783209.34536860
3033840.767622117.730244207.41001160
4043905.212324123.236159207.09834760
\n", + "
" + ], + "text/plain": [ + " geo week sales search_spend tv_spend first_treat\n", + "0 0 0 3543.352739 100.000000 203.578696 60\n", + "1 0 1 3687.103795 106.026834 212.798263 60\n", + "2 0 2 3753.001771 111.965783 209.345368 60\n", + "3 0 3 3840.767622 117.730244 207.410011 60\n", + "4 0 4 3905.212324 123.236159 207.098347 60" + ] + }, + "execution_count": 1, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import warnings\n", + "\n", + "# Scoped suppressions only: framework-churn warnings are hidden, but estimator\n", + "# UserWarnings (diff-diff's loud-failure channel) stay visible.\n", + "warnings.filterwarnings(\"ignore\", category=FutureWarning)\n", + "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n", + "\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "import pandas as pd\n", + "\n", + "from diff_diff import CallawaySantAnna, ImputationDiD\n", + "from diff_diff.mmm import to_pymc_marketing_lift_test\n", + "\n", + "SEED = 2026\n", + "N_GEOS, N_WEEKS = 15, 104\n", + "BETA_SEARCH = 2.0 # true incremental sales per search dollar (the ROI truth)\n", + "SEARCH_BASE = 100.0 # baseline weekly search spend per geo (frozen in-test)\n", + "BOOST = 50.0 # constant weekly boost per treated geo\n", + "FREEZE_WEEK = 52 # budgets frozen from here (test hygiene)\n", + "TREAT_COHORTS = {60: [0, 1, 2], 68: [3, 4, 5], 76: [6, 7, 8]}\n", + "DEMAND_AMP = 300.0 # seasonal demand amplitude (sales units per geo)\n", + "CHASE = 0.5 # how strongly historical spend chased demand\n", + "TV_BASE, TV_BETA, TV_ADSTOCK, TV_SAT = 200.0, 600.0, 0.5, 0.004\n", + "BASE_SALES, GEO_SD, NOISE_SD = 3000.0, 100.0, 15.0\n", + "\n", + "rng = np.random.default_rng(SEED)\n", + "geo_start = {g: w for w, gg in TREAT_COHORTS.items() for g in gg}\n", + "G_TREATED = len(geo_start)\n", + "\n", + "season = np.sin(2 * np.pi * np.arange(N_WEEKS) / 52.0) # common to all geos\n", + "\n", + "rows = []\n", + "for g in range(N_GEOS):\n", + " tv = TV_BASE * (1 + 0.1 * rng.random(N_WEEKS))\n", + " ad = np.zeros(N_WEEKS)\n", + " carry = 0.0\n", + " for t in range(N_WEEKS):\n", + " carry = tv[t] + TV_ADSTOCK * carry # tv keeps adstock ...\n", + " ad[t] = carry\n", + " tvc = TV_BETA * (1 - np.exp(-TV_SAT * ad)) # ... and saturation\n", + " geo_level = rng.normal(0, GEO_SD)\n", + " for t in range(N_WEEKS):\n", + " if t < FREEZE_WEEK:\n", + " search = SEARCH_BASE * (1 + CHASE * season[t]) # demand-chasing history\n", + " else:\n", + " boosted = g in geo_start and t >= geo_start[g]\n", + " search = SEARCH_BASE + (BOOST if boosted else 0.0) # frozen + boost\n", + " sales = (\n", + " BASE_SALES + geo_level + DEMAND_AMP * season[t]\n", + " + BETA_SEARCH * search # linear, zero carryover\n", + " + tvc[t] + rng.normal(0, NOISE_SD)\n", + " )\n", + " rows.append((g, t, sales, search, tv[t], geo_start.get(g, 0)))\n", + "\n", + "panel = pd.DataFrame(\n", + " rows, columns=[\"geo\", \"week\", \"sales\", \"search_spend\", \"tv_spend\", \"first_treat\"]\n", + ")\n", + "panel.head()" + ] + }, + { + "cell_type": "markdown", + "id": "3ae646aa", + "metadata": {}, + "source": [ + "## Measure the experiment with staggered DiD\n", + "\n", + "`CallawaySantAnna` handles the staggered rollout; `aggregate('total')` turns the\n", + "per-geo-week ATT into the campaign's **total incremental sales** - the estimator's\n", + "own finite-masked aggregation mass, no hand-counting. As a robustness habit we\n", + "cross-check the total with `ImputationDiD`, and we verify the total's mass against\n", + "a raw-frame count." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "56db6eaf", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:57:06.440853Z", + "iopub.status.busy": "2026-08-16T19:57:06.440779Z", + "iopub.status.idle": "2026-08-16T19:57:06.463795Z", + "shell.execute_reply": "2026-08-16T19:57:06.463471Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "per-geo-week ATT: 103.64 (se 9.17; truth 100)\n", + "CS total incremental sales: 33,579\n", + "ImputationDiD total (cross-check): 33,217\n", + "simulated truth: 32,400\n" + ] + } + ], + "source": [ + "res = CallawaySantAnna().fit(\n", + " panel, outcome=\"sales\", unit=\"geo\", time=\"week\", first_treat=\"first_treat\"\n", + ")\n", + "simple = res.aggregate(\"simple\")\n", + "total = res.aggregate(\"total\")\n", + "\n", + "imp_total = (\n", + " ImputationDiD()\n", + " .fit(panel, outcome=\"sales\", unit=\"geo\", time=\"week\", first_treat=\"first_treat\")\n", + " .aggregate(\"total\")\n", + ")\n", + "\n", + "# The total's n is the treated geo-week mass - pin it against the raw frame.\n", + "frame_mass = int(((panel.first_treat > 0) & (panel.week >= panel.first_treat)).sum())\n", + "assert total.n[0] == frame_mass\n", + "\n", + "boost_spend_total = BOOST * frame_mass\n", + "true_total = BETA_SEARCH * boost_spend_total\n", + "\n", + "print(f\"per-geo-week ATT: {res.overall_att:.2f} (se {res.overall_se:.2f}; truth {BETA_SEARCH * BOOST:.0f})\")\n", + "print(f\"CS total incremental sales: {total.att[0]:,.0f}\")\n", + "print(f\"ImputationDiD total (cross-check): {imp_total.att[0]:,.0f}\")\n", + "print(f\"simulated truth: {true_total:,.0f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "9a82b59c", + "metadata": {}, + "source": [ + "## Export the lift test - grain reconciliation\n", + "\n", + "PyMC-Marketing scores a lift-test row against\n", + "`saturation(x + delta_x) - saturation(x)`: **every field must describe the same\n", + "observation** - same channel, same population, same period span - at the grain the\n", + "MMM models. Our MMM below is *national-weekly*, so the row must be too:\n", + "\n", + "- `x` - pre-boost national weekly search spend (`SEARCH_BASE * N_GEOS`; well\n", + " defined because budgets were frozen),\n", + "- `delta_x` - the summed weekly boost across treated geos (`BOOST * G_TREATED`),\n", + "- `delta_y` - the *weekly national* incremental: the per-geo-week ATT times the\n", + " number of treated geos.\n", + "\n", + "That last product is exactly what the container route computes:\n", + "`aggregation_result=res.aggregate('simple'), scale=G_TREATED` - the scale is the\n", + "number of treated geos per week, a physical quantity, not a fudge factor.\n", + "\n", + "Two practice notes. With a *carryover* channel you would restrict the aggregation\n", + "to post-ramp horizons before exporting. And mind the staggered rollout: national\n", + "spend moved through two intermediate levels (\\$1,650 and \\$1,800) before reaching\n", + "\\$1,950, so averaging the per-geo-week ATT across those regimes and pairing it\n", + "with the single `1500 -> 1950` transition is exact **here only because this\n", + "simulation's response is linear and homogeneous** (the DGP pins above). With a\n", + "real, possibly *saturating* channel, align the lift row with one actual spend\n", + "transition instead: either boost all test geos simultaneously (the cleaner\n", + "design for lift-test export), or keep the FULL panel in the estimator fit -\n", + "Callaway-Sant'Anna needs every cohort's pre-treatment baseline - and restrict\n", + "only the post-fit aggregation to the full-rollout calendar periods (for\n", + "example, average the `aggregate('calendar')` rows for the weeks where all\n", + "treated geos are boosted), so `delta_y` describes the same\n", + "`x -> x + delta_x` step the row claims." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2e8e7531", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:57:06.464885Z", + "iopub.status.busy": "2026-08-16T19:57:06.464819Z", + "iopub.status.idle": "2026-08-16T19:57:06.467896Z", + "shell.execute_reply": "2026-08-16T19:57:06.467616Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
channelxdelta_xdelta_ysigma
0search1500.0450.0932.76071482.569488
\n", + "
" + ], + "text/plain": [ + " channel x delta_x delta_y sigma\n", + "0 search 1500.0 450.0 932.760714 82.569488" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "x_national = SEARCH_BASE * N_GEOS\n", + "delta_x_national = BOOST * G_TREATED\n", + "\n", + "df_lift = to_pymc_marketing_lift_test(\n", + " channel=\"search\",\n", + " x=x_national,\n", + " delta_x=delta_x_national,\n", + " aggregation_result=simple,\n", + " scale=G_TREATED,\n", + ")\n", + "df_lift" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "a32ae70d", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:57:06.468770Z", + "iopub.status.busy": "2026-08-16T19:57:06.468721Z", + "iopub.status.idle": "2026-08-16T19:57:06.470513Z", + "shell.execute_reply": "2026-08-16T19:57:06.470213Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "ValueError: scale is not accepted with a level='total' container: this container's rows are already totals; scale would double-count - pass the container alone (aggregate('total') already applied the estimator's own finite-masked aggregation mass)\n" + ] + } + ], + "source": [ + "# Guardrail: a 'total' container is already in outcome units - passing any scale\n", + "# with it would double-count, and the exporter refuses loudly.\n", + "try:\n", + " to_pymc_marketing_lift_test(\n", + " channel=\"search\", x=x_national, delta_x=delta_x_national,\n", + " aggregation_result=total, scale=1.0,\n", + " )\n", + "except ValueError as err:\n", + " print(f\"ValueError: {err}\")" + ] + }, + { + "cell_type": "markdown", + "id": "75213ee4", + "metadata": {}, + "source": [ + "## Fit the national MMM - without and with the lift test\n", + "\n", + "The MMM sees only the national weekly series, where the demand-chasing history\n", + "lives. We fit the same model twice: once on the observational data alone, once\n", + "after `add_lift_test_measurements(df_lift)`. The\n", + "`add_original_scale_contribution_variable` call (before sampling) exposes the\n", + "channel-contribution posterior in original sales units, which we need to read the\n", + "ROI back out." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "0b171668", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:57:06.471413Z", + "iopub.status.busy": "2026-08-16T19:57:06.471353Z", + "iopub.status.idle": "2026-08-16T20:03:05.516321Z", + "shell.execute_reply": "2026-08-16T20:03:05.516001Z" + } + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Initializing NUTS using jitter+adapt_diag...\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Multiprocess sampling (4 chains in 4 jobs)\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "NUTS: [adstock_alpha, saturation_lam, saturation_beta, y_sigma, intercept_contribution]\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Sampling 4 chains for 3_000 tune and 500 draw iterations (12_000 + 2_000 draws total) took 179 seconds.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "The effective sample size per chain is smaller than 100 for some parameters. A higher number is needed for reliable rhat and ess computation. See https://arxiv.org/abs/1903.08008 for details\n" + ] + }, + { + "data": { + "text/html": [ + "
\n"
+      ],
+      "text/plain": []
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Initializing NUTS using jitter+adapt_diag...\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Multiprocess sampling (4 chains in 4 jobs)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "NUTS: [adstock_alpha, saturation_lam, saturation_beta, y_sigma, intercept_contribution]\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Sampling 4 chains for 3_000 tune and 500 draw iterations (12_000 + 2_000 draws total) took 175 seconds.\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "
\n"
+      ],
+      "text/plain": []
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "both fits done\n"
+     ]
+    }
+   ],
+   "source": [
+    "nat = panel.groupby(\"week\").agg(\n",
+    "    sales=(\"sales\", \"sum\"), search=(\"search_spend\", \"sum\"), tv=(\"tv_spend\", \"sum\")\n",
+    ").reset_index()\n",
+    "nat[\"date\"] = pd.date_range(\"2024-01-01\", periods=N_WEEKS, freq=\"W-MON\")\n",
+    "X = nat[[\"date\", \"search\", \"tv\"]]\n",
+    "y = nat[\"sales\"].rename(\"y\")\n",
+    "\n",
+    "# Counterfactual spend path with the boost removed (for the ROI readout below).\n",
+    "X_cf = X.copy()\n",
+    "X_cf.loc[X_cf.index >= FREEZE_WEEK, \"search\"] = x_national\n",
+    "\n",
+    "from pymc_marketing.mmm import MMM, GeometricAdstock, LogisticSaturation\n",
+    "\n",
+    "\n",
+    "def fit_mmm(lift=None):\n",
+    "    m = MMM(\n",
+    "        date_column=\"date\",\n",
+    "        channel_columns=[\"search\", \"tv\"],\n",
+    "        adstock=GeometricAdstock(l_max=4),\n",
+    "        saturation=LogisticSaturation(),\n",
+    "    )\n",
+    "    m.build_model(X, y)\n",
+    "    m.add_original_scale_contribution_variable(var=[\"channel_contribution\"])\n",
+    "    if lift is not None:\n",
+    "        m.add_lift_test_measurements(lift)\n",
+    "    m.fit(X, y, chains=4, draws=500, tune=3000, progressbar=False,\n",
+    "          random_seed=SEED, target_accept=0.999,\n",
+    "          nuts_sampler_kwargs={\"max_treedepth\": 14})\n",
+    "    return m\n",
+    "\n",
+    "\n",
+    "mmm_plain = fit_mmm()\n",
+    "mmm_cal = fit_mmm(df_lift)\n",
+    "print(\"both fits done\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c4377462",
+   "metadata": {},
+   "source": [
+    "## Read out the campaign ROI posterior\n",
+    "\n",
+    "The estimand mirrors the experiment exactly: the model-implied **incremental sales\n",
+    "of the boost campaign** - per-draw channel contributions at the observed spend\n",
+    "minus contributions with the boost removed - divided by the boost spend. For this\n",
+    "linear world the truth is `BETA_SEARCH = 2.0`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "fcd7d5b6",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-08-16T20:03:05.520327Z",
+     "iopub.status.busy": "2026-08-16T20:03:05.520268Z",
+     "iopub.status.idle": "2026-08-16T20:03:05.790557Z",
+     "shell.execute_reply": "2026-08-16T20:03:05.790208Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Sampling: []\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Sampling: []\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Sampling: []\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Sampling: []\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      " without lift test: ROI mean 3.52, 90% interval (1.78, 5.36)\n",
+      "    with lift test: ROI mean 2.21, 90% interval (1.91, 2.50)\n",
+      "             truth: 2.00\n",
+      "   DiD measurement: 2.07 ± 0.18\n"
+     ]
+    }
+   ],
+   "source": [
+    "def campaign_roi(m):\n",
+    "    def contrib(Xd):\n",
+    "        ds = m.sample_posterior_predictive(\n",
+    "            Xd, extend_idata=False, combined=False, progressbar=False,\n",
+    "            var_names=[\"channel_contribution_original_scale\"], random_seed=SEED,\n",
+    "        )\n",
+    "        return ds[\"channel_contribution_original_scale\"].sel(channel=\"search\").sum(\"date\")\n",
+    "\n",
+    "    # keeps (chain, draw) dims so ArviZ diagnostics below are chain-aware\n",
+    "    return (contrib(X) - contrib(X_cf)) / boost_spend_total\n",
+    "\n",
+    "\n",
+    "roi_plain = campaign_roi(mmm_plain)\n",
+    "roi_cal = campaign_roi(mmm_cal)\n",
+    "\n",
+    "for name, roi in [(\"without lift test\", roi_plain), (\"with lift test\", roi_cal)]:\n",
+    "    flat = np.asarray(roi).ravel()\n",
+    "    lo, hi = np.quantile(flat, [0.05, 0.95])\n",
+    "    print(f\"{name:>18}: ROI mean {flat.mean():.2f}, 90% interval ({lo:.2f}, {hi:.2f})\")\n",
+    "print(f\"{'truth':>18}: {BETA_SEARCH:.2f}\")\n",
+    "print(f\"{'DiD measurement':>18}: {df_lift.delta_y[0] / df_lift.delta_x[0]:.2f} \"\n",
+    "      f\"± {df_lift.sigma[0] / df_lift.delta_x[0]:.2f}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "c7d904be",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-08-16T20:03:05.791701Z",
+     "iopub.status.busy": "2026-08-16T20:03:05.791632Z",
+     "iopub.status.idle": "2026-08-16T20:03:05.881922Z",
+     "shell.execute_reply": "2026-08-16T20:03:05.881637Z"
+    }
+   },
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxUAAAG4CAYAAAAzAMoTAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAbTZJREFUeJzt3QeUU9X69/Fn6L33jqLSuyigIoogiIpdLCAqihXBio1iQeQqYEVRAcUuAl4VFRGkiCIidhCQXlV6hyHv+u3/e3IzM8lMMsnMZCbfz1phyMnJyT59P7udJJ/P5zMAAAAAyKR8mf0iAAAAABBUAAAAAIgaNRUAAAAAokJQAQAAACAqBBUAAAAAokJQAQAAACAqBBUAAAAAokJQAQAAACAqBBUAAAAAokJQgUxLSkqyW2+9Ne624OrVq13a/vOf/2T5b+l3hgwZ4n8/YcIEN01pCDRy5Eg75phjLH/+/Na8eXPLC2bPnu3W9YMPPsjppCSkOnXqWPfu3XM6GQjhmmuucfsop+i6pPPzn3/+iely33jjDatfv74VLFjQypQpE9Nl5wV5cfvoONbxnPrar7+e008/3Ro3bmyJft4lOoKKBPTee++5C8KUKVPSfNasWTP32axZs9J8VqtWLWvXrl2Wpev33393N8LUGfJQPv300xQZ+nj1xRdf2D333GPt27e38ePH2+OPP24bN250aV+yZElYy/jmm2/c/Dt27MjStCptU6dOzdLfQHzwAmDvVaBAAatevbq7KW/YsCHod3w+n8s0nXbaaS7DVKxYMWvSpIkNGzbM9u7dm2b+eMloxOJ6A7OlS5e64+PYY4+1cePG2csvv8xmyWD77Nu3zx1ngRlwRCfS+yeyD0FFAjrllFPc33nz5qWYvmvXLvv1119d5mL+/PkpPlu3bp17ed/Nqpv80KFDIwoqNH88ufrqq23//v1Wu3Zt/7SvvvrK8uXLZ6+++qr16tXLunXr5i6KSnskQYXmJ6hArCkgUKAwduxY69q1q02aNMk6dOhgBw4cSDFfcnKyXX755e4YFt3UR48e7WredGyefPLJtmXLllyzgyK93kRKmcply5ZZXqKM8dGjR23MmDEu83zppZfmdJLifvsoqNBxlpeCChUq6D6nvzkhvftnXjzvcpMCOZ0AZL9q1apZ3bp10wQVCxYscCWRl1xySZrPvPdZGVTkBWrepFegrVu3WtGiRa1QoUI5li7kDirtL168eLb+pgKJ1q1bu/9ff/31VqFCBRsxYoR99NFHKTKNTz75pKvlvOuuu1xzPs8NN9zg5uvRo4fLSE2fPt0SmbcP1fwlVpRRPXTokBUpUsRykq5lEstmPcp0q8YrL8iK7RNP1wqPCslieSyqAEP3Ry03WrE87xA5aioSlIKDH3/80ZU2eFQ70ahRI5fJ+Pbbb92NLPAzNZFQE57U1FxGTRwKFy7svv/ZZ5+l+HzNmjV288032wknnOAy1+XLl3eBS2AJoZpiaJp07NjR3yQjVOmOMi/PP/+8+39gE47UVP2sqmil7cQTT7Tvv/8+aJX1xRdfbOXKlXMXSmWwlKHKjNR9KvR/NXnSDcBLo+ZRWqRPnz4ppgejEuG7777b/V/BoDd/4PZT6XKrVq3c9tV6qERZNUuBli9fbhdddJFVqVLFrWeNGjXcfDt37vSnVemcOHGi/zcC29GGohLs+++/3y1XN7nzzjsvzW+nbpMb2DxGL9mzZ4/7fv/+/dPMt379ehesDR8+3L0/fPiwK6k67rjj3LromNIxPWPGDP93NI/27aZNmzJch82bN7t9oW2iY6Vq1ap2/vnnpynFVob51FNPdeksWbKknXPOOfbbb7+lmOfnn39266o+NEqbtsu1115r//77b9A27yoxv+KKK6xs2bIpgnbt0zZt2rgMlz5TqaCa0qWmgF/z6bf0m6+//rpFQ+snK1eu9E/TdUKBxPHHH+/fB4HOPfdc6927tzv3de2IlNdM6ocffnBNLHUc61hX7UmwjNt1111nlStXduusJps6ZlN755133Dmh/VSqVCnXTEslyOFeb8LZ19rPJUqUcNtKNZCa78orrwzZtlvn15133mk1a9Z0x5muier7pcKcYP3V3nzzTXdN1bypr6vhUp8KBX3aBjpPdH6lroUK5xqidRk8eLD7f8WKFdP0J3vhhRf8aVXB1S233JKmZjVwP+t41rGta4ccPHjQLb9evXpuGdpGajaq6enRdtI+UHCSWs+ePd35p2uULFq0yLp06eICZ+8Y07mZkWnTprn9r/VS2nRPeeSRR/zLDbV9dAzo/6LrlXecBW63cO4/3n3l66+/dvfSSpUquWtVerSP9Ts6Z7VcXdMuvPDCFOe1jj2dbzoutD20/8PpIxesT4Uno3PY+67OzwcffNA1udRxoJYS27Ztc4UWOle1T3XMKj/y008/pfh+evfPWJx3GeVpEBo1FQlKmRc1efjuu+/8mToFDroY6KWMpppCNW3a1P+ZOp/p4pM6Q/Phhx+6C51uqM8884zLuK5du9Y/rzLyar6jm5QuhMqovfjii+53laHSBUU3mNtvv919XzeZBg0auO96f1O78cYbXRWoMpFaj2Deeust2717t5tXFwuVtOqi+tdff/lLM5RJUKCkC9t9993nMhAqjVWp6+TJk+2CCy6IajsrbQpsFi5caK+88oqbpoywmpw8/PDDrpTXy8SF6q+iNP/555/29ttv26hRo9wNUbyb1WOPPWYPPfSQyziopPnvv/+2Z5991m1TBY4qNVMpp26mukHfdttt7kardvMff/yxu/GXLl3apVXfVwZV6RLdPDOi39f2vffee12GT01iOnXq5KqmdWMJl24i2t7vvvuuPf300ylqfLTuugF4GTbdLJW59dKrG5IyDIsXL7azzjrLzaP10/GjzG6ogM2jY1bHgraNbkhaDx1bOo69G5S2j5al7aiSfGVidBx7Abo3n76nY0w3PG1nLVfHgP4qw506+FXmVseE+rN4NzllQLSOOiZ0rKgUT+eqmtJ17tzZ/90VK1a4DIky2Urba6+95m6qyhzoZpgZXiClQCbwPN++fbvLkKp5ZDBqFqUAWseUmkJFSstXxlzHsTKDOg9vuukmt+5exk/Bja4bWm/d/JVpef/999066zj2AlLtAy3jzDPPdPtK/vjjD3cd0zwZXW/C3ddy5MgRN58+U0YlVKm79q0CbvVX0/5Ss7HPP//cFRjoWNW5HUj7WttA66lzPrOdT7U99V2dLzr+tM7a1oHBZzjXEJ3X+o764mlb6Hz17g86VnXM6rzXPlPzE82ja7+2eWDpsYJrZRR1P7jqqqtccKgCLG0bHWe69mg//PLLL26b6NqXXj+vyy67zBUwffLJJ/5AUbTP/vvf/7pjQ9cSndM6d3Td1LVe66RjXfevjOj6ofUdOHCg+6t9o+u3rjterV2w7aPMsc4FbRNd23QtF2+7RXr/0X1W6ddvB+vD5FGwo0EcZs6c6bazjnndC3Ve6L7uXdcVZGu767qqe4Qy+tqGOocVRGXFOexRUKbpCiJ0X9L/lR/QvlYadG6rOeVLL73kmmPqMwV1OjYiuX9Get6Fk6dBOnxISL/99ptyL75HHnnEvT98+LCvePHivokTJ7r3lStX9j3//PPu/7t27fLlz5/f17dv3xTL0PcLFSrkW7FihX/aTz/95KY/++yz/mn79u1L8/sLFixw873++uv+ae+//76bNmvWrLDW4ZZbbnHzp7Zq1So3vXz58r5t27b5p0+bNs1N/+9//+ufduaZZ/qaNGniO3DggH/a0aNHfe3atfMdd9xxGaZByxs8eLD//fjx4900pcHTu3dvt20Dff/9924+zR+OkSNHplmurF692u2bxx57LMX0X375xVegQAH/9B9//NF9X9s4PUqn0hsO7Scts3r16u4Y8bz33ntu+pgxY/zTateuHXS5HTp0cC/P559/7r47ffr0FPM1bdo0xXzNmjXznXPOOemmzzsOMlqf7du3u/m0jUPZvXu3r0yZMmnOgc2bN/tKly6dYnqw4/3tt992vzFnzhz/NB03mtazZ88U8y5fvtyXL18+3wUXXOBLTk5O8ZmOzcBtmnqZW7du9RUuXNh35513+jLiHatffvml7++///atW7fO98EHH/gqVqzolqH3ntGjR7t5p0yZEnJ5Otc0z4UXXuifpn3WqFGjDNOi+fTdp556yj/t4MGDvubNm/sqVarkO3ToUIp0TJo0yT+fPmvbtq2vRIkS/uOwf//+vlKlSvmOHDkS8jdDXW8i2dc6trSM++67L83y9Zn2kWfq1Klu3kcffTTFfBdffLEvKSkpxXVU8+kY0HU6s7zj67zzzksx/eabb3bTda2O5BoSuEwdL4HHnO4DnTt3TnG8Pvfcc27e1157Lc1+Hjt2bIrfeuONN9z6zp07N8V0zaf558+fH3I9dU7oGnTRRRelmO5dh7zzQ8eu3uvaG6lg5/SNN97oK1asWIp7R7Dto/+nvk9Eev/xztVTTjkl3WPao22u+Z9++uk0nwVeQ1Kvl86lxo0b+84444wU01Nfv71rf+C5E+457H33mGOOSfP72g6pr3m6jut6NGzYsLDun9Ged+HkaRAazZ8SlKJ9Rd1eXwlVL6rkw4v29dfrrK2+Fir5CNafQiVTgaXZKoFRlaVKaj2BpdVqkqKSKlVxq6RIJctZRSVYgaWtXomGlzZVtarESaUqKsVRMwG9lD6VPKq5UKhRcOKFSlRUyqd18NKvl0rIVfrtjeKlmghRCU2wZgLRUAm1SnQ8KjlXVbs60kdKx5NKo9Tsw6OSNTUpUqmmR8eOSvm0j0JR6azuExnVUnj9XVStrpK2YFTCp5Jwlb4FbmeVgJ500kkpRksLPN7VBEHzeSX3wY73fv36pXivkjrtU5XEpW5jnLqWo2HDhv7jWlSKqar9wPMvnG2u76lpgPadSkvV/CKweYXODwncz6l5n6n0NjNUA6JaRY/2id6rhFlNKkTHlI5t7QePSsFV66Dmc2oe4h0fup4FNocLVyT72qPS2Iwo7VqG0hpIzTJ0nKbui6LSWe3faKkZUiDVxnnpieQaEsqXX37pSrnvuOOOFMdr37593b1ANQiB1KREtXiBVNuke5JqwwPTcMYZZ7jP00uDzgmVbGt9dAx4VOOpGgDvvuX1c1ApvO5DkQg8p717hc47XUvVfCkzMnP/0TZN3WcvGNVyqHbL29ehriGB66Vrn1ooaL0ye18O5xz2qCYwdU22jg3vGFKeQ9tCNT66pmU2TZGed+HkaRAaQUWC0oVFgYPXd0IBhNppKrOfOqjw/gYLKjTMbGrKyAdmztRkQRkkrz2jLnbKxOjG7bXnzwqp0+YFGF7a1IRCFxVV+ys9gS+vbazX8S5e6cajddDNP/U6qLmHl35VJavqXk2wtP1101KTgVhsf/126mNLx1FmRtXRDUVV8cpYe8GPAgy1CQ5s2qDqbx0/ai+sJgaqylbgkRk6JtXERTcXNcVQkw81lVM/C48XvCiTk3o7q59D4HGizIKaG2hZumlqHm1/Cba9vc88avOs7RBOhjKj8083Zq1H4EsZwEA6DpSRVltqNV1QxkbbJFjA4AUXwYQTeKRHwWTqjqfav+IdS+qfpeMtdbDlNVvS56KmC/qumtkoOFLTi3DbRUeyr72MVEbt2720aR1Tb5/UaQ91XMTq/FSGSdvP26bhXkPSWy9Rxi+QMpTq45N6vZTRTz1ohdKgQoLUv+/t/4zSoAIk3We8vggKLpSZ1DXDy0QrSFMzFjXT0jVQfabUXC+jPhuitKkpkgpnlMFU2rxCjsxeQzNz/wn3mNA1RPsjVFNFj9dUUddX9enQb6vpVmbXKZxzOL11UV5EzZF0LAbmFXRtz2yaIj3vwsnTIDT6VCQwBQlqc6q2q15/Co/+77U5VG2GTkrdIFILVWoS2AFKpSW6eKskq23btu7CrAu92noGdgaPtYzS5v222nQqkx2MF2TFK62DtqUyxMHWV6U8nqeeesq1L1anQ2WOVHLjtbMOJ1MUjWCd6L1Mb+p0q+ZD7ZQVWKi0WH1j1D7Yq20RZfx14/TWRcGSbkbqFKg24ZHSsanOxvpN1eboRq9to5LEFi1a+I8VtbVXCW5qgTdvlTyqD5HOH7Xf1T7Q988+++ygx3sk/U4iPcbV0Tb1zVulvl4/KlGfFG/0J7Xl1nVBHcfVLt47frwbsG7umicYL6iLRel6tFRAoj492pc6N/TSNUjHVrBO3YEi2depS1djKZrjIpJzMZJrSFatl9KgwgH1pQpGBVLpUcZYNZNqw69jV/c1BRkKNjzegzp1vdPnOjYUbOq6qGmh1lOFFwpIFEyoMENBmTLhKjlXP7LM3sMyc/+J5TExd+5c19dA11J1slftsmr9dJ7ompvVgq2L+pXp2qv9oj4XCnR0bun6nJV5hUjzNAiNoCKBBT6vQkGFTlyPOnrqZqkmIeogqhLMzNKFXFWdungHNgtJPTJIqIxnKJHOn5oXJOlCqirP7BSrddUNThc7ZRy9EqH06Matl0bdUMZXnQSVEX/00UczlS5J3QRJ6VEpnNcZ0SvpCfaMDZUSpQ5WNeqGMvKqoVCwow5y6jSamm44akahl0omdXNUh9HMBBXetlSVuF5aJwUEOmY1Ko5XHa7ManrHikqz1DlSpaGqnfOk10wrWDp0A1XHxGifvq5MceomQBotKRRvhC2NiPTcc8+5zqPetULNR5TZeOCBB4LeeL2Ov5l9yrcGXkg9TKY66YrXSVnPf1Hwou0TmJH3mqAEPh9GpeEKFPXS/Kq9UKdPZVqUWUvvnApnX0dKaVNTIdXoBJaaBkt7LOnYCwwsdW5qe3jbNNJrSGpeuhWEBp7LqhFbtWpVWNtQaVATXHWsz+x1XcG8Oh6r+Z2aPmn9gg0YoGl6qXO6jmfVjKqDcqjrhu6BaoajZmKBz2XQuoUj1Ppk5f1H21P3bTXzCjXEqppIKThScBVYM6mgIrPCOYczyivo2qNnOgXSvcMboEQiOUZy6rxLVDR/SmAqndRFRZk31UgE1lToItOyZUvXNEIXiWieT6EMSOooX5nEwOH4xLsQhfuAt0jnT02ZBpXYKqMRbNhRjYCSVWK1rhpNRNtXmdjU21jvvWFMdaPVKDWBFFwoYxZY/a/fiXR7KjMZ2CxGNwZtTzU9CbzJqTQwsOmNqt5TDz0b+BBB1UBoRBX1/QlclqQenlWljMooBq5LuEPKqplV6iE2lV7dgLzlqSRRJZUqSQvWHts7VrzMdup9ofUIl2oCtF9UKpq6dC7S0jKd38qwBL4C+xkFo3NCtRdKs7ddNKKRSlSVcVRQkZrazavvirZTZkZ+Eh2fOhc9Olb0Xs0fVMghKtxQEy5lGgO/p+uJjgGVKAc7PrQ9vSDX26ehzqlw93WklHZd8xSsBVINmzJJqY/xWPGG3vZ4Abr3e+FeQ0LRMaUATqPkBH5fGUM1WQlnFCEFBLoH6cFlqanGIb2RjjyqldC+VU2UmrqlfjCfAv7U6+cF7ek1gQp2TuvYVOl+OLzRwFIfZ1l5/1EzLzVjTH2sBa6H1kvHXeB9WE2U0htpKxbncKR5BfW3Sd23JJL7Z06dd4mKmooEphuBxntWNaiCiNQnvYIMr3YhmqBCJZdqSqDmK2oaoY7fKjlIPTybLvC6qKh9u25GSpPaNeviG4yXXjXjUUZA31WTqkhvuFo3ZbDVCU6lRxrGTmnUsxECx8eOJWVaVfKrWgJlXnWRVCfQUG1mvXVVhk7rqNInlcBqOaplGDRokLshKEOq5akUTUMbasg9ZQbVjEdDU6qNsUojdfHXPtE20w0o8He0b9QMwXtIotKVHtUYaBuqxkDbTplRZfC1PT0qBVSwoSZAutmr6VJgDUBqasKgMeq1DuoEm7q0TceRbshKr35fw8lq+VpHT7hDyqokTSWkSpeWq+Yt+l2ti3c8KZOptsYKdhRsa7pulKpFUYZaNT66aWk+r0+GMqRqP67gKNxSTdG2035W9b86TSrTp3NBw3NqnwR7TkSsqemWjhVtN68juWotNLyozk+dHzpu1IRBNZ3al9rWGTUtSo/WTcvWcaxjVIGDmjBpOF5v/+t4ViZFzfjU8VOln9rvqmnVceeVROp4U98WXT9U26UaMWWmdY3xmnKld70JZ19HSuerSmG1b7WOqjHSsaEmfKolDmf4ZtG6azvrmAqn9FfzqZmLzj3tN+0rnV9ejVW415BQtG30XQUl+g39loJPZbp1fwkcYCEUbWs1XdKxpuZ52sbKCKpQQNNVmu410QtF+8o7dxQkBDZ9Em0zpUl9I7TOKghREKNzNr2aeN0HFYjrOqJ7jTKiunaGG+DrHNF1Rcezjmtdr1Qbq1dW3X/UzE+FPepHp+HMdR1RYKZru2rs1J9EwZ6u89pnOh7Uf0Pp0TbMbP+0cM7hjPIKKkzRvUTbXU2zVeiZujY7kvtnrM47hCmdkaGQAAYNGuSGS9MQdql9+OGH7rOSJUsGHcZOn2lY19RSDz+nITv79Onjq1Chghv2sUuXLr6lS5cGHWZ03Lhxbqg5DXGY0fCyStNtt93mhsDU0HDe4ewNJRpsiNBgQ/utXLnS16tXL1+VKlV8BQsWdMMTdu/e3Q2vmVVDynpD3DZs2NAN2xjO8LIa/ldp09CLqX9j8uTJbrhB/Y5e9evXd/tm2bJl7vO//vrLd+211/qOPfZYX5EiRXzlypXzdezY0Q0nGkj75bTTTvMVLVo0w+FYvaEBNVyqjiMNG6jvaajXNWvWpJlfQw0q/RoesH379r5FixalGVI2ULdu3dzyv/nmmzSfaXjANm3auKE/9ZtaXw196Q1bGMmQsv/884/bVlqGtp2GDT3ppJPckJTB1lnHr+bRdtT2vOaaa9y6eNavX++Gg1XaNN8ll1zi27hxY5pjJdjwk6mHhWzRooXbXmXLlnXbacaMGf7Pdf4EG1Y3vW0ayDtWgw2xqWEdtW56BZ77mq7vaf9pyFZtAw0ZO3ToUN+ePXuCpiXcIWU1n7ajhofVcrV+GpY0tS1btvivJxr+UUNypj53dO5qiFMdk5qnVq1abgjQTZs2hX29CWdfhzq3gw1t6Q1XO2DAAF+1atXctUbDhuo6FTjMZ3rXVtHQqTrmdV1Nj3d8/f777274TF3HdRzdeuutvv3796eZP6NrSEbHrPaVvqP10pDkN910U5o0pnc86NwdMWKE+9w75lu1auWOrZ07d/rC8cADD7j01atXL81nixcvdsM361jQ8nVs6DofuD9D0ZC2J598stvu2nf33HOPf/jrwGMm1PbRNUzromMx9XUgnPtPeudqKBquVdujbt26brlavo4D/Z7n1Vdfdcegtof2nX7HW4fMDCkbzjnsfTfY8OYaUlbDYVetWtVta11nNPx8sGtaqPtnVpx3oYZER1pJ+ifcAAQAsotKFFVSpTbgyNtU66TmGho+GOnTqGLeYAYAEE/oUwEg7qiNsZqaqFkEgP8Nbao+Bhp1CADiDX0qAMQNteNW+3gNEas2uIEPUgISXaNGjTL9cEEAyGrUVACIG3oismonFFyoY2Ww5wQAAID4Q58KAAAAAFGhpgIAAABAVAgqAAAAAEQl4Tpq6wm1epS8HpgSyaPeAQAAgETj8/ncAyP1gMN8+ULXRyRcUKGAombNmjmdDAAAACDXWLdundWoUSPk5wkXVKiGwtswpUqVyunkADmqfv367pkQVatWtaVLl6b5/I5Ph9j2/TutbNHSNrrbkBxJIwAAyDkayloF8l4eOpSECyq8Jk8KKAgqkOi8akz9DXY+PNrtHkv2HbX8ScE/BwAAiSGjbgMJF1QACF+1UjwnAgAAZIzRnwAAAABEhaACAAAAQFRo/gQgpHlrFtrBI4escIFCdkrtNmwpAHlGcnKyHT58OKeTAeS4ggULWv78+aNeDkEFgJAm/TTFtu3fYeWKliGoAJBnxtzfvHmz7dixI6eTAsSNMmXKWJUqVaJ6hhtBBQAASBheQFGpUiUrVqwYD8KFJXqQvW/fPtu6dat7ryHmM4ugAgAAJEyTJy+gKF++fE4nB4gLRYsWdX8VWOjcyGxTKDpqAwCAhOD1oVANBYD/8c6JaPoZEVQAAICEEk27cSAvSorBOUFQAQAAACAqBBUAAAB5yIQJE9xoPuGUTk+dOtXiwerVq116lixZ4t7Pnj3bvQ8cpUtprVevnmvzf8cdd+RgahEMHbUBAEDCWzV+YrZug7p9emfZsi+77DLr1q2b//2QIUNchtzLsGc3BQdTpkyxHj16hP2ddu3a2aZNm6x06dL+aTfeeKP16dPHbr/9ditZsqRdc801LujIKDA6/fTTrXnz5jZ69GiLlXB/O5EQVAAAAOSx0Xy8EX1yq0KFCrnnJnj27NnjRifq0qWLVatWLUfThuBo/gQgpDJFSrkH3+kvACBnfPzxx645k4bEFdU4qPT/vvvu889z/fXX21VXXZWm+ZP+P3ToUPvpp5/cd/TSNM8///xjF1xwgRv957jjjrOPPvooxW9//fXX1qZNGytcuLB7hoF+88iRI/7P69Spk6YGQLUCqh3xPhf9hn7be5+RwOZP+r9qJuSMM85w01X7MHHiRJs2bZp/vTRfsBoFrcOYMWP886mplfz666/WtWtXK1GihFWuXNmuvvpqtz08H3zwgTVp0sQFaBqCuFOnTrZ37163buH8dqKhpgJASE90HhTR1hk5aVG6n999VWu2NgBE6NRTT7Xdu3fbjz/+aK1bt3aZ5AoVKqTIyGravffeG7QplDLPn332mX355ZduWmCTIgUcTz75pI0cOdKeffZZu/LKK23NmjVWrlw527Bhg2tGpYz566+/bkuXLrW+fftakSJF/EFDRr7//nv37IPx48fb2WefnalnIKgp1LJly+yEE06wyZMnu/cKgpSWXbt2uWWL0pyagok///zTGjdubMOGDXPTKlas6IIVBSgKxkaNGmX79+932+/SSy+1r776yjW96tmzp9s2Coi0/efOneseFnfXXXfZH3/8keFvJxqCCgAAgDimIECl/woiFFTo74ABA1xAoGZBO3futBUrVliHDh3SfFel7CqJL1CgQIrmRB4FDMo8y+OPP27PPPOMLVy40AUAL7zwgtWsWdOee+45Vxpfv35927hxo8t8P/zww5YvX8YNXpSBF9WcBPv9cJtCKTDxMu/ecrRuBw8eTHe52nb6voKQwPm0Ti1atHDr7Hnttdfc+ioI0XZVjcyFF15otWvXdp+r1sITzm8nGpo/AQAAxDkFDAomVFKuEnNldhs0aGDz5s1ztRTqZ6DmS5Fq2rSp///Fixe3UqVKub4LotL4tm3bpniGQfv27V2Ge/369ZabqTnYrFmzXMDlvRQ0ycqVK61Zs2Z25plnukDikksusXHjxtn27dtzOtlxjZoKAACAOKc+BCpJV2a4YMGCLgOsaQo0lNkNVksRDi0rkAKIo0ePhv191VYo0AkUzVOZs4sCo3PPPddGjBiR5jP1HVEzrRkzZtg333xjX3zxhWsa9sADD9h3331ndevWzZE0xztqKgCE9PL3b9rT88e5vwCAnO9Xofb/XgDhBRV66f+hqPmP18k7EqoJWbBgQYqgYf78+a7TdI0aNfzNm9T/wKN+BqtWrUoTuGTm9zMS7noFm69ly5b222+/uY7jevZF4Es1Nl6ApZoZNTNTfxYtR0PjRvLbiYSgAkBIizf9at+uX+z+AgByTtmyZV1TpTfffNMfQJx22mm2ePFi1wcgvZoKZZyV0deoURrdSH0BwnHzzTfbunXr7LbbbnOdtDXa0eDBg23gwIH+/hTq7PzGG2+4Jlm//PKL9e7dO01nbP3+zJkzbfPmzTFtQqTl/vzzz64Tt9YrVA2J5lMNg0Z90nyqibnlllts27Ztrj+JOpOrydPnn3/unoOhYEHzq7/FokWLbO3atfbhhx/a33//7QKtSH47kRBUAAAA5AIKHJTh9YIKdVpu2LCh6yyskZFCueiii1zH644dO7qahbfffjus36tevbp9+umnruO2+hj069fPrrvuOnvwwQf98wwaNMilq3v37nbOOee4B9wde+yxKZbz1FNPuaZE6gStztGxotGftN7qvK71Ui1KMBqtSYGOtpXmU5CgPiiaX9uzc+fOru+EntKtDuUKmNS3ZM6cOW70q+OPP96ts9ZDQ9BG8tuJJMmXuiFcHqdqOY0EoJESdMAAiUzV1xoyUDeOYJ3u+n00yLbt3+GeVTH2vOEZLo8hZQHEswMHDrgSe7WJ17CoADI+N8LNO9NRG0C2IegAACBvovkTAAAAgKgQVAAAAACICkEFAAAAgKgQVAAAAACICkEFAAAAgKgw+hOAkNrXam17D+2z4oWKsZUAAEBIBBUAQrq6+UVsHQAAkCGaPwEAAACICkEFAABAHjJhwgQrU6ZMhvMlJSXZ1KlTI1p2nTp1bPTo0SGXsXTpUjv55JPdU5mbN28eYcqRm9H8CQAAJLyRkxZl6za4+6rWWbbsyy67zLp16+Z/P2TIEJfxX7JkScx/a9OmTVa2bFn/+8GDB1vx4sVt2bJlVqJECRfg3HHHHbZjx450l5MVaQz3txEbBBUAQrrj0yG2ff9OK1u0tI3uNoQtBQC5QNGiRd0rO1SpUiXF+5UrV9o555xjtWvXzpbfR/yg+ROAkA4cOWj7jxxwfwEAOePjjz92zZmSk5Pde5Xmq9nRfffd55/n+uuvt6uuuipN8yf9f+jQofbTTz+57+ilaZ5//vnHLrjgAitWrJgdd9xx9tFHH0WUtsDmT/r/Dz/8YMOGDXP/P/30061Pnz62c+dO/2+rRiK19NKoWgatW8WKFa1UqVJ2xhlnuPk8+n/Hjh2tZMmS7vNWrVrZokWLbPbs2WH9NmKHoAIAACCOnXrqqbZ792778ccf3fuvv/7aKlSo4DLOHk1TJj5YU6g777zTGjVq5Joq6aVpHmXmL730Uvv5559dk6krr7zStm3blql0atn6Hf2e/q8ARf0vlNn3fvuuu+6KKI2XXHKJbd261aZPn+4ClpYtW9qZZ57pT6PSW6NGDfv+++/d5wq0ChYsaO3atQvrtxE7BBUAAABxrHTp0q7TsxdE6O+AAQNckLFnzx7bsGGDrVixwjp06JDmu2oGpb4NBQoUcE2V9ApsGnXNNddYz549rV69evb444+75S1cuDBT6dSy9Tv6Pf1fGXqlXbUE3m/rs3DTOG/ePJeW999/31q3bu1qUv7zn/+4WpgPPvjAfXft2rXWqVMnq1+/vvtcQUizZs2sUKFCYf028khQMXz4cDvxxBNdlVWlSpWsR48ermNPRnRw6eDRyAJNmjSxTz/9NFvSCwAAkBMUMCiY8Pl8NnfuXLvwwgutQYMGLuOtWopq1aq5THWkmjZt6v+/OlgrEFDNQDxQ0yYFOeXLl3cBgfdatWqV67shAwcOdM2jFFg88cQT/ulIsKBCJ8Ett9xi3377rc2YMcMOHz5snTt3tr1794b8zjfffOMi6uuuu85F6ApE9Pr111+zNe0AAADZRU2bFEAoo63mPSpc1TQFGspPBaulCIeWFUgl+0ePHrV4oICiatWqrg9J4EsF0HfffbebR/0kfvvtN9c5/KuvvrKGDRvalClTcjrpCSlHR3/67LPPUrxXpxzVWKhN3GmnnRb0O2PGjLGzzz7bfzA98sgjLiB57rnnbOzYsdmSbgAAgJzoVzFq1Ch/AKGgQqXz27dvd30SQlFTIK+Td3YL97eDzaf+E5s3b3bNovR8jFCOP/5491KTMBU8jx8/3nU+z8n1TkRx1adCPfSlXLlyIedZsGCBq+IK1KVLFzcdAAAgL9KzINRU6c033/R3yFYB7OLFi+3PP/9Mt6ZCGXI1GVIpv0Z7Ongw+0b002+rxmHmzJnut/ft2xd2GpXfa9u2rWuR8sUXX9jq1atdi5UHHnjAjfC0f/9+u/XWW11tzZo1a2z+/Pmuw7aahUXy28hjQYWq2vSAkvbt21vjxo1DzqeItXLlyimm6b2mB6ODcteuXSleAAAAuY0CB5W8e0GFCmHV3EedkE844YSQ37voootcKw8NvaqhWd9+++1sS7NGYerXr58bzUm//eSTT4adRjXFUr9ZBU8aHla1EZdffrkLIJT3y58/v/3777/Wq1cv95lGseratasb0SqS30ZsJPnU4ycO3HTTTW64MLUX1NBgoagqa+LEia56y/PCCy+4A2jLli1p5ldbO+/gSl0ros5IQCLTuaZRQ6pXr27r169P83m/jwbZtv07rFzRMjb2vOFZ/kTarHzCLAAcOHDAlYbXrVvXDfYCIONzQwXyGkkro7xzXDxRW1VXerDLnDlz0g0oRNF46uBB71M/0dEzaNAgNzJA4IapWbNmjFIO5G19W19hh5IPWaH8hXI6KQAAII7laFChSpLbbrvN9dJXezhFRxlR2zq1jVNTKY86amt6MIULF3YvAJFrVa0Jmw0AAMR3UKHhZN966y2bNm2ae1aF1y9CVSzeg1nUTk5NM/RMC+nfv79rU/jUU0+54cPeeecd11nn5ZdfzslVAQAAABJWjnbUfvHFF137LHU40jjE3uvdd9/1z6MnJerR6h51ulEgoiBCT0zUExWnTp2abuduAAAAAHm4+VNGvEfSB9Ij2PUCkLX+2rbGjhxNtgL58tsx5WqzuQEAQPx21AYQn56cNzai0Z8AAEBiipvnVAAAAADInQgqAAAAAESFoAIAAABAVAgqAAAAcrFrrrnGevTokeW/M2TIEGvevLnllKSkJDfiZ07SAEJKx44dOyL+7quvvmqdO3e2rHb55Ze7Ry9kN4IKAACAXGzMmDE2YcIEi0fZGQjoEQWBD0eOp+UdOHDAHnroIRs8eLB/2rhx4+zUU0+1smXLulenTp1s4cKFYQU2LVu2dA93rlevXpp9/+CDD9pjjz3mHtuQnQgqAAAAcjE9NLhMmTI5nYxcQY8zOHLkSLb/7gcffGClSpWy9u3bpwgOevbsabNmzbIFCxZYzZo1XU3Ghg0bQi5n1apV7uHPHTt2tCVLlrig5/rrr7fPP//cP4+e3XbsscfapEmTLDsRVAAAAMQ5ZUqbNGliRYsWtfLly7tS7b179wZt/qQS9ttuu81lOFUCXrlyZVcqrvn79OljJUuWdCXc06dP939Hpd2pAxPVMKimIZTvv//ezjrrLKtQoYILbDp06GCLFy/2f16nTh3394ILLnDL8d7LtGnTXGl7kSJF7JhjjrGhQ4emyOwvX77cTjvtNPd5w4YNbcaMGeluH22Dr7/+2tXa6Lf0Wr16tb+5kta1VatWrnR/3rx5QZuMaXtp26W3PM8PP/xgrVu3tmLFirkHMy9btizd9L3zzjt27rnnppj25ptv2s033+yalNWvX99eeeUVO3r0qM2cOTPkcsaOHWt169Z1zZsaNGhgt956q1188cU2atSoFPPpt/Sb2YmgAgAAII5t2rTJlWhfe+219scff7iM8oUXXpjuQ4QnTpzoMvtqTqMA46abbnIPDlYGWBl/lYhfffXVtm/fvkyna/fu3da7d2+XSf/222/tuOOOs27durnpXtAh48ePd+vgvZ87d6716tXL+vfvb7///ru99NJLLqhRkx1RxlrrV6hQIfvuu+9cRvree+9NNy3K/Ldt29b69u3rfksvlfx77rvvPnviiSfc9mvatGmG65bR8h544AGXsV+0aJEVKFDA7Zv0aBspCEmP9sXhw4etXLlyIedRjYYCykBdunRx0wO1adPG7fuDBw9aduHhdwAAIKEps7d58+Zs/90qVaq4TGlGlKFVKb4y2rVr13bTVGuRnmbNmrm29TJo0CCXoVaQoUyyPPzww/biiy/azz//bCeffHKm0n/GGWekeP/yyy+72g6V8Hfv3t0qVqzopmua1tWjWgll8hWQiGoqHnnkEbvnnntcn4Mvv/zSli5d6pr0VKtWzc3z+OOPW9euXUOmRTUlCkJUcxD4W55hw4a5WpVwZbQ8BUCqmRGtyznnnOP6TahmJTV16lb/Bm9dQlHgpHlSBw2BdJyq5imQ3u/atcv279/varJEyzl06JCb3ztmshpBBYCQRnUdbD7zWZKFrv4GgNxOGa/02rHnNAUIZ555pgskVCqtWgY1eVHTplACS+Pz58/vmkwFBiJexnTr1q2ZTteWLVtc4KKaEy0nOTnZlbavXbs23e/99NNPNn/+fH/NhOi7ypTr+6pNUK1AYCZctQbRyKiWIFKB27dq1arur7ZBrVq10syrzL4ECzg8CvrUXEnbMr35wuUFF9HUREWKoAJASEULRn9hA4B4F6wkOp5+V0GB+hR888039sUXX9izzz7rmt+oaZDa1wdTsGDBFO/VJyBwmtdXQk2NJF++fGmaU6kpTnpU0/Dvv/+6pkIqDVd/BWX+VUKenj179rjaCtW8pBaLDHUwxYsXT/E+M+sbKL1tmZoCOs2zfft2C+Y///mPCypUQ5NR0ywdMwrmAum9OoF7gYRs27bN/fVqi7IDQQUAAEho4TRBymnKlGrkIL3UdEmZ+ClTptjAgQNjsnxlPtUXQp25vQy4RhdKj2obXnjhBdePQtatW2f//PNPmsy3aiECqYO2Ojars3gw6oCsZanZl1cLoD4bGVFzpdS/ld76/vrrrymmaX0Dg4VIlpceLUedzdV/JPVzKp588klXY6OmXuHUpiho+/TTT1NMU8CZuiZH61ajRg3X5C270FEbAAAgjqlGQn0KFPyoadGHH35of//9t8t8x8pJJ53k+g/cf//9tnLlSnvrrbcyfPaFOma/8cYbrrmS0njllVemKC0Xjfik0YzUxMwrqVdQ9Prrr7vait9++819X01/vD4g6lNw/PHHu5oQNZVSx27VzGREv6V0aJQmBTehag68/iDankqHRppSX47UQUYky8uImq2ps3agESNGuGdXvPbaa+63tI30Uk2OR/1h1Knd069fP/vrr79c/xP1O1FQ995779mAAQNSLFvbLDsetBeIoAJASB8v+9Le+/Vj91dGTlqU7gsAEHtq2jJnzhxXI6DMtjLfGnkovY7LkdKIQ3qugUrB1ffi7bffdk/QzugJ0QoUVPOgkaRuv/12q1SpUop5lE6VpKuPRIsWLfwZ7I8//tg15TrxxBNdR3ENiep1KFbTJNXCqC+CRjHScxgC+1+Ectddd7mmYqoVUE1Een07lAZl6JU5VxpUSxOYeY90eRm57rrr3LYNfCCdOsqrqZj6x6hGxnupOZRHtTWBv6vmbp988onbpupro+2roWi1Ph71TdFwwF6n/OyS5EtvPLI8SL3j1aNfO1UnKZDIVDWqzonVq1e39evXp/m830eDbNv+HVauaBkbe97wLA8c7r4qth3pACCQMlt6eJgyZlnVdh8IRUP6KgBT7UNWUrCioExBWyzOjXDzztRUAAAAAFls5MiRVqJEiSzfzuoXos782Y2O2gAAAEAWq1OnjnsQYVZTc7GcQE0FAAAAgKgQVAAAAACICkEFAAAAgKgQVAAAAACICkEFAAAAgKgQVAAAAACICkPKAgipbtmaVr5YWStVOOvH1QYAALkXNRUAQrr31JvtsU73uL8AgNwhKSnJpk6dGvNlrV692r1fsmSJ//P58+dbkyZN3APXevToEZPfRO5EUAEAABDnrrnmGpeh10sZ+MqVK9tZZ51lr732mh09ejTFvJs2bbKuXbv633vf06t48eJ23HHHueX98MMPEaWhZs2abtmNGzf2Txs4cKA1b97cVq1aZRMmTLAhQ4a494gf2i9lypTJ8t8hqAAAAMgFzj77bJepV43B9OnTrWPHjta/f3/r3r27HTlyxD9flSpVrHDhwim+O378ePfd3377zZ5//nnbs2ePnXTSSfb666+H/fv58+d3yy5Q4H+t51euXGlnnHGG1ahRI1syrvHo8OHDOZ2EuEBQAQAAkAsoUFCmvnr16tayZUu7//77bdq0aS7AUGl0es2flOHXd+vUqWOdO3e2Dz74wK688kq79dZbbfv27WH9fmDzJ+////77r1177bXu/0rD0KFD7aeffvLXjASmK5BqStRc6vHHH3e1LkrfsGHDXHB09913W7ly5VygomAo0Lp16+zSSy9182ue888/36XF8/3337sanAoVKljp0qWtQ4cOtnjxYv/nPp/P1abUqlXLbc9q1arZ7bffnuG289bDW+93333XLbtIkSL25ptvus9eeeUVa9CggZtWv359e+GFF9Jsu/fee89OPfVUK1q0qJ144on2559/ujS3bt3aSpQo4WqY/v777xS/H85yP/zwQxdkFitWzJo1a2YLFixwn8+ePdv69OljO3fu9O8TrX9WIKgAENKIuS/YA18+6f4CAOKPagmUiVSmMlIDBgyw3bt324wZMyL+rtcUqlSpUjZ69Gj3/8suu8zuvPNOa9SokXvvTQvlq6++so0bN9qcOXPs6aeftsGDB7tal7Jly9p3331n/fr1sxtvvNHWr1/vrxHo0qWLlSxZ0ubOnev6cygjrhqcQ4cOuXm0Pr1797Z58+bZt99+65p6devWzU2XyZMn26hRo+yll16y5cuXuwBCfUIidd9997laoj/++MOl6c0337SHH37YHnvsMTdNwdJDDz1kEydOTPE9reODDz7oAh3V+FxxxRV2zz332JgxY9w6rVixwi3HE+5yH3jgAbvrrrtcwHf88cdbz549XYDWrl07t3+0n7x9ovmyAqM/AQhp1fZ1tm3/DitXNDGrtAEkjo+XfWkfL5sZ1qh4qQevUMGLrpcZ6X7Cmdb9hE4Wayq9/vnnnzP1PQks6Y+0KZRKvlUjoP+LMvnKLHvv06Oahmeeecby5ctnJ5xwgj355JO2b98+VwMjgwYNsieeeMIFCJdffrmrHVD/EZXc63dFNRmqSVCJvGpgFGQFevnll93nX3/9tQtY1q5d69LWqVMn1zdFNRZt2rSJeP3vuOMOu/DCC1MEC0899ZR/Wt26de333393wYuCHI8y9ApCREGJMv8zZ8609u3bu2nXXXdditqdSJZ7zjnnuP+rtkiBnQIU7WPtH22vcPZJNAgqAABAwtt3+IArRMmIhtlObdfBPWF9V7+RFdSkx8tkR/o9ycx3Y0EZXwUUHjWDCuwErsClfPnytnXrVvdezaqUUVZNRaADBw64vh2yZcsWVxOgIEPfS05OdoGKggm55JJLXMn9Mccc42o4VItx7rnnpugnEg41V/Ls3bvX/b4Cgr59+/qnq6ZAGfpATZs2TbG+ElhTomne+mZ2uVWrVnV/tRwvcMwOBBUAACDhFStYJKxa2WDP7dG0cL6r38gKahajEuzMfE8y891YUE1BIG9kq9TTvNGt1Lm8VatW/j4MgSpWrOj+qvRe/TzUnKh27dqu30Tbtm39zaPUbGvZsmX25ZdfumZfN998s40cOdLVZOi39XtesJVeR2yNouVRumTcuHGu83sgBUaBAtfPC+ZSTwtc32iWm3pUsKxGUAEAABKemiVltmlSTj7LR/0SfvnlF9c/IlJeW3s1BYqVQoUKudqBrKDO6WoCValSJZfuYNTPQh2ZVQPhdez+559/UsyjTtKqndDrlltucaX52oZavoIT9TvwqN+FajrSo9qFatWq2V9//eU6v8dKrJablfskEEEFAABALnDw4EHbvHmzyyCqmc9nn31mw4cPd30FevXqle53d+zY4b6rZWjEIbXJVydlDSkby6FgNbqUnlmhDsMavUlNlVIPb5tZylirVkEjPmmkKC1/zZo1rpO6OjvrvTpmv/HGG6550q5du9xIUgoiPOqvoO2nkn+NlDRp0iT3uWo1RH0ynnvuOVe7ofnuvffeNLUnwQwdOtSNIqVmSWpWpe28aNEiN7KWnuWRWbFYrvaJaj3Ud0Od+rXeesUaoz8BAADkAgoi1F5emURlMGfNmuU6OmtY2dTNYVLTsKL6rkrlb7rpJteheuHChW70oVi66KKLXNo0vKlK/d9+++2YLVsZYY0Upc7V6risYVbV30B9Kryai1dffdVluFXrcPXVV7sMuWo2PAqg1JxIHaPVD0HNoP773/+6vhuiTtFqIqVhX7Vt1AE6nAz49ddf7zqQq+O4+khouFkFMNE2LYvFcjUClEbS0khc2ifqEJ8VknypG47lcYpaFe1pvN5QVWdAolCpzoYNG9yY596QfYH6fTTIP/rT2POG28hJi7I0PXdf9b+ObwAQa8p8qhRdGTKN+Q8g43Mj3LwzzZ8AxI1wghYCDwAA4g/NnwAAAABEhZoKAOk+qEnjqmfVMIgAACBvIKgAEFJWPPkVAADkPTR/AgAACSXBxqgBsuWcIKgAAAAJwXveQEYPMwMSzb7/f06E80yOUGj+BCCk/YcPmM98lmRJVpR+FQByOT3LQc8p2Lp1q3uv5w8kJSXldLKAHK2hUEChc0LnRkbPO0kPQQWAkAZMH5riORUAkNtVqVLF/fUCCwDmAgrv3MgsggoAAJAwVDOhJ0vrKcuHDx/O6eQAOU5NnqKpofAQVAAAgISjTFQsMlIA/g8dtQEAAABEhaACAAAAQFQIKgAAAABEhaACAAAAQFQIKgAAAABEhaACAAAAQFQIKgAAAABEhedUAAjpnlP62ZGjyVYgH2O5AwCA0AgqAIR0TLnabB0AAJAhmj8BAAAAiApBBQAAAICo0PwJQEg/bPzFDiUfskL5C1mrak3YUgAAICiCCgAhjVv0lm3bv8PKFS1jrc4bzpYCAABB0fwJAAAAQFQIKgAAAABEhaACAAAAQFQIKgAAAABEhaACAAAAQFQIKgAAAABEhaACAAAAQO4NKubMmWPnnnuuVatWzZKSkmzq1Knpzj979mw3X+rX5s2bsy3NAAAAAKIMKvbu3RvpV9JdVrNmzez555+P6HvLli2zTZs2+V+VKlWKWZoA/E+RAoWtaIEi7i8AAEDMnqhduXJlu/TSS+3aa6+1U045xaLRtWtX94qUgogyZcpE9dsAMja62xA2EwAAiH1NxaRJk2zbtm12xhln2PHHH29PPPGEbdy40bJT8+bNrWrVqnbWWWfZ/Pnz05334MGDtmvXrhQvAAAAADkYVPTo0cP1fdiwYYP169fP3nrrLatdu7Z1797dPvzwQzty5IhlFQUSY8eOtcmTJ7tXzZo17fTTT7fFixeH/M7w4cOtdOnS/pe+AwAAACB2knw+ny/ahTz77LN2991326FDh6xChQou2LjvvvusWLFi4SckKcmmTJnigpZIdOjQwWrVqmVvvPFGyJoKvTyqqVBgsXPnTitVqlREvwXkNTVq1HAFBNWrV7f169dnOP/ISYssp919VeucTgIAAAlj165drmA+o7xzxH0qPFu2bLGJEyfahAkTbM2aNXbxxRfbdddd5zImI0aMsG+//da++OILy2pt2rSxefPmhfy8cOHC7gUgcm8smWx7D+2z4oWK2dXNL2ITAgCA2AQVauI0fvx4+/zzz61hw4Z2880321VXXZWi43S7du2sQYMGlh2WLFnimkUBiL35axfZtv07rFzRMgQVAAAgdkFFnz597PLLL3cdpE888cSg8+i5Ew888ECGy9qzZ4+tWLHC/37VqlUuSChXrpxr0jRo0CDXNOP11193n48ePdrq1q1rjRo1sgMHDtgrr7xiX331VbbUiAAAAACIUVCh50Jk1FeiaNGiNnjw4AyXtWjRIuvYsaP//cCBA93f3r17u2ZV+q21a9f6P1efjTvvvNMFGkpD06ZN7csvv0yxDAAAAABxHlSULFky6APn/v33XzctOTk57GVp5Kb0+okrsAh0zz33uBcAAACAXDykbKggQCMsFSpUKBZpAgAAAJAXayqeeeYZ/9Cv6stQokQJ/2eqnZgzZ47Vr18/a1IJAAAAIPcHFaNGjfLXVOgBdPnz5/d/phqKOnXquOkAAAAAEkvYQYVGZhJ1itawsmXLls3KdAEAAADIqx21Z82alTUpAQAAAJArhRVUaKjXRx55xIoXL+4f9jWUp59+OlZpA5DDWlZtbHsO7bMShdIfRhoAACS2sIKKH3/80Q4fPuz/fyjqxA0g77jhxCtzOgkAACCvBBWBTZ5o/gQAAAAgqudUpLZr1y6bOnWqLV26NNpFAQAAAEiEoOLSSy+15557zv1///791rp1azetSZMmNnny5KxIIwAAAIC8FFToIXennnqq+/+UKVPccyt27NjhHo736KOPZkUaAeSQ+74Ybv0+GuT+AgAAxCyo2Llzp5UrV879/7PPPrOLLrrIihUrZuecc44tX7480sUBiGM7Duyybft3uL8AAAAxCypq1qxpCxYssL1797qgonPnzm769u3brUiRIpEuDgAAAECiPfzujjvusCuvvNJKlChhtWvXttNPP93fLEr9KgAAAAAkloiDiptvvtnatGlj69ats7POOsvy5fu/yo5jjjmGPhVALjZy0qI003YXOGSWZLZ736GgnwMAAGQqqBCN+KRXIPWpAAAAAJB4Ig4qkpOTbcKECTZz5kzbunWrHT16NMXnX331VSzTBwAAACCvBRX9+/d3QYVqJho3bmxJSUlZkzIAAAAAeTOoeOedd+y9996zbt26ZU2KAAAAAOTtIWULFSpk9erVy5rUAAAAAMj7QcWdd95pY8aMcU/SBpC31UxuY3WOnOL+AgAAxKz507x582zWrFk2ffp0a9SokRUsWDDF5x9++GGkiwQQp8r7js3pJAAAgLwYVJQpU8YuuOCCrEkNAAAAgLwfVIwfPz5rUgIAAAAgMfpUyJEjR+zLL7+0l156yXbv3u2mbdy40fbs2RPr9AHIQftth+237e4vAABAzGoq1qxZY2effbatXbvWDh48aGeddZaVLFnSRowY4d6PHTs20kUCiFPLCky3w0n7rKCvmDU/0jOnkwMAAPJKTYUefte6dWvbvn27FS1a1D9d/Sz0lG0AAAAAiSXimoq5c+faN998455XEahOnTq2YcOGWKYNAAAAQF6sqTh69KglJyenmb5+/XrXDAoAAABAYok4qOjcubONHj3a/z4pKcl10B48eLB169Yt1ukDAAAAkNeaPz311FPWpUsXa9iwoR04cMCuuOIKW758uVWoUMHefvvtrEklAAAAgLwTVNSoUcN++ukne+edd+znn392tRTXXXedXXnllSk6bgMAAABIDAUy9aUCBeyqq66KfWoAAAAA5M2g4qOPPgp7geedd1406QEAAACQF4OKHj16pHivztk+ny/NNAk2MhQAAACABB/9ScPIeq8vvvjCmjdvbtOnT7cdO3a4l/7fsmVL++yzz7I+xQCyTcMj51uzw5e7vwAAADHrU3HHHXfY2LFj7ZRTTvFP02hQxYoVsxtuuMH++OOPSBcJIE4VsmI5nQQAAJAXn1OxcuVKK1OmTJrppUuXttWrV8cqXQAAAADyalBx4okn2sCBA23Lli3+afr/3XffbW3atIl1+gAAAADkteZPr732ml1wwQVWq1Ytq1mzppu2bt06O+6442zq1KlZkUYAOWRr0lI7mnTY8vkKWiVfffYDAACITVBRr14999C7GTNm2NKlS920Bg0aWKdOnfwjQAHIGzbm/9EOJ+2zgr5iVukIQQUAAIjhw+8UPHTu3Nm9AAAAACS2iPtUAAAAAEAgggoAAAAAUSGoAAAAAJB9QcWRI0fs9ddfTzGcLAAAAIDEFlFQUaBAAevXr58dOHAg61IEAAAAIG83f9ID7pYsWZI1qQEAAACQ94eUvfnmm90TtfXAu1atWlnx4sVTfN60adNYpg8AAABAXgsqLr/8cvf39ttvT/HcCp/P5/4mJyfHNoUAckwRX2nLb4WsoK8oewEAAMQuqFi1alWkXwGQS9VP7pbTSQAAAHkxqKhdu3bWpAQAAABAYgQVsnLlShs9erT98ccf7n3Dhg2tf//+duyxx8Y6fQAAAADy2uhPn3/+uQsiFi5c6Dpl6/Xdd99Zo0aNbMaMGVmTSgAAAAB5p6bivvvuswEDBtgTTzyRZvq9995rZ511VizTByAHrcw/247YAStgRezY5NPZFwAAIDY1FWrydN1116WZfu2119rvv/8e6eIAxLHdSZtsV74N7i8AAEDMgoqKFSsGffidplWqVCnSxQEAAABItOZPffv2tRtuuMH++usva9eunZs2f/58GzFihHsoHoD4NHLSojTT9uw7nOIvAABAtgQVDz30kJUsWdKeeuopGzRokJtWrVo1GzJkSIoH4gEAAABIDBEHFXpqtjpq67V79243TUEGAAAAgMSUqedUeAgmAAAAAIQVVLRs2dJmzpxpZcuWtRYtWrjailAWL17MVgUAAAASSFhBxfnnn2+FCxd2/+/Ro0dWpwkAAABAXgsqBg8e7P4mJydbx44d3VO0y5Qpk9VpAwAAAJDXnlORP39+69y5s23fvj3rUgQgblQ8eoJVTm7k/gIAAMSso3bjxo3dMyrq1q0b6VcB5DLVj7bM6SQAAIC8+ETtRx991O666y77+OOPbdOmTbZr164Ur0jMmTPHzj33XPecC3X+njp1aobfmT17tus4rj4e9erVswkTJkS6CgAAAABysqaiW7du7u95552XYhQon8/n3qvfRbj27t1rzZo1s2uvvdYuvPDCDOdftWqVnXPOOdavXz9788033YhU119/vVWtWtW6dOkS6aoAAAAAyImgYtasWRYrXbt2da9wjR071jW70tO8pUGDBjZv3jwbNWoUQQUAAACQW4KKDh06WE5ZsGCBderUKcU01VDccccdOZYmIC9bUuBtO5y0zwr6ilnzIz1zOjkAACCv9KmQuXPn2lVXXWXt2rWzDRs2uGlvvPGGqzXISps3b7bKlSunmKb36suxf//+oN85ePBgVP0+AAAAAMQ4qJg8ebKrHShatKh7erYy7bJz5057/PHHLd4MHz7cSpcu7X/VrFkzp5MEAAAA5CmZGv1JfRvGjRtnBQsW9E9v3769CzKyUpUqVWzLli0ppul9qVKlXJATzKBBg1zA473WrVuXpWkEAAAAEk3EfSqWLVtmp512WprpqgXYsWOHZaW2bdvap59+mmLajBkz3PRQNPSsXgAAAADipKZCtQUrVqxIM139KY455piIlrVnzx5bsmSJe3lDxur/a9eu9dcy9OrVyz+/hpLVg/fuueceW7p0qb3wwgv23nvv2YABAyJdDQAAAAA5FVT07dvX+vfvb9999517LsXGjRvdMyP0QLybbropomUtWrTIWrRo4V4ycOBA9/+HH37YvdfD9bwAQzSc7CeffOJqJ/R8Cw0t+8orrzCcLAAAAJCbmj/dd999dvToUTvzzDNt3759rimUmhcpqLjtttsiWtbpp5/uHpoXSrCnZes7P/74Y6TJBgAAABAvQYVqJx544AG7++67XTMoNWFq2LChlShRImtSCAAAACBvNX+69tprbffu3VaoUCEXTLRp08YFFHv37nWfAQAAAEgsEQcVEydODPqgOU17/fXXY5UuAHHgmOTT7fgjXdxfAACAqJs/6UnU6v+gl2oqihQp4v8sOTnZDfVaqVKlcBcHIBco5aua00kAAAB5KagoU6aM60+h1/HHH5/mc00fOnRorNMHAAAAIK8EFbNmzXK1FGeccYZNnjzZypUr5/9M/Stq165t1apVy6p0AgAAAMjtQUWHDh38D6irVauWq5kAkLftStpkPku2JMtPUygAABC7jtp//PGHzZ8/3//++eeft+bNm9sVV1xh27dvj3RxAOLYX/ln258FPnd/AQAAYvacCj2fYsSIEe7/v/zyi3sK9p133umaR+n/48ePj3SRABC2kZMWpX+Nuqo1WxMAgHgPKtT8Sc+nEPWtOPfcc+3xxx+3xYsXW7du3bIijQAAAADyUvMndcret2+f+/+XX35pnTt3dv9Xx20NOwsAAAAgsURcU3HKKae4Zk7t27e3hQsX2rvvvuum//nnn1ajRo2sSCMAAACAvFRT8dxzz1mBAgXsgw8+sBdffNGqV6/upk+fPt3OPvvsrEgjAAAAgLxUU6HhZD/++OM000eNGhWrNAEAAADIy0GFJCcn29SpU93wstKoUSM777zzLH/+/LFOHwAAAIC8FlSsWLHCjfK0YcMGO+GEE9y04cOHW82aNe2TTz6xY489NivSCQAAACCv9Km4/fbbXeCwbt06N4ysXmvXrrW6deu6zwAAAAAklohrKr7++mv79ttv3RCynvLly9sTTzzhRoQCkHc0P9Izp5MAAADyYk1F4cKFbffu3Wmm79mzxz3DAgAAAEBiiTio6N69u91www323Xffmc/ncy/VXPTr18911gYAAACQWCIOKp555hnXp6Jt27ZWpEgR91Kzp3r16tmYMWOyJpUAAAAA8k6fijJlyti0adNs+fLlbkjZpKQka9CggQsqAOQtG/IttmQ7ZPmtkFU/2jKnkwMAAPLScyrkuOOO8wcSCiwA5D1/51tmh5P2WUFfMYIKAAAQu+ZP8uqrr1rjxo39zZ/0/1deeSUziwIAAACQaDUVDz/8sD399NN22223uX4VsmDBAhswYIB7XsWwYcOyIp0AAAAA8kpQ8eKLL9q4ceOsZ8//jV+vUZ+aNm3qAg2CCgAAACCxRNz86fDhw9a6des001u1amVHjhyJVboAAAAA5NWg4uqrr3a1Fam9/PLLduWVV8YqXQAAAADy8uhP6qj9xRdf2Mknn+ze60F46k/Rq1cvGzhwoH8+9b0AAAAAkLdFHFT8+uuv1rLl/41Xv3LlSve3QoUK7qXPPAwzCwAAACSGiIOKWbNmZU1KAAAAACTWw+8A5H0lfVXtiO+AFbAiOZ0UAAAQxwgqAIR0bPLpbB0AAJA1T9QGAAAAAA9BBQAAAICoEFQAAAAAiAp9KgCEtDT/p3Y4ab8V9BW1+snd2FIAACAoggoAIR1I2mmHk/ZZsh1iKwEAgJBo/gQAAAAgKgQVAAAAAKJCUAEAAAAgKgQVAAAAAKJCUAEAAAAgKgQVAAAAAKJCUAEAAAAgKgQVAAAAAKLCw+8AhFQtuYUdTTps+XwF2UoAACAkggoAIVXy1TfzsYEAAED6aP4EAAAAICoEFQAAAACiQvMnACEdsn32f+2fkqyQFWNLAQCAoAgqAIT0e4FpdjhpnxX0FbPmR3qypQAAQFA0fwIAAAAQFYIKAAAAAFEhqAAAAAAQFYIKAAAAAFEhqAAAAAAQFYIKAAAAAFFhSFkgjxg5aVFOJwEAACQoaioAAAAARIWgAgAAAEBUaP4EIKQTjnQ1M5+ZJeWZZmB3X9U629ICAECiIKgAEFJRK8PWAQAAGaL5EwAAAICoEFQAAAAAiArNnwCE9G/SSjtqRyyfFbDyvmPZUgAAICiCCgAhrcu/0A4n7bOCvmJW/ghBBQAAiOPmT88//7zVqVPHihQpYieddJItXLgw5LwTJkywpKSkFC99DwAAAECCBhXvvvuuDRw40AYPHmyLFy+2Zs2aWZcuXWzr1q0hv1OqVCnbtGmT/7VmzZpsTTMAAACAOAoqnn76aevbt6/16dPHGjZsaGPHjrVixYrZa6+9FvI7qp2oUqWK/1W5cuVsTTMAAACAOAkqDh06ZD/88IN16tTpfwnKl8+9X7BgQcjv7dmzx2rXrm01a9a0888/33777beQ8x48eNB27dqV4gUAAAAgjwQV//zzjyUnJ6epadD7zZs3B/3OCSec4Goxpk2bZpMmTbKjR49au3btbP369UHnHz58uJUuXdr/UiACAAAAIA81f4pU27ZtrVevXta8eXPr0KGDffjhh1axYkV76aWXgs4/aNAg27lzp/+1bt26bE8zAAAAkJfl6JCyFSpUsPz589uWLVtSTNd79ZUIR8GCBa1Fixa2YsWKoJ8XLlzYvQAAAADkwZqKQoUKWatWrWzmzJn+aWrOpPeqkQiHmk/98ssvVrVq1SxMKQAAAIC4ffidhpPt3bu3tW7d2tq0aWOjR4+2vXv3utGgRE2dqlev7vpGyLBhw+zkk0+2evXq2Y4dO2zkyJFuSNnrr78+h9cEyHv00LvAvwAAAHEZVFx22WX2999/28MPP+w6Z6uvxGeffebvvL127Vo3IpRn+/btbghazVu2bFlX0/HNN9+44WgBxFaj5PPZpAAAIP6DCrn11lvdK5jZs2eneD9q1Cj3AgAAABAfct3oTwAAAADiC0EFAAAAgNzf/AlAfFqdb54dSTpoBXyFrc7RU3I6OYihVeMnZjhP3T692eYAgLAQVAAIaUe+dXY4ad//jf50lA0FAACCo/kTAAAAgKhQUwEAyLXNtGiiBQDxgaACAHIhMtsAgHhC8ycAAAAAUSGoAAAAABAVggoAAAAAUaFPBQAgS/AsDABIHNRUAAAAAIgKNRUAQip/9Fj/E7UBAABCIagAEFLNo23YOgAAIEMEFQCQS/sjAAAQL+hTAQAAACAq1FQAQB58WnVuqenILekEAKSPoAJASL8U+MAO2T4rZMWsyZGL2VIAACAoggoAISXbYTuadNiSfYfZSsiV4uFZGfGQBgDIavSpAAAAABAVaioAALlWPPTJiIc0AEBOo6YCAAAAQFSoqQAABEUJPAAgXAQVAJDNyKwDAPIaggoAAHK5vPBsFQC5G0EFAAAgMAEQFYIKAHFj71+rMpyn+DF1LadRKox4O+YAIKcRVAC5xMhJi7L9N+skt7ejlmz5LL/lFWTOEI84LgHkdgQVAEIq46vF1kGeR80TAESP51QAAAAAiAo1FQAApIOmSQCQMYIKACHttX/Ml5RsSb78VtwqsKWAXIrACEBWI6gAENLyAjPscNI+K+grZs2P9GRLAQCAoAgqAABAttSG8BA+IO+iozYAAACAqFBTAQAxRvt1AECiIagAkG1PxI6Hp2EDAIDYo/kTAAAAgKhQUwEAAOICTzcHci+CCgB5qolVRiZasQzn6V1zX1S/AQBAoiGoAAAA2YJBDIC8i6ACQK6pZQAAAPGJoAJASE2OXMTWAeBQywAgPQQVAELKb4XYOgAAIEMEFQDCbprEcyYA5CRGhwLiF0EFkMBBw9HkIyn+AgAAZAZBBYCQ/i271pLzHbH8RwtY+e212FIAACAoggoAIf1bdp0dKXjQChwu7IIKRm8CAADBEFQAAICEEW2/jHjo1xHOSFzZkQ4gUL4U7wAAAAAgQtRUAAAA/H88jwPIHIIKAACAGKFpEhIVQQUAAMgTqGUAcg5BBRAHhgx7P6eTAAAAkGkEFUBclJ4VYz8AQIKgRgV5EaM/AQAAAIgKNRUAQipysKQlHyls+ZMLsZUAAEBIBBUAQqq1oWlCbp2J69Jvjta75r5sSwsAZEY8PKQPiYWgAgkvFhde2scCAIBERp8KAAAAAFGhpgLIALUQAIBEQ/MpRIqgAkBIa6v/bMn5D7mO2onavwIAAGSMoAJASAcK77YjBQ9agcOF2UoAACAkggrk+qZHjGABAACQswgqkOvR5wEAgPi791Lol1gIKgAgQjzHAkBuR4Ec8mRQ8fzzz9vIkSNt8+bN1qxZM3v22WetTZs2Ied///337aGHHrLVq1fbcccdZyNGjLBu3bpla5oBAACQc4FNLJ4jldEyYhF81Y3yN3JLjU+OBxXvvvuuDRw40MaOHWsnnXSSjR492rp06WLLli2zSpUqpZn/m2++sZ49e9rw4cOte/fu9tZbb1mPHj1s8eLF1rhx4xxZB2QeJSUAAAC5X44HFU8//bT17dvX+vTp494ruPjkk0/stddes/vuuy/N/GPGjLGzzz7b7r77bvf+kUcesRkzZthzzz3nvov4ai9J0BBecxkAAIDcLEeDikOHDtkPP/xggwYN8k/Lly+fderUyRYsWBD0O5qumo1AqtmYOnVqlqcXAAAA4cnqgkUKLuNLjgYV//zzjyUnJ1vlypVTTNf7pUuXBv2O+l0Em1/Tgzl48KB7eXbu3On+7tq1y/K61ZPeyvLf+PkFaofCcfCAxSWfz+f/e/DAvjSfH95/0I4cPmS+Ixb0cwT38vL0t0zPGvvZdAAQB7IjH7Mrgzzn7v3p3xNyOs/q/b6XZ4jb5k9ZTX0vhg4dmmZ6zZo1cyQ9QDzas3u7PTH8mgzmIoCMlSditiQAQNy75aac/X6M7N6920qXLh2fQUWFChUsf/78tmXLlhTT9b5KlSpBv6PpkcyvplWBzaWOHj1q27Zts/Lly1tSUlJM1gP/i2QVrK1bt85KlSrFZskF2Ge5D/ss92Gf5T7ss9yHfZZ1VEOhgKJatWrpzpejQUWhQoWsVatWNnPmTDeCk5fp1/tbb7016Hfatm3rPr/jjjv809RRW9ODKVy4sHsFKlOmTEzXAykpoCCoyF3YZ7kP+yz3YZ/lPuyz3Id9ljXSq6GIm+ZPqkXo3bu3tW7d2j2bQkPK7t271z8aVK9evax69equGZP079/fOnToYE899ZSdc8459s4779iiRYvs5ZdfzuE1AQAAABJTjgcVl112mf3999/28MMPu87WzZs3t88++8zfGXvt2rVuRChPu3bt3LMpHnzwQbv//vvdw+808hPPqAAAAAASNKgQNXUK1dxp9uzZaaZdcskl7oX4omZmgwcPTtPcDPGLfZb7sM9yH/ZZ7sM+y33YZzkvyZfR+FAAAAAAkI7/tSsCAAAAgEwgqAAAAAAQFYIKAAAAAFEhqEBMzJkzx84991z3YBQ9VFAjciF+aYjmE0880UqWLGmVKlVyz4lZtmxZTicL6XjxxRetadOm/jHY9Wye6dOns81yiSeeeMJdGwOfsYT4M2TIELefAl/169fP6WQhHRs2bLCrrrrKPdS4aNGi1qRJE/eoAWQ/ggrEhJ4t0qxZM3v++efZornA119/bbfccot9++237uGRhw8fts6dO7v9iPhUo0YNlzH94Ycf3A3zjDPOsPPPP99+++23nE4aMvD999/bSy+95IJCxL9GjRrZpk2b/K958+bldJIQwvbt2619+/ZWsGBBV8jy+++/u+eYlS1blm2WqEPKIvfr2rWreyF30LNgAk2YMMHVWCjDetppp+VYuhCaagIDPfbYY672QoGhMkGIT3v27LErr7zSxo0bZ48++mhOJwdhKFCggFWpUoVtlQuMGDHCatasaePHj/dPq1u3bo6mKZFRUwHAdu7c6bZCuXLl2Bq5QHJysr3zzjuuZknNoBC/VCN4zjnnWKdOnXI6KQjT8uXLXVPeY445xgWEeggv4tNHH31krVu3ds8uU8FYixYtXACPnEFNBZDgjh496tp5qwqZJ9PHt19++cUFEQcOHLASJUrYlClTrGHDhjmdLISgwG/x4sWu+RNyh5NOOsnV3J5wwgmu6dPQoUPt1FNPtV9//dX1QUN8+euvv1yN7cCBA+3+++9359rtt99uhQoVst69e+d08hIOQQWQ4FSSqhsm7YbjnzI6S5YscTVLH3zwgbtpqn8MgUX8WbdunfXv39/1WSpSpEhOJwdhCmzGqz4wCjJq165t7733nl133XVsxzgsFFNNxeOPP+7eq6ZC97OxY8cSVOQAmj8BCezWW2+1jz/+2GbNmuU6AiO+qfStXr161qpVKzeClwZHGDNmTE4nC0Gof9LWrVutZcuWro2+XgoAn3nmGfd/NWFD/CtTpowdf/zxtmLFipxOCoKoWrVqmkKVBg0a0GQth1BTASQgn89nt912m2s+M3v2bDq25eJSuoMHD+Z0MhDEmWee6ZqrBerTp48bnvTee++1/Pnzs91yAXW0X7lypV199dU5nRQEoWa7qYdD//PPP13tErIfQQViduENLMlZtWqVa6ahjr+1atViK8dhk6e33nrLpk2b5toJb9682U0vXbq0G+cb8WfQoEGuaYbOp927d7v9p4Dw888/z+mkIQidV6n7KBUvXtyNpU/fpfh11113uZHWlCnduHGjDR482AWAPXv2zOmkIYgBAwZYu3btXPOnSy+91BYuXGgvv/yyeyH7EVQgJjRufseOHf3v1WlK1OZbnd4QX9SxTU4//fQU0zUs3zXXXJNDqUJ61JSmV69ervOogj+191ZAcdZZZ7HhgBhZv369CyD+/fdfq1ixop1yyilu2Gb9H/FHD3FVjbsKXYYNG+Zq3UePHu1G7UL2S/KpHQQAAAAAZBIdtQEAAABEhaACAAAAQFQIKgAAAABEhaACAAAAQFQIKgAAAABEhaACAAAAQFQIKgAAAABEhaACAAAAQFQIKgDEnSFDhljz5s2z/Hf09PAePXpk+e/kRdm1jzLj6quvtscffzwmy1q9erUlJSXZkiVLLC84/fTT7Y477rDcqk6dOu6JyZk9TmN1zn/22WduuUePHo16WUBeQVABIO7cddddNnPmzCz/nTFjxtiECRMskWVVYOVlxr1XuXLlrEOHDjZ37tw0827bts1ldGvXrm2FChWyatWq2bXXXmtr166NOK0//fSTffrpp3b77bfHfJ0QG7k9sJGzzz7bChYsaG+++WZOJwWIGwQVAOJOiRIlrHz58ln+O6VLl7YyZcpk2fKTk5MTviTzyy+/tE2bNtmcOXNcsNC9e3fbsmVLioDi5JNPdvONHTvWVqxYYe+88477e+KJJ9pff/0V0TZ/9tln7ZJLLnHHUCI7dOhQTichzzp8+LA/yH3mmWdyOjlA3CCoAPIoVcs/+eSTVq9ePStcuLDVqlXLHnvsMf/n9957rx1//PFWrFgxO+aYY+yhhx7y3ywDmw289tpr7rvKpN18880uo6zlVqlSxSpVqpRimaJS6RdffNG6du1qRYsWdcv+4IMPUswT7m97jhw54kqeFQAo2ND3e/funaLUWqWfmueee+5xpeJKn5aTntQl3+EsY8eOHXbjjTda5cqVrUiRIta4cWP7+OOP3Weq9VAaP/roI2vYsKHb7iptP3jwoKt9qV69uhUvXtxOOukkmz17tn+Z3ve0nBNOOMFtl4svvtj27dtnEydOdE0+ypYt69Km7e8Jd7mff/65NWjQwO1DlbAqk+9tZy1/2rRp/hoF7/sZ7aNwaX9pO2o73X///bZr1y777rvv/J8/8MADtnHjRhdU6JjRsXbaaae5NKsk+JZbbgn7t7RtdKyde+65Kaa/8MILdtxxx7n9pf2mbRvYjOWUU07xH1sKelauXJnu7/z6668urdqeWp6aW/3zzz/+z5WGJk2auONfy+zUqZPt3bs36LK0vbXdP/nkE2vatKlLo4Is/UagefPm2amnnuqWWbNmTXcsBC5Tx8gjjzxivXr1slKlStkNN9wQMv06n2699VYXVFeoUMHtW5/P5/98+/btbjk65rT/ta7Lly9PsYzJkydbo0aN3DGu337qqafC2uY6577++mtXS+gdc6rVCmbr1q1uX2qd69atG7RWQOfX+eef7/aF1vvSSy9NEbRmJKP979W4vfvuu66mTevjpUNpW7RoUYbHC5AwfADypHvuucdXtmxZ34QJE3wrVqzwzZ071zdu3Dj/54888ohv/vz5vlWrVvk++ugjX+XKlX0jRozwfz548GBfiRIlfBdffLHvt99+c/MUKlTI16VLF99tt93mW7p0qe+1115TTsT37bff+r+n9+XLl3e/tWzZMt+DDz7oy58/v+/333+P6LebNWvmf//oo4/6ypUr5/vwww99f/zxh69fv36+UqVK+c4//3z/PB06dHDThgwZ4vvzzz99EydO9CUlJfm++OKLkNuod+/eES0jOTnZd/LJJ/saNWrkpq1cudL33//+1/fpp5+6z8ePH+8rWLCgr127dm79tI327t3ru/766920OXPmuH0xcuRIX+HChd1vBH7vrLPO8i1evNj39ddfu23YuXNn36WXXuq2v35H2/+dd97xpzfc5Xbq1Mn3/fff+3744QdfgwYNfFdccYX7fPfu3W75Z599tm/Tpk3udfDgwUzto9T0PR0LP/74o3u/b98+31133eWmTZ8+3b89y5Qp47vhhhuCLuOxxx5z2//ff/8Nur9S07bT8jdv3uyfpvXW8ffWW2/5Vq9e7eYZM2aM//MPPvjAN3nyZN/y5ctdWs8991xfkyZNXNqCrcf27dt9FStW9A0aNMgdi1qe9lvHjh3d5xs3bvQVKFDA9/TTT7vv/vzzz77nn3/ebetgZs2a5Zav/aJjSvN3797dV6dOHd+hQ4fcPNq3xYsX940aNcrtW+2XFi1a+K655hr/cmrXru2O3f/85z9ufr2C0TGu87p///7u+Jw0aZKvWLFivpdfftk/z3nnnefSo+NqyZIl7pyvV6+ePz2LFi3y5cuXzzds2DB3jus4K1q0qPub0TbfsWOHr23btr6+ffv6j7kjR44ETWvXrl3dMbZgwQL3mzrW9TvaDqJ91Lx5c98pp5ziPtd1qFWrVm4dQx2nqY+hcPe/9ofm++uvv9w+9ui88NYbSHQEFUAetGvXLpe5DAwiMqIMqW7IgTdjZTa0LI8yF7q5ejdcOeGEE3zDhw/3v9cNWJn+QCeddJLvpptuiui3AzMCunFrHo8yIbVq1UoTEChzEejEE0/03XvvvREFFekt4/PPP3eZKWWkglHmQuuvjJhnzZo1LoO1YcOGFPOeeeaZLmMa+L3AjOCNN97otn9gZlTbX9OjWa4yuNqeobZBZvdRal5mTJlAZYgVHOi9luFlTpX51zQvk5iagkh9/t1334WV1ilTprhtcvToUf80ZQSV2Q48jtPz999/u9/85ZdfggYVCrYU7AVat26dm0fHhQI3/V+Z6XB4QUVgsKggStvt3Xffde+vu+66NIGXCgl0LO7fv98fVPTo0SPD39MxroAhcBvp+NY0UdCi9Chw8fzzzz8uPe+99557r6BUgVSgu+++29ewYcOwtrnSoKAmPdqWSsfChQv90xTEBR4vCsK0v9euXeufRwF44PcyCirC3f+jR48OOr+COxVCAPD5aP4E5EF//PGHaxpz5plnhpxH1fnt27d3TVPUdODBBx9M0zFWzRpKlizpf69mDGrWky9fvhTT1EwhUNu2bdO8V5oi+W3Pzp07XXOGNm3a+Kflz5/fWrVqlWZeNR8JVLVq1TRpy0h6y9AIQDVq1HDNgkJRR+PAZfzyyy+uWY6+o3X1XmoCEthsQs1Mjj322BTbVds/sG9A4LbO7HLD3SaR7KOMlvPjjz+65jJqiqcmWWrWFCiw6U009u/f75rjqLmK56yzznIdwNWES82U1HRFzco8atbTs2dP97maz2ibS6h1VUfwWbNmpdjm9evXd59puzdr1sydd2r+pL4d48aNc82JMhJ4zqjpnZrBeeeMflPbLfA3u3Tp4po4rlq1yv+91q1bh7Wd1LwqcBvpt7UddDzpNwsUKOCa0nnULCgwPfqrYyOQ3nvLyGibh8NLR+B5ru0c2AdK86gpmF4eXZ80T+D1Jj3h7v9Q21ZNsyJdNyCvKpDTCQAQe7rRpWfBggV25ZVX2tChQ13mRG2r1Tk2dbvo1Jk/ZUSCTYtkWMVwfzszok1bRsvIaLt68wRm2Pbs2eOCoB9++MH9DRQYMES6raNZbkaZ+FjuI2X41LZeL7Xlv+CCC1x/AWX+K1asmG4GUNOVXgUj4VD/AGXw1ElZwZ0oKF68eLHru/DFF1/Yww8/7PqSfP/99+631S5eGWBl/tWRXNtX/T9CdXTWdtd3RowYkeYzBWzaFzNmzLBvvvnG/Z46jqvfiPqRqF9AZug31Y8n2IhW6oPiUb+aeJDRNo8n4e7/UNtWAw3oOAZAR20gT1IGTpnbUMOyKsOjG6kyOyqB0/xr1qyJ2e9/++23ad6ro3BmflsZWpXQK0PiUWmoMi3ZTTUQ69evtz///DPs77Ro0cKlV7UDyhwHvlQLkFmxWq4y34Gdv7Py+FBnXZU+qxOvqMZLHWvfeust27x5c5paB82noEYl9+HwOvf//vvvKabrN9VZWgMM/Pzzz67z7VdffWX//vuvLVu2zNXCqHZBx2hGtQotW7a03377zZVop97uXsZTgZBK7hWUqZZG23jKlClhnzNKg44x75zRb2qdUv+eXl7wFInAjvLeb2sfKyDSbyr4C5zH206qBRDNM3/+/BTL0HvVmnkBbqhtHuqYS021EkqHgmaP0qCBEjxKx7p169zLo+2keby0picz+z/QgQMHXO2UzkUABBVAnqQRSjR6j0Yxev31192NTxmHV1991X2uDISq91X6rM80LGJGmZ5IvP/++27UKGWMBg8ebAsXLnSjzWT2t2+77TYbPny4G6VImYD+/fu7m39gjUB20OgvGpnooosucqXRanoyffp0N4JMKMpoqdRfo+l8+OGH7jvaHlofjfiTWbFarjLHyvRpu2oEI43wlFXHh/aXStufeOIJf5MRPaROQZCazGhbKoOo4WcVTCgtzz//fNjLV4mxMuAaKcmjEbWUfjVdU2Ck80Gl0WrOo9GN1LTn5ZdfdkPYKtM7cODAdH9Do1GpdFpNZhToavtopKo+ffq4jLIy41onjQqkbah98/fff/sDhFCGDRvmCgFUi6MRklTr4o1MpnNZgZ7OIa2HmuzoXPDOqUgpXVpP7fO3337b1abonBLte42m1LdvX7cd1fTqqquuciOMabrceeedLq0abUrnuEYQe+6559xIZBltc++Y03ZSoKFjLlhtoubVSGWqodG8Ci6uv/76FLWFClrUzEzngQoZdPzrfNB5Gk5TsMzs/0C6pqrGLXVzTyBR0acCyKM0TKRu/mp6oAzNZZdd5m9Lf95559mAAQNcpkSlu8qwaP5YUQmtMqQq2VeGQhkXr+QwM7+tTJUyccow6AbutSlX8JTd1DdAz09QerROCtwyKnUdP368S7v2hzJLyiwqQxrYdCUzYrFcZR71XWXClClXiXNWHh8aCljBgjKhokydMmcdO3Z0GUj1/1Dthf5qXdTWPRLKeAYOParmNsrYn3HGGe480LMwdDxqOFTVlOg4VYZVTV60ziNHjkx3+Woio22kfd65c2eXqdWD3PQ7Wp7a5Sso6tatmwv8VAquZmMaljU9CrSUsVcfAtXa/Pe///XXQug8Ul8ZZeA1rKxKxnVeKy2ZoWNGNUHqp6QgSb8bOAStjiulQ8Or6nxTczk9UNBrSqfA7b333nPbTttNaVFQpGAoo20uCj5Uo6HzR8dcqP4rSofWUUHChRde6NKoYawDg1QFVwoOFOwryNDxon484cjM/g+kdVJAo35LAMyS1FudDQEgVnSjV6l2Vjyl2aOSTWVWlPlUaSngUWZZQZIylrmhBFn9DhRQqeYt3vobIDTVsOg4U41UZvvKAHkNHbUBxD01oVCHT5VYalQrlXKruc8VV1yR00lDnFHzGNWOBT6MDog1Nd1Snx8CCuB/CCoAxD01U9CQmmo2ocpVNVXQE5gzaqeOxKQnowNZSc0Fwx3CF0gUNH8CAAAAEBU6agMAAACICkEFAAAAgKgQVAAAAACICkEFAAAAgKgQVAAAAACICkEFAAAAgKgQVAAAAACICkEFAAAAgKgQVAAAAACwaPw/GGcyqFkpPa4AAAAASUVORK5CYII=",
+      "text/plain": [
+       "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig, ax = plt.subplots(figsize=(8, 4.5))\n", + "bins = np.linspace(0.5, 6.5, 80)\n", + "ax.hist(np.asarray(roi_plain).ravel(), bins=bins, density=True, alpha=0.55,\n", + " label=\"without lift test\", color=\"#c44e52\")\n", + "ax.hist(np.asarray(roi_cal).ravel(), bins=bins, density=True, alpha=0.65,\n", + " label=\"with lift test\", color=\"#4c72b0\")\n", + "ax.axvline(BETA_SEARCH, color=\"black\", lw=2, label=\"simulated truth (2.0)\")\n", + "did_roi = df_lift.delta_y[0] / df_lift.delta_x[0]\n", + "ax.axvline(did_roi, color=\"#55a868\", lw=2, ls=\"--\", label=\"DiD lift measurement\")\n", + "ax.set_xlabel(\"campaign incremental ROI (sales per boost dollar)\")\n", + "ax.set_ylabel(\"posterior density\")\n", + "ax.set_title(\"What the lift test buys: search-ROI posterior, before vs after calibration\")\n", + "ax.legend()\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "4326b1bb", + "metadata": {}, + "source": [ + "The uncalibrated posterior is wide and biased high - the model still credits\n", + "search with demand it chased historically. The calibrated posterior concentrates\n", + "near the truth: the lift likelihood anchors the response curve at the one spend\n", + "change we actually experimented on." + ] + }, + { + "cell_type": "markdown", + "id": "a83be6fe", + "metadata": {}, + "source": [ + "## Acceptance checks\n", + "\n", + "Hard checks on the claims this notebook makes: sampler health first, then the\n", + "calibration claims with slack margins against draw noise. The saturation\n", + "parameters sit on a ridge (`beta` and `lam` trade off along the fitted curve), so\n", + "the demanding sampler settings above (`target_accept=0.999`, deeper trees, long\n", + "tuning) are what it takes to traverse that geometry cleanly - with them, BOTH fits\n", + "sample with zero divergences, zero maximum-tree-depth events, and whole-model\n", + "maximum R-hat under PyMC's own 1.01 warning bar, so the wide uncalibrated\n", + "posterior is a faithful picture of the model's uncertainty rather than a sampling\n", + "artifact. ArviZ's chain-aware `az.rhat` and bulk/tail `az.ess` are additionally\n", + "checked on the derived campaign-ROI posterior - the quantity the notebook\n", + "reports." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "64294128", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T20:03:05.883166Z", + "iopub.status.busy": "2026-08-16T20:03:05.883107Z", + "iopub.status.idle": "2026-08-16T20:03:06.202559Z", + "shell.execute_reply": "2026-08-16T20:03:06.202135Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "plain: divergences=0, treedepth events=0, whole-model max Rhat=1.0069, Rhat(campaign ROI)=1.0016, ESS bulk=458 tail=366\n", + "calibrated: divergences=0, treedepth events=0, whole-model max Rhat=1.0062, Rhat(campaign ROI)=1.0014, ESS bulk=2104 tail=1197\n", + "error: 1.52 -> 0.21; 90% width: 3.59 -> 0.59\n", + "all acceptance checks passed\n" + ] + } + ], + "source": [ + "import arviz as az\n", + "\n", + "ess_floors = {\"plain\": 300, \"calibrated\": 800}\n", + "for name, m, roi in [(\"plain\", mmm_plain, roi_plain), (\"calibrated\", mmm_cal, roi_cal)]:\n", + " div = int(m.idata.sample_stats[\"diverging\"].sum())\n", + " mtd = int(m.idata.sample_stats[\"reached_max_treedepth\"].sum())\n", + " with warnings.catch_warnings():\n", + " # constant/deterministic posterior variables have undefined R-hat;\n", + " # arviz emits RuntimeWarnings computing them - excluded below.\n", + " warnings.simplefilter(\"ignore\", RuntimeWarning)\n", + " rh_all = az.rhat(m.idata.posterior)\n", + " finite_rhats = np.concatenate(\n", + " [np.asarray(rh_all[v]).ravel() for v in rh_all.data_vars]\n", + " )\n", + " max_rhat_model = float(finite_rhats[np.isfinite(finite_rhats)].max())\n", + " rhat_roi = float(az.rhat(roi).values)\n", + " ess_bulk = float(az.ess(roi).values)\n", + " ess_tail = float(az.ess(roi, method=\"tail\").values)\n", + " print(f\"{name}: divergences={div}, treedepth events={mtd}, \"\n", + " f\"whole-model max Rhat={max_rhat_model:.4f}, \"\n", + " f\"Rhat(campaign ROI)={rhat_roi:.4f}, ESS bulk={ess_bulk:.0f} tail={ess_tail:.0f}\")\n", + " assert div == 0, f\"{name} fit: {div} divergences - sampling is unhealthy here\"\n", + " assert mtd == 0, f\"{name} fit: {mtd} maximum-tree-depth events - raise max_treedepth\"\n", + " assert max_rhat_model < 1.01, (\n", + " f\"{name} fit: some model parameter has not converged (max Rhat {max_rhat_model:.4f})\"\n", + " )\n", + " assert rhat_roi < 1.01, f\"{name} fit: campaign-ROI chains disagree (Rhat {rhat_roi:.4f})\"\n", + " assert ess_bulk >= ess_floors[name], (\n", + " f\"{name} fit: campaign-ROI bulk ESS too low ({ess_bulk:.0f})\"\n", + " )\n", + " assert ess_tail >= 300, f\"{name} fit: campaign-ROI tail ESS too low ({ess_tail:.0f})\"\n", + "\n", + "flat_plain = np.asarray(roi_plain).ravel()\n", + "flat_cal = np.asarray(roi_cal).ravel()\n", + "err_plain = abs(float(flat_plain.mean()) - BETA_SEARCH)\n", + "err_cal = abs(float(flat_cal.mean()) - BETA_SEARCH)\n", + "width_plain = float(np.quantile(flat_plain, 0.95) - np.quantile(flat_plain, 0.05))\n", + "width_cal = float(np.quantile(flat_cal, 0.95) - np.quantile(flat_cal, 0.05))\n", + "print(f\"error: {err_plain:.2f} -> {err_cal:.2f}; 90% width: {width_plain:.2f} -> {width_cal:.2f}\")\n", + "\n", + "assert err_cal < err_plain, (\n", + " \"calibrated ROI mean is not closer to the simulated truth - the lift test did \"\n", + " \"not correct the bias in this environment\"\n", + ")\n", + "assert width_cal < 0.8 * width_plain, (\n", + " \"calibrated 90% interval is not clearly narrower - the lift test did not add \"\n", + " \"information in this environment\"\n", + ")\n", + "print(\"all acceptance checks passed\")" + ] + }, + { + "cell_type": "markdown", + "id": "f7ee1dbf", + "metadata": {}, + "source": [ + "## Where to go next\n", + "\n", + "- **Meridian**: notebook 30 runs the same hand-off into Google Meridian's ROI\n", + " priors, using a geo-holdout *launch* experiment (the other classic design) and\n", + " `to_meridian_roi_prior` + `meridian_calibration_mask`.\n", + "- **API reference**: the `diff_diff.mmm` module documents both exporters'\n", + " contracts - including the two input routes (a pre-scaled `aggregate('total')`\n", + " container, or `aggregate('simple')`/`aggregate('group')` plus an explicit\n", + " `scale=`) and the additive-outcome caveat.\n", + "- **Estimator choices**: the DiD side works with any diff-diff estimator that\n", + " supports `aggregate('total')` (CallawaySantAnna, EfficientDiD, ImputationDiD,\n", + " TwoStageDiD)." + ] + } + ], + "metadata": { + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/tutorials/30_mmm_calibration_meridian.ipynb b/docs/tutorials/30_mmm_calibration_meridian.ipynb new file mode 100644 index 00000000..927af01d --- /dev/null +++ b/docs/tutorials/30_mmm_calibration_meridian.ipynb @@ -0,0 +1,970 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "d86bf740", + "metadata": {}, + "source": [ + "# MMM Calibration with Meridian ROI Priors\n", + "\n", + "Google [Meridian](https://developers.google.com/meridian) calibrates a marketing\n", + "mix model by putting an experiment-informed prior on a channel's ROI. diff-diff\n", + "closes the loop from a geo experiment to that prior:\n", + "`results.aggregate('total')` gives the campaign's total incremental outcome in\n", + "the estimator's own units, `to_meridian_roi_prior` turns it into Meridian's\n", + "lognormal `roi_m` prior, and `meridian_calibration_mask` scopes the prior to the\n", + "experiment window. The prior object even generates a ready-to-paste Meridian\n", + "snippet - which this notebook executes verbatim.\n", + "\n", + "The experiment here is the second classic geo design (notebook 29 showed the\n", + "spend-boost lift test): a **geo-holdout launch**. The channel runs *only* in\n", + "launched geos; never-launched geos are the holdout. That matters for the\n", + "estimand: Meridian's `roi_m` is the return on a channel's *full spend against a\n", + "zero-spend counterfactual*, and a launch experiment measures exactly that - the\n", + "DiD total *is* the incremental outcome versus zero spend.\n", + "\n", + "**Requirements.** This notebook needs `google-meridian==1.8.0` (Python 3.12,\n", + "TensorFlow-based), which is *not* a diff-diff dependency:\n", + "`pip install \"google-meridian==1.8.0\" diff-diff`. It cannot share an environment\n", + "with the PyMC-Marketing tutorial (notebook 29) - the two frameworks pin\n", + "incompatible `arviz` ranges - so each tutorial gets its own environment/kernel.\n", + "Runtime is a few minutes (two small MCMC fits).\n", + "\n", + "**Reproducibility.** MCMC draws depend on framework and transitive-dependency\n", + "versions and on the platform. The committed outputs were produced with\n", + "google-meridian 1.8.0; re-executing under other versions may shift posterior\n", + "numbers. The `assert` cells check the qualitative claims with slack margins - if\n", + "one trips on your machine, it means exactly that claim did not reproduce in your\n", + "environment." + ] + }, + { + "cell_type": "markdown", + "id": "7e0b5e03", + "metadata": {}, + "source": [ + "## A launch the national model cannot see\n", + "\n", + "Twelve geos, 104 weeks. The **search** channel launches in eight geos (staggered\n", + "cohorts at weeks 30, 44, 58) at a constant \\$150/week and never runs in the four\n", + "holdout geos; its true response is linear with `BETA_SEARCH = 2.5` - the true ROI.\n", + "The **tv** channel runs everywhere with adstock and saturation.\n", + "\n", + "Two realistic wrinkles create the calibration problem:\n", + "\n", + "- the launches ride a **demand ramp** (the brand expanded during its growth\n", + " season), and\n", + "- weekly **national shocks** (macro, weather, promos) hit every geo at once.\n", + "\n", + "The geo-level DiD is immune to both - common time effects difference out against\n", + "the holdout geos. But the MMM below is *national* (as most production MMMs are),\n", + "and aggregation destroys the holdout contrast: nationally, search spend simply\n", + "ramps up while sales ramp up, so the model overcredits search - and the common\n", + "shocks it must absorb make its evidence noisy too. The experiment's information\n", + "survives only if we carry it in as a prior." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "c0d3841c", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:52:58.412094Z", + "iopub.status.busy": "2026-08-16T19:52:58.412020Z", + "iopub.status.idle": "2026-08-16T19:52:59.370306Z", + "shell.execute_reply": "2026-08-16T19:52:59.369837Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
geoweeksalessearch_spendtv_spendfirst_treat
0005288.8831910.0274.98489230
1015442.5187300.0265.64266330
2025560.9412830.0272.25190830
3035546.2330270.0265.02713830
4045442.6057040.0274.29027630
\n", + "
" + ], + "text/plain": [ + " geo week sales search_spend tv_spend first_treat\n", + "0 0 0 5288.883191 0.0 274.984892 30\n", + "1 0 1 5442.518730 0.0 265.642663 30\n", + "2 0 2 5560.941283 0.0 272.251908 30\n", + "3 0 3 5546.233027 0.0 265.027138 30\n", + "4 0 4 5442.605704 0.0 274.290276 30" + ] + }, + "execution_count": 1, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import warnings\n", + "\n", + "# Scoped suppressions only: framework-churn warnings are hidden, but estimator\n", + "# UserWarnings (diff-diff's loud-failure channel) stay visible.\n", + "warnings.filterwarnings(\"ignore\", category=FutureWarning)\n", + "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n", + "\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "import pandas as pd\n", + "\n", + "from diff_diff import CallawaySantAnna, ImputationDiD\n", + "from diff_diff.mmm import meridian_calibration_mask, to_meridian_roi_prior\n", + "\n", + "SEED = 30301\n", + "N_GEOS, N_WEEKS = 12, 104\n", + "BETA_SEARCH = 2.5 # true ROI: incremental sales per search dollar\n", + "LAUNCH_SPEND = 150.0 # constant weekly spend per launched geo\n", + "LAUNCH_COHORTS = {30: [0, 1, 2], 44: [3, 4, 5], 58: [6, 7]} # 8 launch, 4 holdout\n", + "DEMAND_AMP = 80.0 # demand-ramp amplitude (sales units per geo)\n", + "TV_BASE, TV_BETA, TV_ADSTOCK, TV_SAT = 250.0, 800.0, 0.4, 0.003\n", + "BASE_SALES, GEO_SD, NOISE_SD, SHOCK_SD = 5000.0, 150.0, 20.0, 60.0\n", + "\n", + "rng = np.random.default_rng(SEED)\n", + "geo_start = {g: w for w, gg in LAUNCH_COHORTS.items() for g in gg}\n", + "\n", + "# Common demand ramp through the second half - the launches ride it.\n", + "ramp = np.clip((np.arange(N_WEEKS) - 26) / 52.0, 0, None)\n", + "# Common weekly national shocks - hit every geo equally, so the geo DiD\n", + "# differences them out while the national model must absorb them.\n", + "shock = rng.normal(0, SHOCK_SD, N_WEEKS)\n", + "\n", + "rows = []\n", + "for g in range(N_GEOS):\n", + " tv = TV_BASE * (1 + 0.1 * rng.random(N_WEEKS))\n", + " ad = np.zeros(N_WEEKS)\n", + " carry = 0.0\n", + " for t in range(N_WEEKS):\n", + " carry = tv[t] + TV_ADSTOCK * carry # tv keeps adstock ...\n", + " ad[t] = carry\n", + " tvc = TV_BETA * (1 - np.exp(-TV_SAT * ad)) # ... and saturation\n", + " geo_level = rng.normal(0, GEO_SD)\n", + " for t in range(N_WEEKS):\n", + " launched = g in geo_start and t >= geo_start[g]\n", + " search = LAUNCH_SPEND if launched else 0.0 # launched-geos-only spend\n", + " sales = (\n", + " BASE_SALES + geo_level + DEMAND_AMP * ramp[t] + shock[t]\n", + " + BETA_SEARCH * search # linear, zero carryover\n", + " + tvc[t] + rng.normal(0, NOISE_SD)\n", + " )\n", + " rows.append((g, t, sales, search, tv[t], geo_start.get(g, 0)))\n", + "\n", + "panel = pd.DataFrame(\n", + " rows, columns=[\"geo\", \"week\", \"sales\", \"search_spend\", \"tv_spend\", \"first_treat\"]\n", + ")\n", + "panel.head()" + ] + }, + { + "cell_type": "markdown", + "id": "c7066241", + "metadata": {}, + "source": [ + "## Measure the launch with staggered DiD\n", + "\n", + "`aggregate('total')` is the natural route here: the ROI is a ratio of totals, so\n", + "no grain arithmetic is needed - total incremental sales over total spend. We\n", + "cross-check with `ImputationDiD` and pin the total's mass and window against the\n", + "raw frame (any mismatch would mean a mis-scaled prior, so these are hard\n", + "asserts, not prose)." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "d65de63d", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:52:59.371420Z", + "iopub.status.busy": "2026-08-16T19:52:59.371320Z", + "iopub.status.idle": "2026-08-16T19:52:59.398548Z", + "shell.execute_reply": "2026-08-16T19:52:59.397984Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "per-geo-week ATT: 373.0 (truth 375)\n", + "CS total incremental sales: 184,281\n", + "ImputationDiD total (cross-check): 184,870\n", + "simulated truth: 185,250\n", + "total experiment spend: 74,100\n" + ] + } + ], + "source": [ + "res = CallawaySantAnna().fit(\n", + " panel, outcome=\"sales\", unit=\"geo\", time=\"week\", first_treat=\"first_treat\"\n", + ")\n", + "total = res.aggregate(\"total\")\n", + "\n", + "imp_total = (\n", + " ImputationDiD()\n", + " .fit(panel, outcome=\"sales\", unit=\"geo\", time=\"week\", first_treat=\"first_treat\")\n", + " .aggregate(\"total\")\n", + ")\n", + "\n", + "total_spend = float(panel.search_spend.sum())\n", + "true_total = BETA_SEARCH * total_spend\n", + "\n", + "# Alignment pins: in a launch design the channel's ENTIRE spend is experiment\n", + "# spend, so the total's mass must equal the spend-positive geo-week count, and\n", + "# every spend-positive week must lie inside the calibration window below.\n", + "spend_positive = int((panel.search_spend > 0).sum())\n", + "assert total.n[0] == spend_positive, (total.n[0], spend_positive)\n", + "first_launch = min(geo_start.values())\n", + "assert panel.loc[panel.search_spend > 0, \"week\"].min() >= first_launch\n", + "\n", + "print(f\"per-geo-week ATT: {res.overall_att:.1f} (truth {BETA_SEARCH * LAUNCH_SPEND:.0f})\")\n", + "print(f\"CS total incremental sales: {total.att[0]:,.0f}\")\n", + "print(f\"ImputationDiD total (cross-check): {imp_total.att[0]:,.0f}\")\n", + "print(f\"simulated truth: {true_total:,.0f}\")\n", + "print(f\"total experiment spend: {total_spend:,.0f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "16734e20", + "metadata": {}, + "source": [ + "## Export the ROI prior and the calibration mask\n", + "\n", + "`to_meridian_roi_prior` consumes the total container directly (`scale` is neither\n", + "needed nor accepted - the total is already in outcome units) plus the spend total,\n", + "and returns the lognormal `roi_m` prior Meridian expects. The mask restricts the\n", + "prior to the experiment window; the search column is `True` exactly on the window\n", + "and every other channel keeps all periods, per Meridian's documented convention." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "d6f48973", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:52:59.399843Z", + "iopub.status.busy": "2026-08-16T19:52:59.399759Z", + "iopub.status.idle": "2026-08-16T19:52:59.403525Z", + "shell.execute_reply": "2026-08-16T19:52:59.403076Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "experiment ROI: 2.487 ± 0.048 (truth 2.5)\n", + "lognormal prior: mu=0.9109, sigma=0.0195\n", + "mask shape: (104, 2); search weeks in window: 74\n" + ] + } + ], + "source": [ + "prior = to_meridian_roi_prior(aggregation_result=total, spend=total_spend)\n", + "print(f\"experiment ROI: {prior.roi_mean:.3f} ± {prior.roi_sd:.3f} (truth {BETA_SEARCH})\")\n", + "print(f\"lognormal prior: mu={prior.mu:.4f}, sigma={prior.sigma:.4f}\")\n", + "\n", + "dates = pd.date_range(\"2024-01-01\", periods=N_WEEKS, freq=\"W-MON\")\n", + "times = [str(d.date()) for d in dates]\n", + "\n", + "mask = meridian_calibration_mask(\n", + " media_times=times,\n", + " media_channels=[\"search\", \"tv\"],\n", + " channel=\"search\",\n", + " window=(times[first_launch], times[-1]),\n", + ")\n", + "print(f\"mask shape: {mask.shape}; search weeks in window: {int(mask[:, 0].sum())}\")" + ] + }, + { + "cell_type": "markdown", + "id": "3ef93f94", + "metadata": {}, + "source": [ + "## The generated snippet, executed verbatim\n", + "\n", + "`prior.to_code()` emits a complete Meridian `ModelSpec` construction - vectorized\n", + "`mu`/`sigma` in the model's channel order (the experiment channel carries the\n", + "prior, every other channel keeps Meridian's default), the serialized calibration\n", + "mask, and `media_prior_type=\"roi\"`. We print it, then execute it exactly as\n", + "generated." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "a89cc8f6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:52:59.404551Z", + "iopub.status.busy": "2026-08-16T19:52:59.404487Z", + "iopub.status.idle": "2026-08-16T19:52:59.406297Z", + "shell.execute_reply": "2026-08-16T19:52:59.405920Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "# Generated by diff_diff.mmm.to_meridian_roi_prior (template pinned to Meridian 1.7.0)\n", + "# TensorFlow-substrate snippet; for JAX-backed Meridian use\n", + "# `import tensorflow_probability.substrates.jax as tfp` instead.\n", + "# Channel order MUST match your Meridian InputData media channel order exactly:\n", + "# ['search', 'tv']\n", + "# 'search' carries the experiment-informed prior; the other channels keep\n", + "# Meridian's default roi_m prior LogNormal(0.2, 0.9).\n", + "import tensorflow_probability as tfp\n", + "from meridian.model import prior_distribution, spec\n", + "\n", + "import numpy as np\n", + "\n", + "roi_calibration_period = np.ones((104, 2), dtype=bool)\n", + "roi_calibration_period[:, [0]] = False\n", + "roi_calibration_period[np.ix_([30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103], [0])] = True\n", + "\n", + "mu = [0.9108575376065811, 0.2]\n", + "sigma = [0.019461139350044197, 0.9]\n", + "roi_prior = tfp.distributions.LogNormal(mu, sigma, name=\"roi_m\")\n", + "prior = prior_distribution.PriorDistribution(roi_m=roi_prior)\n", + "model_spec = spec.ModelSpec(\n", + " prior=prior,\n", + " media_prior_type=\"roi\",\n", + " # Mask restricting the prior to the experiment window.\n", + " roi_calibration_period=roi_calibration_period,\n", + ")\n", + "\n" + ] + } + ], + "source": [ + "code = prior.to_code(\n", + " channel=\"search\", media_channels=[\"search\", \"tv\"], roi_calibration_period=mask\n", + ")\n", + "print(code)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "dac9062f", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:52:59.407205Z", + "iopub.status.busy": "2026-08-16T19:52:59.407123Z", + "iopub.status.idle": "2026-08-16T19:53:02.436319Z", + "shell.execute_reply": "2026-08-16T19:53:02.435918Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "ModelSpec(prior=PriorDistribution(knot_values=, tau_g_excl_baseline=, beta_m=, beta_rf=, beta_om=, beta_orf=, eta_m=, eta_rf=, eta_om=, eta_orf=, gamma_c=, gamma_n=, xi_c=, xi_n=, alpha_m=, alpha_rf=, alpha_om=, alpha_orf=, ec_m=, ec_rf=, ec_om=, ec_orf=, slope_m=, slope_rf=, slope_om=, slope_orf=, sigma=, roi_m=, roi_rf=, mroi_m=, mroi_rf=, contribution_m=, contribution_rf=, contribution_om=, contribution_orf=, contribution_n=), media_effects_dist='log_normal', hill_before_adstock=False, max_lag=8, unique_sigma_for_each_geo=False, media_prior_type='roi', rf_prior_type=None, paid_media_prior_type=None, roi_calibration_period=array([[False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [False, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True],\n", + " [ True, True]]), rf_roi_calibration_period=None, organic_media_prior_type='contribution', organic_rf_prior_type='contribution', non_media_treatments_prior_type='contribution', non_media_baseline_values=None, knots=None, baseline_geo=None, holdout_id=None, control_population_scaling_id=None, non_media_population_scaling_id=None, adstock_decay_spec='geometric', saturation_spec='hill', enable_aks=False)" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "namespace = {}\n", + "exec(code, namespace)\n", + "model_spec_calibrated = namespace[\"model_spec\"]\n", + "assert np.array_equal(np.asarray(model_spec_calibrated.roi_calibration_period), mask)\n", + "model_spec_calibrated" + ] + }, + { + "cell_type": "markdown", + "id": "1df0ff91", + "metadata": {}, + "source": [ + "## Fit the national Meridian model - default vs calibrated priors\n", + "\n", + "The national data is the geo panel summed across geos. We build Meridian's\n", + "`InputData` with its DataFrame builder and fit twice: once with the default\n", + "`ModelSpec()` (Meridian's default `roi_m` prior, `LogNormal(0.2, 0.9)`), once\n", + "with the calibrated spec from the generated snippet." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "8f903e93", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:53:02.437982Z", + "iopub.status.busy": "2026-08-16T19:53:02.437881Z", + "iopub.status.idle": "2026-08-16T19:54:21.815732Z", + "shell.execute_reply": "2026-08-16T19:54:21.815009Z" + } + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/data/input_data_builder.py:706: UserWarning: The `population` argument is ignored in a nationally aggregated model. It will be reset to [1, 1, ..., 1]\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/data/input_data.py:575: UserWarning: Revenue from the `kpi` data is used when `kpi_type`=`revenue`. `revenue_per_kpi` is ignored.\n", + " warnings.warn(\n", + "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", + "I0000 00:00:1786909982.522983 32840500 service.cc:153] XLA service 0x912c26b00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n", + "I0000 00:00:1786909982.522997 32840500 service.cc:161] StreamExecutor [0]: Host, Default Version (Driver: 0.0.0; Runtime: 0.0.0; Toolkit: 0.0.0; DNN: 0.0.0)\n", + "I0000 00:00:1786909982.538321 32840500 device_compiler.h:208] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/model.py:103: UserWarning: In a nationally aggregated model, the `media_effects_dist` will be reset to `normal`.\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. tau_g_excl_baseline has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_m has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_rf has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_om has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_orf has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. xi_c has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. xi_n has been automatically set to Deterministic(0).\n", + " warnings.warn(\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "I0000 00:00:1786910002.193695 32840500 dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "W0000 00:00:1786910002.596307 32840500 assert_op.cc:39] Ignoring Assert operator mcmc_retry_init/assert_equal_1/Assert/AssertGuard/Assert\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "W0000 00:00:1786910010.912318 32840719 cpp_gen_intrinsics.cc:74] Empty bitcode string provided for eigen. Optimizations relying on this IR will be disabled.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "WARNING:tensorflow:5 out of the last 5 calls to .f at 0x12c2f3740> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "WARNING:tensorflow:6 out of the last 6 calls to .f at 0x12c2f3ec0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/arviz/data/inference_data.py:157: UserWarning: trace group is not defined in the InferenceData scheme\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/arviz/data/inference_data.py:1647: UserWarning: trace group is not defined in the InferenceData scheme\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/model.py:103: UserWarning: In a nationally aggregated model, the `media_effects_dist` will be reset to `normal`.\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. tau_g_excl_baseline has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_m has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_rf has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_om has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_orf has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. xi_c has been automatically set to Deterministic(0).\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1391: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. xi_n has been automatically set to Deterministic(0).\n", + " warnings.warn(\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "W0000 00:00:1786910040.502335 32840500 assert_op.cc:39] Ignoring Assert operator mcmc_retry_init/assert_equal_1/Assert/AssertGuard/Assert\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "both fits done\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/arviz/data/inference_data.py:157: UserWarning: trace group is not defined in the InferenceData scheme\n", + " warnings.warn(\n", + "~/.venvs/diffdiff-mmm-meridian/lib/python3.12/site-packages/arviz/data/inference_data.py:1647: UserWarning: trace group is not defined in the InferenceData scheme\n", + " warnings.warn(\n" + ] + } + ], + "source": [ + "nat = panel.groupby(\"week\").agg(\n", + " sales=(\"sales\", \"sum\"),\n", + " search_spend=(\"search_spend\", \"sum\"),\n", + " tv_spend=(\"tv_spend\", \"sum\"),\n", + ").reset_index()\n", + "nat[\"time\"] = nat[\"week\"].map(dict(enumerate(times)).get)\n", + "nat[\"geo\"] = \"national\"\n", + "nat[\"population\"] = 12_000_000.0\n", + "\n", + "from meridian.data import data_frame_input_data_builder as dfb\n", + "from meridian.model import model as m_model, spec as m_spec\n", + "\n", + "builder = (\n", + " dfb.DataFrameInputDataBuilder(kpi_type=\"revenue\")\n", + " .with_kpi(nat, kpi_col=\"sales\")\n", + " .with_population(nat)\n", + " .with_media(\n", + " nat,\n", + " media_cols=[\"search_spend\", \"tv_spend\"],\n", + " media_spend_cols=[\"search_spend\", \"tv_spend\"],\n", + " media_channels=[\"search\", \"tv\"],\n", + " )\n", + ")\n", + "input_data = builder.build()\n", + "\n", + "\n", + "def fit_meridian(model_spec):\n", + " mer = m_model.Meridian(input_data=input_data, model_spec=model_spec)\n", + " mer.sample_posterior(n_chains=4, n_adapt=500, n_burnin=500, n_keep=500, seed=SEED)\n", + " return mer\n", + "\n", + "\n", + "mer_default = fit_meridian(m_spec.ModelSpec())\n", + "mer_calibrated = fit_meridian(model_spec_calibrated)\n", + "print(\"both fits done\")" + ] + }, + { + "cell_type": "markdown", + "id": "309f825d", + "metadata": {}, + "source": [ + "## Read out the ROI posterior\n", + "\n", + "Meridian's `Analyzer.roi()` returns the posterior ROI per channel in KPI units." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "62cf511d", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:54:21.817640Z", + "iopub.status.busy": "2026-08-16T19:54:21.817543Z", + "iopub.status.idle": "2026-08-16T19:54:22.579406Z", + "shell.execute_reply": "2026-08-16T19:54:22.578982Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " default prior: ROI mean 3.29, 90% interval (3.02, 3.57)\n", + " calibrated prior: ROI mean 2.54, 90% interval (2.47, 2.62)\n", + " truth: 2.50\n", + " DiD measurement: 2.49 ± 0.05\n" + ] + } + ], + "source": [ + "import arviz as az\n", + "from meridian.analysis import analyzer as m_analyzer\n", + "\n", + "\n", + "def roi_posterior(mer):\n", + " roi = np.asarray(m_analyzer.Analyzer(mer).roi()) # (chains, draws, channels)\n", + " return roi[..., 0] # search is channel 0\n", + "\n", + "\n", + "roi_default = roi_posterior(mer_default)\n", + "roi_calibrated = roi_posterior(mer_calibrated)\n", + "\n", + "for name, roi in [(\"default prior\", roi_default), (\"calibrated prior\", roi_calibrated)]:\n", + " flat = roi.ravel()\n", + " lo, hi = np.quantile(flat, [0.05, 0.95])\n", + " print(f\"{name:>17}: ROI mean {flat.mean():.2f}, 90% interval ({lo:.2f}, {hi:.2f})\")\n", + "print(f\"{'truth':>17}: {BETA_SEARCH:.2f}\")\n", + "print(f\"{'DiD measurement':>17}: {prior.roi_mean:.2f} ± {prior.roi_sd:.2f}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "1ca40c00", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:54:22.580665Z", + "iopub.status.busy": "2026-08-16T19:54:22.580579Z", + "iopub.status.idle": "2026-08-16T19:54:22.711622Z", + "shell.execute_reply": "2026-08-16T19:54:22.711064Z" + } + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAG4CAYAAADYN3EQAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAbDJJREFUeJzt3QecE9X6//Fn6b1KlSIC0hUpKoKKIKBYwI4XFIELig1EUbAhFxUQC3YUFVAU9SJYUEBAQEVBaYoFFATpYKHXZTf/1/fc/+SX3c3uJptdspt83q9XCDuZJCdnJpPzzHnOmQSfz+czAAAAAIhAvkieDAAAAAAEFgAAAACyBT0WAAAAACJGYAEAAAAgYgQWAAAAACJGYAEAAAAgYgQWAAAAACJGYAEAAAAgYgQWAAAAACJGYIE8ISEhwW677TbLbTZs2ODK9sQTT0S7KLnaww8/7OoJabVt29bd4vl7FKvbbMGCBa7Ode+58cYb7aSTTsrhEsbWceOvv/6KdlEQ4v4NEFggR7333nvuwDN9+vQ0j5122mnusfnz56d5rEaNGnb22WfnWLl+/vln96OlwCAUn376qVsfse2xxx6zDz74INrFQIgNGt0mT54cdJ3WrVu7xxs3bkx9xjFvP/FupUqVsvPOO88++eSTdJ/z008/WY8ePezEE0+0woULW9WqVa179+5ueWoTJ050r7t06VLLjTim4XgjsECOatOmjbv/6quvUizfu3ev/fjjj1agQAFbtGhRisc2bdrkbt5zcyqwGD58eFiBhdZH1jzwwAN26NChXF990fgR/uyzz9wN4StSpIi9/fbbaZbre/3111+7x3PbNhs/frytWbMm28uE9HXo0MHefPNNe+ONN+yee+6xtWvX2qWXXmqzZ89Os+60adOsWbNmNm/ePOvVq5e9+OKL1qdPH3cCTMuDnSSL12Paueee647rugc8Bfz/A3KAzvTUqlUrTWDxzTffmM/ns6uvvjrNY97fORlY4Pg4cOCAFS9e3AWQuuH/HDx40IoVK2aFChXKtmo5duyYJScnZ+tr5madO3e2jz76yKXKnHDCCf7lCjYqVapkdevWtV27duWqbVawYEGLZ9HYR0855RTXA+G58sorrWHDhvbMM89Yp06d/MvXrVtn119/vZ188sn2xRdfWIUKFfyPDRgwwM455xz3+A8//ODWiVeHDx922y9fvnzZGrx7vxfI2+ixQI5TgLBixYoUZ6zVS9GoUSO76KKLbPHixe6HJvAxdS0rlSE1nXlRaoO6p/X8WbNmpXj8jz/+sFtuucXq1atnRYsWtfLly7vgJbBnQl3XWibnn3++v4s8vTxR5US/8MIL7v+BXeqpvfLKK1a7dm1XtpYtW9p3332XZp3Vq1fbVVddZeXKlXMH5BYtWriGUShUR2PHjnWfW89Vw+mmm25K0XAaNmyYO9jrbFugfv36uR+C77//PkUqybvvvmv33XefVa5c2R3QL7vsMtdblNqSJUvswgsvtNKlS7uGlVIJUvc0efnQ6g3617/+ZWXLlvUHh8HGWHj5/v/973/dj7y2V6tWrWzVqlXu8Zdfftnq1KnjPqvy2YP1LoVTLp2l1LYsU6aMW19nI9VQDCyPftgmTZrk38ZaPz3h1KHKr/122bJl7uyeyqrnpJevv3PnTneWVNtYn19pgypXeuN7tF94+57qPzNvvfWW+47otZs3b+4aUYHSGweQejuqvlW2YPT6gY22d955x71XyZIlXTpKkyZNXMMukBp2uoWqS5cu7jNrHwqkwOKaa66x/PnzB32e0qdUFu1z+i5269Yt4m22efNm69q1q9sHKlasaHfeeacdOXIkzXsHq1ttQ6V+6nilMqlsU6dOTfNc7zuT2XEwPc8995xbX59F308df1L3+GzZssV69+7t9j3v9V9//fUU6xw9etQeeughV059l/SZ1ehOndaa2T6q46G2kxrw+tzaZ+6///405d69e3eG391wNWjQwAWiqfe1MWPGuNfVsTwwqBCtr2OSjhGPP/542O/ppUzpu6bjtra1vgc33HBD0OBXPSWqey8V69Zbb3X1EOi3335zQZKOPfouV6tWze3Le/bsCemYFsq29o5z+v6q51npYdp/lHWQ3hgLfR+975fqTUGd3iuQylGiRAm3DXSCQMcFpZsh7+MUInKcGpfqhlYj0PsxVuNPP6S66SCotKhTTz3V/1j9+vXdgTd1T4a6qRU46CD07LPPuoPqxo0b/euqMa8UCB1cdZDVD9tLL73k3lc/ZjogqpFwxx13uOeroaAfGfHuU9OPwNatW23OnDnucwSjH+d9+/a5dXWg1Q/PFVdcYb///rv/DKXycxUs6cA8ZMgQ92OsMShqjLz//vt2+eWXZ1iPem39OOlHVeVfv369Pf/88y5oU53pfXTg//jjj12jVA101ZO6+5V+MWLEiDSNwEcffdSV995773WNWf34X3DBBbZy5Ur3oyCff/65CwD1Q+EFLhMmTLB27drZl19+aWeccUaK11TQpjPF6oJXr1RG9HwFVvrRlJEjR9oll1zi0hX0w6ptrR9d1ad+AFUWT7jlUgNGvWd6j+XLl9urr77qGoCjR492j2vb/vvf/3bPUyAmaghlJpQ6lL///tuVV/umfmj1Yx6MAnDtrwqE1IhUmfVDrR9iNSx05jSQPrPOIKrMahyooZyRhQsXumBI+5DWVz0rOPv222/DHo+gs7d9+/Z139/A5+p7+Ouvv7r9UfTdue6666x9+/b++v7ll1/cfhv4efS4hJqiqO+zgospU6ZY//793TIFz/quafvqzHKw7fXggw+6/UHb+88//3QNbh0X9F1S4zUr20xl17FI9aqGoPanwP01IwqwFJCqYaVGuxpx+h7NmDHDLr744rCPg8HoGKCy6cSG6lz7jOpHx2WdCJAdO3bYWWed5Q9g1LieOXOmO56oITlw4EC3nv6v+tU21fbXse+1115zgaT2o6ZNm2a6j+q9FYzouKXlCrbUyNTxS9sonO9uuPSbo+NK6u+33lvlULmC0T6ixzMan5EZ1av2MQXpSonT75NOiHmNdNFjSr3VcUT7tbeevlfesV77iepbwevtt9/uggs13rXP6DihACyjY1qo29qj3w+dnLr77rvde6bX4+T9RunkmraX3kf7t8qd+vul3it9BrURFHzq+4wY4ANy2E8//aTWpW/EiBHu78TERF/x4sV9kyZNcn9XqlTJ98ILL7j/792715c/f35f3759U7yGnl+oUCHf2rVr/cu+//57t/y5557zLzt48GCa9//mm2/cem+88YZ/2X//+1+3bP78+SF9hltvvdWtn9r69evd8vLly/v++ecf//IPP/zQLf/444/9y9q3b+9r0qSJ7/Dhw/5lycnJvrPPPttXt27dDN//yy+/dK/31ltvpVg+a9asNMtXrVrl6urf//63b9euXb4TTzzR16JFC1fvHn1uPU+Pqc497733nlv+zDPP+MunsnXq1Mn9P7Cea9Wq5evQoYN/2bBhw9xzr7vuujTl9x4LpL8LFy7s6tDz8ssvu+WVK1dOUa6hQ4e65d66WSlX7969U7z/5Zdf7rZbIO2XPXv29IUi1DqU8847zy0bN25cmtfRY7p5xo4d69adPHmyf9nRo0d9rVq18pUoUcL/Xt6+V6pUKd/OnTtDKrPW123p0qX+ZX/88YevSJEirj48qoOaNWtmuh13797tnnvvvfemWO+OO+5wdbl//37394ABA1w5jx07lmH59J7B3je9utf3eMaMGb6EhATfxo0b3WODBw/2nXzyye7/qtdGjRr5n7dhwwZ3fHn00UdTvJ6+MwUKFEixPCvbTNvec+DAAV+dOnXSHGeC1W3q45a2d+PGjX3t2rXL0nEwmC5duqSoi2D69Onjq1Kliu+vv/5Ksbxbt26+0qVL+8up7XjkyJEU6+hYo2N54Pcso3303HPP9ZUsWdLtf4ECv8/hfHfTo+frc/3555+uDNr3L7zwQrd8zJgxKfZlLVM9ZeSyyy5z63nfwwkTJri/v/vuuwyf563XvHlzt309jz/+uFuu3wxRGbWNO3bs6EtKSvKv9/zzz7v1Xn/9dff3ihUr/N+BjKR3TAt1W3vfNX2nUu+n3mPe/q3PVbFiRbfvHjp0yL+evqNa76GHHvIvU5m0bMiQIRmWH3kPqVDIceoJ0Jk0b+yEziiqe9ab9Un3XvqKxl4kJSUFHV+hszeBZ5jUw6GuZPUKeALPECcmJrozjkqn0VkSnenKKddee61LLfB4Z7y8sv3zzz/u7KXOvOnsnnLCdVP5dMZGXdqpu4oD6Yy1zkBpEKL3XN10tl7dyYEpCDpzrLNdOqun19Z66goPNsZB3fA66+nR2cwqVaq4weqis+4qm85oqqze+2r76QytuvUD09jk5ptvDrne9BqBaSFnnnmmu9cZ2MByecu9+syOcmkb6bk6OxeJzOrQozO1OpOXGT1PZx91NtijM5Q627x//37X4xBIdZU6bSMjSjfTfhM4A5vO+qtnS9+9cGif9HoMvN4pvYZ6RLy0INH3T9tGPRcZUU9FqL0Vno4dO7oz4DrLrzLoPrDuAulMv/YLfQ8Dv0eqb/WypU7lCWebaZtr23t09tU7S5yZwOOWzqTrjLr2z2DHrFCOg8FoGyhdK1iKpqju1HOqQc36f2D96DiiMnnlUYqZd8Za9anjm84+K7UqWJlT76PqJdJ3VL2Q2v8CBUszjfS7q94Uvb96OVRGpYqqV3TQoEH+dXRclsDvcjDe41k9bmifCBxnox4JHZu948XcuXNdb4R6DNQL61HPkLaz11ui757oextuWlg429rTs2fPFPtpMJoZS7226k0LHHuhXjdlIQTr6fF6GhE7SIVCjtMPhYIHr7GnIEIHeDX4RY8ppUe8ACNYYJH6B0jUmA/MT1VKgrpf1fWuhnpgKo6Xd5oTUpfNCzK8simtRWVRCoZuweiArDSpYNSIVvlVb+k9N9DgwYNdA0tpCUpJ0hiGYNSYSr2ttF28xp3e1/tRSY/KFRhUKWUhq/Xm/VhWr1496HKvPrNSroy2kX6wsyqzOvRo24YyYFVpEXrNwEZFYKqeHg8UTn0HK683uFWNEzX41MgON7BSIKH0M6WKqGGk9AelSXnU0FDan9KKVA8KBtS4VwpWpNRIU9qQ0hGV8qGxEl5qT2rab/Q9DFYH3mtldZtpm6duFGvMQCiUvvLII4+4gDlwXEawRnYox8FglKqnbaM6Ulm1DVRP3lg2bXul0Gh8gW6ZHWd0suLJJ5904yR0Eiej/TH1Mi8ICjX1LrPvrgIbNcY9agB7xwxR8Kt0H62jwErHRO3vgd8xL2DwAoz0hBqApCf1vqcTQwpKveOF9/1Ove9oP9SAce9x1akCo6eeesqNmVKwpXQ6pewFfvZgwt3W3vtlJr2yiwKL1BO1KKBSyjJiC4EFjgsFCspfVd6/N77Co/+rIaxAQAce5ScHm3EjvYGYgcGDck0VVOhsj87M6gCrH2flSKc+g52dMiub997KTw0c0BrIC7SC0fMVVOgHJJjUZ6z1w+01vr3B0FnhlVuDGlPnTQf+MAbK7KxWKPUWan2GU65Q9p+cFE69RPt107uYYbAeDe3PGnugAdEKLHSv4ERn1j3ad9Vo1tlV5XHrpu+pgpLUg9KzQg3kcePGudx0jSNKL5DWfqPPpvcPtj9Esi9nlQIyNQhVdxrvokamAhzVT7CpdLO6HyswVa6+ghgN9tYZa72fBmGrh9P7Tqlhml7A7o2D0zbWmB/1SunYre2rcumkTrDB95HWY2afWePZAnvyVH7l+nvUePX2Rw0U1oBiBRqavEPPFf1WqO6DjcsJpMcVcEZyMiK7KLDTdvjwww/d9Mfq1dQ20IQoGTXYw9nWOfldUI9g6hMoyPsILHDcr2ehwCJwYJjSMnSA0eA1DSTUgT+rNJOKDpQ64Ho0aDD1bBrhXgU60qtGe4GSGgyBDa5QKfVBZxt1djGzA7x+NPRjox8+1bPOzilFw/sBDeQFH4E/1Opd8X5UvJQLvVZWyp1TcqpcWdnOmdVhuGrWrOkaL9qOgT+6OjPsPR6J1OUVDbRW6o4XoOqMcOrvTLDeEq/Rp4a9GnIaTKsZi5S2kboxqDOuSr3QTZ9NvRiaZUc9eBkF1aEeX3RWW8eQjAb0ar/R9tHZV/XSZBdtEw1g12sH7kOhXK9CDXyljSjo0nHQo8Aiuyk1TWmbuunsvY4JGig9dOhQt+11Fl7BY2bfKR1ndUxTalng59UkCuEcD1Vn2UHH+8AeG52cymwijKefftpNLqBJM7zPoIkjNMhdv1PBes0VBKpnQc+P5PungMaj9MZt27b5f/e877f2ncATbNpemrAj9bbR7Gq66bNo4hL9RijIVg9Yese0cLZ1OALLrkk0AmlZpMcu5A2EijgulNeqH0+dcVfPRGCPhX5MdeEhTemqPOxIrl+hxkzqM3ea8SX1mVYv9ztY4ymYcNdPTWf0NNOPGlL6EQnWNZ0RpY3oM2hmjtSU2xxYLnWN6wdGXdxaX3WtPFblz6amC0YFdv2rwaDyKWXFC/rUGNOMHfoBDLfcOSWnyqXtHO42zqwOw6UGxvbt2116UeA21n6sM+qa4jUSGscUmD+t1CGd8VRqjBcMqG6VShZ49lafKb2LgyntSQ07Nbi0PQKvGSDKhw+kgMkLvAJTf8KdbtajxpNmR1LDNjAFKzU1pPUZdYY+9XFCf6cuZzjbTDPHBU4R601bmhmVR+UPPEap8ZrdFzVL/dkU6KlnR59bqUwqh8ZCKNAJ1uAP/E55+0lgHeqkkPatUKhhqx4aTW2q2awi7UHU8UANZO+WXo9VYArOXXfd5WYm077vUe+LTtxoP05dX0q30lgPBeBaL6u0TwSmjmm2J32/veOFyq9to/05sC40TkTfSW+WMI3x0PMCKcDQdyvwOxXsmBbOtg73d16/dQpsAsugHkLVdeoZzhCb6LHAcaEDpaaf0xkfBRKBg0dFjV+vlyGSwEJnnDTFnrq19eOiHzqd6U89DaPSZ3Rw1dlNHaxVJp1hSW8Mg1dedTUr9UPPVXpVOBQ46bPp4K8zujobpVx0lVGDKr1rTASjxqR+7NTNrZQSNQLV+6GzXxrYren81Cuhg7fOAKvHQmeGRWeS9Xm9PPdAGvSqMmmAqsqiqVJ19ljlE/1IaRC4fvQ0x7nWUxqAgkMNdFWPgVLcjrecKpe2s/YXBWfexR29gePpyawOszK4UwGotqGuoaDB7WqwqqdPr53V3G6P8tq1DwdONyuBV5bXvq2cfJ3N1XpqJKsBpLP8wQbnnn766e51tS8q5UYnCgJpyks1zPQdU4qGej4UKGm/DJzmOdzpZgMpj163jChg0plcnaHXeyiVR/WpM8EKmlT3SlcMl7a1xokptUvbTCk1Og6FMn2mGlva3zTeRD0/ym3XsUL7UGZpOeHQMUMpajqjrdQ1HStUZr2/t0+NGjXKfX+0z+sz6Riq7aZtru+F/u8dZ9Vbof1Dz1f9qTGp9YMF+sGo4azvjfYV1bu+a9omGuCrY1xO0/dLaWD6DdB+4I1/UGqepv3VcVpTr3rlUsNeJ2c0UUEo01CnRz0P2s91skhn8fX9Uz0oHc4LurR/6vuofULLvfX0G+oF7ZoMROlcGl+k76WCDO1zXtCQ2TEt1G0dDv0mqT51LNRvliZR8Kab1XFM13ZBHIj2tFSIH96UoZpeNbVp06a5xzT9YLApKfWYpnxNTdM2Bk6lpykPe/Xq5TvhhBPc1JyajnT16tVp1pPx48e7KfQ0/WRmU8+qTLfffruvQoUKbmpL76vjTacYOG1hYJk1XWKgdevW+W644QY3nWrBggXdVKWXXHKJb+rUqb5QvPLKK266wqJFi7q60vS199xzj2/r1q2ujC1btvRVq1bNTZ0YSFOfqjzvvvtuimkCp0yZ4raLpgjUa1588cVppn/0pja84oor3BSPmiJW9XnNNdf45s2bl2ZqSE3rGOp0s6m3aXr1GTi9aHaVy5v+MXC6W+0rmgZTdaHHMpp6Npw6TD3taUZTl8qOHTv8+7GmntR2VnlDqauMeHWuqWw1Xa/q7PTTTw+673/22Wdu2ki9f7169dxzgm3H1NNmPvbYY2ke0/6t6TNVR3q9GjVq+G666Sbftm3bIp5uNiPp1fv777/va9OmjZuKU7f69eu7elmzZk2mz01vm2mbayrSYsWKue2mKXa96aAzm272tdde828PlUXbOtTvjAQ7vqWmqZy1b3vfldq1a7upeffs2ZNm39N7VK9e3R2jdKzSVNk69gROCavtrPf19iFNKZr6s2W2j/74449u6tgyZcq4aYu1nz344INZ+u6mJ706k4cffjjosf+HH35w02ZrOlavDvS3piVOLdzpZhcuXOjr16+fr2zZsu43qnv37r6///47zfqaXlb7gt5f0/j279/f/b55fv/9dzcNr7aj6q5cuXK+888/3zd37twUr5PRMS2UbZ3Rdy31dLMe/c5on9C+oXLpM27evDnFOiqHvnuIPQn6J9rBDYDjS7noyvPVGebAKTJBHWaVzkrqjKTO7gabuQiIZ96F4zQrlVKGgFjFGAsAQER0fkqpIkp/IKgAgPjFGAsAQJZosoWPPvrI5WprWuPAgbAAgPhDYAEAyBLNHqMBx7qq83333ecfgAoAiE+MsQAAAAAQMcZYAAAAAIgYgQUAAACA+B5jkZyc7K52qov7BLtsPQAAAIDIZv7bt2+fu8iiLlAbs4GFgorq1atHuxgAAABATNu0aZNVq1YtdgML9VR4H7RUqVLRLg4QNQM/fdh2HdpjZYuWtrGdH3bL6tevb9u2bbMqVarY6tWr2ToAACBse/fudSfyvXZ3zAYWXvqTggoCC8SzRzrfY0m+ZMufkM//XfC6K3XP9wMAAEQilGEHeTqwAPA/VUtVpioAAEBUMSsUAAAAgIgRWAAAAACIGKlQQAz46o9v7cixo1a4QCFrU/OMaBcHAHK1pKQkS0xMjHYxgFyhYMGClj9//mx5LQILIAZM/n66/XNot5UrWobAAgAymI9/+/bttnv3buoICFCmTBmrXLlyxNeFI7AAAABxwQsqKlasaMWKFePiuoh7Pp/PDh48aDt37nR1oSnqI0FgAQAA4iL9yQsqypcvH+3iALlG0aJF3b2CC30/IkmLYvA2AACIed6YCvVUAEjJ+15EOvaIwAIAAMSNSHPIgViUkE3fCwILAAAAABEjsAAAAMhj2rZtawMHDgzrOR988IHVqVPH5dCH+9zMznbrtY+HBQsWuPfLKzN7nXTSSTZ27FiLFwzeBgAAcW39hEnH7b1q9epp0XLTTTdZr1697I477rCSJUvmyHts2LDBatWqZStWrLCmTZtm++ufffbZtm3bNitdurTlBd99950VL17c4gWBBQAAQIzbv3+/m/WnU6dOVrVqVcuN055q5q4CBTJumhYqVMhdbyG3O3r0qCtrhQoVsuV18gpSoYAYUKZIKXdxPN0DAGLLgQMH7IYbbrASJUq46ww8+eSTadY5cuSI3X333XbiiSe6M+RnnnmmSxsS3Xs9FO3atXOpRFr2999/23XXXeeeo1mBmjRpYlOmTMk0lUc9EQ8//HDQsqq3Qk4//XT3PkrZyiilaebMmda8eXMrXLiwffXVV+5zqEdF054WKVLE2rRp4876ZyUVauLEie7CbzNmzLB69eq5z3jVVVe56zZMmjTJfbayZcu691NQkx59Vn3ml19+2apXr+5e55prrrE9e/b417nxxhuta9eu9uijj7rATe8XrP42btxoXbp0cduyVKlS7nV27NiR5r1effVVV5eqg7yEHgsgBozqODSi54+ZvDTTdQb3aBHRewAAsmbw4MG2cOFC+/DDD12D+7777rPly5enSDW67bbb7Oeff7Z33nnHNWynT59uF154oa1atcqlD61Zs8Y1dt9//333d7ly5ezPP/90jfp7773XNXI/+eQTu/7666127dp2xhlnZKms3377rXvu3LlzrVGjRpmebR8yZIg98cQTdvLJJ7tG/j333OPKqIZ/zZo17fHHH3e9LGvXrnVlDpeCiGeffdbVy759++yKK66wyy+/3AUcn376qf3+++925ZVXWuvWre3aa69N93X0/u+99559/PHHtnfvXuvTp4/dcsst9tZbb/nXmTdvnqvHOXPmBH2N5ORkf1Ch7Xns2DG79dZb3ft6QaD3XqqDadOmRXRNiWggsAAAAMjFKUyvvfaaTZ482dq3b++WqdFdrVq1FGfBJ0yY4O69NCf1XsyaNcstf+yxx1xAImqce6lE6qnQep7bb7/dZs+e7RrQWQ0svNQfXYQwlJSl//znP9ahQwd/z8xLL73kehouuugit2z8+PGuoa46UIAVLl2XQa+pYEnUY/Hmm2+6XgI18Bs2bGjnn3++zZ8/P8PA4vDhw/bGG2+4OpPnnnvOLr74Ytd75H3O4sWLu56G9IIpBR4K9NavX+96PkSvqQBMvTItW7b0pz9peaRpVNFAYAEAAJBLrVu3zjU0ldrkUXDgpdqIGqtK5TnllFNSPFdpRRldZVzPUdChQGLLli3uffSc43kRwRYtWqT4rAoE1HvgKViwoAtyfvnllyy9vj6LF1RIpUqVXHqSgorAZRp/kpEaNWr4gwpp1aqV64FQT5AXWDRp0iTDHhp9BgUUXlAhCmzUe6LHvMBCPTV5MagQAgsAAIA83quhlJlly5alSZ0JbECnNmbMGHvmmWfcGAA1inXGXdPQKsDw5MuXzw2sDhTp1ZkD5fSMSQpMAml8RrBlChJyy2cpnodnkWLwNhADXvnuLXtq0Xh3DwCIHTrbrobwkiVL/Mt27dplv/76q/9vDZRW74POuus6FYG3jNKRFi1a5HL+e/ToYaeddpob5xD4uqIz55re1aPxBUrlSY93xj6jwdAZfVY9X+UKDGKUJqQz+9GkNLOtW7f6/168eLELugJ7jjLToEED27Rpk7t5NC5GA9Gj/fmyCz0WQAxYvu1H++fQbjczFAAgdqjHQQOFNb5AaU0aK3H//fe7Rq1HKVDdu3d3M0cp51+BhgZmK6f/1FNPdWMBgqlbt65NnTrVvv76azdw+qmnnnJjDwIbuZpFSmMeLr30Upey89BDD2U4oFjlK1q0qBvfoXEgmtUo1GtO6Ex9//793WdVupfSjzR4WwOwVQfRpM/Rs2dPN9BcwZVmktKMTuFMfXvBBRe4niFtK/USafC2BoCfd955KVLC8jJ6LAAAAHIxpSydc845rnGvxqmmYNVsToE0SFuBxV133eXOomvqU53pV+M8PQ888IA1a9bMzbqkaWHVSNbzAg0dOtQ1fC+55BIXoOjxwDELqek6FJqFSVOzaiC5ekTCMWrUKDdLk2anUtk0Q5IGlCvwiSb1/mhGqc6dO1vHjh1dwPbiiy+G9RoJCQluZi99lnPPPddtS/USvfvuuxYrEnypE+fyEEWMioI1j7Cm9wLi1c0fDfX3WIy7bKRbpjNFGoynwWabN2/O8PlMNwsg1mlWH6Xw5MVrAyC6dG2JDz74wFauXBmX34+9YbS3o9pjofy7Bx980H0IdZspAh4xYkSaQUIAAAAAcreoBhajR492cws///zzbpot/a1cOs0NDAAAAKRH17rQGJRgN02jizgbvK3BQsq98wYVaV5hXUpeV20EAAAA0qOL0R06dCjoY1m5SndGqVC6IZcHFrqk/CuvvOKmNtOMBt9//7199dVXblYCAAAAID2BF6xD7hDVwGLIkCFuQEj9+vXd1GUac/Hoo4+6abiC0dUgdfPouQAAAADifIyFLiH/1ltv2dtvv23Lly+3SZMmufmBdR/MyJEj3ah07xZ4SXQAAAAAcdpjoQugqNeiW7du7m9dNOSPP/5wAYQuQpKa5lIeNGhQih4LggvArHWNFnbg6EErXqgY1QEAAOIvsNCVFAOvHClKiUpOTg66fuHChd0NQErXN72SKgEAAPEbWOgKkhpToatCNmrUyFasWOEGbvfu3TuaxQIAAACQl8ZY6HoVV111ld1yyy3WoEEDu/vuu+2mm25yF8kDAABA5DRVatOmTf1/33jjjda1a1f/323btrWBAwfmuc+RXRYsWGAJCQm2e/duywtOOukkGzt2rOVGUe2xKFmypKuY3Fo5AAAg9o2ZvPS4vdfgHi0s2p555hnz+XzHLRj44IMPbOXKlZZb6fIH27ZtcxMD5QXfffedFS9e3HKjqAYWALLHwE8ftl2H9ljZoqVtbGcu4gMASF92NKCPHj1qhQoVytXVrOBJlzIoUCDj5q4+R+XKlS23O/r/67xChQrZ8joxlwoFIHscPnbEDh077O4BALFFk9o8/vjjVqdOHTeJjcamaoyq595773UXGi5WrJidfPLJ9uCDD1piYmK6r5c6FUqOHTtmt912mws6TjjhBPcagb0aSr9RqvoNN9xgpUqVsn79+mX63hMnTrThw4e7CyAr1Ug3LROlHf373/92jWS9Xrt27dx6gUaNGmWVKlVyGS59+vSxw4cPh5TSNHPmTGvevLmrK114WddAu+OOO6xixYpWpEgRa9OmjTvrn5VUKJW/TJkyNmPGDKtXr5773Err14REkyZNcvVUtmxZ934KajJL63r55ZfdDKd6nWuuucb27NmTZjtpW1etWtW9n7ctArN9Nm7caF26dLESJUq4utTr7NixI8176UrltWrVcnWQU+ixAAAAyMU03f748ePt6aefdo1ipe2sXr3a/7ga3mrwqvG5atUq69u3r1t2zz33hPweahSr8f7tt9/a0qVLXeCgAEav5dG1xh566CEbNmxYSO997bXX2o8//mizZs2yuXPnpugtufrqq61o0aIuCNAyNbDbt29vv/76q5UrV85d60wN4hdeeMF95jfffNOeffZZF7xkRpcyUFm1rhr5Ksv777/vPmPNmjVdkNapUydbu3ate69wKYhQWd555x3bt2+fXXHFFXb55Ze7gOPTTz+133//3a688kpr3bq1q4P06P31OT/++GN3CQXVv8Yd6xpvnnnz5rlgYc6cOekGnV5QsXDhQhcg3nrrre59FTAFvpfqYNq0aW4G1pxCYAEAAJBLqeGqMRHPP/+8/xpftWvXdo1tzwMPPOD/v85mazIcNXrDCSx01lyBi87c68y4ggT9HRhYqFfhrrvuSvG8jN5bgYMavEpFCkw1Ui+CApidO3f6LyOgQEBjMaZOneqCGp2RV0NbN3nkkUdccJJZr4X85z//sQ4dOrj/HzhwwF566SUX/Fx00UVumYI0NdRfe+01d021cKlHRq+p7SDqsVDgs2PHDvd5GzZsaOeff77Nnz8/w8BCn+WNN96wE0880T+p0cUXX2xPPvmkv740lkI9DemlLinw0LZav369/9puek3NtqpemZYtW/rTn7Q80jSqzJAKBQAAkEv98ssvLpVHZ/PT8+6777qz42qMqmGrxr7SY8Jx1llnuaDC06pVK/vtt99SpPO0aNEiW95bKU/79++38uXLu+d4NzWO161b5//cZ555ZornqUyhCCynXk+BgMroKViwoJ1xxhnuPbJCaUteUCFK11JQVaJEiRTLFDhlRD1CXlDhfT71QKxZs8a/TBePzmg8hD6DAorAC0YrsFHvSeDnU09NTgcVQo8FAABALqWz/hn55ptvrHv37m4sg9J7lFakHgOd9c5uqWciyup7K6ioUqVKilQdjxrE2V3O7KbAJJACsmDLktO54HM0PsvxmkWKHgsAAIBcqm7dui64UMpLMF9//bU7G33//fe7M/Va/48//gj7fZYsWZLi78WLF7vXyigfP5T31tn21IOYmzVrZtu3b3cpUhqQHnjTwHHR9c2ClSlc6llQGRYtWuRfph4MpQnpzH40bdy40bZu3Zri8+XLl88/SDsUqqdNmza5m+fnn392A9Gj8fnosQAAAMilNIOPZl7SmAU1kJXS8+eff9pPP/3kxh+oMa8GqnoKlE//ySef2PTp08N+H73GoEGD3IWKly9f7vL9M+t5COW9lSKkFCddx6JatWpuYPcFF1zg0n4045EGUmtWKTWw9XwNglaQMmDAADcrkv6vz6wBzfrMoQzeTn2mvn///m4shQZqK/1I76kB2N74jWhu2549e7rxJRq8rZmkNKNTOFPfqi6VLqWeI41L0eBtDQA/77zzgqau5TR6LAAAAHIxTeGqQdOakUlnqDUg2Mvfv+yyy+zOO+90U8VqSlH1Imj9cGka2UOHDrmxB5pVSA17b0rZ9ITy3pod6cILL3SDmZXjP2XKFJcmpNmTzj33XOvVq5cLLLp16+Z6OzQ2QfQZ9VoKqDR1rB5TgJAVmrZW5bj++utdb4lmSJo9e7abMSqa6tSp42aU6ty5s3Xs2NFOPfVUe/HFF8N6DdXlhx9+6D6L6lOBhoIvjX2JhgTf8br0Yg5QdKd8Ps35q6m4gHh180dD7Z9Du61c0TI27rKRbpnODG3ZssUNDNu8eXPEV53NDVeLBYCs0gw8OnOe0/P4A3nxiuQZfT/CaW+TCgXEgL4t/mVHk45aofy5+yqoAAAgdpEKBcSA5lWbWKvqzd09AADIGl3rInAK3MDbY489RrVmgh4LAAAAwMxdjE5jTYLJylW6M0qF0i3WEFgAAAAAZikuWIfwEVgAMeD3f/6wY8lJViBffju5XM1oFwcAAMQhAgsgBjz+1bg0s0IBAAAcTwzeBgAAABAxAgsAAAAAESOwAAAAABAxAgsAAAAAESOwAAAAyKNuvPFG69q1a46/j6650LRpU4uWhIQE++CDDyyaFixY4Mqxe/fusJ/72muvWceOHS2ndevWzZ588kmLFgILAACAPOqZZ56xiRMnWm50PIOBtm3b2sCBA3Pl6x0+fNgefPBBGzZsmH/Z+PHj7ZxzzrGyZcu62wUXXGDffvttSIFN6tv27dv96zzwwAP26KOP2p49eywaCCwAAADyqNKlS1uZMmWiXYw8wefz2bFjx477+06dOtVKlSplrVu3ThEkXHfddTZ//nz75ptvrHr16q5HY8uWLZm+3po1a2zbtm3+W8WKFf2PNW7c2GrXrm2TJ0+2aCCwAAAAyMXUMG3SpIkVLVrUypcv785uHzhwIGgqlM6033777e5su86EV6pUyZ0d1/q9evWykiVLWp06dWzmzJn+56jHI3Vwop4GnQ1Pz3fffWcdOnSwE044wQU35513ni1fvtz/+EknneTuL7/8cvc63t/y4YcfWrNmzaxIkSJ28skn2/Dhw1M0+H/77Tc799xz3eMNGza0OXPmZFg/qoOFCxe63hvvLP6GDRv8Z/j1WZs3b26FCxe2r776Kmj6mOpLdZfR63mWLVtmLVq0sGLFitnZZ5/tGvoZeeedd+zSSy9Nseytt96yW265xaWX1a9f31599VVLTk62efPmWWYUSFSuXNl/y5cvZXNe76X3jAYukAcAAOKWGoiBqSTHgxqDS5cuDWldnZHWme3HH3/cNdL37dtnX375pTv7np5JkybZPffc41Jr3n33Xevfv79Nnz7dPf++++6zp59+2q6//nrbuHGjaxxnhcrRs2dPe+6551xZlNffuXNnFxQoeFHgoQbwhAkT7MILL7T8+fO756nsN9xwgz377LMuFWjdunXWr18/95hShdS4vuKKK1xAtGTJEpfSk1lKkgKAX3/91Z2t/89//uOWVahQwR8MDBkyxJ544gkXxCjYykxmr3f//fe7z6tlN998s/Xu3dsWLVqU7uspmFF9Z+TgwYOWmJho5cqVy7R8CkaOHDniyqexL4E9IXLGGWe4dCito2DqeCKwAGLA0xcNM5/5LMHSP7sEAEhLQUUo6SfRosBCZ/PV2K5Zs6Zbpt6LjJx22mku116GDh1qo0aNcj0Lffv2dcseeughe+mll+yHH36ws846K0vlateuXYq/X3nlFdfroTP9l1xyiWt0i5YpkPKod0INfQUlosb+iBEjXCCkwGLu3Lm2evVqmz17tlWtWtWt89hjj9lFF12UblnUY1KoUCEXJAW+l0fBgXpXQpXZ66nRrh4a0We5+OKL3TiKIkWKpFlXA70VHHmfJT333nuvW0e9UempUqWKjRs3zgXDChrUy6FeFgVg6gHy6HWOHj3q9m1vnzleCCyAGFC0YNqDGQAgc8EajrnpPRUktG/f3gUTnTp1cnn4V111VYZn3k899VT//9VToPSpwGBEvQGyc+fOLH+GHTt2uOBF6UZ6naSkJHfWXb0gGfn+++/d2X01zj16rhrmev4vv/zixhsENsRbtWplkVBDPDsF1q8a+6I6qFGjRpp1Dx065O6DBR0eBX5KXVJdZrRevXr13M2jNCz1+KgH6s033/QvV8qcqD6PNwILAAAQt0JNSYoWBQYaY/D111/bZ5995lKPlIqjs9S1atUK+pyCBQum+FtjBAKXeWMnlHYkytFPnVqltJyMqMfh77//dmlDOiuulBsFADpTnpH9+/e7Xgv1wKSWUaM6EsWLF0/xd1Y+b6CM6jI1BXVaZ9euXRaMUrQUWKinJjBgCZXSnpRqFeiff/5x916v0fHE4G0AAIBcTA1T5dGrQb5ixQqXpqMxE9lFDVCNmfAGhMvKlSszfI56He644w43rqJRo0YusPjrr7/SNMDVGxFIKTsa7KwB5KlvavA3aNDANm3a5FLAPIsXL870M6hOUr9XRp838PWDfd5wXi+zcjVs2NB+/vnnNI9p3IzSwGbNmpXlXhWV2+s18fz4449WrVo1l/52vNFjAcSAGWvm2sHEw1asYBG7pF76+ZkAgLxFPROaKUgpUBoMrb///PNP1wDPLmeeeaYbT6CB3QoW9B6ZXRujbt26Lv1GDeK9e/fa4MGD/Sk4Hs0EpbIrKFLgofQtje/QGAylDSmlS8GE0qPUGH7kkUfcGINTTjnF9YiMGTPGvbZ6aDKj91K5NcC6RIkSGQ6C1vgQvfYbb7zhelk0Nave//TTT8/S62WmU6dOrlchcBD66NGjXV28/fbb7r28CQT0Xrp542M0/kfllLFjx7peKgVySh3TGIvPP//c9WQF0gD543ExvlzXY6GKDHahj1tvvTWaxQLynBlr5tnUnz5x9wCA2KHrH3zxxReuZ0ANbo1r0IxEGQ1mDpcazWpcf/rpp24sxpQpU9xsQ5ldSVrpPeqB0IxHCkgCr6cgKqfSuDRmwmu0q5E9Y8YM1xhu2bKlGzyuMQLeIGMFGuqN0dgEpfn8+9//TjEeIz133323SxtT74B6JDIa66Ey6IJ1GjCuMqi3RjNVZfX1MtOnTx9Xt4EXrdPgeaWNKbhSj4N3U2qUR70qge+r9e+66y63jTR4XAGZUqg0BsejgENTBXsD9Y+3BF9G85XlMEXcgd1MihY1al8XC/HmEs6IoliN3NeG0hcPiFc3fzTU/jm028oVLWPjLhvplqkbVGc6TjzxRNu8eXOGzx8zOfMc48E9snfwGwAcT2pwrV+/3p3xzalcfiA9V199tQvC1AuRkxSwKDBL3YsRyfcjnPZ2VHssFAEGXuBDEayuFuhN4QUAAADkdWPGjPGnOOUkjWvRAP9oyTVjLNS9o264QYMGZXilRwAAACAvOemkk9wV0XOaUseiKdcEFsoH00VEdBn19OhiILoFds0AAAAAiL5cM92sBgFpIFJGVyYcOXKky/HybhoMBAAAACD6ckVg8ccff7hR7Zl132jAiwaOeDfNcwwAAAAg+nJFKtSECRPcFGUXX3xxhutpDmTdAAAAAOQuUe+x0CXQFVjoQigFCuSKOAcAAABAmKLeklcKlC7+0bt372gXBcizapWtbuWLlbVShXN+KjsAAIBcGVjokuNRvEYfEBPuPeeWaBcBAJALaMp+XSCta9euFssmTpxoAwcOdDOKIveIeioUgJynK2tndAMA5F6ail8Bg266AFqlSpWsQ4cO9vrrr7uU8kDbtm1zs2x6vOfpVrx4catbt657vWXLllledu2119qvv/5quUHbtm1dkBPKet620NWtTznlFDfjabAT7JMmTbKWLVtasWLFrGTJku7i0bqQdKAFCxa418pNwRWBBQAAQC534YUXuqBhw4YNNnPmTDv//PNtwIABdskll9ixY8f861WuXDnNRDcay6rn/vTTT/bCCy/Y/v377cwzz7Q33njD8qLExEQrWrSom/gnr+nbt6/bFmvWrHGznT700EM2bty4FOvcfffddtNNN7ng6YcffrBvv/3W2rRpY126dLHnn3/ecjMCCwAhodcDAKJHwYKChhNPPNGaNWtm9913n3344YcuyFBakEdnsHXR4UBlypRxz9XVn5WCPnXqVOvevbvddttttmvXrnTfU2fCdSmAChUqWKlSpaxdu3b2/fffu8f+/PNP95qPPfaYf/2vv/7aChUqZPPmzXN/P/zww9a0aVN7+eWX3bXHdPb9mmuucZcMCPTqq69agwYN3Fn8+vXr24svvuh/TIGUPtO7777rztprnbfeest9Zn0uj/de6sWpUaOGlShRwm655RZLSkqyxx9/3JVVgcijjz4a8mcMfN0333zT1Z+uo9atWzfbt2+fe1y9PwsXLrRnnnnG3xuhMqdHdaCy1KxZ03r16mWnnnqqzZkzx//44sWL7cknn7QxY8a4AKNOnTqublRu9YoMGjQoV19ugcACiAGjv3zR7p/7uLsHAMQHNYJPO+00mzZtWtjPvfPOO13jOLBRm9rVV19tO3fudMGLUqcU0LRv397++ecf1xBXI14N76VLl7rXuv76612wonU8a9eutffee88+/vhjmzVrlq1YscI1+D0KEnTWXg3nX375xQUqDz74oEsFCjRkyBDXQ6N1OnXqFLS869atc2XV+0yZMsVdfFmXMti8ebNr/I8ePdoeeOABW7JkSUifMfB1FawpFUk3vdaoUaPcYwooWrVq5e+J2LZtW0gXcFb605dffmmrV692wZhH5VZQpB6L1O666y7XW/P+++9bbhX1wdsAIrd+1yb759BuK1f0/87eAABCM2PNXJux5n9n2TObgS/1ZBk6oaNjcGYuqdfeLql3QbZvEp3hV7pMVp4n6Z1d/+qrr1wKjhrdXmrVE0884RrY6vHo16+fde7c2TWo1fvRokULN4ZDYwYCHT582KVcqadFnnvuOdfY11l5nbkfNmyY+/8VV1zhHq9Vq5b9/PPPrpdDlyLw6Gy9t056NN5EwY7GJDRs2NCliynl6NNPP7V8+fJZvXr1XHAxf/58lwoWymf0Xlc9JHpdUQClXhkFQ+rBUGDg9URkRr0x6qE5evSoCxLUA3PHHXf4H9e4kdq1a6cINjxVq1Z1vSq5ZWxJMAQWAAAgrh1MPOxOzmRG03qntvfI/pCeq/fICTrzrfSbrDxP0nuu0oE0FqN8+fIplh86dMidwfeoId64cWP773//6874px7fobQkL6gQnd1XQ10NfjXU9Vp9+vRxAYpHY0bUYA+kwCUzSlXyGv+iQe758+d3QUXgMgUS4XzG1K9bpUoV/2uEq3v37nb//fe7FDQFVWeffba7BcrLs6USWAAAgLhWrGCRkHp8g10rSMtCea7eIycoNUhn+bPyPEnvuWpwqwGtmYdSCxzboAb41q1bXbCg3o8mTZqEXAa9h4wfP971IARSQBBIvSGZ0YxZgbxZtFIv82bSCvUzZvQa4SpdurQbNyFKEdP/zzrrLLvggv/1ZmmmKPWkqEcjda+F6nnv3r1undyKwAIAAMQ1pShlNU0pmtcR+vzzz23VqlVuvES4xo4d69JqvAZtahprsH37ditQoIA7Yx+MGr89evRwsxcpzUiDoFWewNmadBFkNYiVxuMNTvbSktR7oOW///67O5N/vIXyGUOhAECDxMOlsRQaN6JB2hp7ooBFA8OfffZZlwp2++23p1hfvUMKcq688krLrQgsAAAAcrkjR464RrAasDt27HADlDWeQdPN3nDDDRk+VzMf6bl6DeXnq9GqcQQa+xB4Zj6QAg6lLelCe5pVSWfJFSB88skndvnll7vUJKX0aIYnNYTVSNZYht69e6e43oLGEGishBrFOtuu8QSaGcobjzB8+HC3TGfyNaWuyqjB4EoV0gxIOSmUzxgKBSUaEL5hwwZXD+XKlUuRfpURDdIeMWKEG5B91VVXufIo2Bg8eLAL3FQ2jcWYPHmyGyiugDCUweHRQmABAACQyymQUNqOzq6XLVvWzQalBr0a7Zk1YjWtqdfI13gHXRNBg5Z1xj49OnuuQEHBg57vTS977rnnup4GpQ+pkauB0Or5EE3JqnK99NJL1r9/f7dMqT4adK2B3pppSYFQ4HSy6uXQwGdNr6rGtFKelE4VygXnIpXZZwyVehy0HRo2bOjGZ6xfvz7kHhAFIQoMNbuW6knbUvWqaWhVT5rFSmlh2lYKBi+99FLLzRJ8eXiEiCJfRbiKlr2dGohHN3801D8r1LjL/jcjR7Vq1WzLli3uR2TAqJRzmueEwT1CO7MDANGg2YnU4NOYAjWwkfPUWFZjeOXKlVR3Hv5+hNPe5joWAAAAACJGYAEAAAAgYoyxAGKALrykOdJzajpDAACykgqlG+IHgQUQA3Liaq4AAADhIBUKAAAAQMQILAAAQNzIw5NhArn+e0FgAcSAQ4mH7WDiIXcPAEhLVyyWgwcPUj1AKt73wvueZBVjLIAYcOfM4WmuYwEA+D+6yJiuMr1z5073ty7KpgukAfHeU3Hw4EH3vdD3Q9+TSBBYAACAuKCrKosXXAD4HwUV3vcjEgQWAAAgLqiHokqVKlaxYkVLTEyMdnGAXEHpT5H2VHgILAAAQFxRIyq7GlIA/g+DtwEAAABEjMACAAAAQMQILAAAAABEjMACAAAAQMQILAAAAABEjMACAAAAQMSYbhaIAfe0udmOJSdZgXxMnwgAAKKDwAKIASeXqxntIgAAgDgX9VSoLVu2WI8ePax8+fJWtGhRa9KkiS1dujTaxQIAAACQV3osdu3aZa1bt7bzzz/fZs6caRUqVLDffvvNypYtG81iAQAAAMhLgcXo0aOtevXqNmHCBP+yWrVqRbNIQJ60bOsqO5p01ArlL2TNqzaJdnEAAEAcimoq1EcffWQtWrSwq6++2ipWrGinn366jR8/Pt31jxw5Ynv37k1xA2A2funb9vTXr7p7AACAuAssfv/9d3vppZesbt26Nnv2bOvfv7/dcccdNmnSpKDrjxw50kqXLu2/qbcDAAAAQJwHFsnJydasWTN77LHHXG9Fv379rG/fvjZu3Lig6w8dOtT27Nnjv23atOm4lxkAAABALgssqlSpYg0bNkyxrEGDBrZx48ag6xcuXNhKlSqV4gYAAAAgzgMLzQi1Zs2aFMt+/fVXq1mTOfkBAACAvCSqgcWdd95pixcvdqlQa9eutbffftteeeUVu/XWW6NZLAAAAAB5KbBo2bKlTZ8+3aZMmWKNGze2ESNG2NixY6179+7RLBYAAACAvHQdC7nkkkvcDQAAAEDeFdUeCwAAAABxGlgcOHAgZ0oCIMuKFChsRQsUcfcAAAB5IhWqUqVKds0111jv3r2tTZs2OVMqAGEZ2/lhagwAAOStHovJkyfbP//8Y+3atbNTTjnFRo0aZVu3bs2Z0gEAAACIzcCia9eu9sEHH9iWLVvs5ptvdlPE6roTGoA9bdo0O3bsWM6UFAAAAEDsDd6uUKGCDRo0yH744Qd76qmnbO7cuXbVVVdZ1apV7aGHHrKDBw9mb0kBAAAAxN50szt27LBJkybZxIkT7Y8//nBBRZ8+fWzz5s02evRod+G7zz77LHtLCyCoN1e+bweOHrTihYrZ9U2vpJYAAEDuDyyU7jRhwgSbPXu2NWzY0G655Rbr0aOHlSlTxr/O2WefbQ0aNMjusgJIx6KNS+2fQ7utXNEyBBYAACBvBBa9evWybt262aJFi9yVs4NROtT999+fHeUDAAAAEIuBxbZt26xYsWIZrlO0aFEbNmxYJOUCAAAAEMuBRcmSJV1wUbFixRTL//77b7csKSkpO8sHII8YM3lppusM7tHiuJQFAADkgVmhfD5f0OVHjhyxQoUKZUeZAAAAAMRqj8Wzzz7r7hMSEuzVV1+1EiVK+B9TL8UXX3xh9evXz5lSAgAAAIiNwOLpp5/291iMGzfO8ufP739MPRUnnXSSWw4AAAAg/oQcWKxfv97dn3/++W7K2bJly+ZkuQAAAADE8uDt+fPn50xJAAAAAMR2YDFo0CAbMWKEFS9e3P0/I0899VR2lQ1AiJpVaWz7jx60EoUyngoaAAAgqoHFihUrLDEx0f//9GhgN4Djr1/L7lQ7AADI/YFFYPoTqVAAAAAAIr6ORWp79+61Dz74wFavXh3pSwEAAACIl8Dimmuuseeff979/9ChQ9aiRQu3rEmTJvb+++/nRBkBAAAAxFpgoQvhnXPOOe7/06dPd9e12L17t7uA3iOPPJITZQSQiSGfjbSbPxrq7gEAAPJEYLFnzx4rV66c+/+sWbPsyiuvtGLFitnFF19sv/32W06UEUAmdh/ea/8c2u3uAQAA8kRgUb16dfvmm2/swIEDLrDo2LGjW75r1y4rUqRITpQRAAAAQKxdIG/gwIHWvXt3K1GihNWsWdPatm3rT5HSOAsAAAAA8SfswOKWW26xM844wzZt2mQdOnSwfPn+1+lx8sknM8YCAAAAiFNhBxaimaB0C6QxFgAAAADiU9iBRVJSkk2cONHmzZtnO3futOTk5BSPf/7559lZPgAAAACxGFgMGDDABRbqoWjcuLElJCTkTMkAAAAAxG5g8c4779h7771nnTt3jvjNH374YRs+fHiKZfXq1eMq3gAAAECsBxaFChWyOnXqZFsBGjVqZHPnzv2/AhXI0rAPAAAAAFEUdiv+rrvusmeeecaef/75bEmDUiBRuXLliF8HiFdjJi+1MglNrZQds3z7Cri/Zf/BxBT3AAAAuSqw+Oqrr2z+/Pk2c+ZM19tQsGDBFI9PmzYtrNfT1bqrVq3qLq7XqlUrGzlypNWoUSPcYgFxrbyvdrSLAAAA4lzYgUWZMmXs8ssvz5Y3P/PMM91AcI2r2LZtmxtvcc4559iPP/5oJUuWTLP+kSNH3M2zd+/ebCkHAAAAgOMcWEyYMMGyy0UXXeT//6mnnuoCDV3NW4PD+/Tpk2Z99WakHuwNAAAAIPr+d9nsMB07dswNuH755Zdt3759btnWrVtt//79ERVGvSGnnHKKrV27NujjQ4cOtT179vhvuvo3ALNDttsO2S53DwAAkCd6LP744w+78MILbePGjS4tqUOHDi5tafTo0e7vcePGZbkwCkzWrVtn119/fdDHCxcu7G4AUlpTYKYlJhy0gr5i1vTYdVQPAADI/T0WukBeixYtbNeuXVa0aFH/co270NW4w3H33XfbwoULbcOGDfb111+718ifP79ddx0NIwAAACCmeyy+/PJLFwToehaBTjrpJNuyZUtYr7V582YXRPz9999WoUIFa9OmjS1evNj9HwAAAEAMBxbJycmWlJQUNEgINpNTZlfxBgAAABCHqVAdO3a0sWPH+v/WRfI0NmLYsGHWuXPn7C4fAAAAgFjssXjyySetU6dO1rBhQzt8+LD961//che5O+GEE2zKlCk5U0oAAAAAsRVYVKtWzb7//nuXxvTDDz+43gpdc6J79+4pBnMDAAAAiB8FsvSkAgWsR48e2V8aAAAAALEbWHz00Uchv+Bll10WSXkAAAAAxGpg0bVr1xR/a8C2z+dLs0yCzRgFAAAAILblC3WKWe/22WefWdOmTW3mzJm2e/dud9P/mzVrZrNmzcr5EgNIo+GxLnZaYjd3DwAAkCfGWAwcONDGjRvnLmbn0SxRxYoVs379+tkvv/yS3WUEkIlCVow6AgAAees6FuvWrbMyZcqkWV66dGnbsGFDdpULAAAAQCwHFi1btrRBgwbZjh07/Mv0/8GDB9sZZ5yR3eUDAAAAEIupUK+//rpdfvnlVqNGDatevbpbtmnTJqtbt6598MEHOVFGAJnYmbDakhMSLZ+voFX01ae+AABA7g8s6tSp4y6MN2fOHFu9erVb1qBBA7vgggv8M0MBOL625l9hiQkHraCvmFU8RmABAADyyAXyFEB07NjR3QAAAAAg7DEWAAAAAJAagQUAAACAiBFYAAAAADi+gcWxY8fsjTfeSDHVLAAAAACEFVgUKFDAbr75Zjt8+DA1BwAAACDrqVC6CN7KlSvDfRoAAACAGBb2dLO33HKLu/K2LorXvHlzK168eIrHTz311OwsHwAAAIBYDCy6devm7u+4444U17Xw+XzuPikpKXtLCCBTRXylLb8VsoK+otQWAADIG4HF+vXrc6YkALKsflJnag8AAOStwKJmzZo5UxIAAAAA8RNYyLp162zs2LH2yy+/uL8bNmxoAwYMsNq1a2d3+QAAAADE4qxQs2fPdoHEt99+6wZq67ZkyRJr1KiRzZkzJ2dKCQAAACC2eiyGDBlid955p40aNSrN8nvvvdc6dOiQneUDEIJ1+RfYMTtsBayI1U5qS50BAIDc32Oh9Kc+ffqkWd67d2/7+eefs6tcAMKwL2Gb7c23xd0DAADkicCiQoUKQS+Qp2UVK1bMrnIBAAAAiOVUqL59+1q/fv3s999/t7PPPtstW7RokY0ePdpdOA8AAABA/Ak7sHjwwQetZMmS9uSTT9rQoUPdsqpVq9rDDz+c4qJ5AAAAAOJH2KlQurq2Bm9v3rzZ9uzZ4276v6ab1WNZpcHgev7AgQOz/BoAAAAA8tB1LDzqucgO3333nb388stu6loAAAAAMRpYNGvWzObNm2dly5a1008/PcOeieXLl4dVgP3791v37t1t/Pjx9sgjj4T1XAAAAAB5KLDo0qWLFS5c2P2/a9eu2VqAW2+91S6++GK74IILMg0sjhw54m6evXv3ZmtZAAAAAORgYDFs2DB3n5SUZOeff75LWSpTpoxF6p133nE9HEqFCsXIkSNt+PDhEb8vAAAAgCgO3s6fP7917NjRdu3aFfEbb9q0yQ34fuutt6xIkSIhPUezUHkDxnXTawAwq5BczyolNXL3AAAAeWLwduPGjd01LGrVqhXRGy9btsx27tzpxm941CPyxRdf2PPPP+9SnhTIBFI6lpeSBeD/nJj8f98jAACAPBFYaBzE3XffbSNGjLDmzZtb8eLFUzxeqlSpkF6nffv2tmrVqhTLevXqZfXr17d77703TVABAAAAIPcKO7Do3Lmzu7/ssstSzA7l8/nc3+p1CHWqWvV+BFKQUr58+TTLAQAAAMRYYDF//vycKQkAAACA+AkszjvvvJwpiZktWLAgx14biGUrC0yxxISDVtBXzJoeuy7axQEAAHEorFmhPF9++aX16NHDzj77bNuyZYtb9uabb9pXX32V3eUDAAAAEIuBxfvvv2+dOnWyokWLumtQeBes0/Svjz32WE6UEQAAAECsBRaaFWrcuHE2fvx4K1iwoH9569atXaABAAAAIP6EHVisWbPGzj333DTLS5cubbt3786ucgEAAACI5cCicuXKtnbt2jTLNb7i5JNPzq5yAQAAAIjlwKJv3742YMAAW7JkibtuxdatW+2tt95yF83r379/zpQSAAAAQGxNNztkyBBLTk52V84+ePCgS4sqXLiwCyxuv/32nCklAAAAgNgKLNRLcf/999vgwYNdStT+/futYcOGVqJEiZwpIQAAAIDYS4Xq3bu37du3zwoVKuQCijPOOMMFFQcOHHCPAQAAAIg/YQcWkyZNskOHDqVZrmVvvPFGdpULQBhOTmprpxzr5O4BAABydSrU3r17zefzuZt6LIoUKeJ/LCkpyT799FOrWLFiTpUTQAZK+apQPwAAIG8EFmXKlHHjK3Q75ZRT0jyu5cOHD8/u8gEAAACIpcBi/vz5rreiXbt29v7771u5cuX8j2m8Rc2aNa1q1ao5VU4AAAAAsRBYnHfeee5+/fr1VqNGDddDASB32JuwzXyWZAmWn7QoAACQNwZv//LLL7Zo0SL/3y+88II1bdrU/vWvf9muXbuyu3wAQvB7/gX2a4HZ7h4AACBPBBa6foUGcsuqVats0KBB1rlzZ9eTof8DAAAAiD9hXyBPAYSuXyEaa3HppZfaY489ZsuXL3cBBgAAAID4E3aPhQZqHzx40P1/7ty51rFjR/d/Deb2ejIAAAAAxJeweyzatGnjUp5at25t3377rb377rtu+a+//mrVqlXLiTICAAAAiLUei+eff94KFChgU6dOtZdeeslOPPFEt3zmzJl24YUX5kQZAQAAAMRaj4Wmmp0xY0aa5U8//XR2lQkAAABArAcWkpSUZB988IGbelYaNWpkl112meXPnz+7ywcAAAAgFgOLtWvXutmftmzZYvXq1XPLRo4cadWrV7dPPvnEateunRPlBAAAABBLYyzuuOMOFzxs2rTJTTGr28aNG61WrVruMQAAAADxJ+wei4ULF9rixYvd9LKe8uXL26hRo9xMUQCOv6bHrqPaAQBA3uqxKFy4sO3bty/N8v3797trXAAAAACIP2EHFpdccon169fPlixZYj6fz93Ug3HzzTe7AdwAAAAA4k/YgcWzzz7rxli0atXKihQp4m5KgapTp44988wzOVNKAAAAALE1xqJMmTL24Ycf2m+//eamm01ISLAGDRq4wAJAdGzJt9yS7Kjlt0J2YnIzNgMAAMj9PRaeunXr2qWXXupSo7IaVOjK3aeeeqqVKlXK3dQLoit4AwjPn/nW2I78P7l7AACAPBNYvPbaa9a4cWN/KpT+/+qrr4b9OtWqVXOzSS1btsyWLl1q7dq1sy5duthPP/2UlWIBAAAAyCupUA899JA99dRTdvvtt7seBvnmm2/szjvvdNez+M9//hPya6nHI9Cjjz7qejE0GFxX8wYAAAAQo4GFGv7jx4+36677v3nzNRuUUpoUbIQTWARKSkqy//73v3bgwAF/wAIAAAAgRgOLxMREa9GiRZrlzZs3t2PHjoVdgFWrVrlA4vDhw1aiRAmbPn26NWzYMOi6R44ccTfP3r17w34/AAAAALlgjMX111/vei1Se+WVV6x79+5hF6BevXq2cuVKd12M/v37W8+ePe3nn38Ouu7IkSOtdOnS/lv16tXDfj8AAAAAuaDHwhu8/dlnn9lZZ53l/lZQoPEVN9xwgw0aNMi/nsZiZEZX6/ZmlVKvx3fffeeuh/Hyyy+nWXfo0KEpXl89FgQXAAAAQB4MLH788Udr1ux/8+SvW7fO3Z9wwgnupsc8ur5FViQnJ6dIdwpUuHBhdwMAAACQxwOL+fPnZ9ubqwfioosusho1ati+ffvs7bfftgULFtjs2bOz7T0AAAAA5NJUqOyyc+dOlz61bds2N2ZCM0spqOjQoUM0iwXkOSV9VeyY77AVsCLRLgoAAIhTUQ0sNFYDQORqJ7WlGgEAQN678jYAAAAABCKwAAAAABAxAgsAAAAAeXuMBYDssTr/p5aYcMgK+opa/aTOVCsAADjuCCyAGHA4YY8lJhy0JDsa7aIAAIA4RSoUAAAAgIgRWAAAAACIGKlQAADEmPUTJmX4eK1ePY9bWQDED3osAAAAAESMwAIAAABAxAgsAAAAAESMMRZALjdm8tJoFwEAACBTBBYAAMTZwGsGdwPICQQWQAyomnS6JSckWj5fwWgXBQAAxCkCCyAGVPTVN/NFuxQAACCeMXgbAAAAQMTosQAAIA/JbHwEAEQLgQUQA47aQftfLlSCFbJi0S4OAACIQwQWQAz4ucCHlphw0Ar6ilnTY9dFuzgAACAOMcYCAAAAQMQILAAAAABEjMACAAAAQMQYYwEAwHHCjE4AYhk9FgAAAAAiRmABAAAAIGIEFgAAAAAiRmABAAAAIGIEFgAAAAAixqxQQAyod+wiM/OZWUK0iwIAAOIUgQUQA4pamWgXAQAAxLmoBhYjR460adOm2erVq61o0aJ29tln2+jRo61evXrRLBYAAHEtlOtt1OrV87iUBUDeEdUxFgsXLrRbb73VFi9ebHPmzLHExETr2LGjHThwIJrFAgAAAJCXeixmzZqV4u+JEydaxYoVbdmyZXbuuedGrVxAXvN3wjpLtmOWzwpYeV/taBcHQBzIrFeDHg0g/uSqMRZ79uxx9+XKlYt2UYA8ZVP+by0x4aAV9BWz8scILADEBoIXIG/JNYFFcnKyDRw40Fq3bm2NGzcOus6RI0fczbN3797jWEIAAAAAuf46Fhpr8eOPP9o777yT4WDv0qVL+2/Vq1c/rmUEAAAAkIsDi9tuu81mzJhh8+fPt2rVqqW73tChQ126lHfbtGnTcS0nAAAAgFyYCuXz+ez222+36dOn24IFC6xWrVoZrl+4cGF3A5A3jZm8NMPHB/docdzKAgAAYiiwUPrT22+/bR9++KGVLFnStm/f7pYrzUnXtQAAAHkT18IA4k9UA4uXXnrJ3bdt2zbF8gkTJtiNN94YpVIBAIDcEnwAyDuingoFAAAAIO/LFYO3AQAAAORtueY6FgCyThfGC7wHEB2k9gCIZwQWQAxolNQl2kUAAABxjlQoAAAAABGjxwIAgBCQ5gQAGaPHAgAAAEDE6LEAYsCGfF/ZsYQjVsBX2E5KbhPt4gAAgDhEYAHEgN35NlliwsH/zQqVHO3SAACAeEQqFAAAAICIEVgAAAAAiBiBBQAAAICIEVgAAAAAiBiBBQAAAICIEVgAAAAAiBiBBQAAAICIEVgAAAAAiBgXyANiQPnk2v4rbwMAAEQDgQUQA6onnxHtIgAAgDhHYAEAAPKk9RMmZbpOrV49j0tZABBYAADiRGaNUBqgABAZBm8DAAAAiBipUEAMWFVgqh21g1bIilmTY1dFuzgAACAOEVgAMSDJEi05IdGSfInRLgoQ0/n6AID0kQoFAAAAIGIEFgAAAAAiRmABAAAAIGIEFgAAAAAiRmABAAAAIGIEFgAAAAAiRmABAAAAIG8HFl988YVdeumlVrVqVUtISLAPPvggmsUBAAAAkBcDiwMHDthpp51mL7zwQjSLAeR5JyW1ttrH2rl7AACAuLvy9kUXXeRuACJTxleDKgQAAPEbWITryJEj7ubZu3dvVMsDAAAAIA8O3h45cqSVLl3af6tevXq0iwQAAAAgr/VYDB061AYNGpSix4LgAjA7YH+ZLyHJEnz5rbidkGerZMzkpRk+PrhHi+NWFgCxYf2ESRk+XqtXz+NWFiDW5anAonDhwu4GIKXfCsyxxISDVtBXzJoeu47qAQAAx12eCiwAAACOZ4+G0KsB5IHAYv/+/bZ27Vr/3+vXr7eVK1dauXLlrEYNZrkBAAAA8oqoBhZLly61888/3/+3N36iZ8+eNnHixCiWDAAAAECeCSzatm1rPp8vmkUAAOQBDMAFgNwvT003CwAAACB3IrAAAAAAEDFmhQJy+bUbgFgXyqw8x+M1AACRoccCAAAAQMQILAAAAABEjFQoIAY0OXZltIsAAADiHIEFEAPyW6FoFwEAYhbTHQOhIRUKAAAAQMTosQAAAMhh9HogHhBYADFge75VlmSJlt8KWuXkJtEuDgAAiEMEFkAM2J7vR0tMOGgFfcUILADgOOM6KsD/MMYCAAAAQMQILAAAAABEjMACAAAAQMQILAAAAABEjMACAAAAQMQILAAAAABEjOlmAQA5Os1mrV49qWEAiAMEFgCAHMUc/wAQHwgsgBhQ3HeCJfoOWUErGu2iAAByAL2DyAsILIAYUDepQ7SLAADI5cEHaYnIaQQWAAAAUUbKIGIBgQUAIF00dgAAoSKwAHLQmMlLqV/kagQOAIDsQmABxIDf8s+xRPvf4O1YHm8RSqA2uEeL41IWAACQEoEFEAMOJPxliQkHraCvWLSLAgAA4hRX3gYAAAAQMQILAAAAABEjsAAAAAAQG2MsXnjhBRszZoxt377dTjvtNHvuuefsjDPOiHaxACBPY8YnAEBcBRbvvvuuDRo0yMaNG2dnnnmmjR071jp16mRr1qyxihUrRrt4AAAAMSE7TjZw9W7k6sDiqaeesr59+1qvXr3c3wowPvnkE3v99ddtyJAh0S4egBibkja3TEdLbwKAvCiUYxfBR/yKamBx9OhRW7ZsmQ0dOtS/LF++fHbBBRfYN998E82iASHhAngAAGTviRMCk7wrqoHFX3/9ZUlJSVapUqUUy/X36tWr06x/5MgRd/Ps2bPH3e/du/c4lBbx5pl3l1tekZj/iCUmHDXz5bfDSfvdMp8v2X9/+ND/lsHshxfHZVoNJ/X4V4aPb5j8dkTPl32HDrE5ACCHjtPIPl472+fzZbpugi+UtXLI1q1b7cQTT7Svv/7aWrVq5V9+zz332MKFC23JkiUp1n/44Ydt+PDhUSgpAAAAEL82bdpk1apVy709FieccILlz5/fduzYkWK5/q5cuXKa9ZUypYHenuTkZPvnn3+sfPnylpCQYNGK4qpXr+4qu1SpUlEpQyyhPqnL3Ij9krrMjdgvqcvcin0ztupSfRD79u2zqlWrZrpuVAOLQoUKWfPmzW3evHnWtWtXf7Cgv2+77bY06xcuXNjdApUpU8ZyA21sAgvqMzdi36QucyP2S+oyN2K/pD5zq1JRbmeWLl06b8wKpR6Inj17WosWLdy1KzTd7IEDB/yzRAEAAADI/aIeWFx77bX2559/2kMPPeQukNe0aVObNWtWmgHdAAAAAHKvqAcWorSnYKlPeYFSs4YNG5YmRQvUZ7Sxb1KXuRH7JXWZG7FfUp+5VeE81s6M6qxQAAAAAGJDvmgXAAAAAEDeR2ABAAAAIGIEFgAAAAAiRmARYOTIkdayZUsrWbKkVaxY0V1bY82aNZlW4n//+1+rX7++FSlSxJo0aWKffvppisc1jEWzXlWpUsWKFi1qF1xwgf32228Wy7JSl+PHj7dzzjnHypYt626qp2+//TbFOjfeeKO7GGLg7cILL7RYlpW6nDhxYpp60v4Z7/tlVuuzbdu2aepTt4svvjiu982XXnrJTj31VP/86q1atbKZM2dm+ByOl9lTlxwvs2+/5HiZfXXJsTJ0o0aNcr8TAwcOjKljJoFFgIULF9qtt95qixcvtjlz5lhiYqJ17NjRXVcjPV9//bVdd9111qdPH1uxYoVrpOj2448/+td5/PHH7dlnn7Vx48bZkiVLrHjx4tapUyc7fPiwxaqs1OWCBQtcXc6fP9+++eYbd6VJPWfLli0p1lNjbdu2bf7blClTLJZlpS5FPwKB9fTHH3+keDwe98us1ue0adNS1KW+3/nz57err746rvfNatWquR/HZcuW2dKlS61du3bWpUsX++mnn4Kuz/Ey++qS42X21aVwvMyeuuRYGZrvvvvOXn75ZRe0ZSRPHjM1KxSC27lzp2bM8i1cuDDdKrrmmmt8F198cYplZ555pu+mm25y/09OTvZVrlzZN2bMGP/ju3fv9hUuXNg3ZcqUuKn6UOoytWPHjvlKlizpmzRpkn9Zz549fV26dPHFs1DqcsKECb7SpUun+zj7ZXj1mdrTTz/t9s39+/f7l7Fv/k/ZsmV9r776atB643iZfXWZGsfLrNclx8uc2y85Vqa1b98+X926dX1z5szxnXfeeb4BAwb40pMXj5n0WGRgz5497r5cuXLprqMz6+p2CqRIUctl/fr17sJ/gevosuhnnnmmf514EEpdpnbw4EF3Njn1c3SmTiks9erVs/79+9vff/9t8STUuty/f7/VrFnT9fykPsPEfhl+fQZ67bXXrFu3bu7MUKB43jeTkpLsnXfecT0/SpcIhuNl9tVlahwvI6tLjpc5s19yrExLPeZKo03ddoyVY2auuEBebpScnOzy3lq3bm2NGzdOdz1t0NRXCdffWu497i1Lb51YF2pdpnbvvfda1apVU3xhlGpyxRVXWK1atWzdunV233332UUXXeS+QEpNiXWh1qUatq+//rrrZlXD+YknnrCzzz7bBRfq2ma/DK8+A2ncj7qh9YMZKF73zVWrVrlGhrrdS5QoYdOnT7eGDRsGXZfjZfbVZWocL7Nelxwvc2a/5FiZlgKz5cuXu1SoUOTFYyaBRQYRpRoPX3311fHdIjEoK3WpnE59AXUGOHDQsc4SezSISQ3n2rVru/Xat29vsS7UutSPQOAZJQUVDRo0cDmdI0aMOA4ljd19UwGF9r0zzjgjxfJ43TfVKFu5cqULYKdOnWo9e/Z041hCbRAj8rrkeBnZfsnxMmf2S46VKW3atMkGDBjgxvalnkwllpAKFcRtt91mM2bMcIOIdXY3I5UrV7YdO3akWKa/tdx73FuW3jqxLJy69Ojsun4oP/vss0wHNp188sl2wgkn2Nq1ay3WZaUuPQULFrTTTz/dX0/xvl9mtT6VAqCAVwPpMhMv+2ahQoWsTp061rx5czfj1mmnnWbPPPNM0HU5XmZfXXo4XmZfXXo4XkZelxwr09IA+J07d1qzZs2sQIEC7qYATQOv9X+lmsXCMZPAItWUXWpsqJvv888/dykNmdGZjnnz5qVYpmjUO1us19DGDVxn7969buR+qDmK8VKX3uwGOqM+a9Ysa9GiRabrb9682eWxa5q1WJXVugykA5a6s716itf9MtL61LR/R44csR49emS6bjzsm+mll6mOguF4mX11KRwvs68uA3G8jLwuOVampZ5r/Q6r98e7qZ3TvXt39/9gKbN58pgZlSHjuVT//v3dTDoLFizwbdu2zX87ePCgf53rr7/eN2TIEP/fixYt8hUoUMD3xBNP+H755RffsGHDfAULFvStWrXKv86oUaN8ZcqU8X344Ye+H374wc1qVKtWLd+hQ4d8sSordal6KlSokG/q1KkpnqMZFET3d999t++bb77xrV+/3jd37lxfs2bN3OwKhw8f9sWqrNTl8OHDfbNnz/atW7fOt2zZMl+3bt18RYoU8f30009xvV9mtT49bdq08V177bVplsfrvqk60mxa+szah/R3QkKC77PPPnOPc7zMubrkeJl9dcnxMvvq0sOxMjSpZ4WKhWMmgUVgZZgFvWkqusCdQNNKBnrvvfd8p5xyimsUN2rUyPfJJ5+keFzTgT344IO+SpUquSnA2rdv71uzZo0vlmWlLmvWrBn0OfoiiRp+HTt29FWoUMF9sbR+3759fdu3b/fFsqzU5cCBA301atRw+6T2u86dO/uWL1/ui/f9MpLv+erVq9163g9qoHjdN3v37u0+q/YzfXbtQ4H1w/Ey5+qS42X21SXHy+z9jnOszHpgEQvHzAT9E52+EgAAAACxgjEWAAAAACJGYAEAAAAgYgQWAAAAACJGYAEAAAAgYgQWAAAAACJGYAEAAAAgYgQWAAAAACJGYAEAAAAgYgQWABBlEydOtDJlylhecf3119tjjz2WLa+1YcMGS0hIsJUrV1qsatu2rQ0cODDL+8PDDz9sTZs2jbgcP//8s1WrVs0OHDgQ8WsBQDAEFgCQB3kNcu9Wrlw5O++88+zLL79Ms+4///zjGrY1a9a0QoUKWdWqVa137962cePGFOvdeOON1rVr1wzf9/vvv7dPP/3U7rjjjmz/TMhZDRs2tLPOOsueeuopqhpAjiCwAIAcdPTo0Ryt37lz59q2bdvsiy++cAHDJZdcYjt27EgRVKgxqfXGjRtna9eutXfeecfdt2zZ0n7//few3u+5556zq6++2kqUKGHxLKe3a3ZLTEx097169bKXXnrJjh07Fu0iAYhBBBYA4srUqVOtSZMmVrRoUStfvrxdcMEFKVJDXn31VWvQoIEVKVLE6tevby+++GKK59977712yimnWLFixezkk0+2Bx980N9oC0xb0evUqlXLvY7s3r3bbrrpJqtUqZJb1rhxY5sxY0aK1549e7Z7bzXaL7zwQhcwZEafoXLlyu717rvvPtu7d68tWbLE//j9999vW7dudYHFRRddZDVq1LBzzz3XvVfBggXt1ltvDbnukpKSXP1deumlKZarjurWres+lz7fVVdd5X9s1qxZ1qZNG5fao7Iq8Fm3bl2G7/Pjjz+6sqoe9HpKvfrrr79C3oaBFixY4Hp0PvnkEzv11FNdGRVo6T0CffXVV3bOOee416xevbrrkQl8zZNOOslGjBhhN9xwg5UqVcr69esX9P30HK2jslepUsWefPLJNOvs2rXLrVO2bFm3H+mz/vbbbxaq7777zjp06GAnnHCClS5d2vVULV++PMU6+swKIC677DIrXry4Pfroo265nqdgc+HChSG/HwCEisACQNxQQ/26665zaUC//PKLa3ReccUV5vP53ONvvfWWPfTQQ64Rpsc1jkCBw6RJk/yvUbJkSZcDr3z1Z555xsaPH29PP/10ivdRb8D7779v06ZNc2MHkpOTXeNx0aJFNnnyZPfcUaNGWf78+f3POXjwoD3xxBP25ptvut4HpSndfffdIX+2Q4cO2RtvvOH+r3Qn0fuqd6J79+4u+AikBvQtt9ziAgw1NEPxww8/2J49e6xFixb+ZUuXLnWN8P/85z+2Zs0aF0gocAlsaA8aNMitN2/ePMuXL59dfvnlrmzBKABr166dnX766e45ej31wFxzzTUhbcP0DB482DXy1SivUKGCC468gFCBjgK5K6+80n3Gd9991wUat912W4rX0PY57bTTbMWKFW6/SO991Gj/8MMP7bPPPnPlS93oV8qZPttHH31k33zzjSt7586dUwSoGdm3b5/17NnTlXHx4sUuqNPztTyQglzV9apVq1x9efuGAt9gKXMAEDEfAMSJZcuWqfXp27BhQ9DHa9eu7Xv77bdTLBsxYoSvVatW6b7mmDFjfM2bN/f/PWzYMF/BggV9O3fu9C+bPXu2L1++fL41a9YEfY0JEya4cq1du9a/7IUXXvBVqlQp3fddv369e07RokV9xYsX9yUkJLi/VZajR4+6dbZv3+6WPf3000FfY9q0ae7xJUuWuL979uzp69KlS7rvOX36dF/+/Pl9ycnJ/mXvv/++r1SpUr69e/f6QvHnn3+691y1alWKz7FixQp/fXfs2DHFczZt2uTWUf1ltg1Tmz9/vlv/nXfe8S/7+++/Xb29++677u8+ffr4+vXrl+J5X375pdtmhw4dcn/XrFnT17Vr1wzfa9++fb5ChQr53nvvvTTvNWDAAPf3r7/+6sqzaNEi/zp//fWXW8d7nvaH0qVLp9inTjvttHTfNykpyVeyZEnfxx9/7F+m9xg4cGDQ9S+//HLfjTfemOFnAYCsoMcCQNzQ2eb27du7NBqNE1Bvg9JSvDPrOnPdp08fl8bi3R555JEUqTs6m926dWvXA6DHH3jggTSDoDVIWmfFPeq10Gw8SqFKj1Jiateu7f9baTQ7d+7M9DOpPDqDrh6SOnXquN4UpTgFyuxsfji9IoULF3ZpNh6l1ujzKi1MKUvq9VHvi0cpPuph0ONKIVJKkaSus8DB4fPnz0+xDZSSJtoOGW3DjLRq1cr/fw10r1evnuvx8N5T9Rb4np06dXK9KuvXr/c/L7CnJhiVT2MvzjzzzDTv5dF7FihQIMU6SucKLE9m1IPTt29f11OhVCjV6/79+9PUaXrlVW9V4DYCgOxSINteCQByOaUezZkzx77++muXpqKByBqDoDEJatiLGqqBjT7veaK0FaUVDR8+3DU81ahTqlHqPHrltKduyGUmdTCgxnsoAYHGA6iBqZsG5Cr1ReMHFAAouNHYhvQarFqu91FAEgrl9KtBqsazl26l1DCl+ijlR3WqVDKl4CjlSO+tlCMFHqpXDS5XY13jQdIb/KwGsp4zevToNI8p2MpoG2pMS1boPTX+JdhMVxqTkt52jRalQf39998uFU91q22twCl1naZXXqW+BQaxAJBd6LEAEFfUkFaPg4IDnelXA3n69OlukLAavpolSQ3twJvXYFVjVg05NWR1NliN+T/++CPT99Sg4c2bN9uvv/6ao59Ng6Z1NtwbcK7xDBqb8Pbbb9v27dvT9D5oPQVIOqseCu9aChojEkjvqQHUjz/+uBujoKlwP//8c9f41bgL9eqol0ED0zPrXWjWrJn99NNPrmcj9XbwGsrpbcOMaCyCR2XQtlB5vPfUZ0r9frp5AVQo1FhXgBg4eN57L4/eUwFg4DpePWk62FBorI6CII2raNSokQssAge3Z0aBp8awAEB2I7AAEDfUmNOAbA2cVdqIBlf/+eef/gamGqojR460Z5991jUGNeh1woQJ/nn/FUjoeeqlUNqL1susQSuatUcDmjU4WGfblV4zc+ZMNzA5O6nBrQanBoZ7qS76vErbUsqS3nPTpk1ucLgCCg0WfuGFF0J+ffWAqBGuQcMezWylelC6l4IsDSBXr4RSezTrkdJ8XnnlFTegXcGGBnJnRLNU6Yy60qfU66F61gBzTZOqWaky24bp0eByDR5Xo1qDp9X74l2zQzN9KWjUYG19DqVvafB16sHbmVEKlVLpNIBbn9V7LwV4Hu1DXbp0calMqkelYfXo0cNOPPFEtzwUeg0N8lePk+pDvWih9IqJgr4tW7a4QBAAshuBBYC4oVx0Nap1plfjHXQmXWlMmrFJ/v3vf7tpYhVMKIdfAYFy770eC03deeedd7oGp87eqzGa3uxAqWkMhK4boQazzkzfc889rqGcE2kyChief/5597ca9jpbf/7557t0H51VVy+G7tVw19iHcKiONI7Co3QnNe41k5Ma97pWxpQpU9yZdDWoFYQtW7bMpT+p7saMGZPh66vXSGfkVTcdO3Z020EX99P76PUy24bpUbA1YMAAa968ueu9+fjjj/29EepR0kxOCiY15azO5iulS2UJlz6fXkPpXGq8a6pdvWcg7V9apql3lcKklDdddDB1Olx6XnvtNdcToiBP41oUTFasWDGk52rbqF7V8wYA2S1BI7iz/VUBADFJKVTqjdCg8cAB0bmVxn4oqFJDXMFJPNMYDPV2KDVOqWQAkN3osQAAhEwpN0p3CienH7mDUsd0EUWCCgA5hVmhAABhadu2LTWWB3kD0gEgp5AKBQAAACBipEIBAAAAiBiBBQAAAICIEVgAAAAAiBiBBQAAAICIEVgAAAAAiBiBBQAAAICIEVgAAAAAiBiBBQAAAICIEVgAAAAAsEj9P+exS1nTkNUgAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig, ax = plt.subplots(figsize=(8, 4.5))\n", + "bins = np.linspace(2.0, 4.0, 80)\n", + "ax.hist(roi_default.ravel(), bins=bins, density=True, alpha=0.55,\n", + " label=\"default roi_m prior\", color=\"#c44e52\")\n", + "ax.hist(roi_calibrated.ravel(), bins=bins, density=True, alpha=0.65,\n", + " label=\"calibrated roi_m prior\", color=\"#4c72b0\")\n", + "ax.axvline(BETA_SEARCH, color=\"black\", lw=2, label=\"simulated truth (2.5)\")\n", + "ax.axvline(prior.roi_mean, color=\"#55a868\", lw=2, ls=\"--\", label=\"DiD experiment ROI\")\n", + "ax.set_xlabel(\"search ROI (sales per dollar)\")\n", + "ax.set_ylabel(\"posterior density\")\n", + "ax.set_title(\"What the experiment prior buys: Meridian search-ROI posterior\")\n", + "ax.legend()\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "204a3490", + "metadata": {}, + "source": [ + "With the default prior the national model overcredits search - the launches rode\n", + "the demand ramp, and nationally there is no holdout to say otherwise. The\n", + "calibrated prior carries the geo experiment's information into the model: the\n", + "posterior centers on the truth and tightens." + ] + }, + { + "cell_type": "markdown", + "id": "ed4f381a", + "metadata": {}, + "source": [ + "## Acceptance checks\n", + "\n", + "Sampler health first - the whole-model maximum R-hat (Meridian's own health-check\n", + "convention treats R-hat < 1.2 as converged) plus a tighter bar on the reported\n", + "`roi_m` for the experiment channel - then the calibration claims with slack\n", + "margins against draw noise." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "fb43c031", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-16T19:54:22.712911Z", + "iopub.status.busy": "2026-08-16T19:54:22.712807Z", + "iopub.status.idle": "2026-08-16T19:54:22.823473Z", + "shell.execute_reply": "2026-08-16T19:54:22.822974Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "default: whole-model max Rhat=1.0038, Rhat(roi_m, search)=0.9998\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "calibrated: whole-model max Rhat=1.0175, Rhat(roi_m, search)=1.0045\n", + "error: 0.79 -> 0.04; 90% width: 0.55 -> 0.15\n", + "all acceptance checks passed\n" + ] + } + ], + "source": [ + "for name, mer in [(\"default\", mer_default), (\"calibrated\", mer_calibrated)]:\n", + " post = mer.inference_data.posterior\n", + " with warnings.catch_warnings():\n", + " # constant/deterministic posterior variables have undefined R-hat;\n", + " # arviz emits RuntimeWarnings computing them - excluded below.\n", + " warnings.simplefilter(\"ignore\", RuntimeWarning)\n", + " rh_all = az.rhat(post)\n", + " finite_rhats = np.concatenate(\n", + " [np.asarray(rh_all[v]).ravel() for v in rh_all.data_vars]\n", + " )\n", + " max_rhat_model = float(finite_rhats[np.isfinite(finite_rhats)].max())\n", + " rhat_roi = float(az.rhat(post[\"roi_m\"]).sel(media_channel=\"search\")[\"roi_m\"])\n", + " print(f\"{name}: whole-model max Rhat={max_rhat_model:.4f}, \"\n", + " f\"Rhat(roi_m, search)={rhat_roi:.4f}\")\n", + " # Meridian's own health-check convention treats Rhat < 1.2 as converged;\n", + " # the reported ROI parameter is held to a tighter bar.\n", + " assert np.isfinite(max_rhat_model) and max_rhat_model < 1.2, (\n", + " f\"{name} fit: some model parameter has not converged \"\n", + " f\"(max Rhat {max_rhat_model:.4f})\"\n", + " )\n", + " assert np.isfinite(rhat_roi) and rhat_roi < 1.05, (\n", + " f\"{name} fit: roi_m chains disagree (Rhat {rhat_roi:.4f}) - sampling is \"\n", + " \"unhealthy here\"\n", + " )\n", + "\n", + "flat_d, flat_c = roi_default.ravel(), roi_calibrated.ravel()\n", + "err_default = abs(flat_d.mean() - BETA_SEARCH)\n", + "err_calibrated = abs(flat_c.mean() - BETA_SEARCH)\n", + "width_default = float(np.quantile(flat_d, 0.95) - np.quantile(flat_d, 0.05))\n", + "width_calibrated = float(np.quantile(flat_c, 0.95) - np.quantile(flat_c, 0.05))\n", + "print(f\"error: {err_default:.2f} -> {err_calibrated:.2f}; \"\n", + " f\"90% width: {width_default:.2f} -> {width_calibrated:.2f}\")\n", + "\n", + "assert err_calibrated < err_default, (\n", + " \"calibrated ROI mean is not closer to the simulated truth - the experiment \"\n", + " \"prior did not correct the bias in this environment\"\n", + ")\n", + "assert width_calibrated < 0.8 * width_default, (\n", + " \"calibrated 90% interval is not clearly narrower - the experiment prior did \"\n", + " \"not add information in this environment\"\n", + ")\n", + "print(\"all acceptance checks passed\")" + ] + }, + { + "cell_type": "markdown", + "id": "4aa3ca94", + "metadata": {}, + "source": [ + "## Where to go next\n", + "\n", + "- **PyMC-Marketing**: notebook 29 runs the same hand-off as a *lift test* for a\n", + " spend-boost experiment - the other classic geo design - including the grain\n", + " reconciliation a non-total export needs.\n", + "- **API reference**: the `diff_diff.mmm` module documents `to_meridian_roi_prior`\n", + " (spend-weighted pooling of multiple experiments, `se_widening`, `mroi_m`),\n", + " `meridian_calibration_mask`, and `MeridianROIPrior.to_code` - including why\n", + " channel order must match your `InputData` exactly, and Meridian's own guidance\n", + " on when to prefer `full_model_window=True` over a calibration mask.\n", + "- **Estimator choices**: the DiD side works with any diff-diff estimator that\n", + " supports `aggregate('total')` (CallawaySantAnna, EfficientDiD, ImputationDiD,\n", + " TwoStageDiD)." + ] + } + ], + "metadata": { + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/tutorials/README.md b/docs/tutorials/README.md index 4adbb65b..9ee2ebb1 100644 --- a/docs/tutorials/README.md +++ b/docs/tutorials/README.md @@ -159,6 +159,20 @@ A loyalty program looks dead on mean DiD ($0.22, p = 0.90) but in truth lifted t - Covariate-composition confounding fixed with `covariates=` (quantile-regression conditioning, qte `xformla` parity), and the `practitioner_next_steps()` close - Companion drift-test file (`tests/test_t27_cic_distributional_effects_drift.py`); fully self-contained (runs live, no committed data files) +### 29. MMM Calibration with PyMC-Marketing Lift Tests (`29_mmm_calibration_pymc.ipynb`) +Executed end-to-end against the real framework: a staggered geo spend-boost experiment measured with `CallawaySantAnna`, exported via `to_pymc_marketing_lift_test`, and fed to `MMM.add_lift_test_measurements` - the demand-chasing-biased MMM's search-ROI posterior moves to the simulated truth and tightens: +- Grain reconciliation for a national-weekly lift row (`aggregate('simple')` + `scale=G_treated`), the `aggregate('total')` campaign headline, and the double-count guardrail +- With/without-calibration money plot plus in-notebook sampler-health and posterior acceptance asserts +- Requires `pip install "pymc-marketing==1.0.0"` on Python 3.12 (this tutorial only); cannot share an environment with the Meridian tutorial (conflicting `arviz` pins); fully self-contained data +- Companion drift-test file (`tests/test_t29_mmm_calibration_pymc_drift.py`) + +### 30. MMM Calibration with Meridian ROI Priors (`30_mmm_calibration_meridian.ipynb`) +Executed end-to-end against the real framework: a geo-holdout launch experiment (the `roi_m` estimand exactly), exported via `to_meridian_roi_prior` from the `aggregate('total')` container, with `meridian_calibration_mask` and the generated `to_code()` snippet run verbatim: +- A national Meridian model that cannot see the geo holdout overcredits the launch; the experiment prior centers its ROI posterior on the truth and tightens it +- Window/spend/mass alignment asserts, default-vs-calibrated money plot, and sampler-health checks +- Requires `pip install "google-meridian==1.8.0"` on Python 3.12 (this tutorial only); cannot share an environment with the PyMC-Marketing tutorial (conflicting `arviz` pins); fully self-contained data +- Companion drift-test file (`tests/test_t30_mmm_calibration_meridian_drift.py`) + ## Running the Notebooks 1. Install diff-diff with dependencies: diff --git a/docs/tutorials/index.rst b/docs/tutorials/index.rst index f484960d..e91c47da 100644 --- a/docs/tutorials/index.rst +++ b/docs/tutorials/index.rst @@ -1,5 +1,5 @@ .. meta:: - :description: Hands-on diff-diff tutorials — 28 Jupyter notebooks covering basic 2x2 DiD, staggered adoption, synthetic DiD, power analysis, and business applications. + :description: Hands-on diff-diff tutorials — 31 Jupyter notebooks covering basic 2x2 DiD, staggered adoption, synthetic DiD, power analysis, and business applications. :keywords: DiD tutorial, difference-in-differences examples, causal inference notebooks Tutorials @@ -79,6 +79,20 @@ Practitioner walkthroughs built around marketing and policy scenarios. See which quantiles moved when the average hides the action, with Changes-in-Changes. + .. grid-item-card:: MMM Calibration: PyMC-Marketing + :link: 29_mmm_calibration_pymc + :link-type: doc + + Turn a geo spend-boost experiment into a PyMC-Marketing lift test and + watch the MMM's ROI posterior correct - executed end-to-end. + + .. grid-item-card:: MMM Calibration: Meridian + :link: 30_mmm_calibration_meridian + :link-type: doc + + Export a geo-holdout launch as Meridian's ROI prior and calibration + mask, then fit the real model - executed end-to-end. + .. toctree:: :maxdepth: 1 :caption: Business Applications @@ -93,6 +107,8 @@ Practitioner walkthroughs built around marketing and policy scenarios. Spillover DiD (TVA) <23_spillover_tva> Composition Drift & Calibration <26_composition_drift_calibration> Distributional Effects (CiC) <27_cic_distributional_effects> + MMM Calibration: PyMC-Marketing <29_mmm_calibration_pymc> + MMM Calibration: Meridian <30_mmm_calibration_meridian> Fundamentals ------------ diff --git a/tests/test_guides.py b/tests/test_guides.py index a2ba3694..ef6eb953 100644 --- a/tests/test_guides.py +++ b/tests/test_guides.py @@ -39,6 +39,25 @@ def test_content_stability_practitioner_workflow(): assert "8-step" in get_llm_guide("practitioner").lower() +def test_practitioner_step8_mmm_handoff_route_qualified(): + """The Step-8 MMM hand-off must stay ROUTE-QUALIFIED: both exporters named, + the total-container route AND the scaled simple/group route both present + inside the Step-8 slice - an unqualified total-first claim (or a total-route + deletion) must fail here, not ship silently.""" + guide = get_llm_guide("practitioner") + start = guide.index("## Step 8") + end = guide.index("## Common Pitfalls") + step8 = guide[start:end] + for needle in ( + "to_pymc_marketing_lift_test", + "to_meridian_roi_prior", + "aggregate('total')", + "scale=", + "SAME\n observation", + ): + assert needle in step8, f"Step-8 MMM hand-off lost {needle!r}" + + def test_content_stability_self_reference_after_rewrite(): assert "get_llm_guide" in get_llm_guide("concise") diff --git a/tests/test_mmm_interop_meridian.py b/tests/test_mmm_interop_meridian.py new file mode 100644 index 00000000..953106cb --- /dev/null +++ b/tests/test_mmm_interop_meridian.py @@ -0,0 +1,148 @@ +"""Meridian interop smoke tests (schema/defaults-drift canary). + +These tests exercise :func:`diff_diff.mmm.to_meridian_roi_prior`, +:meth:`MeridianROIPrior.to_code`, and :func:`diff_diff.mmm.meridian_calibration_mask` +against the REAL installed google-meridian API - no sampling anywhere. Skipped +unless google-meridian >= 1.8 is installed (NOT a diff-diff dependency); the +dedicated ``mmm-interop.yml`` CI job installs it and runs this file, with an +import-canary step so an install failure cannot be silently skipped into a green +run. +""" + +import numpy as np +import pandas as pd +import pytest + +pytest.importorskip( + "meridian", + minversion="1.8", + reason="google-meridian>=1.8 required (installed only in the mmm-interop CI job)", +) + +import tensorflow_probability as tfp # noqa: E402 +from meridian.data import data_frame_input_data_builder as m_dfb # noqa: E402 +from meridian.model import model as m_model # noqa: E402 +from meridian.model import prior_distribution, spec # noqa: E402 + +from diff_diff.mmm import ( # noqa: E402 + _MERIDIAN_PARAM_DEFAULTS, + meridian_calibration_mask, + to_meridian_roi_prior, +) + +TIMES = [str(d.date()) for d in pd.date_range("2024-01-01", periods=8, freq="W-MON")] +CHANNELS = ["search", "tv"] + + +def _prior(): + return to_meridian_roi_prior( + incremental_outcome=5000.0, incremental_outcome_se=800.0, spend=2000.0 + ) + + +def _mask(): + return meridian_calibration_mask( + media_times=TIMES, + media_channels=CHANNELS, + channel="search", + window=(TIMES[4], TIMES[-1]), + ) + + +class TestMeridianDefaultsPin: + def test_param_defaults_match_live_meridian(self): + # The multi-channel to_code() snippet keeps non-experiment channels on what + # it BELIEVES are Meridian's default LogNormal params - if Meridian changes + # its defaults, this canary must fail rather than silently rot the snippet. + # Meridian stores float32 tensors: compare after float32 rounding. + live = prior_distribution.PriorDistribution() + for param, (mu, sigma) in _MERIDIAN_PARAM_DEFAULTS.items(): + dist = getattr(live, param) + live_mu = np.float32(np.asarray(dist.parameters["loc"])) + live_sigma = np.float32(np.asarray(dist.parameters["scale"])) + assert (live_mu, live_sigma) == (np.float32(mu), np.float32(sigma)), ( + f"Meridian default drift for {param}: " + f"ours=({mu}, {sigma}) live=({live_mu}, {live_sigma})" + ) + + +class TestToCodeExecutes: + def test_multi_channel_snippet_execs_with_value_retention(self): + prior = _prior() + mask = _mask() + code = prior.to_code(channel="search", media_channels=CHANNELS, roi_calibration_period=mask) + ns: dict = {} + exec(code, ns) # noqa: S102 - executing our own generated snippet is the test + model_spec = ns["model_spec"] + assert isinstance(model_spec, spec.ModelSpec) + # Value retention: the exec'd prior's roi_m LogNormal carries our mu/sigma + # for the experiment channel and Meridian's defaults for the other channel. + dist = model_spec.prior.roi_m + loc = np.asarray(dist.parameters["loc"], dtype=float) + scale = np.asarray(dist.parameters["scale"], dtype=float) + default_mu, default_sigma = _MERIDIAN_PARAM_DEFAULTS["roi_m"] + np.testing.assert_allclose(loc, [prior.mu, default_mu], rtol=1e-6) + np.testing.assert_allclose(scale, [prior.sigma, default_sigma], rtol=1e-6) + # And the serialized mask round-trips exactly. + assert np.array_equal(np.asarray(model_spec.roi_calibration_period), mask) + + def test_single_channel_snippet_execs(self): + prior = _prior() + code = prior.to_code(single_channel=True, full_model_window=True) + ns: dict = {} + exec(code, ns) # noqa: S102 + model_spec = ns["model_spec"] + assert isinstance(model_spec, spec.ModelSpec) + dist = model_spec.prior.roi_m + assert float(np.asarray(dist.parameters["loc"])) == pytest.approx(prior.mu) + assert float(np.asarray(dist.parameters["scale"])) == pytest.approx(prior.sigma) + assert model_spec.roi_calibration_period is None + + +class TestModelAcceptsExports: + def test_mask_and_prior_accepted_by_meridian(self): + prior = _prior() + mask = _mask() + roi_prior = tfp.distributions.LogNormal( + [prior.mu, _MERIDIAN_PARAM_DEFAULTS["roi_m"][0]], + [prior.sigma, _MERIDIAN_PARAM_DEFAULTS["roi_m"][1]], + name="roi_m", + ) + model_spec = spec.ModelSpec( + prior=prior_distribution.PriorDistribution(roi_m=roi_prior), + media_prior_type="roi", + roi_calibration_period=mask, + ) + # Build InputData through the PUBLIC DataFrameInputDataBuilder (the + # route notebook 30 teaches) rather than Meridian's internal test-utils + # fixtures, so upstream fixture churn cannot break this canary. + rng = np.random.default_rng(1) + rows = [] + for geo in ["g0", "g1", "g2", "g3"]: + for t in TIMES: + rows.append( + { + "geo": geo, + "time": t, + "sales": 1000.0 + 50.0 * rng.random(), + "population": 1_000_000.0, + "search_spend": 100.0 + 10.0 * rng.random(), + "tv_spend": 200.0 + 10.0 * rng.random(), + } + ) + frame = pd.DataFrame(rows) + input_data = ( + m_dfb.DataFrameInputDataBuilder(kpi_type="revenue") + .with_kpi(frame, kpi_col="sales") + .with_population(frame) + .with_media( + frame, + media_cols=["search_spend", "tv_spend"], + media_spend_cols=["search_spend", "tv_spend"], + media_channels=CHANNELS, + ) + .build() + ) + mer = m_model.Meridian(input_data=input_data, model_spec=model_spec) + assert mer is not None + assert np.array_equal(np.asarray(mer.model_spec.roi_calibration_period), mask) diff --git a/tests/test_mmm_interop_pymc.py b/tests/test_mmm_interop_pymc.py new file mode 100644 index 00000000..812f9720 --- /dev/null +++ b/tests/test_mmm_interop_pymc.py @@ -0,0 +1,174 @@ +"""PyMC-Marketing interop smoke tests (schema-drift canary). + +These tests exercise :func:`diff_diff.mmm.to_pymc_marketing_lift_test` against the +REAL installed pymc-marketing API - no sampling anywhere. They are skipped unless +pymc-marketing >= 1.0 is installed (it is NOT a diff-diff dependency); the dedicated +``mmm-interop.yml`` CI job installs it and runs this file, with an import-canary +step so an install failure cannot be silently skipped into a green run. + +Value-retention oracle: pymc-marketing 1.0 stores an added lift test as a +``lift_measurements`` potential inside the PyMC model graph and retains no +DataFrame attribute, so retention is asserted FUNCTIONALLY - the measured values +enter the likelihood (perturbing ``delta_y``/``sigma`` changes the model's logp at +a fixed parameter point; an identical DataFrame reproduces it exactly). +""" + +import numpy as np +import pandas as pd +import pytest + +pytest.importorskip( + "pymc_marketing", + minversion="1.0", + reason="pymc-marketing>=1.0 required (installed only in the mmm-interop CI job)", +) + +from pymc_marketing.mmm import MMM, GeometricAdstock, LogisticSaturation # noqa: E402 + +from diff_diff.mmm import to_pymc_marketing_lift_test # noqa: E402 + +LIFT_COLUMNS = ["channel", "x", "delta_x", "delta_y", "sigma"] + + +def _national_frame(n: int = 40) -> tuple[pd.DataFrame, pd.Series]: + rng = np.random.default_rng(7) + dates = pd.date_range("2024-01-01", periods=n, freq="W-MON") + search = (1500 + 450 * (np.arange(n) > n // 2)).astype(float) + tv = 3000 + 100 * rng.random(n) + y = pd.Series(45000 + 2.0 * search + 0.5 * tv + rng.normal(0, 200, n), name="y") + return pd.DataFrame({"date": dates, "search": search, "tv": tv}), y + + +def _build_mmm(X: pd.DataFrame, y: pd.Series, **kwargs) -> MMM: + mmm = MMM( + date_column="date", + channel_columns=["search", "tv"], + adstock=GeometricAdstock(l_max=2), + saturation=LogisticSaturation(), + **kwargs, + ) + mmm.build_model(X, y) + return mmm + + +def _model_logp(mmm: MMM) -> float: + model = mmm.model + return float(model.compile_logp()(model.initial_point())) + + +class TestLiftTestSchema: + def test_exporter_frame_accepted(self): + X, y = _national_frame() + mmm = _build_mmm(X, y) + df = to_pymc_marketing_lift_test( + channel="search", x=1500.0, delta_x=450.0, delta_y=930.0, sigma=110.0 + ) + assert list(df.columns) == LIFT_COLUMNS + mmm.add_lift_test_measurements(df) + assert "lift_measurements" in mmm.model.named_vars + + def test_wrong_schema_rejected(self): + # Regression pin: the API validates the exporter's column names - a renamed + # column must FAIL, otherwise the schema contract is no longer exercised. + X, y = _national_frame() + mmm = _build_mmm(X, y) + df = to_pymc_marketing_lift_test( + channel="search", x=1500.0, delta_x=450.0, delta_y=930.0, sigma=110.0 + ).rename(columns={"delta_y": "lift"}) + with pytest.raises(KeyError, match="delta_y"): + mmm.add_lift_test_measurements(df) + + def test_multiple_rows_accepted(self): + X, y = _national_frame() + mmm = _build_mmm(X, y) + df = to_pymc_marketing_lift_test( + channel=["search", "tv"], + x=[1500.0, 3000.0], + delta_x=[450.0, 300.0], + delta_y=[930.0, 240.0], + sigma=[110.0, 60.0], + ) + mmm.add_lift_test_measurements(df) + assert "lift_measurements" in mmm.model.named_vars + + +class TestLiftTestDims: + def test_dims_row_round_trips(self): + # dims= is a first-class exporter parameter emitting real DataFrame + # columns; verify a dims-carrying row against the real multidimensional API. + rng = np.random.default_rng(11) + n = 30 + dates = pd.date_range("2024-01-01", periods=n, freq="W-MON") + frames = [] + for geo in ["east", "west"]: + search = (700 + 200 * (np.arange(n) > n // 2)).astype(float) + tv = 1500 + 50 * rng.random(n) + frames.append( + pd.DataFrame( + { + "date": dates, + "geo": geo, + "search": search, + "tv": tv, + "y": 20000 + 2.0 * search + 0.5 * tv + rng.normal(0, 100, n), + } + ) + ) + panel = pd.concat(frames, ignore_index=True) + X = panel[["date", "geo", "search", "tv"]] + y = panel["y"] + mmm = MMM( + date_column="date", + channel_columns=["search", "tv"], + dims=("geo",), + adstock=GeometricAdstock(l_max=2), + saturation=LogisticSaturation(), + ) + mmm.build_model(X, y) + df = to_pymc_marketing_lift_test( + channel="search", + x=700.0, + delta_x=200.0, + delta_y=410.0, + sigma=55.0, + dims={"geo": "east"}, + ) + assert list(df.columns) == ["channel", "geo", "x", "delta_x", "delta_y", "sigma"] + mmm.add_lift_test_measurements(df) + assert "lift_measurements" in mmm.model.named_vars + + +class TestLiftTestValueRetention: + """Functional retention: the exported values actually enter the likelihood.""" + + def test_identical_frames_identical_logp(self): + X, y = _national_frame() + df = to_pymc_marketing_lift_test( + channel="search", x=1500.0, delta_x=450.0, delta_y=930.0, sigma=110.0 + ) + logps = [] + for _ in range(2): + mmm = _build_mmm(X, y) + mmm.add_lift_test_measurements(df.copy()) + logps.append(_model_logp(mmm)) + assert logps[0] == pytest.approx(logps[1], rel=0, abs=0) + + @pytest.mark.parametrize( + "column,changed", + [("x", 3000.0), ("delta_x", 900.0), ("delta_y", 1860.0), ("sigma", 220.0)], + ) + def test_perturbed_values_change_logp(self, column, changed): + X, y = _national_frame() + base = to_pymc_marketing_lift_test( + channel="search", x=1500.0, delta_x=450.0, delta_y=930.0, sigma=110.0 + ) + perturbed = base.copy() + perturbed[column] = changed + mmm_a = _build_mmm(X, y) + mmm_a.add_lift_test_measurements(base) + mmm_b = _build_mmm(X, y) + mmm_b.add_lift_test_measurements(perturbed) + assert _model_logp(mmm_a) != _model_logp(mmm_b), ( + f"changing {column} did not change the model logp - the lift value is " + "being ignored or remapped by the framework" + ) diff --git a/tests/test_openai_review.py b/tests/test_openai_review.py index 468f5f2f..afac8902 100644 --- a/tests/test_openai_review.py +++ b/tests/test_openai_review.py @@ -2203,6 +2203,7 @@ def test_rust_test_yml_pr_filter_covers_ai_review_surfaces(self, workflow_paths) ".github/workflows/notebooks.yml", ".github/workflows/ci-gate.yml", ".github/workflows/release-build-check.yml", + ".github/workflows/mmm-interop.yml", ) def test_rust_test_yml_push_filter_covers_guarded_workflows(self, workflow_paths): @@ -2269,6 +2270,7 @@ class TestCiWorkflowLabelEventGuard: EXPECTED_JOBS = { "rust-test.yml": ("rust-tests", "python-tests", "python-fallback"), "notebooks.yml": ("execute-notebooks", "interop-notebooks"), + "mmm-interop.yml": ("mmm-pymc-interop", "mmm-meridian-interop"), "docs-tests.yml": ("doc-snippets", "sphinx-build", "docs-deps-py39-smoke"), # release-build-check.yml is a single reusable-workflow caller job gated on # ready-for-ci (it build-tests the PyPI release path on PRs); lock its guard too. diff --git a/tests/test_t29_mmm_calibration_pymc_drift.py b/tests/test_t29_mmm_calibration_pymc_drift.py new file mode 100644 index 00000000..73daa008 --- /dev/null +++ b/tests/test_t29_mmm_calibration_pymc_drift.py @@ -0,0 +1,250 @@ +"""Drift guards for docs/tutorials/29_mmm_calibration_pymc.ipynb. + +The notebook is CI-excluded from execution (it needs pymc-marketing, which +conflicts with the Meridian tutorial's environment), so these tests protect its +committed surface with diff-diff-only recomputation: + +- DGP constant + formula-line sync (the t26 pattern): the notebook's DGP must + equal the copy recomputed here, or stale committed outputs would stay green; +- sampler-configuration needles: sampling cannot be weakened or de-seeded + silently; +- integration-content presence: the framework fits and acceptance asserts cannot + be deleted while CI stays green; +- rendered-number cross-check: the DiD-side numbers the notebook prints are + recomputed with diff-diff alone and matched against the committed outputs + (loose bands per the t27 normal-suite convention - never bit-exact floats). +""" + +import re + +import numpy as np +import pandas as pd +import pytest + +from diff_diff import CallawaySantAnna +from diff_diff.mmm import to_pymc_marketing_lift_test +from tests._tutorial_drift import ( + _read_notebook, + assert_quotes_in_rendered, + notebook_output_text, +) + +NB = "docs/tutorials/29_mmm_calibration_pymc.ipynb" + +# ---- locked DGP constants (must equal the notebook's copy verbatim) ---- +SEED = 2026 +N_GEOS, N_WEEKS = 15, 104 +BETA_SEARCH, SEARCH_BASE, BOOST = 2.0, 100.0, 50.0 +FREEZE_WEEK = 52 +TREAT_COHORTS = {60: [0, 1, 2], 68: [3, 4, 5], 76: [6, 7, 8]} +DEMAND_AMP, CHASE = 300.0, 0.5 +TV_BASE, TV_BETA, TV_ADSTOCK, TV_SAT = 200.0, 600.0, 0.5, 0.004 +BASE_SALES, GEO_SD, NOISE_SD = 3000.0, 100.0, 15.0 + +DGP_CONSTANT_NEEDLES = [ + "SEED = 2026", + "N_GEOS, N_WEEKS = 15, 104", + "BETA_SEARCH = 2.0", + "SEARCH_BASE = 100.0", + "BOOST = 50.0", + "FREEZE_WEEK = 52", + "TREAT_COHORTS = {60: [0, 1, 2], 68: [3, 4, 5], 76: [6, 7, 8]}", + "DEMAND_AMP = 300.0", + "CHASE = 0.5", + "TV_BASE, TV_BETA, TV_ADSTOCK, TV_SAT = 200.0, 600.0, 0.5, 0.004", + "BASE_SALES, GEO_SD, NOISE_SD = 3000.0, 100.0, 15.0", +] + +# Mechanism lines: linear zero-carryover experiment channel, tv adstock, the +# demand-chasing history, and the frozen baseline + constant boost. +DGP_FORMULA_NEEDLES = [ + "+ BETA_SEARCH * search", + "carry = tv[t] + TV_ADSTOCK * carry", + "search = SEARCH_BASE * (1 + CHASE * season[t])", + "search = SEARCH_BASE + (BOOST if boosted else 0.0)", + "DEMAND_AMP * season[t]", +] + +SAMPLER_NEEDLES = [ + "chains=4, draws=500, tune=3000", + "target_accept=0.999", + 'nuts_sampler_kwargs={"max_treedepth": 14}', + "random_seed=SEED", +] + +INTEGRATION_NEEDLES = [ + "add_lift_test_measurements(", + "mmm_plain = fit_mmm()", + "mmm_cal = fit_mmm(df_lift)", + "aggregation_result=simple,\n scale=G_TREATED,", + "assert div == 0", + "assert mtd == 0", + "assert max_rhat_model < 1.01", + "assert rhat_roi < 1.01", + "assert ess_bulk >= ess_floors[name]", + "assert ess_tail >= 300", + 'ess_floors = {"plain": 300, "calibrated": 800}', + "assert err_cal < err_plain", + "assert width_cal < 0.8 * width_plain", +] + + +def _notebook_source() -> str: + nb = _read_notebook(NB) + return "\n".join( + "".join(c["source"]) if isinstance(c["source"], list) else c["source"] + for c in nb["cells"] + if c["cell_type"] == "code" + ) + + +def _build_panel() -> pd.DataFrame: + rng = np.random.default_rng(SEED) + geo_start = {g: w for w, gg in TREAT_COHORTS.items() for g in gg} + season = np.sin(2 * np.pi * np.arange(N_WEEKS) / 52.0) + rows = [] + for g in range(N_GEOS): + tv = TV_BASE * (1 + 0.1 * rng.random(N_WEEKS)) + ad = np.zeros(N_WEEKS) + carry = 0.0 + for t in range(N_WEEKS): + carry = tv[t] + TV_ADSTOCK * carry + ad[t] = carry + tvc = TV_BETA * (1 - np.exp(-TV_SAT * ad)) + geo_level = rng.normal(0, GEO_SD) + for t in range(N_WEEKS): + if t < FREEZE_WEEK: + search = SEARCH_BASE * (1 + CHASE * season[t]) + else: + boosted = g in geo_start and t >= geo_start[g] + search = SEARCH_BASE + (BOOST if boosted else 0.0) + sales = ( + BASE_SALES + + geo_level + + DEMAND_AMP * season[t] + + BETA_SEARCH * search + + tvc[t] + + rng.normal(0, NOISE_SD) + ) + rows.append((g, t, sales, search, tv[t], geo_start.get(g, 0))) + return pd.DataFrame( + rows, columns=["geo", "week", "sales", "search_spend", "tv_spend", "first_treat"] + ) + + +def _parse_number(output: str, label_regex: str) -> float: + match = re.search(label_regex, output) + assert match, f"notebook output lost the line matching {label_regex!r}" + return float(match.group(1).replace(",", "")) + + +class TestNotebookSourceSync: + def test_dgp_constants_match(self): + src = _notebook_source() + missing = [n for n in DGP_CONSTANT_NEEDLES if n not in src] + assert not missing, f"notebook DGP constants drifted from the test copy: {missing}" + + def test_dgp_formula_lines_present(self): + src = _notebook_source() + missing = [n for n in DGP_FORMULA_NEEDLES if n not in src] + assert not missing, f"notebook DGP mechanism lines drifted: {missing}" + + def test_sampler_config_pinned(self): + src = _notebook_source() + missing = [n for n in SAMPLER_NEEDLES if n not in src] + assert not missing, f"notebook sampler configuration drifted: {missing}" + + def test_integration_cells_present(self): + src = _notebook_source() + missing = [n for n in INTEGRATION_NEEDLES if n not in src] + assert not missing, f"notebook integration/acceptance cells missing: {missing}" + + def test_no_kernelspec_metadata(self): + nb = _read_notebook(NB) + assert "kernelspec" not in nb["metadata"], ( + "committed notebooks stay kernelspec-free; local execution names the " + "venv kernel explicitly" + ) + + +class TestRenderedOutputs: + def test_acceptance_checks_passed_in_committed_run(self): + assert_quotes_in_rendered( + NB, ["all acceptance checks passed", "both fits done"], surface="output" + ) + + def test_money_plot_output_present(self): + nb = _read_notebook(NB) + has_image = any( + "image/png" in (out.get("data") or {}) + for c in nb["cells"] + if c["cell_type"] == "code" + for out in c.get("outputs", []) + ) + assert has_image, "the committed money-plot image output is missing" + + +@pytest.fixture(scope="module") +def fitted(): + panel = _build_panel() + res = CallawaySantAnna().fit( + panel, outcome="sales", unit="geo", time="week", first_treat="first_treat" + ) + return panel, res + + +class TestDidNumbersRecompute: + """Recompute the DiD/export side with diff-diff only; match printed numbers.""" + + def test_total_and_att_match_notebook(self, fitted): + panel, res = fitted + total = res.aggregate("total") + out = notebook_output_text(NB) + printed_att = _parse_number(out, r"per-geo-week ATT: ([\d.,-]+)") + printed_total = _parse_number(out, r"CS total incremental sales:\s+([\d.,-]+)") + np.testing.assert_allclose(printed_att, res.overall_att, rtol=2e-3) + np.testing.assert_allclose(printed_total, total.att[0], rtol=2e-3) + mass = int(((panel.first_treat > 0) & (panel.week >= panel.first_treat)).sum()) + assert total.n[0] == mass + + def test_lift_row_matches_notebook(self, fitted): + # Every lift-row FIELD is pinned individually (not just the ROI ratio, + # where the treated-geo multiplier would cancel): x/delta_x against the + # frozen-baseline national-weekly grain, delta_y/sigma against the + # aggregate('simple') container scaled by the treated-geo count. + panel, res = fitted + simple = res.aggregate("simple") + g_treated = int((panel.first_treat > 0).sum() / N_WEEKS) + assert g_treated == 9 + df = to_pymc_marketing_lift_test( + channel="search", + x=SEARCH_BASE * N_GEOS, + delta_x=BOOST * g_treated, + aggregation_result=simple, + scale=g_treated, + ) + assert float(df.x[0]) == SEARCH_BASE * N_GEOS == 1500.0 + assert float(df.delta_x[0]) == BOOST * g_treated == 450.0 + np.testing.assert_allclose(float(df.delta_y[0]), res.overall_att * g_treated, rtol=1e-12) + np.testing.assert_allclose(float(df.sigma[0]), res.overall_se * g_treated, rtol=1e-12) + out = notebook_output_text(NB) + printed_roi = _parse_number(out, r"DiD measurement:\s+([\d.,-]+)") + np.testing.assert_allclose(printed_roi, df.delta_y[0] / df.delta_x[0], atol=5e-3) + + def test_notebook_states_linearity_scope_of_lift_row(self): + # The staggered-average -> single-transition construction is exact only + # under this simulation's linear homogeneous response; the notebook must + # keep saying so (round-4 review: an unscoped "generalizes" claim would + # transport a mixed-regime average onto one spend step of a saturating + # curve). + from tests._tutorial_drift import notebook_markdown + + md = notebook_markdown(NB) + assert "linear and homogeneous" in md + assert "boost all test geos simultaneously" in md + # The nonlinear-case guidance must keep the estimator fit on the FULL + # panel (a post-only refit strips every cohort's pre-treatment baseline + # and leaves Callaway-Sant'Anna unidentified - round-5 review) and + # restrict only the post-fit aggregation. + assert "keep the FULL panel in the estimator fit" in md + assert "re-fit the DiD restricted" not in md diff --git a/tests/test_t30_mmm_calibration_meridian_drift.py b/tests/test_t30_mmm_calibration_meridian_drift.py new file mode 100644 index 00000000..304d22b7 --- /dev/null +++ b/tests/test_t30_mmm_calibration_meridian_drift.py @@ -0,0 +1,215 @@ +"""Drift guards for docs/tutorials/30_mmm_calibration_meridian.ipynb. + +Same protection pattern as the t29 twin (see its module docstring): the notebook +is CI-excluded from execution (google-meridian conflicts with the PyMC tutorial's +environment), so its committed surface is guarded here with diff-diff-only +recomputation - DGP constant/formula sync, sampler-config needles, +integration-content presence, and rendered-number cross-checks with loose bands. +""" + +import re + +import numpy as np +import pandas as pd +import pytest + +from diff_diff import CallawaySantAnna +from diff_diff.mmm import meridian_calibration_mask, to_meridian_roi_prior +from tests._tutorial_drift import ( + _read_notebook, + assert_quotes_in_rendered, + notebook_output_text, +) + +NB = "docs/tutorials/30_mmm_calibration_meridian.ipynb" + +# ---- locked DGP constants (must equal the notebook's copy verbatim) ---- +SEED = 30301 +N_GEOS, N_WEEKS = 12, 104 +BETA_SEARCH, LAUNCH_SPEND = 2.5, 150.0 +LAUNCH_COHORTS = {30: [0, 1, 2], 44: [3, 4, 5], 58: [6, 7]} +DEMAND_AMP = 80.0 +TV_BASE, TV_BETA, TV_ADSTOCK, TV_SAT = 250.0, 800.0, 0.4, 0.003 +BASE_SALES, GEO_SD, NOISE_SD, SHOCK_SD = 5000.0, 150.0, 20.0, 60.0 + +DGP_CONSTANT_NEEDLES = [ + "SEED = 30301", + "N_GEOS, N_WEEKS = 12, 104", + "BETA_SEARCH = 2.5", + "LAUNCH_SPEND = 150.0", + "LAUNCH_COHORTS = {30: [0, 1, 2], 44: [3, 4, 5], 58: [6, 7]}", + "DEMAND_AMP = 80.0", + "TV_BASE, TV_BETA, TV_ADSTOCK, TV_SAT = 250.0, 800.0, 0.4, 0.003", + "BASE_SALES, GEO_SD, NOISE_SD, SHOCK_SD = 5000.0, 150.0, 20.0, 60.0", +] + +# Mechanism lines: linear zero-carryover experiment channel, launched-geos-only +# spend, the demand ramp, and the common national shocks. +DGP_FORMULA_NEEDLES = [ + "+ BETA_SEARCH * search", + "carry = tv[t] + TV_ADSTOCK * carry", + "search = LAUNCH_SPEND if launched else 0.0", + "DEMAND_AMP * ramp[t] + shock[t]", + "shock = rng.normal(0, SHOCK_SD, N_WEEKS)", +] + +SAMPLER_NEEDLES = [ + "n_chains=4, n_adapt=500, n_burnin=500, n_keep=500, seed=SEED", +] + +INTEGRATION_NEEDLES = [ + "exec(code, namespace)", + "mer_default = fit_meridian(m_spec.ModelSpec())", + "mer_calibrated = fit_meridian(model_spec_calibrated)", + "to_meridian_roi_prior(aggregation_result=total, spend=total_spend)", + "assert np.isfinite(max_rhat_model) and max_rhat_model < 1.2", + "assert np.isfinite(rhat_roi) and rhat_roi < 1.05", + "assert err_calibrated < err_default", + "assert width_calibrated < 0.8 * width_default", + "assert total.n[0] == spend_positive", +] + + +def _notebook_source() -> str: + nb = _read_notebook(NB) + return "\n".join( + "".join(c["source"]) if isinstance(c["source"], list) else c["source"] + for c in nb["cells"] + if c["cell_type"] == "code" + ) + + +def _build_panel() -> pd.DataFrame: + rng = np.random.default_rng(SEED) + geo_start = {g: w for w, gg in LAUNCH_COHORTS.items() for g in gg} + ramp = np.clip((np.arange(N_WEEKS) - 26) / 52.0, 0, None) + shock = rng.normal(0, SHOCK_SD, N_WEEKS) + rows = [] + for g in range(N_GEOS): + tv = TV_BASE * (1 + 0.1 * rng.random(N_WEEKS)) + ad = np.zeros(N_WEEKS) + carry = 0.0 + for t in range(N_WEEKS): + carry = tv[t] + TV_ADSTOCK * carry + ad[t] = carry + tvc = TV_BETA * (1 - np.exp(-TV_SAT * ad)) + geo_level = rng.normal(0, GEO_SD) + for t in range(N_WEEKS): + launched = g in geo_start and t >= geo_start[g] + search = LAUNCH_SPEND if launched else 0.0 + sales = ( + BASE_SALES + + geo_level + + DEMAND_AMP * ramp[t] + + shock[t] + + BETA_SEARCH * search + + tvc[t] + + rng.normal(0, NOISE_SD) + ) + rows.append((g, t, sales, search, tv[t], geo_start.get(g, 0))) + return pd.DataFrame( + rows, columns=["geo", "week", "sales", "search_spend", "tv_spend", "first_treat"] + ) + + +def _parse_number(output: str, label_regex: str) -> float: + match = re.search(label_regex, output) + assert match, f"notebook output lost the line matching {label_regex!r}" + return float(match.group(1).replace(",", "")) + + +class TestNotebookSourceSync: + def test_dgp_constants_match(self): + src = _notebook_source() + missing = [n for n in DGP_CONSTANT_NEEDLES if n not in src] + assert not missing, f"notebook DGP constants drifted from the test copy: {missing}" + + def test_dgp_formula_lines_present(self): + src = _notebook_source() + missing = [n for n in DGP_FORMULA_NEEDLES if n not in src] + assert not missing, f"notebook DGP mechanism lines drifted: {missing}" + + def test_sampler_config_pinned(self): + src = _notebook_source() + missing = [n for n in SAMPLER_NEEDLES if n not in src] + assert not missing, f"notebook sampler configuration drifted: {missing}" + + def test_integration_cells_present(self): + src = _notebook_source() + missing = [n for n in INTEGRATION_NEEDLES if n not in src] + assert not missing, f"notebook integration/acceptance cells missing: {missing}" + + def test_no_kernelspec_metadata(self): + nb = _read_notebook(NB) + assert "kernelspec" not in nb["metadata"], ( + "committed notebooks stay kernelspec-free; local execution names the " + "venv kernel explicitly" + ) + + +class TestRenderedOutputs: + def test_acceptance_checks_passed_in_committed_run(self): + assert_quotes_in_rendered( + NB, ["all acceptance checks passed", "both fits done"], surface="output" + ) + + def test_money_plot_output_present(self): + nb = _read_notebook(NB) + has_image = any( + "image/png" in (out.get("data") or {}) + for c in nb["cells"] + if c["cell_type"] == "code" + for out in c.get("outputs", []) + ) + assert has_image, "the committed money-plot image output is missing" + + def test_generated_snippet_shown(self): + assert_quotes_in_rendered( + NB, + ["Generated by diff_diff.mmm.to_meridian_roi_prior"], + surface="output", + ) + + +@pytest.fixture(scope="module") +def fitted(): + panel = _build_panel() + res = CallawaySantAnna().fit( + panel, outcome="sales", unit="geo", time="week", first_treat="first_treat" + ) + return panel, res + + +class TestDidNumbersRecompute: + """Recompute the DiD/export side with diff-diff only; match printed numbers.""" + + def test_total_and_prior_match_notebook(self, fitted): + panel, res = fitted + total = res.aggregate("total") + total_spend = float(panel.search_spend.sum()) + prior = to_meridian_roi_prior(aggregation_result=total, spend=total_spend) + out = notebook_output_text(NB) + printed_total = _parse_number(out, r"CS total incremental sales:\s+([\d.,-]+)") + printed_roi = _parse_number(out, r"experiment ROI: ([\d.,-]+)") + np.testing.assert_allclose(printed_total, total.att[0], rtol=2e-3) + np.testing.assert_allclose(printed_roi, prior.roi_mean, atol=5e-3) + # launch-design alignment: the total's mass is the spend-positive count + assert total.n[0] == int((panel.search_spend > 0).sum()) + + def test_mask_window_matches_notebook(self, fitted): + panel, _ = fitted + geo_start = {g: w for w, gg in LAUNCH_COHORTS.items() for g in gg} + first_launch = min(geo_start.values()) + times = [str(d.date()) for d in pd.date_range("2024-01-01", periods=N_WEEKS, freq="W-MON")] + mask = meridian_calibration_mask( + media_times=times, + media_channels=["search", "tv"], + channel="search", + window=(times[first_launch], times[-1]), + ) + out = notebook_output_text(NB) + printed_true = _parse_number(out, r"search weeks in window: (\d+)") + assert int(printed_true) == int(mask[:, 0].sum()) + # every spend-positive week is inside the window (the time-axis + # containment the TIME x CHANNEL mask expresses) + assert panel.loc[panel.search_spend > 0, "week"].min() >= first_launch