Skip to content

Add PTB-XL dataset support (ECG) - #1187

Open
AxelNoun wants to merge 12 commits into
sunlabuiuc:masterfrom
AxelNoun:feat/ptbxl-dataset
Open

Add PTB-XL dataset support (ECG)#1187
AxelNoun wants to merge 12 commits into
sunlabuiuc:masterfrom
AxelNoun:feat/ptbxl-dataset

Conversation

@AxelNoun

@AxelNoun AxelNoun commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Contributor: AxelNoun (GitHub: @AxelNoun) — external contributor, no NetID
Contribution Type: New Dataset + Task
Description: Adds PTB-XL (12-lead ECG, PhysioNet v1.0.3, open access) as a
PyHealth dataset, with a 5-superclass multi-label diagnosis task and a
split helper using the official stratified folds. Motivated by the CPBench
effort: it gives conformal prediction a multi-label ECG benchmark that needs
no credentialing, with metadata suited to covariate/label-shift splits.
Based on Wagner et al., Scientific Data 2020
(https://www.nature.com/articles/s41597-020-0495-6).

Output label order: MultiLabelProcessor.fit sorts observed labels
alphabetically. On the full v1.0.3 corpus the multi-hot vector is
CD, HYP, MI, NORM, STTC. The task does not pin a custom vocabulary.

Files to Review:

  • pyhealth/datasets/ptbxl.py — dataset implementation
  • pyhealth/datasets/configs/ptbxl.yaml — table config (v1.0.3 template)
  • pyhealth/tasks/ptbxl.py — 5-superclass multi-label task
  • pyhealth/datasets/splitter.pysplit_by_strat_fold helper
  • tests/core/test_ptbxl.py — tests with synthetic fixtures
  • docs/api/datasets/pyhealth.datasets.PTBXLDataset.rst, docs/api/tasks/pyhealth.tasks.ptbxl.rst
  • examples/ecg/ptbxl/ptbxl_superclass_quickstart.py

Implementation notes:

  • wfdb is an optional extra (pip install 'pyhealth[ptbxl]'), lazily imported.
  • PTBXLDataset.root is the user-provided data path. Derived metadata CSVs
    live under metadata_cache_dir (not root) so read-only mounts stay
    untouched. BaseDataset.load_table concatenates root / file_path and
    cannot take an absolute path, so load_table is overridden to scan the
    cache directory. Cache identity still hashes data-root path + source CSV
    bytes into dataset_name (EEGBCI pattern) so two roots or an in-place
    ptbxl_database.csv replace cannot share global_event_df.
  • Resolved cache YAML: BaseDataset only accepts config_path, so PTB-XL
    writes a rate/root/source-keyed YAML next to the derived CSV before
    super().__init__(). CSV and YAML writes use tmp+os.replace under
    filelock.FileLock.
  • Unrelated pre-existing # noqa: E402 on the EEGBCI import line in
    pyhealth/datasets/__init__.py — left untouched (out of diff).
  • Signals returned as (n_leads, n_samples), matching EEGBCI's convention.
  • Sample age is always an int: missing → -1 (age_is_missing=True);
    HIPAA-censored raw 300 is clipped to 90 (age_is_censored=True). The
    raw 300 remains in the derived metadata CSV.
  • Empty superclass sets (≈400 records on v1.0.1, Wagner et al. Table 9;
    mainly pacemaker ECGs) are dropped by default via drop_empty_labels.
  • split_by_strat_fold(..., folds=) avoids decoding 12-lead signals just to
    read an integer. check_patient_disjoint=True raises if a patient_id
    appears in more than one split (official PTB-XL folds are patient-disjoint;
    the synthetic fixture leaks patient 15709 across folds 1 and 9 to exercise
    this).
  • Fixture under test-resources/core/ptbxl/ is synthetic CSVs; WFDB
    waveforms are generated at test time (not committed).

Testing / CI:

  • Local: python -m unittest discover -t tests -s tests/core -p 'test_ptbxl.py' -v28 passed (with wfdb installed).
  • Without wfdb / pyhealth[ptbxl], waveform I/O and set_task e2e tests skip.
  • pixi test feature uses pyhealth = { extras = ["ptbxl"] } so make testall under pixi covers waveform I/O.

Follow-up: 71-SCP-code classification and age regression are structured
for but not implemented here — planned as a separate PR.

Test plan

  • python -m unittest discover -t tests -s tests/core -p 'test_ptbxl.py' -v — 28 passed (with wfdb)
  • python tools/check_pr_rules.py --base upstream/master --head HEAD
  • CI make testall / contribution-rules on the PR

Recreates #1186, which was closed automatically on 12 Aug when the fork
hosting its branch was detached during an account cleanup on my side —
my mistake, apologies for the noise.

All prior context and discussion: #1186

AxelNoun and others added 12 commits August 19, 2026 00:13
Introduce PTBXLDataset (BaseDataset + YAML) with optional wfdb extra,
root-keyed metadata cache, and a resolved cache YAML so file_path is
correct before BaseDataset init.

Co-authored-by: Cursor <cursoragent@cursor.com>
Add PTBXLSuperclassClassification and split_by_strat_fold (folds 1-8 /
9 / 10) in separate modules so they can move to benchmarks later.

Co-authored-by: Cursor <cursoragent@cursor.com>
Cover helpers, metadata cache keyed by root, event reads via
PTBXLDataset, waveform IO (wfdb), and the 5-superclass task path.

Co-authored-by: Cursor <cursoragent@cursor.com>
Document the dataset/task Overview RST pages and a minimal ECG
superclass example for CPBench users.

Co-authored-by: Cursor <cursoragent@cursor.com>
…cally

BaseDataset keys global_event_df on root+dataset_name; hash data-root path
and CSV bytes into dataset_name (EEGBCI pattern) and write CSV/YAML via
tmp+os.replace under FileLock so DDP cannot read a truncated file.

Co-authored-by: Cursor <cursoragent@cursor.com>
Move the SCP class map to functools.cache so vars(task) is unchanged
after __call__. Emit age=-1 when missing and clip HIPAA-censored 300
to 90 so litdata can serialize the field.

Co-authored-by: Cursor <cursoragent@cursor.com>
Accept a precomputed folds sequence and a generic fold_field so callers
need not materialize 12-lead signals just to read an integer. Error
messages no longer mention PTB-XL.

Co-authored-by: Cursor <cursoragent@cursor.com>
Exercise the real litdata path without mocking wfdb, assert
MultiLabelProcessor vocab order, and pin cache-dir uniqueness plus
vars(task) stability.

Co-authored-by: Cursor <cursoragent@cursor.com>
v1.0.3 is 21,799 records / 18,869 patients. Document that
MultiLabelProcessor emits CD, HYP, MI, NORM, STTC. Harmonize the
datasets toctree and reuse the ptbxl extra in the pixi test env.

Co-authored-by: Cursor <cursoragent@cursor.com>
The redundant-looking 'as' is required by ruff's unused-import rule for
module re-exports, same as EEGBCIDataset / PTBXLDataset in this file.

Co-authored-by: Cursor <cursoragent@cursor.com>
BaseDataset.load_table concatenates root/file_path and rejects absolute
paths, so override load_table to read the derived CSV from
metadata_cache_dir while leaving self.root equal to the constructor path.

Co-authored-by: Cursor <cursoragent@cursor.com>
Official PTB-XL folds are patient-disjoint; this flag verifies that
property via patient_to_index. Default False keeps current callers
working. The synthetic fixture leaks patient 15709 across folds 1 and 9
so the check is actually exercised.

Co-authored-by: Cursor <cursoragent@cursor.com>
@AxelNoun
AxelNoun force-pushed the feat/ptbxl-dataset branch from 47aa3a4 to fe1007f Compare August 18, 2026 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant