Add PTB-XL dataset support (ECG) - #1187
Open
AxelNoun wants to merge 12 commits into
Open
Conversation
Introduce PTBXLDataset (BaseDataset + YAML) with optional wfdb extra, root-keyed metadata cache, and a resolved cache YAML so file_path is correct before BaseDataset init. Co-authored-by: Cursor <cursoragent@cursor.com>
Add PTBXLSuperclassClassification and split_by_strat_fold (folds 1-8 / 9 / 10) in separate modules so they can move to benchmarks later. Co-authored-by: Cursor <cursoragent@cursor.com>
Cover helpers, metadata cache keyed by root, event reads via PTBXLDataset, waveform IO (wfdb), and the 5-superclass task path. Co-authored-by: Cursor <cursoragent@cursor.com>
Document the dataset/task Overview RST pages and a minimal ECG superclass example for CPBench users. Co-authored-by: Cursor <cursoragent@cursor.com>
…cally BaseDataset keys global_event_df on root+dataset_name; hash data-root path and CSV bytes into dataset_name (EEGBCI pattern) and write CSV/YAML via tmp+os.replace under FileLock so DDP cannot read a truncated file. Co-authored-by: Cursor <cursoragent@cursor.com>
Move the SCP class map to functools.cache so vars(task) is unchanged after __call__. Emit age=-1 when missing and clip HIPAA-censored 300 to 90 so litdata can serialize the field. Co-authored-by: Cursor <cursoragent@cursor.com>
Accept a precomputed folds sequence and a generic fold_field so callers need not materialize 12-lead signals just to read an integer. Error messages no longer mention PTB-XL. Co-authored-by: Cursor <cursoragent@cursor.com>
Exercise the real litdata path without mocking wfdb, assert MultiLabelProcessor vocab order, and pin cache-dir uniqueness plus vars(task) stability. Co-authored-by: Cursor <cursoragent@cursor.com>
v1.0.3 is 21,799 records / 18,869 patients. Document that MultiLabelProcessor emits CD, HYP, MI, NORM, STTC. Harmonize the datasets toctree and reuse the ptbxl extra in the pixi test env. Co-authored-by: Cursor <cursoragent@cursor.com>
The redundant-looking 'as' is required by ruff's unused-import rule for module re-exports, same as EEGBCIDataset / PTBXLDataset in this file. Co-authored-by: Cursor <cursoragent@cursor.com>
BaseDataset.load_table concatenates root/file_path and rejects absolute paths, so override load_table to read the derived CSV from metadata_cache_dir while leaving self.root equal to the constructor path. Co-authored-by: Cursor <cursoragent@cursor.com>
Official PTB-XL folds are patient-disjoint; this flag verifies that property via patient_to_index. Default False keeps current callers working. The synthetic fixture leaks patient 15709 across folds 1 and 9 so the check is actually exercised. Co-authored-by: Cursor <cursoragent@cursor.com>
AxelNoun
force-pushed
the
feat/ptbxl-dataset
branch
from
August 18, 2026 22:57
47aa3a4 to
fe1007f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contributor: AxelNoun (GitHub: @AxelNoun) — external contributor, no NetID
Contribution Type: New Dataset + Task
Description: Adds PTB-XL (12-lead ECG, PhysioNet v1.0.3, open access) as a
PyHealth dataset, with a 5-superclass multi-label diagnosis task and a
split helper using the official stratified folds. Motivated by the CPBench
effort: it gives conformal prediction a multi-label ECG benchmark that needs
no credentialing, with metadata suited to covariate/label-shift splits.
Based on Wagner et al., Scientific Data 2020
(https://www.nature.com/articles/s41597-020-0495-6).
Output label order:
MultiLabelProcessor.fitsorts observed labelsalphabetically. On the full v1.0.3 corpus the multi-hot vector is
CD, HYP, MI, NORM, STTC. The task does not pin a custom vocabulary.Files to Review:
pyhealth/datasets/ptbxl.py— dataset implementationpyhealth/datasets/configs/ptbxl.yaml— table config (v1.0.3 template)pyhealth/tasks/ptbxl.py— 5-superclass multi-label taskpyhealth/datasets/splitter.py—split_by_strat_foldhelpertests/core/test_ptbxl.py— tests with synthetic fixturesdocs/api/datasets/pyhealth.datasets.PTBXLDataset.rst,docs/api/tasks/pyhealth.tasks.ptbxl.rstexamples/ecg/ptbxl/ptbxl_superclass_quickstart.pyImplementation notes:
wfdbis an optional extra (pip install 'pyhealth[ptbxl]'), lazily imported.PTBXLDataset.rootis the user-provided data path. Derived metadata CSVslive under
metadata_cache_dir(notroot) so read-only mounts stayuntouched.
BaseDataset.load_tableconcatenatesroot / file_pathandcannot take an absolute path, so
load_tableis overridden to scan thecache directory. Cache identity still hashes data-root path + source CSV
bytes into
dataset_name(EEGBCI pattern) so two roots or an in-placeptbxl_database.csvreplace cannot shareglobal_event_df.BaseDatasetonly acceptsconfig_path, so PTB-XLwrites a rate/root/source-keyed YAML next to the derived CSV before
super().__init__(). CSV and YAML writes use tmp+os.replaceunderfilelock.FileLock.# noqa: E402on the EEGBCI import line inpyhealth/datasets/__init__.py— left untouched (out of diff).(n_leads, n_samples), matching EEGBCI's convention.ageis always an int: missing →-1(age_is_missing=True);HIPAA-censored raw 300 is clipped to
90(age_is_censored=True). Theraw 300 remains in the derived metadata CSV.
mainly pacemaker ECGs) are dropped by default via
drop_empty_labels.split_by_strat_fold(..., folds=)avoids decoding 12-lead signals just toread an integer.
check_patient_disjoint=Trueraises if apatient_idappears in more than one split (official PTB-XL folds are patient-disjoint;
the synthetic fixture leaks patient 15709 across folds 1 and 9 to exercise
this).
test-resources/core/ptbxl/is synthetic CSVs; WFDBwaveforms are generated at test time (not committed).
Testing / CI:
python -m unittest discover -t tests -s tests/core -p 'test_ptbxl.py' -v— 28 passed (withwfdbinstalled).wfdb/pyhealth[ptbxl], waveform I/O andset_taske2e tests skip.pyhealth = { extras = ["ptbxl"] }somake testallunder pixi covers waveform I/O.Follow-up: 71-SCP-code classification and age regression are structured
for but not implemented here — planned as a separate PR.
Test plan
python -m unittest discover -t tests -s tests/core -p 'test_ptbxl.py' -v— 28 passed (withwfdb)python tools/check_pr_rules.py --base upstream/master --head HEADmake testall/ contribution-rules on the PRRecreates #1186, which was closed automatically on 12 Aug when the fork
hosting its branch was detached during an account cleanup on my side —
my mistake, apologies for the noise.
All prior context and discussion: #1186