Fix iceberg partition pruning by nanoseconds - #2239
Conversation
Identity partition pruning on `timestamp_ns` over-prunes matching files. Min/max pruning also returns empty results when Iceberg `timestamp` (`DateTime64(6)`) lower/upper bounds are stored as nanoseconds, which is the customer `toDateTime64(..., 6)` case. Co-authored-by: Cursor <cursoragent@cursor.com>
Identity partition values for `timestamp_ns` arrive as Avro Int64, and some writers store nanosecond min/max bytes on Iceberg `timestamp` (`DateTime64(6)`). Both were compared at the wrong scale and dropped matching files. Write `DateTime64(9)` as `timestamp_ns` so ClickHouse does not recreate that layout. Co-authored-by: Cursor <cursoragent@cursor.com>
ianton-ru
changed the title
`1e16` microseconds is ~year 2286, so spec-correct Spark `9999-12-31` and `DateTime64` max `2299-12-31` bounds were divided by 1000 and over-pruned. Convert only `|ticks| > 1e18` (true nanosecond band); fail open in between. Co-authored-by: Cursor <cursoragent@cursor.com>
`timestamp_ns` is v3-only. Mapping `DateTime64(9)` without checking `iceberg_format_version` wrote invalid v2 metadata (default is 2). Throw instead of emitting v3 types into v1/v2 tables. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 381b43d5e2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (scale == 6 && magnitudeExceeds(unscaled, microseconds_ambiguous_threshold)) | ||
| { | ||
| if (!magnitudeExceeds(unscaled, nanoseconds_lower_threshold)) | ||
| return std::nullopt; | ||
|
|
||
| unscaled = nanosecondsToMicrosecondsForBound(unscaled, lower_bound); |
There was a problem hiding this comment.
When an Iceberg timestamp manifest contains a magnitude above this threshold, the schema still identifies the value as microseconds; magnitude alone cannot prove that the writer actually used nanoseconds. Dividing the value and then trusting the resulting range allows mayBeTrueInRange to prune a file whose corrupt or differently encoded bounds do not cover its data, producing missing query results. Treat these suspicious bounds as unavailable and skip min/max pruning rather than taking a consequential fallback path.
AGENTS.md reference: AGENTS.md:L153-L153
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This is not a P1. The overlapping band already fail-opens; converting |ticks| > 1e18 is the customer ns-stats case, not a guess inside the µs/ns overlap.
Iceberg timestamp is still microseconds in the schema. Magnitude cannot prove nanoseconds, which is why |ticks| in (1e16, 1e18] already returns nullopt. Callers then skip that hyperrectangle (if (!left || !right) continue) and min/max pruning for that column is skipped. That covers spec-correct far-future microseconds: Spark 9999-12-31 (~2.53e17 µs) and ClickHouse DateTime64 max 2299-12-31 (~1.04e16 µs).
Conversion runs only for |ticks| > 1e18. As microseconds that is ~year 33658, which is outside ClickHouse DateTime64(6) (max ~2299, ~1.04e16 µs) and outside normal Iceberg timestamp sentinels. As nanoseconds it is the customer layout (~1.76e18 for 2026) that previously over-pruned every file.
Fail-opening that band as well would keep query results correct (files scanned) but would disable min/max pruning again for those tables (IcebergMinMaxIndexPrunedFiles: 0). That undoes the point of this heuristic.
A corrupt 8-byte value > 1e18 that is not nanoseconds could still be converted into a fake in-range window. That is a residual, not a realistic P1 on this path.
3 participants
Solved #2240
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fix partition and min/max pruning for Iceberg tables by
timestamp_nstype, fix writingDateTime(9)astimestamp_ns.Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: