Use min/max stats for raw parquet INT64 columns read as DateTime - #117017
Use min/max stats for raw parquet INT64 columns read as DateTime#117017wudidapaopao wants to merge 6 commits into
INT64 columns read as DateTime#117017Conversation
Stats outside the `DateTime` range are not used as bounds because the cast saturates.
|
Workflow [PR], commit [bf68630] AI ReviewSummaryThis PR extends parquet min/max pruning to raw physical PR Metadata
Final VerdictNon-blocking review. No new inline code findings; only the changelog wording above should be corrected before merge. |
Build profile diff (arm_release)Comparing ✅ No significant changes. Binary sizes
Only the stripped binary is compared: the official master build keeps debug symbols while PR builds strip them, so the other binaries differ by construction. Compile time of recompiled translation units17 translation units recompiled, 143 s compile time in total, 17 of them have a recent master baseline. |
Only the `UInt32` to `IPv4` cast is supported; stats on other inputs could hide the unsupported-cast error when all row groups are pruned.
|
📊 Cloud Performance Report ✅ AI verdict: no significant changes detected. K_source=6 K_base=30 flagged=0/65 clickbench🟢 No significant changes tpch_adapted_1_official🟢 No significant changes Debug info
|
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Use min/max statistics for pruning when a parquet
INT64column is read asDateTime, including rawINT64columns without a logical type annotation. Also stop using out-of-rangeINTEGER(64)statistics asDateTimebounds, which could previously prune matching row groups incorrectly.Previously, predicates on such columns scanned all row groups. Statistics values outside the
DateTimerange are not used as bounds, because the cast saturates.IPv4statistics are additionally restricted to unsigned 32-bit parquet columns — the only input whose cast is supported — so that pruning cannot hide an unsupported-cast error behind an empty result.Columns of this shape are common in files from older parquet writers, e.g.
EventTime,ClientEventTimeandLocalEventTimein the ClickBenchhits.parquet.Workflow [PR]
Sync PR [sync-upstream/pr/117017]