Fix sort order violation for TTL GROUP BY with SET on a sorting key column by groeneai · Pull Request #108550 · ClickHouse/ClickHouse

groeneai · 2026-06-25T21:22:36Z

Changelog category (leave one):

Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Fixed wrong results and a Sort order of blocks violated logical error for MergeTree tables that use TTL ... GROUP BY ... SET. Assigning a column the sorting key depends on no longer writes a part whose primary index disagrees with the data; every MATERIALIZED column that reads an assigned column is now recomputed instead of keeping its pre-SET value (so skip indices and projections over it are no longer built from stale data); and a table with several GROUP BY TTLs no longer loses or fragments groups when an earlier SET rewrites a later TTL's GROUP BY key or expiry input. Both new paths are bounded by new MergeTree settings, each spilling to disk past its threshold instead of holding the whole part in memory: ttl_resort_max_bytes_before_external_sort (default 256 MiB) for the repair sort, and ttl_group_by_unsorted_max_bytes_before_external_group_by (default 256 MiB) for the hash aggregation a later GROUP BY TTL performs when an earlier SET rewrote its grouping key.

Description

Found via the AST fuzzer / stress tests (Stress test (arm_debug), AST fuzzer (amd_debug, targeted, old_compatibility)), recurring under STID 3413-350b. Example CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=100173&sha=5cfd0f5aeabe1f357125fd015ebbaaf90717b30e&name_0=PR&name_1=Stress%20test%20%28arm_debug%29

Reproducer (master, any build with assertions aborts; release writes a corrupt part):

CREATE TABLE t (k Float64, ts DateTime, v Float64)
ENGINE = MergeTree ORDER BY (k, toStartOfDay(ts))
TTL ts + toIntervalDay(1) GROUP BY k, toStartOfDay(ts)
    SET ts = max(ts) + interval 100 years, k = max(v)
SETTINGS min_bytes_for_full_part_storage = 128;

SYSTEM STOP MERGES t;
INSERT INTO t VALUES (1.0, '2000-06-09 10:00', 96827);
INSERT INTO t VALUES (1.0, '2000-06-10 10:00', 41302);
SYSTEM START MERGES t;
OPTIMIZE TABLE t FINAL;   -- Logical error: 'Sort order of blocks violated for column number 0, left: Float64_96827, right: Float64_41302...'

Root cause. TTLAggregationAlgorithm::finalizeAggregates emits aggregated groups in the input (already-sorted) order. The merge writer trusts the stream order: it reads the primary-key columns by name (including the materialized sort-key expression column such as toStartOfDay(ts), which still holds its pre-SET value) and writes them without re-sorting. When the SET clause rewrites a sort-key column, the produced part is therefore no longer ordered by the sorting key. A debug build catches this with the merge's CheckSortedTransform and aborts; a release build silently writes a part whose primary index does not match the data, which can return wrong results for primary-key-filtered queries.

Fix. After the TTL step, when a GROUP BY TTL assigns a column the sorting key depends on, recompute the sort-key expression columns from the post-SET values (overwriting the now-stale materialized ones) and re-sort by the sorting key. This is done in both the merge pipeline (MergeTask) and the mutation pipeline (MutateTask, e.g. ALTER TABLE ... MATERIALIZE TTL), where the rebuilt primary index and the skip-index expressions must see the same order. Every repair is gated on the SET targets of the GROUP BY TTLs that can actually fire in the part (getFiringGroupByTTLSetTargets), so it is a no-op for every other merge, and a not-yet-expired GROUP BY ... SET on the sorting key does not cost a whole-part re-sort.

The re-sort of a whole part must not be unbounded in a background merge, so it is bounded by the new MergeTree setting ttl_resort_max_bytes_before_external_sort (default 256 MiB, 0 disables spilling). Background contexts inherit max_bytes_before_external_sort = 0, which would disable spilling entirely; buildTTLResortSortingSettings applies the table-level threshold and zeroes the query-memory gate derived from max_bytes_ratio_before_external_sort, so the threshold is the real bound. The temporary-data scope comes from the global context (getSharedTempDataOnDisk), which background merges and mutations have.

Two further correctness gaps of TTL ... GROUP BY ... SET, both pre-existing on master, are fixed here as well, because the first shares the MATERIALIZED dependency analysis this change introduces and the second would otherwise be worsened by the repair sort — the shape stops failing with the loud Sort order of blocks violated and starts silently producing wrong aggregates instead:

Stale MATERIALIZED columns. TTLAggregationAlgorithm carries every column that is not a GROUP BY key or a SET target forward as any(...), so a MATERIALIZED column keeps its pre-SET value even though its source was rewritten (d Date MATERIALIZED toDate(ts) with ... SET ts = ...). The stored value, and any skip index or projection rebuilt from it, was written stale — a minmax index over d then pruned away rows that a brute-force scan returns. Every MATERIALIZED column that transitively reads a firing SET target is now recomputed from its default expression on both the merge and the mutation path, matching what ALTER TABLE ... UPDATE already does through column_to_affected_materialized. A MATERIALIZED column that mixes an EPHEMERAL input with a SET target cannot be recomputed during a merge (ephemeral columns are never stored), so each one is reported with a warning instead of silently written stale, mirroring MutationsInterpreter::prepare.
Several GROUP BY TTLs in one table. TTLTransform runs each TTLAggregationAlgorithm sequentially on the same block, and each assumes its input is contiguous in its own GROUP BY keys. An earlier SET that rewrites a later TTL's key (TTL ts1 + 1d GROUP BY k SET k = max(v), ts2 + 1d GROUP BY k SET payload = sum(payload)) split one logical group into several partial aggregates, and a derived key or expiry input (a computed key, a subcolumn, or a MATERIALIZED column) was even read at its pre-SET value. Such a TTL is now detected by mapping its keys and expiry inputs back to their physical storage columns (groupByKeysAffectedByEarlierSet, groupByTTLExpiryAffectedByEarlierSet), told to aggregate the whole stream instead of taking the streaming flush-on-key-change path, and its stale derived columns are refreshed before it runs (buildRefreshGroupByKeysDAG). The lost-order state cascades, since finalizing an unsorted aggregation emits groups in hash-table order.

That unsorted path holds every expired key of the part at once, so it is bounded by the new ttl_group_by_unsorted_max_bytes_before_external_group_by. Making the bound effective exposed a second defect in the same code: Aggregator can only spill a two-level hash table, so with both two-level thresholds at 0 the bound was dead letter, and had it ever fired, finalizeAggregates used convertToChunks on the in-memory state alone and would have silently dropped the spilled groups from the written part. group_by_two_level_threshold_bytes is now tied to the bound so the spill sites are reachable, and finalizeAggregates flushes the in-memory remainder and merges the spilled generations back bucket by bucket via Aggregator::mergeBlocks, keeping the merge-back high-water mark at one bucket. Both halves are new in this pull request, so no release ever had a live spill here.

Tests. 03545_number_of_rows_in_ttltransform, which intentionally exercises SET on sort-key columns while the output stays ordered, passes unchanged. New:

04327_ttl_group_by_set_sort_key_resort — Float64, String and LowCardinality(String) sort keys with a non-monotonic SET, subcolumn and wrapped-subcolumn sorting keys, the MATERIALIZE TTL mutation path with skip-index rebuild, MATERIALIZED sort keys (direct, chained and over a subcolumn), the mixed-EPHEMERAL shape, and a control case that only sets a non-sort-key column. Each asserts the part is physically ordered by the sorting key.
04511_ttl_multi_group_by_set_rewrites_key — several GROUP BY TTLs whose keys and expiry inputs are rewritten by an earlier SET, including derived keys, the cascade, the not-yet-expired case that must keep the fast path, and stale non-sort-key MATERIALIZED columns with a minmax index over them.
04661_ttl_group_by_set_resort_external_sort_spill — forces ttl_resort_max_bytes_before_external_sort = 1 and asserts via ExternalSortWritePart in system.part_log that both the merge and the MATERIALIZE TTL path really spill and still produce a correct, physically sorted part.
04327_ttl_group_by_set_sort_key_resort also forces ttl_group_by_unsorted_max_bytes_before_external_group_by = 1 and asserts via ExternalAggregationWritePart that the unsorted aggregation really spills while the group count is preserved.

Workflow [PR]
Sync PR [sync-upstream/pr/108550]

groeneai · 2026-06-25T21:23:15Z

Pre-PR validation gate (click to expand)

#	Question	Answer
a	Deterministic repro?	Yes. `CREATE ... ORDER BY (k, toStartOfDay(ts)) TTL ... GROUP BY k, toStartOfDay(ts) SET ts = max(ts)+interval 100 years, k = max(v)`, two single-row inserts in different days, then `OPTIMIZE TABLE ... FINAL` aborts every time. Reproduced on the unfixed master snapshot (26.7.1.1) and the pre-edit worktree binary; 30/30 deterministic.
b	Root cause explained?	`TTLAggregationAlgorithm::finalizeAggregates` emits aggregated groups in input (already-sorted) order. The `SET` rewrites a sort-key column (`k`, and `ts` feeding `toStartOfDay(ts)`), so the produced stream is no longer ordered by the sorting key. The merge writer reads PK columns by name (incl. the materialized `toStartOfDay(ts)`, still holding its pre-SET value) and writes without re-sorting → part with a primary index inconsistent with the data. Debug `CheckSortedTransform` aborts; release writes silently.
c	Fix matches root cause?	Yes. After the TTL step, when a GROUP BY TTL assigns a sorting-key dependency column, the sort-key expression columns are recomputed from the post-SET values and the stream is re-sorted by the sorting key — directly fixing the unsorted output and the stale materialized sort-key column. Not a band-aid (no widened bounds, no tag, no data reduction).
d	Test intent preserved / new tests added?	Yes. Existing `03545_number_of_rows_in_ttltransform` (which intentionally exercises `SET` on sort-key columns) passes unchanged. New `04327_ttl_group_by_set_sort_key_resort` asserts both correct results and that the part is physically sorted by the sorting key.
e	Both directions demonstrated?	Yes. Unfixed master snapshot: aborts with `Sort order of blocks violated for column number 0, left: Float64_96827, right: Float64_41302`. Fixed binary (Build ID verified, differs from pre-edit): passes, part physically sorted, 30/30.
f	Fix is general across code paths?	Yes. GROUP BY TTL forces the Horizontal merge algorithm (`canVerticalTTLDelete` returns false on `hasAnyGroupByTTL`), so the single-stream merge pipeline is the only affected path; the re-sort is placed there. The fix targets the produced-unsorted-stream root cause, not a single crash site.
g	Fix generalizes across inputs (params/datatypes/wrappers)?	Yes. Verified with `Float64`, `String`, `LowCardinality(String)` and `Nullable(Float64)` sort keys, monotonic vs non-monotonic SET, SET on the first vs a deeper sort column. Control case (SET only a non-sort-key column) confirmed to be a no-op (gate does not fire). Float64/String/LowCardinality variants are in the regression test.
h	Backward compatible? (maintainer-approved exception only)	Yes. No setting default, on-disk/wire format, or validation change. Behaviour only changes for the previously-broken case (it now produces a correctly sorted part instead of aborting / corrupting); the already-correct case is byte-for-byte unchanged (`03545` reference unchanged).
i	Invariants and contracts preserved?	Yes. The fix restores the MergeTree invariant the writer relies on: the stream handed to the writer is sorted by the sorting key and the materialized sort-key expression column is consistent with the storage columns. The re-sort step is gated and only adds a recompute+sort; it does not alter the merge/aggregation semantics, row counts, or the TTL result values.

Session id: cron:clickhouse-worker-slot-4:20260625-200200

groeneai · 2026-06-25T21:23:48Z

cc @CurtizJ @KochetovNicolai — could you review this? It fixes a sort-order violation (silent primary-index corruption in release, LOGICAL_ERROR abort in debug) when a TTL ... GROUP BY ... SET clause assigns a sorting-key column: TTLAggregationAlgorithm emits groups in input order, so the merge writes an unsorted part. The fix recomputes the sort-key expression from the post-SET values and re-sorts the TTL merge output, gated so it is a no-op for all other merges. Closes #108514.

clickhouse-gh · 2026-06-25T21:59:29Z

Workflow [PR], commit [98b5743]

Summary: ❌

Performance Comparison: Performance dashboard

job_name	test_name	status	info	comment
Integration tests (arm_binary, distributed plan, 2/4)		FAIL
	test_storage_nats/test_nats_jet_stream.py::test_nats_restore_failed_connection_without_losses_on_write	FAIL	cidb, issue

AI Review

Summary

This PR broadens the TTL ... GROUP BY ... SET fix to cover sort-key repair, stale MATERIALIZED columns, chained GROUP BY TTLs, and bounded spilling in both merge and MATERIALIZE TTL paths. After reading the current PR head and the full prior thread history, I did not find any new line-level correctness blockers or majors that still hold.

PR Metadata

Changelog category: Bug Fix matches the change.

Changelog entry: required, but the current text overstates the final behavior. [src/Storages/MergeTree/TTLResortUtils.cpp:475-481] explicitly skips recomputing non-deterministic MATERIALIZED defaults, so "every MATERIALIZED column that reads an assigned column is now recomputed" is no longer accurate.

Exact replacement:
Fixes wrong results and a \Sort order of blocks violated` logical error for `MergeTree` tables that use `TTL ... GROUP BY ... SET`. Assigning a column the sorting key depends on no longer writes a part whose primary index disagrees with the data, deterministic affected `MATERIALIZED` columns are recomputed consistently, and tables with several `GROUP BY` TTLs no longer lose or fragment groups when an earlier `SET` rewrites a later TTL's key or expiry input.`

Final Verdict

No new blocking or major code issues found in the current PR head. Tighten the PR text so it matches the final non-deterministic MATERIALIZED behavior.

LLVM Coverage Report

Measured on commit 98b5743.

Metric	Baseline	Current	Δ
Lines	88.40%	88.40%	+0.00%
Functions	92.00%	92.00%	+0.00%
Branches	80.70%	80.80%	+0.10%

Changed lines: Changed C/C++ lines covered: 848/895 (94.75%) · Uncovered code

Full report · Diff report

groeneai · 2026-06-25T23:41:57Z

Fixed in e779631.

The gate now maps each sorting-key dependency to its storage column via getColumnNameInStorage before comparing with set_part.column_name, the same way extractMergingAndGatheringColumns does. So ORDER BY t.a + SET t = ... now resolves t.a -> t and the gate fires.

One more place needed the same treatment: the re-sort step built createSubcolumnsExtractionActions from the current header, which still holds the stale materialized t.a. The extractor saw t.a as available and skipped re-extraction, so the re-sort would have keyed on the pre-SET value. The step now hides the stale computed sort-key columns from the extractor's input header, so t.a is recomputed from the post-SET physical t.

Validated both directions on a debug build: ORDER BY t.a TTL ... GROUP BY t.a SET t = argMax(candidate, v) fails before the change (NOT_FOUND_COLUMN_IN_BLOCK for t.a in createMergedStream) and produces a correctly sorted part after it. Added a Tuple subcolumn case to 04327_ttl_group_by_set_sort_key_resort; a JSON subcolumn sort key was verified manually. 03545_number_of_rows_in_ttltransform still passes unchanged.

groeneai · 2026-06-26T04:00:34Z

CI finish ledger — `e779631`

Every failed check below has an owner (a fixing PR). Mergeable Check / PR are aggregators that only reflect the rows below.

Check / test	Reason	Owner / fixing PR
Stress test (amd_msan) / Hung check failed, possible deadlock found	deadlock, chronic shutdown/teardown family (not PR-caused)	#101680 / #105905 (ours, open)
Stress test (arm_debug) / Logical error 'Unexpected exception in refresh scheduling' (STID 2508-34af)	RefreshTask shutdown crash (trunk bug); the Cannot-start-server and Check-failed rows on this lane are collateral of the same abort	#105588 (ours, open)

Not PR-caused: this PR only changes src/Storages/MergeTree/MergeTask.cpp plus a TTL regression test; both failures are pre-existing chronic stress-test issues unrelated to the TTL merge re-sort.

Session id: cron:our-pr-ci-monitor:20260626-033000

cv4g · 2026-06-30T09:35:46Z

@groeneai This fixes the MergeTask path at src/Storages/MergeTree/MergeTask.cpp:3208, but the same issue looks reachable via MATERIALIZE TTL in MutateTask (src/Storages/MergeTree/MutateTask.cpp lines 2171 and 2484) and possibly between multiple GROUP BY TTL algorithms executed by one TTLTransform (src/Processors/Transforms/TTLTransform.cpp lines 88 and 190). Should this PR cover these as well?

groeneai · 2026-06-30T11:20:03Z

@cv4g Good catch on MutateTask / MATERIALIZE TTL. Confirmed reachable and fixed in bcd95db.

MutateTask (MATERIALIZE TTL): yes, fixed. A MATERIALIZE TTL mutation with a GROUP BY TTL goes through MutateAllPartColumnsTask (the GROUP BY forces isAffectingAllColumns()), which materializes the sorting-key expression before the TTLTransform (MutateTask.cpp:2154) and writes via MergedBlockOutputStream with rewrite_primary_key=true, rebuilding the index from the stream. The CheckSortedTransform added in MutationsInterpreter::execute() sits before the appended TTLTransform, so it never sees the post-SET order; nothing else guards it. So unlike the merge path, even a debug build does not abort here, it silently writes a corrupt part.

Reproduced on this branch (with the merge fix already in) using a single level-0 part so the mutation, not a background merge, applies the TTL:

CREATE TABLE t (k Float64, ts DateTime, v Float64)
ENGINE = MergeTree ORDER BY (k, toStartOfDay(ts))
TTL ts + toIntervalDay(1) GROUP BY k, toStartOfDay(ts)
    SET ts = max(ts) + interval 100 years, k = max(v)
SETTINGS min_bytes_for_full_part_storage = 128, materialize_ttl_recalculate_only = 0;
SYSTEM STOP TTL MERGES t;
INSERT INTO t VALUES (1.0, '2000-06-09 10:00', 96827), (1.0, '2000-06-10 10:00', 41302);
ALTER TABLE t MATERIALIZE TTL SETTINGS mutations_sync = 2;
-- before bcd95db: physical read order is k = 96827, 41302 (descending) -> corrupt index
-- after  bcd95db: k = 41302, 96827 (ascending) -> sorted

The fix extracts the gate and the recompute-and-resort into TTLResortUtils (shared by both paths) and applies it after the TTL step in MutateAllPartColumnsTask, mirroring the merge path. MutateSomePartColumnsTask is not affected: a GROUP BY TTL never routes there, and it writes via MergedColumnOnlyOutputStream with rewrite_primary_key=false (reuses the source index). Added MATERIALIZE TTL regressions to 04327 for a plain Float64 key and a Tuple subcolumn key.

Multiple GROUP BY TTL algorithms in one TTLTransform (TTLTransform.cpp:88/190): not a separate gap. The re-sort runs after the whole TTL step (the complete TTLTransform), so it covers the final stream regardless of how many algorithms ran inside it or in what internal order. The corruption only depends on the part written after the TTL step, which both paths now re-sort.

groeneai · 2026-06-30T13:47:22Z

Addressed the secondary-index finding in f977bca (full analysis in the inline thread). Net scope of this PR now: the sort-order re-sort covers both the merge path (MergeTask) and the mutation path (MutateTask / MATERIALIZE TTL), and the mutation path additionally rebuilds skip indices on subcolumns of TTL-rewritten columns instead of hardlinking them stale.

Pre-PR validation gate (click to expand)

#	Question	Answer
a	Deterministic repro?	Yes. `MATERIALIZE TTL` with `GROUP BY ... SET t = argMax(cand, v)` on a Wide part with `INDEX idx t.a` (subcolumn of `t`, not in the sorting key). Before the fix the index is hardlinked from the source part (same inode), `secondary_indices_compressed_bytes` = 0 in the new part, and it stops pruning; deterministic every run.
b	Root cause explained?	`MutationsInterpreter::prepare` compares `index.expression->getRequiredColumns()` (`t.a`) literally against the changed columns. A GROUP BY TTL records the rewritten physical column as the `TTL_TARGET` `t`, so `t.a` != `t`, the index is not added to `materialized_indices`, and on a full Wide part it is hardlinked unchanged (`need_recalculate` is false). Separately, the index expression was materialized before the TTL step, but the GROUP BY aggregation drops that column (zero rows), which aborts the TTL transform once the index is actually rebuilt.
c	Fix matches root cause?	Yes. (1) Map each index/projection required column to its storage column (`Nested::tryGetColumnNameInStorage`, same mapping as the sorting-key gate) before the comparison, so `t.a -> t` and the index is rebuilt. (2) Compute skip-index expressions AFTER the TTL step (and re-sort), mirroring the merge path, so they reflect post-SET values and are not dropped by the aggregation.
d	Test intent preserved / new tests added?	Yes. Existing 04327 cases unchanged and still pass. New case added: `MATERIALIZE TTL` + subcolumn skip index, asserting the rebuilt index returns the same rows with and without skip indexes (a stale index would prune rewritten values away).
e	Both directions demonstrated?	Yes. Before the fix: the index is hardlinked stale (proven by shared inode + `secondary_indices_compressed_bytes` = 0) and, once forced to rebuild, the pre-step expression aborts the mutation (LOGICAL_ERROR in TTLTransform). After: mutation succeeds, index registered, prunes against post-SET values.
f	Fix is general across code paths?	Yes. The mapping is applied to both the secondary-index and projection rebuild loops. Projections cannot reference subcolumns (`Projections cannot contain individual subcolumns`), so their whole-column case was already matched; the mapping keeps both paths consistent. The merge path already rebuilds all indices from the stream and was correct.
g	Fix generalizes across inputs?	Verified across: index column in the sorting key vs not in it (both rebuild and prune correctly), plain physical key column (already matched), `Tuple` subcolumn, value present only after SET vs absent. The pre-TTL materialization split (primary key before, skip indices after) holds whether or not the SET rewrites a sort-key column.
h	Backward compatible?	Yes. No setting, format, or on-disk change. Only the in-mutation decision of which indices to rebuild vs hardlink, and the order in which expression columns are computed in the pipeline.
i	Invariants and contracts preserved?	Yes. The written part's data, primary index, and skip indices are now mutually consistent. The GROUP BY TTL aggregation still receives its sort-key grouping columns before the step (primary-key expression is kept pre-step). No new throw paths across noexcept boundaries; gated to a no-op for non-GROUP-BY-TTL mutations.

Session id: cron:clickhouse-worker-slot-8:20260630-124500

groeneai · 2026-06-30T18:07:00Z

CI finish ledger — `f977bca`

CI fully finished (Finish Workflow: success). This PR fixes the TTL GROUP BY ... SET sort-order violation (STID 3413-350b) across the merge + mutate paths. Every failure below has an owner; none is PR-caused (this PR touches only the TTL-aggregation sort-key handling, none of the failing areas).

Check / test	Reason	Owner / fixing PR
Unit tests (asan_ubsan + msan, function_prop_fuzzer) / FunctionsStress.stress + AllTests	crash (`reinterpret(... AS Decimal256(N))` returns wrong scale: `Const(Decimal256) instead of Decimal(76, N)`, INCORRECT_DATA)	#108878 (ours, open) — "Fix reinterpret to Decimal/DateTime64 with a different scale"
Stateless tests (amd_tsan, s3 storage, parallel, 2/2) / 00612_shard_count	flaky (Code 102 Unexpected packet, connection-pool desync; 30 reruns all pass)	#108854 (external, open) — fixes the distributed TablesStatus desync
Stress test (amd_tsan) + (arm_tsan) / "Hung check failed, possible deadlock found"	deadlock (chronic hung-check family)	#108212 (ours, merged) / #105905 (ours, open)
Bugfix validation (functional tests, amd64) / Environment setup (amd_msan)	infra (msan re-prepare of stateful data failed in setup, not a test result)	a fix task is moved to pending (investigating at full effort; fixing PR link to follow here)
Integration tests (arm_binary, distributed plan, 4/4)	flaky (no per-test FAIL row; job-level only)	a fix task is created (investigating; fixing PR link to follow here)
CH Inc sync	-	CH Inc sync (private, not actionable by us)

Session id: cron:our-pr-ci-monitor:20260630-173000

groeneai · 2026-06-30T18:29:19Z

Re: the Integration tests (arm_binary, distributed plan, 4/4) line on the CI finish ledger — root cause was a transient Docker registry HTTP 500 on image pre-pull (Failed to pre-pull Docker images needed by the test batch), unrelated to this PR. Fixing PR for the pre-pull retry robustness: #108975.

A MergeTree table with `TTL ... GROUP BY ... SET <col> = <agg>` where the SET clause rewrites a column the table's sorting key depends on (directly, or via an expression such as toStartOfDay(ts)) produced an out-of-order data part during the TTL merge. TTLAggregationAlgorithm::finalizeAggregates emits aggregated groups in the input (already-sorted) order, and the merge writer trusts the stream order: it reads the primary-key columns by name (including the materialized sort-key expression column, which still holds its pre-SET value) and writes them without re-sorting. When the SET changes a sort-key column the produced part is therefore not sorted by the sorting key. In a debug build the merge's CheckSortedTransform aborts the server with "Sort order of blocks violated"; in a release build the part is written silently with a primary index inconsistent with the data, which can return wrong results for primary-key-filtered queries. Fix: after the TTL step in the merge pipeline, when a GROUP BY TTL assigns a sorting-key dependency column, recompute the sort-key expression columns from the post-SET values (overwriting the now-stale materialized ones) and re-sort by the sorting key. The step is gated so it is a no-op for every other merge. This preserves the existing tested behaviour where SET targets a sort-key column but the output happens to stay ordered (03545_number_of_rows_in_ttltransform). Closes ClickHouse#108514 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…n sorting keys The re-sort gate added for the TTL GROUP BY SET sort-order fix compared the sorting-key dependency names against the SET target column name directly. A sorting-key dependency can be a subcolumn (ORDER BY t.a requires t.a), while a SET target is always a physical storage column (SET t = ...). The names never matched for a subcolumn sorting key, so the gate did not fire and the part was built from the stale pre-SET subcolumn value. Map each sorting-key dependency to its storage column with getColumnNameInStorage before comparing, the same way extractMergingAndGatheringColumns does. Also hide the stale materialized sort-key columns from the subcolumn extractor in the re-sort step so the subcolumn is recomputed from the post-SET physical column instead of reusing the stale value. Adds a Tuple subcolumn regression to 04327_ttl_group_by_set_sort_key_resort. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…n path The same sort-order violation the merge path fixes is also reachable through ALTER TABLE ... MATERIALIZE TTL. A MATERIALIZE TTL mutation with a GROUP BY TTL runs MutateAllPartColumnsTask, which materializes the sorting-key expression before the TTLTransform and writes through MergedBlockOutputStream (rewrite_primary_key=true). When the GROUP BY ... SET rewrites a column the sorting key depends on, the aggregation emits groups in input order, so the stream is no longer ordered by the sorting key and the materialized sort-key expression is stale. The mutation writer rebuilds the primary index from that stream, so it gets an index inconsistent with the data; there is no CheckSortedTransform after the TTL step in the mutation pipeline, so a debug build does not even catch it and a release build silently writes a corrupt part. Extract the gate (groupByTTLAssignsSortKeyColumn) and the recompute-and-resort logic into TTLResortUtils, shared by both paths. After the TTL step in MutateAllPartColumnsTask, when the gate fires, recompute the sorting-key expression columns from the post-SET values and re-sort by the sorting key, the same way the merge path does. Gated to a no-op for every other mutation. MutateSomePartColumnsTask is not affected: a GROUP BY TTL forces the full rewrite path, and that task writes through MergedColumnOnlyOutputStream with rewrite_primary_key=false, which reuses the source part index. Extends the regression test 04327_ttl_group_by_set_sort_key_resort with MATERIALIZE TTL cases for a plain Float64 sort key and a Tuple subcolumn sort key, asserting the produced part is physically ordered by the sorting key. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A `MATERIALIZE TTL` mutation with a `GROUP BY` TTL records the rewritten physical columns as `TTL_TARGET` (for example `t`), while a skip index can depend on a subcolumn of one of them (for example `INDEX idx t.a ...`). The rebuild decision in MutationsInterpreter compared the index/projection required columns against the changed columns literally, so `t.a` did not match the changed physical column `t`: the index was not added to materialized_indices and on a full Wide part it was hardlinked from the source part unchanged, keeping its pre-SET minmax values. Queries using that index then prune against stale values (missing rows) or, since the index was never registered in the new part's checksums, stop pruning entirely. Map each required column to its storage column (the same way the sorting-key gate does) before comparing, so subcolumn indices and projections of a rewritten column are rebuilt. Rebuilding the index exposed a second problem in the mutation pipeline. The primary-key and skip-index expressions were materialized before the TTL step, but a `GROUP BY` TTL aggregates the stream: the pre-step skip-index expression columns are computed from pre-aggregation, pre-SET rows, and the aggregation can drop them entirely, leaving a zero-row column that aborts the TTL transform. Mirror the merge path: when a `GROUP BY` TTL runs, materialize only the primary-key expression before the step (the aggregation groups by sort-key columns and needs them present) and compute the skip-index expressions after it. When the SET also rewrites a sort-key column, re-sort first so the rebuilt primary index matches the written row order, then recompute the skip-index expressions from the post-SET stream. Extends 04327_ttl_group_by_set_sort_key_resort with a MATERIALIZE TTL case for a skip index on a subcolumn of a TTL-rewritten column that is not in the sorting key (so the primary index cannot mask a stale skip index): the rebuilt index must return the same rows with and without skip indexes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A TTL GROUP BY SET can rewrite a column that a MATERIALIZED sort-key column is computed from (e.g. d Date MATERIALIZED toDate(ts), ORDER BY d, ... GROUP BY d SET ts = ...). The aggregation updates ts but leaves the stored d on its pre-SET value, so the part is written with stale sort-key data and pruning/filtering on d uses the old value. groupByTTLAssignsSortKeyColumn previously only detected a SET target that is itself a sort-key dependency. Extend detection to also flag a SET target that is a source of a MATERIALIZED sort-key column (analyzed the same way the UPDATE mutation path builds column_to_affected_materialized), and recompute those columns from their default expression before recomputing the sorting key and re-sorting. Applied to both the merge (MergeTask) and mutation (MutateTask, MATERIALIZE TTL) paths via shared TTLResortUtils helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

groeneai · 2026-07-11T13:04:20Z

Follow-up commit f2aaf8e addresses the bot's MATERIALIZED sort-key finding.

Pre-PR validation gate (click to expand)

#	Question	Answer
a	Deterministic repro?	Yes. `CREATE TABLE (ts DateTime, d Date MATERIALIZED toDate(ts)) ENGINE=MergeTree ORDER BY d; INSERT 10 rows; ALTER MODIFY TTL ts + toIntervalDay(1) GROUP BY d SET ts = max(ts) + interval 100 year; ALTER MATERIALIZE TTL` -> every row has stale `d` (0/10 consistent). Merge variant with an order-reversing SET aggregate writes an unsorted part.
b	Root cause explained?	`groupByTTLAssignsSortKeyColumn` only compared SET targets against sort-key dependencies, so a SET on `ts` (a SOURCE of MATERIALIZED `d`) was not detected -> no resort/recompute. Even when the resort ran, it recomputed only the sort-key EXPRESSION, reading the stale stored `d`. So `d` kept its pre-SET value and pruning/order on `d` used it.
c	Fix matches root cause?	Yes. Detection now also flags a SET target that is a source of a MATERIALIZED sort-key column (analyzed like the UPDATE path's `column_to_affected_materialized`), and those columns are recomputed from their default expression (`evaluateMissingDefaults` after dropping the stale value) before the sort-key recompute + resort. Applied to both merge (MergeTask) and mutation (MutateTask) via shared TTLResortUtils.
d	Test intent preserved / new tests added?	Yes. Added merge + mutation regression cases in 04327 for a MATERIALIZED sort-key column whose source is SET, asserting `d = toDate(ts)` and physical sort order, `optimize_sorting_by_input_stream_properties = 1` kept on. Existing 04327 cases unchanged and passing.
e	Both directions demonstrated?	Yes. Baseline binary (db34aa9c): mutation 0/10 and merge 0/10 consistent, merge part unsorted by `d`. Fixed binary (045483e1): both 10/10 consistent and physically sorted.
f	Fix is general across code paths?	Yes. Both TTL sort-key rewrite paths are covered: merge (MergeTask.cpp) and mutation MATERIALIZE TTL (MutateTask.cpp via resortPipelineAfterTTLGroupBySet), sharing the same detection + recompute helpers. Not a symptom guard: the stale value is recomputed at its source (the MATERIALIZED default expression).
g	Fix generalizes across inputs?	Verified: direct-SET sort-key col (existing cases) still work; MATERIALIZED sort-key col whose source is NOT set is correctly NOT triggered (no false positive) and stays sorted; subcolumn sort keys unaffected. Scope matches the UPDATE path: single-level materialized dependency. A materialized-of-materialized chain is a pre-existing shared limitation (the UPDATE path's `column_to_affected_materialized` is also single-level), noted in-thread; not expanded here to avoid scope creep.
h	Backward compatible?	Yes. No setting default, on-disk/wire format, or new validation change. Purely corrects the data written for an already-buggy TTL GROUP BY SET case; gated to a no-op for every other merge/mutation.
i	Invariants and contracts preserved?	Yes. The written part is now ordered consistently with its primary index (the invariant `CheckSortedTransform` enforces). The recompute DAG drops only the stale target columns and preserves all others (`save_unneeded_columns`), so block structure is unchanged for the rest of the stream.

Session id: cron:clickhouse-worker-slot-0:20260711-122100

…GROUP BY SET Extend the MATERIALIZED-sort-key recompute for TTL GROUP BY SET in two ways: 1. Transitive closure: a SET on a base column can invalidate a MATERIALIZED sort-key column several hops away (x -> y MATERIALIZED toDate(x) -> z MATERIALIZED toYYYYMM(y), ORDER BY z). The affected-column detection now takes the fixpoint over the materialized-dependency graph and recomputes the intermediate hops (y before z), instead of a one-hop check that left z stale. 2. Subcolumn sources: buildRecomputeMaterializedColumnsDAG now prepends a subcolumn-extraction DAG (as AddingDefaultsTransform does) so a default expression reading a subcolumn (d MATERIALIZED toDate(tup.ts)) can be recomputed; previously it failed with NOT_FOUND_COLUMN_IN_BLOCK. Both paths (merge and MATERIALIZE TTL) go through the shared helper. Regression cases added to 04327 for the transitive chain and the subcolumn source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

groeneai · 2026-07-11T14:11:44Z

Addressed both clickhouse-gh[bot] findings on TTLResortUtils.cpp in 3071816.

Pre-PR validation gate (click to expand)

#	Question	Answer
a	Deterministic repro?	Yes. (#2 transitive) `x->y MATERIALIZED toDate(x)->z MATERIALIZED toYYYYMM(y)`, `ORDER BY z`, `SET x`: stored `z` was stale (`consistent=0` all rows). (#1 subcolumn) `d MATERIALIZED toDate(tup.ts)`, `ORDER BY d`, SET over unrelated plain col: `NOT_FOUND_COLUMN_IN_BLOCK: tup.ts`. Both reproduce on demand via `MATERIALIZE TTL`.
b	Root cause explained?	(#2) `groupByTTLAssignsSortKeyColumn`/affected-detection was one-hop: `getMaterializedColumnSourceColumns(z)=[y]`, SET target=`x` not in `[y]` -> not affected -> no recompute -> `z` written stale. (#1) `buildRecomputeMaterializedColumnsDAG` fed `header_after_drop` (physical `tup` only) to `evaluateMissingDefaults`, whose action requires the subcolumn `tup.ts`, which was never extracted.
c	Fix matches root cause?	Yes. (#2) `getMaterializedColumnsAffectedBySet` takes the fixpoint over the materialized-dependency graph; the recompute set includes intermediate hops in physical-column order (y before z). (#1) prepend a `createSubcolumnsExtractionActions` DAG before the `evaluateMissingDefaults` DAG, exactly as `AddingDefaultsTransform` does.
d	Test intent preserved / new tests added?	Yes. Existing 04327 cases unchanged and still pass. Added `mat chain` (two-hop) and `mat subcol` (subcolumn source) cases asserting both consistency (`d = toDate(...)` for all rows) and physical sort order.
e	Both directions demonstrated?	Yes. Pre-fix: #2 `consistent=0`, #1 `NOT_FOUND tup.ts`. Post-fix (Build ID ea48213b): #2 all `consistent=1` and sorted; #1 recomputed correctly and sorted.
f	Fix is general across code paths?	Yes. Both fixes live in the shared TTLResortUtils helpers used by both the merge (MergeTask) and mutation (MutateTask MATERIALIZE TTL) paths.
g	Fix generalizes across inputs?	Transitive closure is arbitrary-depth (not just two-hop). Subcolumn extraction is generic (Tuple element here; same mechanism handles other subcolumn kinds). Note: a separate, pre-existing bug (TTL SET aggregate whose argument is itself a subcolumn, e.g. `max(tup.ts)`) is orthogonal to this PR and reproduces on master with no MATERIALIZED column; reported in-thread, out of scope here.
h	Backward compatible?	N/A. No setting, format, or on-disk change. Recompute only fires for the GROUP BY TTL SET affected-materialized case; no-op otherwise.
i	Invariants and contracts preserved?	Yes. Recompute runs before the sort-key recompute + re-sort, so the written part remains ordered consistently with its primary index. Pass-through columns preserved via `save_unneeded_columns`; recompute order respects the dependency chain.

Session id: cron:clickhouse-worker-slot-1:20260711-133400

groeneai · 2026-08-19T00:10:31Z

CI finish ledger - `7c031d1`

Every failure below has an owner: a fixing PR (ours or external), or a full-effort fix task
whose fixing-PR link will be posted here when it opens. Only CH Inc sync is exempt.

Check / test	Reason	Owner / fixing PR
Fast test / `04327_ttl_group_by_set_sort_key_resort`	PR-caused - `Code: 47 UNKNOWN_IDENTIFIER: Identifier 's.m' cannot be resolved from table with name s. Maybe you meant: ['s.x']` at line 347. `t_future_ttl_after_modify_saved` is created with `ENGINE = Memory AS SELECT * FROM t_future_ttl_after_modify`, and `SELECT *` does not include the MATERIALIZED column `m`, so the saved table has only `ts`/`x`. 30-day breadth: 5 hits, all on this PR, 0 on any other PR and 0 on master.	PR-caused -> fixed in this PR
Finish Workflow / Post Hooks	Consequence of the Fast test red gating the matrix - not an independent defect	PR-caused -> fixed in this PR

Config Workflow, Style check, Build (arm_tidy) and Build profile diff are green on this head;
the rest of the matrix never ran because Fast test gates it.

Session id: cron:our-pr-ci-monitor:20260819-000000

… values The `t_future_ttl_after_modify` arm saved a reference copy with `ENGINE = Memory AS SELECT * FROM t_future_ttl_after_modify`, but `SELECT *` does not expand to MATERIALIZED columns, so the saved table held only `ts` and `x`. The following assertion reads `s.m` and therefore failed to analyze: Code: 47. DB::Exception: Identifier 's.m' cannot be resolved from table with name s. Maybe you meant: ['s.x']. (UNKNOWN_IDENTIFIER) Naming `x, m` in the projection materialises the column into the snapshot. The arm's intent is unchanged: it still compares the stored `m` before and after `MATERIALIZE TTL` to prove a future GROUP BY TTL that never fires does not recompute a non-deterministic MATERIALIZED default. Verified the assertion is a live oracle rather than merely well-formed: with the `hasNonDeterministic()` guard in `getGroupByTTLSetAffectedMaterializedColumns` removed, the arm reports 0 (2 joined rows, 0 matching); with the guard present it reports 1 (2 of 2 matching). 20/20 passes with CI's settings randomization enabled, and 4 concurrent copies pass, so the file keeps working without a no-parallel tag.

groeneai · 2026-08-19T07:51:50Z

CI finish ledger - `b2b5033`

Every failure below has an owner: a fixing PR (mine or external), or a full-effort fix task
whose fixing-PR link will be posted here when it opens.

Check / test	Reason	Owner / fixing PR
Stress test (amd_tsan) / `Found patch part ... intersects mutation with version` (STID 2781-51e9)	pre-existing trunk logical error in the ReplicatedMergeTree patch-part queue; 24 rows / 21 PRs / 4 master in 30 days, and this PR's diff does not touch `ReplicatedMergeTreeQueue`	#113998 (mine, open)
Stress test (amd_tsan) / `Cannot start clickhouse-server`, `Check failed`	same job, cascade of the row above (server was restarted after the abort, `Connection reset by peer` on the follow-up probe)	#113998 (mine, open)

169 of 170 checks are green (152 success, 17 skipped). No other failures.

Session id: cron:our-pr-ci-monitor:20260819-073000

groeneai · 2026-08-19T19:15:50Z

CI finish ledger - `3614f3c`

Every failure below has an owner: a fixing PR (ours or external), or a full-effort fix task
whose fixing-PR link will be posted here when it opens. Only CH Inc sync is exempt.

Check / test	Reason	Owner / fixing PR
Fast test / Build ClickHouse	PR-caused: `TTLAggregationAlgorithm.cpp:35` `unknown type name 'MergeTreeSettingsUInt64'` and `:104` `'const DB::MergeTreeSettings' does not provide a subscript operator`	PR-caused -> fixed in this PR
Build (arm_tidy) / Build ClickHouse	PR-caused: same two errors, plus `:103` `variable 'max_bytes_before_external_group_by' is not initialized` (`cppcoreguidelines-init-variables`) downstream of them	PR-caused -> fixed in this PR
Style check / various	PR-caused: `TTLResortUtils.cpp:51` uses the banned phrase "new analyzer"	PR-caused -> fixed in this PR

Build profile diff was still queued when this was written, so it is not covered here.

Both build failures are one omission, not a wrong setting name. The new block at
TTLAggregationAlgorithm.cpp:33-36 declares
extern const MergeTreeSettingsUInt64 ttl_group_by_unsorted_max_bytes_before_external_group_by;
and :104 consumes it as (*storage_.getSettings())[MergeTreeSetting::ttl_group_by_unsorted_...],
but the file's includes (:1-11) carry <Core/Settings.h> and not
<Storages/MergeTree/MergeTreeSettings.h>, so neither the MergeTreeSettingsUInt64 alias nor the
MergeTreeSettings::operator[] overload is visible in that translation unit. The setting itself is
declared correctly at MergeTreeSettings.cpp:1816 at this head, so the remedy is to add the include;
src/Storages/MergeTree/TTLResortUtils.cpp in this same diff is the shape to copy. For the style
failure, the checker asks for "the analyzer" or "Analyzer" since the analyzer has been the default
since 24.3.

All three jobs concluded success at the two preceding heads b2b5033c164d and 0ac5ad8294d8
(171 check-runs each), so the regression is confined to the commits pushed after that green run.

Session id: cron:our-pr-ci-monitor:20260819-190000

… bound TTLAggregationAlgorithm.cpp reads ttl_group_by_unsorted_max_bytes_before_external_group_by, but the file only included Core/Settings.h. Neither the MergeTreeSettingsUInt64 alias nor the MergeTreeSettings subscript operator was visible, so the extern declaration did not parse and the subscript at the call site was rejected. Both are generated by MergeTreeSettings.h, which the sibling TTLResortUtils.cpp already includes for the same reason. Adding it in the .cpp keeps Settings.h and MergeTreeSettings.h out of headers, which the settings style check requires. The clang-tidy init-variables report on the same declaration was downstream of the unparsed type and goes away with the include. The query level max_bytes_before_external_group_by and max_bytes_ratio_before_external_group_by externs are dropped: the per-table setting replaced both, leaving no consumer in this translation unit. The style check rejects the wording "new analyzer", since the analyzer has been enabled by default since 24.3. Drop the adjective in the TTLResortUtils.cpp comment.

groeneai · 2026-08-19T20:02:30Z

Fixed the CI break from 3614f3caa8ee in e62720a0173651e, keeping the per-table bound as designed.

TTLAggregationAlgorithm.cpp included only Core/Settings.h, so neither the MergeTreeSettingsUInt64 alias nor the MergeTreeSettings subscript operator was visible and the extern for ttl_group_by_unsorted_max_bytes_before_external_group_by did not parse. Added <Storages/MergeTree/MergeTreeSettings.h>, matching what TTLResortUtils.cpp already does; in the .cpp, so the settings style check that forbids it in headers stays satisfied. The cppcoreguidelines-init-variables report on the same declaration was downstream of the unparsed type and is gone.

Verified both directions: removing the include reproduces the two Fast test / Build (arm_tidy) errors verbatim, and with it the translation unit compiles clean.

Also dropped the now-unused max_bytes_before_external_group_by and max_bytes_ratio_before_external_group_by externs, which the per-table setting replaced, and removed "new" from the analyzer comment in TTLResortUtils.cpp:51 for the style check ("new analyzer" is rejected since the analyzer became the default in 24.3). The wording is the only change to that comment.

…History ttl_group_by_unsorted_max_bytes_before_external_group_by was added to MergeTreeSettings without a corresponding history entry, so 03999_stateless_settings_history reported PLEASE ADD and the compatibility mechanism could not restore the pre-26.8 value for it. The recorded new_value equals the compiled default (268435456), which is what the test's value-drift arm compares against. Verified both directions on a debug build: with the entry the test output is empty (matches the reference); removing only this line and rebuilding reproduces the PLEASE ADD line verbatim.

groeneai · 2026-08-19T21:40:49Z

Pushed f14aa64c for the first of the two Fast test failures at e62720a0.

03999_stateless_settings_history - fixed. ttl_group_by_unsorted_max_bytes_before_external_group_by had no SettingsChangesHistory.cpp row, so the test printed PLEASE ADD and compatibility could not restore the pre-26.8 value. Recorded in the 26.8 MergeTree block with new_value = 268435456, matching the compiled default (that is what the test's value-drift arm compares against). Verified both ways on a debug build: with the row the output is empty; removing only that line and rebuilding reproduces the PLEASE ADD line verbatim.

04327 / t_unsorted_group_by_spill - I have not touched this one, because I think it needs a decision from you rather than a patch from me. Two separate things:

The DDL is rejected before anything runs: ORDER BY a with a second GROUP BY b hits TTLDescription.cpp:1287, Code: 450 ... GROUP BY key should be a prefix of primary key b a. That validator is untouched by this branch, so this is not a regression here. GROUP BY a SET a = max(b), ... GROUP BY a SET b = max(b) is accepted and still puts the later clause on the unsorted path, and it produces your expected unsorted group by count 100.
With that fixed, the spill assertion still returns 0, and I do not think any test-side change can make it pass. All three spill sites in Aggregator::executeOnBlock need either result.isTwoLevel() (Aggregator.cpp:1988, :4355) or the adaptive aggregator (:1919, :1952). TTLAggregationAlgorithm passes enable_adaptive_aggregator = false and both group_by_two_level_threshold and ..._threshold_bytes as 0, and worthConvertToTwoLevel (:104) is false when both are 0 - so the table never converts and the spill branch is dead whatever the setting says. Measured with 200000 distinct keys at a 1-byte limit: 0 spill parts, merge peak 37.6 MB.

Control isolating the mechanism to the thresholds (ordinary query, max_bytes_before_external_group_by=1, only the thresholds vary):

thresholds	ExternalAggregationWritePart	peak
default (nonzero)	7	84.85 MiB
both 0 (the TTL aggregator's shape)	0	320.15 MiB

So making the new setting effective means enabling two-level conversion for this aggregator, which changes output order - the property this PR exists to protect - and the adjacent comment states an exactness guarantee. That is your design call on your lines, so I would rather ask than guess: enable two-level for the unsorted path, or drop the spill assertion and keep the setting as a documented no-op for now? Happy to implement whichever you pick. Note the two 0 threshold arguments are pre-existing (6cc668bd), not from 3614f3ca.

groeneai · 2026-08-19T23:58:53Z

CI finish ledger - `f14aa64`

Every failure below has an owner: a fixing PR (ours or external), or a full-effort fix task
whose fixing-PR link will be posted here when it opens. Only CH Inc sync is exempt.

Check / test	Reason	Owner / fixing PR
Fast test / 04327_ttl_group_by_set_sort_key_resort	`Code: 450 BAD_TTL_EXPRESSION`, then an unsatisfiable spill assertion	design question posted at #108550 (issuecomment-5348331393), awaiting your call
Finish Workflow / Post Hooks	cascade: `new_tests_check.py` sees the Bugfix validation jobs `DROPPED` because `Fast test` failed	resolves with the row above; no separate owner

03999_stateless_settings_history, the other red at e62720a0, is fixed at this head by the
SettingsChangesHistory.cpp row for ttl_group_by_unsorted_max_bytes_before_external_group_by.

The Finish Workflow red is not independent. The Bugfix validation (functional tests, amd64)
node on this head carries ext.notes = "Dropped due to previous failure [Fast test]", so
praktika names Fast test as the cause itself; the hook then reports "no per-arch Bugfix
Validation job validated the bug" because all four per-arch jobs are DROPPED or SKIPPED.
Answer the 04327 question and this row clears with it.

Coverage at this head is therefore 4 OK and 149 DROPPED of 172, all downstream of the one
Fast test failure. Config Workflow and Style check are both green, so this is a genuine
fail-fast cascade rather than a dropped matrix.

Session id: cron:our-pr-ci-monitor:20260819-233000

The `ttl_group_by_unsorted_max_bytes_before_external_group_by` bound was dead code: `Aggregator` can spill only a two-level hash table, and `TTLAggregationAlgorithm` passed both two-level thresholds as 0, so the table never converted and no spill site was reachable. Worse, if a spill had ever happened, `finalizeAggregates` used `convertToChunks` on the in-memory state only, so the spilled data would have been silently dropped. Now the two-level conversion threshold is tied to the external bound (and the bound applies only to the unsorted path - the sorted path flushes per key run and never outgrows one group), and `finalizeAggregates` merges the spilled generations back bucket by bucket via `Aggregator::mergeBlocks`, flushing the in-memory remainder to disk first, so the merge-back memory high-water mark is one bucket rather than the whole part - the same scheme the external aggregation of a query uses. Also fix the `t_unsorted_group_by_spill` scenario in `04327_ttl_group_by_set_sort_key_resort`: its DDL declared `GROUP BY b` under `ORDER BY a`, which `TTLDescription::getTTLFromAST` rejects with `BAD_TTL_EXPRESSION` before anything runs. The valid shape that still forces the later TTL onto the unsorted path is an earlier `SET a = max(b)` rewriting the key itself; the spill assertion is now genuinely satisfiable (verified locally: 200000 distinct keys at a 1-byte bound produce 25 spill generations and exactly preserve count, key set and sums; the written part passes CHECK TABLE and is sorted). CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=108550&sha=f14aa64c51964620cd0cd6f2d473fa6c99081514&name_0=PR&name_1=Fast%20test PR: ClickHouse#108550 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

alexey-milovidov · 2026-08-21T18:37:16Z

🕵 Answering the 04327 / t_unsorted_group_by_spill question from the comment above: the spill must be real, not just the setting — pushed 7c406d64ef84 implementing it.

DDL: fixed as you suggested — GROUP BY a SET a = max(b), ... GROUP BY a SET b = max(b). The earlier SET rewriting the key itself is the valid shape that still puts the later clause on the unsorted path, and it produces the expected unsorted group by count 100.
Spill: your analysis was right — all Aggregator spill sites need a two-level table, and with both thresholds at 0 the bound was dead letter. And it was worse than a dead setting: finalizeAggregates used convertToChunks on the in-memory state only, so if the spill had ever fired, the spilled groups would have been silently dropped from the written part. The fix does both halves:
- group_by_two_level_threshold_bytes is tied to ttl_group_by_unsorted_max_bytes_before_external_group_by (and the bound now applies only to the unsorted path — the sorted path flushes per key run and never outgrows one group), so the conversion happens no later than the bound and the spill sites are reachable;
- finalizeAggregates now detects temporary data, flushes the in-memory remainder to disk too, and merges the spilled generations back bucket by bucket via Aggregator::mergeBlocks — the same scheme a query's external aggregation uses, so the merge-back high-water mark is one bucket, not the whole part.

Verified locally on this head: 04327 passes including unsorted group by spilled 1; a harder carrier (200000 distinct keys at a 1-byte bound) produced 25 spill generations and preserved count, key set and both sums exactly; the written part passes CHECK TABLE and is sorted by the rewritten key. 04511/04661/04691/04692/03545/04044/03999 all green against the same server.

groeneai · 2026-08-21T19:43:18Z

Verified both halves at 7c406d64ef84, and the spill is real rather than just reachable.

Bucket-wise merge-back preserves rows. The decisive check is spilled-vs-not on the same input: ttl_group_by_unsorted_max_bytes_before_external_group_by = 1 against = 0, 50000 distinct keys through the unsorted path, EXCEPT in both directions returns 0 rows either way and the row hashes are equal. Same at scale: 200000 keys over 600000 rows produced 74 spill generations and still gave count = 200000, uniqExact(a) = 200000, an exact sum, a clean CHECK TABLE, and a part physically ordered by the rewritten key. Non-trivial states survive the serialize/deserialize round trip too (uniqExact, groupUniqArray identical across the two arms), and so does the nested-wrapper coercion that now lives in the shared appendAggregatedBlock: the STID 2508-3698 shape (Tuple(LowCardinality(UInt32), UInt32) into Tuple(UInt32, UInt32)) plus a LowCardinality(String) SET target come out identical on the spilled branch, at 18 generations.

The bound is live, not just the spill sites. 400000 keys with a 200-byte payload: merge peak memory goes 738.27 MiB at = 0 to 518.57 MiB at = 4Mi with 50 generations, results identical. Before tying group_by_two_level_threshold_bytes to it the setting could not move that number at all.

Scoping the bound to the unsorted path holds. Measured rather than assumed, since it is the load-bearing assumption: with the bound at 1, the sorted path emits 0 spill generations even at 400000 distinct keys. Holding the row count and output width fixed and varying only the key count 1000 to 20000, peak memory goes 4.08 MiB to 8.19 MiB, i.e. it tracks the output rows and not the hash table, which is what flushing per key run predicts.

All five tests on the branch pass against this head (04327, 04511, 04661, 04691, 04692), including unsorted group by spilled 1.

Two gaps in the PR description, which I will fix:

the changelog entry names ttl_resort_max_bytes_before_external_sort but not ttl_group_by_unsorted_max_bytes_before_external_group_by, which is also new here and also ships with a non-zero default (256 MiB);
the finalizeAggregates defect you found is not described. It belongs in the description rather than a separate issue, since it is inside this PR's own diff: the spill sites were unreachable before this change, so the dropped-groups path was never live on any release.

groeneai · 2026-08-23T05:22:55Z

CI finish ledger - `7c406d6`

Every failure below has an owner: a fixing PR (mine or external), or a full-effort fix task
whose fixing-PR link will be posted here when it opens.

Check / test	Reason	Owner / fixing PR
Upgrade check (amd_release) / `Unknown job error`, `Check failed`	infra: the previous-release `git clone` died on a network error with no retry, so no `test_results.tsv` was produced	#115935 (mine, open)

Detail on that row:

Both reported leaves are one event. job.log shows
git clone ... --branch=v26.7.5.10-stable ... previous_release_repository reaching 32 percent of
68,627 objects and then failing with error: RPC failed; curl 56 GnuTLS recv error (-54),
fetch-pack: unexpected disconnect while reading sideband packet, fatal: early EOF and
ERROR: command failed after 1/1 attempt(s), exit code: 128. Because the runner exits there, no
test_results.tsv is ever written, which is the Unknown job error
(Cannot parse test_results.tsv), and Check failed with exit code 128 is the same 128 restated.
No upgrade test ran, so there is no test verdict on this lane for this commit.
Retry the previous-release clone in the upgrade check #115935 ("Retry the previous-release clone in the upgrade check") is the owner: it wraps that
clone in run_with_retry 3, adds http.lowSpeedLimit and http.lowSpeedTime so a stalled
transfer is abandoned rather than hanging, removes the .git-only directory a killed clone leaves
behind, and writes a test_results.tsv plus check_status.tsv when the clone genuinely cannot be
completed. The 1/1 attempt(s) in the log above is precisely the missing retry it adds.
Not caused by this PR. Over 14 days the Cannot parse test_results.tsv signature appears 80
times across 63 distinct pull requests and 6 different check names, onset
2026-08-12 12:58:03Z, with a negative control on a nonexistent needle returning 0. This diff is
the TTL GROUP BY ... SET resort work: 13 source files and 12 stateless test files, with 0
occurrences of upgrade_runner, docker_scripts or previous_release against a positive control
of 775 for TTL. Nothing in it can reach the upgrade runner's clone step.
No master merge was made: the owner is open, so there is nothing to import yet.
⚠️ The breadth needle was taken from a row I had already read out of the column, not from the log.
My own row shows Cannot parse test_results.tsv at position 1 of test_context_raw but
early EOF at position 0, so keying the query on the more specific clone error would have returned
an unmeasured zero rather than a real one.

Config Workflow, Style check and Finish Workflow all completed successfully, so this is a real
verdict rather than a dropped run, and of 172 check-run names this lane is the only failure.

Session id: cron:our-pr-ci-monitor:20260823-050000

…oup-by-set-sort-key-resort

… limit The `CH Inc sync` job reported `04327_ttl_group_by_set_sort_key_resort` as `FAIL` with `Test runs too long` in `Stateless tests (amd_asan_ubsan, flaky check, s3 storage, meta in keeper)`: 181.58 s and 180.44 s against the 180 s limit. The file had grown to 19 independent scenarios, each doing its own `CREATE`/`INSERT`/`OPTIMIZE FINAL`/`DROP`, and with metadata in Keeper and data on S3 the per-DDL round trips dominate the run time (the whole file takes about 8 s locally, so that configuration inflates it roughly 22x). Split into three files with nothing dropped and no scenario weakened: - `04327_ttl_group_by_set_sort_key_resort` keeps the plain sorting-key cases (`Float64`, `String`, `LowCardinality`, subcolumn, nested-wrapper coercion, the `MATERIALIZE TTL` mutation path, the skip-index rebuild, and the non-sort-key control); - `05044_ttl_group_by_set_materialized_resort` holds the cases where the sorting key is a `MATERIALIZED` column that must be recomputed after the `SET` (direct, mutation, transitive chain, tuple chain, tuple subcolumn source, and the `EPHEMERAL`-sourced column); - `05045_ttl_group_by_set_future_and_folded` holds the non-firing future-TTL cases, the const-folded recompute cases, and the unsorted `GROUP BY` spill case. The 48 reference lines are partitioned 25/13/10, matching the local split of the run time (about 39/21/30 percent), so the slowest part should land near 70 s in that configuration. Verified locally on an aarch64 release build: all three files pass, as do `04511_ttl_multi_group_by_set_rewrites_key`, `04661_ttl_group_by_set_resort_external_sort_spill`, `04691_ttl_multi_group_by_set_materialized_recompute`, `04692_ttl_multi_group_by_set_chained_fire`, `03545_number_of_rows_in_ttltransform` and `04044_mutation_ephemeral_materialized`.

alexey-milovidov · 2026-08-25T20:37:15Z

🕵 Pushed 5759f112c838 — a master merge plus a fix for the one real red on 7c406d64ef84.

CH Inc sync = tests failed (5 new, 2 known) — one of the two failing lanes was genuinely ours.

Stateless tests (amd_asan_ubsan, flaky check, s3 storage, meta in keeper): 04327_ttl_group_by_set_sort_key_resort FAIL with Test runs too long — 181.58 s and 180.44 s against the 180 s limit (the flaky check runs the new/changed test repeatedly, hence the 5 leaves plus Too many test failures). This is the same shape as the earlier 04511 split: the file had grown to 19 independent scenarios, each with its own CREATE/INSERT/OPTIMIZE FINAL/DROP, and with metadata in Keeper and data on S3 the per-DDL round trips dominate. Locally the whole file runs in about 8 s, so that configuration inflates it roughly 22x.

Fixed by splitting into three files with nothing dropped and no scenario weakened: 04327_ttl_group_by_set_sort_key_resort keeps the plain sorting-key cases (Float64, String, LowCardinality, subcolumn, nested-wrapper coercion, the MATERIALIZE TTL mutation path, the skip-index rebuild, the non-sort-key control); 05044_ttl_group_by_set_materialized_resort holds the cases where the sorting key is a MATERIALIZED column recomputed after the SET (direct, mutation, transitive chain, tuple chain, tuple-subcolumn source, EPHEMERAL-sourced); 05045_ttl_group_by_set_future_and_folded holds the non-firing future-TTL cases, the const-folded recompute cases and the unsorted GROUP BY spill case. The 48 reference lines partition 25/13/10, matching the local run-time split of about 39/21/30 percent, so the slowest part should land near 70 s in that lane.
Integration tests (amd_asan_ubsan, db disk, old analyzer, 4/7): test_shared_merge_tree_backup_from_snapshot/test.py::test_resumable_backup_keeps_progress_when_open_cannot_remove_its_lock. Not ours — a SharedMergeTree backup-snapshot lock test, in private-only code this diff does not reach (the diff is the TTL GROUP BY ... SET resort work in src/Processors/TTL/*, MergeTask, MutateTask and TTLResortUtils, plus stateless tests).

Upgrade check (amd_release) was the infra clone failure tracked in the previous ledger; its owner #115935 merged into master on 2026-08-23, so this push merges master (zero conflicts) to pull the retry in.

Verified locally on an aarch64 release build: the three split files pass, as do 04511_ttl_multi_group_by_set_rewrites_key, 04661_ttl_group_by_set_resort_external_sort_spill, 04691_ttl_multi_group_by_set_materialized_recompute, 04692_ttl_multi_group_by_set_chained_fire, 03545_number_of_rows_in_ttltransform and 04044_mutation_ephemeral_materialized.

The AI review's remaining Major is the reverse-direction ttl_finished finding, already dismissed in-thread with measured evidence: it is a pre-existing master gap that this diff neither introduces nor widens, and turning the warning into a throw is a design call that would make today-valid tables unmergeable. All review threads are resolved.

The master merge 589b55e rolled the version to 26.9, so the records for `ttl_resort_max_bytes_before_external_sort` and `ttl_group_by_unsorted_max_bytes_before_external_group_by` were left in the 26.8 block of a release that never had them. The `settings_changes_history` style check rejects that, and `compatibility` needs the same thing: both settings first appear in 26.9, so `compatibility = '26.8'` has to hand out the old value 0. The reason text goes from pre-26.8 to pre-26.9 for the same reason. Recorded values are unchanged, so the newest recorded value per setting is still 268435456 and `03999_stateless_settings_history`, which compares the compiled default against `argMax(new_value, (version, index))`, is unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

groeneai · 2026-08-26T06:02:03Z

Pushed 4df059e3fae42f4: the two new merge-tree settings moved from the 26.8 history block to 26.9. The master merge 589b55e89a58 rolled the version, so they were left recorded against a release that never had them, and the settings_changes_history style check rejected exactly that. It gated the whole matrix, so 150 jobs were dropped.

Values are unchanged (268435456), so the newest recorded value per setting is the same and 03999_stateless_settings_history is unaffected; the reason text now reads pre-26.9. Verified by driving the real check locally: it reproduces the CI message before the move and returns empty after.

…oup-by-set-sort-key-resort

… new master test `master` added `05044_authentication_method_grants_execute_as_source_rights` in the same merge, so the split files move to `05046`/`05047`.

alexey-milovidov · 2026-08-27T22:29:06Z

🕵 Pushed d82cc4434dae — the PR was CONFLICTING against master, so this is a master merge plus one follow-up.

Conflict resolved. git merge origin/master into the branch produced zero textual conflicts; the CONFLICTING state was purely the two days of drift since 589b55e89a58. Merge commit 9e1ecd062a6b.
Test-number collision fixed. That same merge brought in 05044_authentication_method_grants_execute_as_source_rights from master, which collides with the 05044 prefix the 08-25 split had taken. The two split files move up: 05044_ttl_group_by_set_materialized_resort → 05046_ttl_group_by_set_materialized_resort, 05045_ttl_group_by_set_future_and_folded → 05047_ttl_group_by_set_future_and_folded. Content is untouched (pure git mv; neither file refers to its own name).
Settings history is still correct after the merge. master is still 26.9, and both new merge-tree settings (ttl_resort_max_bytes_before_external_sort, ttl_group_by_unsorted_max_bytes_before_external_group_by) stay in the single 26.9 merge_tree_settings_changes_history block that 4df059e3fae4 put them in — no second 26.9 block, no version roll to undo.

Verified locally (aarch64 release, incremental build clean): all seven PR tests plus the neighbours pass — 04327_ttl_group_by_set_sort_key_resort, 04511_ttl_multi_group_by_set_rewrites_key, 04661_ttl_group_by_set_resort_external_sort_spill, 04691_ttl_multi_group_by_set_materialized_recompute, 04692_ttl_multi_group_by_set_chained_fire, 05046_…, 05047_…, 03545_number_of_rows_in_ttltransform, 04044_mutation_ephemeral_materialized, 04691_ttl_expression_rebuild_after_constant_fold, and 03999_stateless_settings_history.

No unresolved review threads on the PR. The one standing AI-review finding (the reverse-direction ttl_finished metadata, TTLTransform.cpp / TTLColumnAlgorithm.cpp) remains refuted with measured evidence and is deliberately not implemented — it is a pre-existing master gap and separate-PR material. Awaiting CI on d82cc4434dae and a human merge.

…SET path A MATERIALIZED column may be defined over an ALIAS column, e.g. `a Int32 ALIAS x + 100, m Int32 MATERIALIZED a + 1`. The four places that analyze such a default here fed it to TreeRewriter against getAllPhysical() plus the EPHEMERAL columns, which by construction excludes ALIAS columns, so `a` could not be resolved and a firing `TTL ... GROUP BY ... SET` failed the merge with UNKNOWN_IDENTIFIER. The mutation path stops with the same error wrapped in Code 341 and then retries forever. Neither needs the alias-backed column to have anything to do with the SET: the dependency map is built over every MATERIALIZED column in the table. An ALIAS is computed on read and never stored, so resolving the name would not help either - the recompute stage would demand a column no part holds. The reference has to be replaced by the expression it stands for, cast to the alias type, which is what MaterializedColumnDependencies::findNode already does for the UPDATE mutation path (04869_materialized_over_alias_column_mutation). Measured on 04691 with the pre-fix and post-fix binaries: the new F3 arm stops the merge with `Missing columns: 'a'` before the fix and reproduces the reference after it. A master binary merges the same table and returns a stale `m`, which the recompute now corrects (109 -> m = 110). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

groeneai · 2026-08-28T07:24:53Z

CI finish ledger - `98b5743`

Every failure below has an owner: a fixing PR (ours or external), or a full-effort fix task
whose fixing-PR link will be posted here when it opens. Only CH Inc sync is exempt.

Check / test	Reason	Owner / fixing PR
Integration tests (arm_binary, distributed plan, 2/4) / `test_storage_nats/test_nats_jet_stream.py::test_nats_restore_failed_connection_without_losses_on_write` (`ClickHouse lost some messages: 77126`)	chronic flaky, not PR-caused: 177 rows over 133 distinct branches and 7 rows on `master` in 30 days, across 12 check flavours, onset 2026-07-30	#115343 (external, open)

The causal link to #115343 was verified rather than assumed: its diff edits this exact test function at tests/integration/test_storage_nats/test_nats_jet_stream.py:1151, the line the failure context cites, alongside the NATS consumer and connection sources, and its own Bugfix validation (integration tests) runs show this test failing against the master binary as that gate requires.

Nothing else is red at this commit: Fast test, Style check, all ten Stress test flavours and the rest of the integration matrix are green, Mergeable Check is success, and Finish Workflow completed successfully.

Session id: cron:our-pr-ci-monitor:20260828-070000

alexey-milovidov added the can be tested Allows running workflows for external contributors label Jun 25, 2026

clickhouse-gh Bot added the pr-bugfix Pull request with bugfix, not backported by default label Jun 25, 2026

clickhouse-gh Bot reviewed Jun 25, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/MergeTask.cpp Outdated

cv4g self-assigned this Jun 30, 2026

clickhouse-gh Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/MutateTask.cpp Outdated

clickhouse-gh Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/MergeTask.cpp

groeneai mentioned this pull request Jun 30, 2026

Retry integration-test image pre-pull with exponential backoff #108975

Closed

groeneai mentioned this pull request Jul 1, 2026

Fix use-after-free reading String .size subcolumn across granules #109020

Closed

groeneai and others added 4 commits July 11, 2026 11:22

groeneai force-pushed the groeneai/fix-ttl-group-by-set-sort-key-resort branch from f977bca to a0d5af7 Compare July 11, 2026 11:37

clickhouse-gh Bot reviewed Jul 11, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/TTLResortUtils.cpp

clickhouse-gh Bot reviewed Jul 11, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/TTLResortUtils.cpp Outdated

Comment thread src/Storages/MergeTree/TTLResortUtils.cpp Outdated

clickhouse-gh Bot reviewed Jul 11, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/MergeTask.cpp Outdated

Comment thread src/Storages/MergeTree/TTLResortUtils.cpp Outdated

groeneai added the groeneai-origin-request PR origin: a maintainer pinged or directed groeneai label Aug 19, 2026

Merge remote-tracking branch 'origin/master' into HEAD

0ac5ad8

clickhouse-gh Bot reviewed Aug 19, 2026

View reviewed changes

Comment thread src/Processors/TTL/TTLAggregationAlgorithm.cpp

Comment thread src/Storages/MergeTree/TTLResortUtils.cpp

alexey-milovidov added 2 commits August 19, 2026 18:19

Merge remote-tracking branch 'origin/master' into HEAD

6170600

Address TTL GROUP BY review feedback

3614f3c

groeneai mentioned this pull request Aug 23, 2026

Require a join subquery alias only when it removes a real ambiguity #109368

Open

1 task

alexey-milovidov added 2 commits August 25, 2026 20:10

Merge remote-tracking branch 'origin/master' into groeneai/fix-ttl-gr…

589b55e

…oup-by-set-sort-key-resort

alexey-milovidov added 2 commits August 27, 2026 21:05

Merge remote-tracking branch 'origin/master' into groeneai/fix-ttl-gr…

9e1ecd0

…oup-by-set-sort-key-resort

Renumber the two split TTL GROUP BY tests to avoid a collision with a…

d82cc44

… new master test `master` added `05044_authentication_method_grants_execute_as_source_rights` in the same merge, so the split files move to `05046`/`05047`.

clickhouse-gh Bot reviewed Aug 27, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/TTLResortUtils.cpp

Uh oh!

Conversation

groeneai commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Description

Uh oh!

groeneai commented Jun 25, 2026

Uh oh!

groeneai commented Jun 25, 2026

Uh oh!

clickhouse-gh Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI Review

Summary

PR Metadata

Final Verdict

LLVM Coverage Report

Uh oh!

Uh oh!

groeneai commented Jun 25, 2026

Uh oh!

groeneai commented Jun 26, 2026

CI finish ledger — e779631

Uh oh!

cv4g commented Jun 30, 2026

Uh oh!

groeneai commented Jun 30, 2026

Uh oh!

Uh oh!

groeneai commented Jun 30, 2026

Uh oh!

Uh oh!

groeneai commented Jun 30, 2026

CI finish ledger — f977bca

Uh oh!

groeneai commented Jun 30, 2026

Uh oh!

Uh oh!

groeneai commented Jul 11, 2026

Uh oh!

Uh oh!

Uh oh!

groeneai commented Jul 11, 2026

Uh oh!

Uh oh!

Uh oh!

groeneai commented Aug 19, 2026

CI finish ledger - 7c031d1

Uh oh!

groeneai commented Aug 19, 2026

CI finish ledger - b2b5033

Uh oh!

Uh oh!

Uh oh!

groeneai commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI finish ledger - 3614f3c

Uh oh!

groeneai commented Aug 19, 2026

Uh oh!

groeneai commented Aug 19, 2026

Uh oh!

groeneai commented Aug 19, 2026

CI finish ledger - f14aa64

Uh oh!

alexey-milovidov commented Aug 21, 2026

Uh oh!

groeneai commented Aug 21, 2026

Uh oh!

groeneai commented Aug 23, 2026

CI finish ledger - 7c406d6

Uh oh!

alexey-milovidov commented Aug 25, 2026

Uh oh!

groeneai commented Aug 26, 2026

Uh oh!

alexey-milovidov commented Aug 27, 2026

Uh oh!

groeneai commented Jun 25, 2026 •

edited

Loading

clickhouse-gh Bot commented Jun 25, 2026 •

edited

Loading

CI finish ledger — `e779631`

CI finish ledger — `f977bca`

CI finish ledger - `7c031d1`

CI finish ledger - `b2b5033`

groeneai commented Aug 19, 2026 •

edited

Loading

CI finish ledger - `3614f3c`

CI finish ledger - `f14aa64`

CI finish ledger - `7c406d6`

CI finish ledger - `98b5743`