Push join key filters into MergeTree index during recursive CTE evaluation by alexey-milovidov · Pull Request #97254 · ClickHouse/ClickHouse

alexey-milovidov · 2026-02-18T07:38:36Z

When a recursive CTE joins against a MergeTree table, each recursion step previously scanned the entire table, because the join condition (e.g. ON e.from_id = t.current_id) was not pushed into MergeTree's key condition.

This change analyzes the recursive query's join tree to find equi-join conditions between the CTE working table and real tables. Before each step it reads the join-key values from the working table and injects them as an IN (...) predicate directly into the WHERE clause of each recursive QueryNode that joins against the CTE (the original WHERE/HAVING/QUALIFY clauses are snapshotted at construction and restored after each step, so nothing accumulates across steps). The planner then lowers this predicate into ReadFromMergeTree's key condition. The injected set is bounded by the new setting recursive_cte_max_in_filter_cardinality, and injection fails closed to a plain scan whenever it could otherwise change results — when the generated set would exceed max_rows_in_set/max_bytes_in_set, cannot be materialized under max_memory_usage, or the IN predicate cannot be resolved for the join-key type. Parallel replicas are disabled for the recursive step queries to avoid stale cached GLOBAL JOIN tables returning incorrect results; the forcing mode allow_experimental_parallel_reading_from_replicas = 2 raises SUPPORT_IS_DISABLED for the recursive part rather than silently downgrading.

For a table with ~1M rows and 10 recursion steps, read_rows drops from ~10M to ~120.

Closes: #75026

Changelog category (leave one):

Performance Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

WITH RECURSIVE queries that use ON or comma equi-joins against MergeTree tables can now use the primary key index, reducing the number of rows read per recursion step. The optimization is controlled by the new setting recursive_cte_max_in_filter_cardinality.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

No separate documentation page is needed — this is a transparent performance improvement. The only user-visible addition is the setting recursive_cte_max_in_filter_cardinality, which is documented at the source level in src/Core/Settings.cpp.

Note

Medium Risk
Touches recursive CTE execution and query settings, and dynamically injects additional_table_filters, which could affect correctness/performance for some JOIN patterns despite being scoped to recursive steps.

Overview
Improves recursive CTE performance by detecting equi-join conditions between the CTE working table and real tables, then pushing join-key values into additional_table_filters (built as IN (...) predicates) on each recursive step so MergeTree reads can use the primary-key index; user-provided additional_table_filters are merged rather than overwritten.

Also disables parallel replicas for recursive CTE step queries to avoid incorrect results from reused cached GLOBAL JOIN tables, and adds a stateless test (03924_recursive_cte_join_index) asserting both correct output and low read_rows for explicit INNER JOIN and comma-join forms.

^{Written by Cursor Bugbot for commit 37f2618. This will update automatically on new commits. Configure here.}

Workflow [PR]
Sync PR [sync-upstream/pr/97254]

…ation When a recursive CTE joins against a MergeTree table, each recursion step previously scanned the entire table because the join condition was not pushed into the MergeTree key condition. This change analyzes the recursive query's join tree to find equi-join conditions between the CTE table and real tables. Before each recursive step, it reads the join key values from the working (CTE) table and injects them as `additional_table_filters` on the query context. The existing planner infrastructure applies the filter as a `FilterStep`, which the optimizer pushes into `ReadFromMergeTree`'s key condition. For a table with ~1M rows and 10 recursion steps, this reduces `read_rows` from ~10M to ~120. Closes #75026 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

clickhouse-gh · 2026-02-18T07:39:12Z

Workflow [PR], commit [2d2d38b]

Summary: ❌

Performance Comparison: Performance dashboard

job_name	test_name	status	info
Stateless tests (amd_llvm_coverage, old analyzer, s3 storage, DBReplicated, WasmEdge, parallel, 2/3)		FAIL
	04926_recursive_cte_merge_view_pruning	FAIL	cidb
Stress test (arm_debug)		FAIL
	Logical error: Not-ready Set is passed as the second argument for function 'A (STID: 0250-4e52)	FAIL	cidb

AI Review

Summary

This PR adds recursive-CTE join-key pushdown into MergeTree index conditions and a large amount of fail-closed logic around parallel replicas and generated IN filters. The remaining problems are both in the final parallel-replica eligibility rewrite: one previously fixed leftmost-VIEW regression is still present in the current code, and the view(...) table-function path still silently downgrades a genuinely forced parallel-replica query.

Findings

⚠️ Majors

[src/Processors/Sources/RecursiveCTESource.cpp:1432] rewrite_context still zeroes every legacy recursive context even when context_may_engage_parallel_replicas is empty. That reintroduces the leftmost-VIEW regression discussed on the existing review thread: shapes that the outer planner keeps local still lose their safe inner-view parallel-replica read because StorageView::getViewContext inherits allow_experimental_parallel_reading_from_replicas = 0 from the rewritten recursive context. The query still runs, but it silently falls back to plain execution on a path the non-recursive planner would keep parallelized. Suggested fix: return early from rewrite_context when the current legacy recursive context cannot engage parallel replicas at all, instead of copying it and forcing the setting to 0.
[src/Processors/Sources/RecursiveCTESource.cpp:756] The TableFunctionNode path never mirrors the outer storage-level eligibility rule that the TableNode path uses. For view(SELECT * FROM mt) with parallel_replicas_allow_view_over_mergetree = 1, mayEngageParallelReplicasForView intentionally zeroes the inner view context because the outer planner can unwrap the table function directly to the MergeTree, so this code records no engagement and mode 2 is silently downgraded instead of throwing. That breaks the documented force-or-throw contract for an actually eligible recursive step. Suggested fix: add the same outer StorageView/MergeTree eligibility handling here that canUseTableForParallelReplicas provides for TableNode, and keep mayEngageParallelReplicasForView only for the cases where the outer join tree is ineligible but the inner view can still use parallel replicas.

Tests

⚠️ Add a focused forced-mode regression for FROM view(SELECT * FROM edges) AS e LEFT JOIN rec AS t ... with parallel_replicas_allow_view_over_mergetree = 1; it should raise SUPPORT_IS_DISABLED rather than succeed.
⚠️ Strengthen the leftmost-view "run plainly" regressions (VIEW / view(...) over Distributed) to assert whether the inner read still used parallel replicas via ParallelReplicasUsedCount, not just that the final result is correct. The current tests would miss the unconditional context-zeroing regression above.

Final Verdict

Status: ⚠️ Request changes

Minimum required actions:

Fix the unconditional recursive-context rewrite so leftmost VIEW / view(...) cases that cannot engage parallel replicas keep their existing safe inner-view execution path.
Make the TableFunctionNode StorageView path enforce the same forced parallel-replica contract as the TableNode path for eligible view(...) over MergeTree.
Add the focused regressions above so both contracts are proven.

LLVM Coverage Report

Measured on commit 2d2d38b.

Metric	Baseline	Current	Δ
Lines	88.40%	88.40%	+0.00%
Functions	92.00%	92.00%	+0.00%
Branches	80.70%	80.70%	+0.00%

Changed lines: Changed C/C++ lines covered: 464/576 (80.56%) · Uncovered code

Full report · Diff report

…YSTEM FLUSH LOGS` Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

When parallel replicas is enabled, JOINs are rewritten to GLOBAL JOINs and the right-side subquery is materialized into a cached external table keyed by tree hash. Since the recursive CTE temporary table has the same tree structure across steps (only the data changes), the hash stays identical and stale cached data is reused, producing wrong results. Also fix the test to use a fixed `index_granularity` so the `read_rows` threshold check is stable across randomized MergeTree settings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

alexey-milovidov · 2026-02-19T14:46:04Z

Fix for ParallelReplicas CI failure

Root cause: When parallel replicas is enabled, JOINs are rewritten to GLOBAL JOINs by rewriteJoinToGlobalJoin. The right-side subquery (the CTE temporary table) is then materialized into a cached external table by executeSubqueryNode in buildQueryTreeForShard.cpp, using the subquery's tree hash as the cache key (_data_{hash}).

Since the recursive CTE temporary table has the same tree structure across all recursive steps (only the data changes between steps), the hash stays identical. On step 2+, the cache returns stale data from step 1, causing the JOIN to find no matches and the recursion to terminate early (producing 1, 2 instead of 1, 2, ..., 10).

Fix: Disable parallel replicas for the recursive CTE step queries by setting allow_experimental_parallel_reading_from_replicas = 0 on the recursive query context. This prevents the GLOBAL JOIN rewrite entirely for recursive steps, while still allowing the additional_table_filters optimization to work. The non-recursive (initial) step is unaffected since its query node has its own context.

Also fixed index_granularity in the test to 8192 so the read_rows < 100000 threshold is stable when the test runner randomizes MergeTree settings.

…index-optimization

Address review feedback: instead of overwriting the `additional_table_filters` setting on each recursive step, save the original user-specified value at construction time and merge CTE-derived filters with it. For tables that appear in both, filters are combined with AND. User-specified filters for unrelated tables are preserved. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

asad-awadia · 2026-02-28T01:39:13Z

Really interested in this optimization. I am traversing trees using recursive CTEs and it never finishes just continues processing rows and chewing up memory [300 million rows in the table].

Any idea when this might land? and will this be used if the join condition is on a column that is not in the primary/order-by key?

@alexey-milovidov

alexey-milovidov · 2026-03-05T22:20:26Z

@asad-awadia, a high chance it will be in 26.3 LTS.

and will this be used if the join condition is on a column that is not in the primary/order-by key?

It's likely it will be used for any indices, but let's see...

alexey-milovidov · 2026-03-05T22:22:04Z

The implementation is a bit wacky - need more thoughts in review...

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

geldot · 2026-03-05T22:34:13Z

The implementation is a bit wacky - need more thoughts in review...

My sense is that the approach is obvious technical debt but it's not user-facing so could be improved later on the go. I think it'd be OK to land something like this but there should be a concrete plan to replace it with a dynamic planner stage, so it ends up sharing implementation with existing parts of the analyzer. I think the stage would benefit potential MV optimizations, distributed queries, dynamic push down to other engines etc., which are all variations on partially runtime constructed execution plans.

…index-optimization

- Deduplicate join key values using `std::set` before building the IN filter expression (review thread by cursor) - Add `MAX_IN_FILTER_CARDINALITY` (10000) limit: if the working table has more distinct values than this, skip the optimization for that step and fall back to unfiltered scans, avoiding unbounded SQL expressions that could exceed `max_query_size` (review thread by clickhouse-gh) - Return `std::nullopt` from `readColumnValuesFromMemoryStorage` to signal cardinality overflow, handled gracefully in `buildAdditionalTableFiltersForRecursiveStep` Note: the concern about overwriting user-specified `additional_table_filters` (review thread by geldot) was already addressed — `original_additional_table_filters` is saved at construction time and merged on each step via `buildAdditionalTableFiltersForRecursiveStep`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…index-optimization

- Remove redundant `static` from `MAX_IN_FILTER_CARDINALITY` inside anonymous namespace (fixes `readability-static-definition-in-anonymous-namespace` clang-tidy error in arm_tidy build). - Match user-specified `additional_table_filters` by both full name (`db.table`) and short name (`table` when current database matches), mirroring the matching logic in `PlannerJoinTree::buildAdditionalFiltersIfNeeded`. Previously, user filters keyed by short table name would not be found during merge and could be silently shadowed by the CTE-generated filter, since `PlannerJoinTree` takes the first matching entry and breaks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…index-optimization

…ight

clickhouse-gh · 2026-08-19T01:44:06Z

+/// what `mayEngageParallelReplicasForRemoteStorage` checks positively. Remote engines that
+/// read directly (`MongoDB`, `MySQL`, ...) never consult the setting and are not eligible.
+///
+/// An ordinary non-inlined `VIEW` is eligible when its inner query is: the view's read


This overstates when the recursive step itself can engage parallel replicas. For an ordinary VIEW, the outer join tree is never a PR candidate: PlannerJoinTree::allowParallelReplicasForJoinTree() returns false as soon as the leftmost table storage is a view (left_table->getStorage()->isView()), so WITH RECURSIVE ... INNER JOIN edges_view AS e ... keeps the outer join local even under allow_experimental_parallel_reading_from_replicas = 2.

StorageView::readImpl may still use parallel replicas inside the view's own inner query, but that inner read never references the recursive working table, so the stale cached GLOBAL JOIN hazard this rewrite is guarding does not arise there. Marking the whole subtree eligible here therefore turns a previously runnable query into SUPPORT_IS_DISABLED (and mode 1 also disables the safe inner-view PR path). This needs to be gated on the same outer join-tree eligibility the planner uses for ordinary VIEW leaves, rather than on the view inner query's PR capability alone.

🕵 Fixed in 5fb7366. The recursive-step preflight now mirrors allowParallelReplicasForJoinTree for an ordinary VIEW in the leftmost join position, so forced mode preserves the safe inner-view parallel-replica path. The regression cases now verify both values of parallel_replicas_allow_view_over_mergetree.

Still present on the current head. The new allowParallelReplicasForJoinTree mirror only suppresses the SUPPORT_IS_DISABLED branch, but rewrite_context still unconditionally forces every legacy recursive context to allow_experimental_parallel_reading_from_replicas = 0 at src/Processors/Sources/RecursiveCTESource.cpp:1381-1383. StorageView::getViewContext decides whether the inner query can keep its parallel-replica path from context->canUseParallelReplicasOnInitiator(), so once the outer recursive context is zeroed the inner edges_view read can no longer use that safe path. In other words, view_pr stopped throwing, but it still falls back to a plain local read instead of preserving the inner-view parallelism this thread was about.

🕵 Fixed in a0b5c77. Contexts whose recursive step cannot engage parallel replicas are now left untouched, preserving an ordinary leftmost VIEW\047s independent inner parallel-replica read.

This is still present on the current head. rewrite_context now suppresses the throw when engagement.any() is false, but it still unconditionally copies the legacy recursive context and sets allow_experimental_parallel_reading_from_replicas = 0 at src/Processors/Sources/RecursiveCTESource.cpp:1432-1434.

That means the leftmost-VIEW case from this thread still loses its safe inner-view parallel-replica path: StorageView::getViewContext inherits the zeroed setting from the rewritten recursive context, so the inner read runs plainly instead of keeping the independent parallel-replica execution that the non-recursive planner would use.

…index-optimization # Conflicts: # ci/jobs/copilot_review_job.py # ci/jobs/scripts/generate_diff_coverage_report.sh # ci/jobs/sqlancer_job.sh # ci/praktika/cidb.py # ci/tests/test_functional_tests_results.py # contrib/antlr4-grammars-cmake/generated/antlr4_grammars/PromQLParser.cpp # contrib/antlr4-grammars-cmake/generated/antlr4_grammars/PromQLParser.h # contrib/antlr4-grammars-cmake/generated/antlr4_grammars/PromQLParser.interp # contrib/antlr4-grammars-cmake/generated/antlr4_grammars/PromQLParserBaseListener.h # contrib/antlr4-grammars-cmake/generated/antlr4_grammars/PromQLParserBaseVisitor.h # contrib/antlr4-grammars-cmake/generated/antlr4_grammars/PromQLParserListener.h # contrib/antlr4-grammars-cmake/generated/antlr4_grammars/PromQLParserVisitor.h # contrib/aws-c-http # docker/keeper/Dockerfile # docker/server/Dockerfile.alpine # programs/client/Client.h # programs/server/webterminal.html # src/AggregateFunctions/AggregateFunctionSequenceMatch.cpp # src/AggregateFunctions/TimeSeries/AggregateFunctionLast2Samples.h # src/AggregateFunctions/TimeSeries/AggregateFunctionTimeSeriesGroupArray.cpp # src/Analyzer/Passes/CrossToInnerJoinPass.cpp # src/Backups/BackupFactory.h # src/Backups/BackupImpl.h # src/Backups/BackupSettings.cpp # src/Backups/BackupSettings.h # src/Backups/IBackup.h # src/Client/BuzzHouse/AST/SQLProtoStr.cpp # src/Client/BuzzHouse/Generator/RandomGenerator.cpp # src/Client/BuzzHouse/Generator/RandomGenerator.h # src/Client/ClientBase.h # src/Columns/ColumnDynamic.h # src/Columns/IColumnUnique.h # src/Common/ColumnsHashing.h # src/Common/ColumnsHashingImpl.h # src/Common/Fiber.h # src/Common/HashTable/TwoLevelHashTable.h # src/Common/OpenTelemetryTraceContext.cpp # src/Common/PoolBase.h # src/Common/ThreadStatus.h # src/Compression/CompressionFactory.h # src/Coordination/KeeperStateMachine.h # src/Coordination/tests/gtest_coordination_snapshot.cpp # src/Core/PostgreSQL/PoolWithFailover.cpp # src/Core/PostgreSQLProtocol.h # src/DataTypes/DataTypeAggregateFunction.h # src/DataTypes/Serializations/SerializationJSON.cpp # src/Databases/MySQL/DatabaseMySQL.cpp # src/Disks/DiskObjectStorage/DiskObjectStorage.cpp # src/Disks/DiskObjectStorage/DiskObjectStorageTransaction.cpp # src/Disks/DiskObjectStorage/DiskObjectStorageTransaction.h # src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureBlobStorageCommon.cpp # src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h # src/Functions/CastOverloadResolver.cpp # src/Functions/FunctionFQDN.cpp # src/Functions/FunctionShowCertificate.cpp # src/Functions/FunctionStringOrArrayToT.h # src/Functions/FunctionsJSON.cpp # src/Functions/FunctionsStringDistance.cpp # src/Functions/FunctionsTransactionCounters.cpp # src/Functions/TimeSeries/timeSeriesThrowDuplicateSeriesIf.cpp # src/Functions/UTCTimestampTransform.cpp # src/Functions/aiEmbed.cpp # src/Functions/array/arrayJoin.cpp # src/Functions/castOrDefault.cpp # src/Functions/filesystem.cpp # src/Functions/formatRow.cpp # src/Functions/match.cpp # src/Functions/queryID.cpp # src/Functions/rand.cpp # src/Functions/rand64.cpp # src/IO/ReadBufferFromFileDescriptor.cpp # src/IO/S3/Client.cpp # src/Interpreters/Access/InterpreterShowCreateAccessEntityQuery.cpp # src/Interpreters/AggregationMethod.h # src/Interpreters/AsynchronousMetricLog.cpp # src/Interpreters/CancellationChecker.cpp # src/Interpreters/ClusterFunctionReadTask.h # src/Interpreters/HashJoin/KeyGetter.h # src/Interpreters/InterpreterHypotheticalIndexQuery.cpp # src/Interpreters/MetricLog.cpp # src/Interpreters/ProcessList.cpp # src/Interpreters/SpillingHashJoin.cpp # src/Interpreters/SpillingHashJoin.h # src/Parsers/Access/ParserCreateQuotaQuery.cpp # src/Parsers/Access/ParserCreateUserQuery.cpp # src/Parsers/ExpressionElementParsers.cpp # src/Parsers/ExpressionListParsers.cpp # src/Parsers/Kusto/ParserKQLOperators.cpp # src/Parsers/Lexer.cpp # src/Parsers/ParserAlterQuery.cpp # src/Parsers/ParserCreateFunctionQuery.cpp # src/Parsers/ParserCreateQuery.cpp # src/Parsers/ParserExplainQuery.cpp # src/Parsers/ParserInsertQuery.cpp # src/Parsers/ParserOptimizeQuery.cpp # src/Parsers/ParserQueryWithOutput.cpp # src/Parsers/ParserSelectQuery.cpp # src/Parsers/ParserSelectWithUnionQuery.cpp # src/Parsers/ParserSetQuery.cpp # src/Parsers/ParserSystemQuery.cpp # src/Parsers/ParserTablesInSelectQuery.cpp # src/Processors/Executors/PipelineExecutor.cpp # src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp # src/Processors/Formats/Impl/NativeORCBlockInputFormat.h # src/Processors/Formats/Impl/ProtobufListInputFormat.cpp # src/Processors/IProcessor.h # src/Processors/QueryPlan/AggregatingStep.cpp # src/Processors/QueryPlan/ArrayJoinStep.h # src/Processors/QueryPlan/CreatingSetsStep.cpp # src/Processors/QueryPlan/CreatingSetsStep.h # src/Processors/QueryPlan/ExpressionStep.cpp # src/Processors/QueryPlan/Optimizations/optimizeLazyMaterialization.cpp # src/Processors/QueryPlan/Optimizations/optimizeReadInOrder.h # src/Processors/QueryPlan/Optimizations/topKThroughJoin.cpp # src/Processors/QueryPlan/ReadFromRemote.cpp # src/Processors/Sources/RecursiveCTESource.cpp # src/Processors/Sources/ShellCommandSource.cpp # src/Processors/Transforms/DistinctTransform.cpp # src/Processors/Transforms/FilterTransform.h # src/Processors/tests/gtest_update_pipeline.cpp # src/QueryPipeline/BlockIO.h # src/QueryPipeline/Pipe.h # src/QueryPipeline/QueryPipelineBuilder.h # src/QueryPipeline/RemoteQueryExecutorReadContext.cpp # src/Server/DistributedQuery/StreamingExchangeSource.cpp # src/Server/HTTPHandlerFactory.h # src/Storages/KVStorageUtils.cpp # src/Storages/MergeTree/LoadedMergeTreeDataPartInfoForReader.h # src/Storages/MergeTree/MergeTask.h # src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp # src/Storages/MergeTree/MergeTreeDataWriter.h # src/Storages/MergeTree/MergeTreeIndexMinMax.h # src/Storages/MergeTree/MergeTreeIndexReader.cpp # src/Storages/MergeTree/MergeTreeIndicesSerialization.h # src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp # src/Storages/MergeTree/Streaming/CursorUtils.cpp # src/Storages/MergeTree/Streaming/CursorUtils.h # src/Storages/MergeTree/Streaming/MergeTreeBoundsSubscription.cpp # src/Storages/MergeTree/Streaming/MergeTreeBoundsSubscription.h # src/Storages/MergeTree/Streaming/MergeTreeCommitOrderSequentialSource.cpp # src/Storages/MergeTree/Streaming/MergeTreeCommitOrderSequentialSource.h # src/Storages/MergeTree/Streaming/StreamingChunkCursor.cpp # src/Storages/MergeTree/Streaming/StreamingChunkCursor.h # src/Storages/MergeTree/Streaming/SubscriptionEnrichment.cpp # src/Storages/MergeTree/Streaming/SubscriptionEnrichment.h # src/Storages/MergeTree/Streaming/tests/gtest_bounds_subscription.cpp # src/Storages/MergeTree/TextIndexUtils.h # src/Storages/MergeTree/WhatIfIndexEstimator.cpp # src/Storages/MergeTree/WhatIfIndexEstimator.h # src/Storages/NATS/NATS_fwd.h # src/Storages/ObjectStorage/DataLakes/DataLakeStorageSettings.cpp # src/Storages/ObjectStorage/DataLakes/DataLakeStorageSettings.h # src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp # src/Storages/ObjectStorage/IObjectIterator.h # src/Storages/ObjectStorage/StorageObjectStorage.h # src/Storages/ObjectStorage/StorageObjectStorageConfiguration.h # src/Storages/ObjectStorage/StorageObjectStorageSink.cpp # src/Storages/ObjectStorageQueue/ObjectStorageQueueMetadata.cpp # src/Storages/PostgreSQL/MaterializedPostgreSQLConsumer.cpp # src/Storages/Statistics/tests/gtest_stats.cpp # src/Storages/StorageAlias.h # src/Storages/StorageBuffer.h # src/Storages/StorageMemory.h # src/Storages/StorageMerge.cpp # src/Storages/StorageMergeTreeTextIndex.cpp # src/Storages/StorageStripeLog.cpp # src/Storages/StorageTimeSeries.cpp # src/Storages/System/StorageSystemMutations.cpp # src/Storages/TimeSeries/PrometheusQueryToSQL/NodeEvaluationRangeGetter.cpp # src/Storages/TimeSeries/PrometheusQueryToSQL/applyBinaryOperatorAnd.cpp # src/Storages/TimeSeries/PrometheusQueryToSQL/applyBinaryOperatorOr.cpp # src/Storages/TimeSeries/PrometheusQueryToSQL/applyBinaryOperatorUnless.cpp # src/Storages/TimeSeries/PrometheusQueryToSQL/applySimpleBinaryOperator.cpp # tests/clickhouse-test # tests/integration/test_ai_functions/test.py # tests/integration/test_broken_projections/test.py # tests/integration/test_prometheus_protocols/test_insert_select.py # tests/integration/test_replicated_users/test.py # tests/integration/test_storage_azure_blob_storage/test.py # tests/integration/test_storage_rabbitmq/test.py # tests/integration/test_storage_s3_queue/test_0.py # tests/integration/test_text_index_upgrade/test.py # tests/performance/README.md # tests/performance/scripts/README.md # tests/queries/0_stateless/00826_cross_to_inner_join.oldanalyzer.reference # tests/queries/0_stateless/00849_multiple_comma_join_2.reference # tests/queries/0_stateless/00908_analyze_query.oldanalyzer.reference # tests/queries/0_stateless/00940_order_by_read_in_order_query_plan.reference # tests/queries/0_stateless/01029_early_constant_folding.oldanalyzer.reference # tests/queries/0_stateless/01083_cross_to_inner_with_like.oldanalyzer.reference # tests/queries/0_stateless/01323_redundant_functions_in_order_by.oldanalyzer.reference # tests/queries/0_stateless/01372_wrong_order_by_removal.oldanalyzer.reference # tests/queries/0_stateless/01376_GROUP_BY_injective_elimination_dictGet.oldanalyzer.reference # tests/queries/0_stateless/01415_table_function_view.oldanalyzer.reference # tests/queries/0_stateless/01470_columns_transformers2.oldanalyzer.reference # tests/queries/0_stateless/01471_with_format.oldanalyzer.reference # tests/queries/0_stateless/01495_subqueries_in_with_statement_4.oldanalyzer.reference # tests/queries/0_stateless/01582_move_to_prewhere_compact_parts.sql # tests/queries/0_stateless/01622_constraints_simple_optimization.oldanalyzer.reference # tests/queries/0_stateless/01622_constraints_where_optimization.oldanalyzer.reference # tests/queries/0_stateless/01623_constraints_column_swap.oldanalyzer.reference # tests/queries/0_stateless/01625_constraints_index_append.reference # tests/queries/0_stateless/01655_plan_optimizations.reference # tests/queries/0_stateless/01655_plan_optimizations.sh # tests/queries/0_stateless/01706_optimize_normalize_count_variants.oldanalyzer.reference # tests/queries/0_stateless/01732_explain_syntax_union_query.oldanalyzer.reference # tests/queries/0_stateless/01883_with_grouping_sets.oldanalyzer.reference # tests/queries/0_stateless/02000_join_on_const.sql # tests/queries/0_stateless/02004_intersect_except_distinct_operators.oldanalyzer.reference # tests/queries/0_stateless/02004_intersect_except_operators.oldanalyzer.reference # tests/queries/0_stateless/02117_show_create_table_system.reference # tests/queries/0_stateless/02125_constant_if_condition_and_not_existing_column.oldanalyzer.reference # tests/queries/0_stateless/02220_array_join_format.oldanalyzer.reference # tests/queries/0_stateless/02226_analyzer_or_like_combine.oldanalyzer.reference # tests/queries/0_stateless/02253_empty_part_checksums.sh # tests/queries/0_stateless/02315_replace_multiif_to_if.oldanalyzer.reference # tests/queries/0_stateless/02346_text_index_direct_read.sql # tests/queries/0_stateless/02346_text_index_tokenizer_partially_materialized.sql # tests/queries/0_stateless/02353_order_by_tuple.reference # tests/queries/0_stateless/02377_executable_function_settings.sql # tests/queries/0_stateless/02428_parameterized_view.oldanalyzer.reference # tests/queries/0_stateless/02428_parameterized_view.sh # tests/queries/0_stateless/02477_logical_expressions_optimizer_low_cardinality.oldanalyzer.reference # tests/queries/0_stateless/02496_remove_redundant_sorting.sh # tests/queries/0_stateless/02500_remove_redundant_distinct.reference # tests/queries/0_stateless/02500_remove_redundant_distinct_analyzer.reference # tests/queries/0_stateless/02500_remove_redundant_distinct_analyzer.sh # tests/queries/0_stateless/02516_join_with_totals_and_subquery_bug.reference # tests/queries/0_stateless/02554_fix_grouping_sets_predicate_push_down.oldanalyzer.reference # tests/queries/0_stateless/02554_fix_grouping_sets_predicate_push_down.reference # tests/queries/0_stateless/02785_date_predicate_optimizations_ast_query_tree_rewrite.oldanalyzer.reference # tests/queries/0_stateless/02800_clickhouse_local_default_settings.reference # tests/queries/0_stateless/02815_join_algorithm_setting.sql # tests/queries/0_stateless/02864_statistics_ddl.sql # tests/queries/0_stateless/02868_distinct_to_count_optimization.reference # tests/queries/0_stateless/03001_matview_columns_after_modify_query.sh # tests/queries/0_stateless/03071_fix_short_circuit_logic.sql # tests/queries/0_stateless/03161_cnf_reduction.reference # tests/queries/0_stateless/03381_query_result_cache_old_analyzer.sql # tests/queries/0_stateless/03393_non_constant_second_argument_for_in.reference # tests/queries/0_stateless/03393_non_constant_second_argument_for_in.sql # tests/queries/0_stateless/03444_explain_asterisk.reference # tests/queries/0_stateless/03560_parallel_replicas_memory_bound_merging_projection.reference # tests/queries/0_stateless/03628_subcolumns_of_columns_with_dot_in_name.sql # tests/queries/0_stateless/03707_analyzer_convert_outer_any_to_inner.reference # tests/queries/0_stateless/03724_filter_assume_not_null_materialize.sql # tests/queries/0_stateless/03773_deterministic_functions_key_condition_explain.reference # tests/queries/0_stateless/03779_time_series_tags_functions.reference # tests/queries/0_stateless/03779_time_series_tags_functions.sql # tests/queries/0_stateless/03988_setting_use_partition_pruning.sql # tests/queries/0_stateless/04019_pipeline_stuck_sort_overflow.sql # tests/queries/0_stateless/04033_except_transformer_with_alias.sql # tests/queries/0_stateless/04093_text_index_separate_analysis.sql # tests/queries/0_stateless/04098_row_policy_disjunction_optimization.reference # tests/queries/0_stateless/04098_row_policy_disjunction_optimization.sql # tests/queries/0_stateless/04105_explain_syntax_parameterized_view.oldanalyzer.reference # tests/queries/0_stateless/04105_explain_syntax_parameterized_view.sql # tests/queries/0_stateless/04105_view_positional_args_via_local_plan.sh # tests/queries/0_stateless/04201_inverse_dictionary_lookup_edge_cases.reference # tests/queries/0_stateless/04207_pr_additional_filters.reference # tests/queries/0_stateless/04207_pr_additional_filters.sql # tests/queries/0_stateless/04209_statistics_retry_load.reference # tests/queries/0_stateless/04209_top_k_through_join_read_in_order_gate.reference # tests/queries/0_stateless/04209_top_k_through_join_read_in_order_gate.sql # tests/queries/0_stateless/04234_top_k_through_join_final_desc_gate.sql # tests/queries/0_stateless/04280_udf_normalizer_dangling_source_columns.sql # tests/queries/0_stateless/04320_distributed_plan_read_rejects.sql # tests/queries/0_stateless/04323_text_index_marks_empty_part.sh # tests/queries/0_stateless/04335_low_cardinality_mixed_key_join.sql # tests/queries/0_stateless/04338_analyzer_unknown_table_cross_database_hint.reference # tests/queries/0_stateless/04338_analyzer_unknown_table_cross_database_hint.sh # tests/stress/keeper/framework/core/cluster.py # utils/list-versions/version_date.tsv

…light Addresses the two open review findings on `plannerDisablesParallelReplicasForJoinTreeShape`: * A `StorageView` reached through a table function (`view(...)`, `viewIfPermitted(...)`) was not recognized by the leftmost-view escape, which only looked at `TableNode`. `allowParallelReplicasForJoinTree` decides a leftmost `TableFunctionNode` with `parallelReplicasEnabledForStorage`, which accepts a view only when it unwraps to an eligible `MergeTree` table - so a `view(...)` over a `Distributed` table leaves the outer join tree ineligible exactly like a plain view table, and the forced mode must not reject such a step. The new `parallelReplicasEnabledForViewStorage` mirrors that rule, so a view that *does* unwrap to an eligible `MergeTree` table still reports a genuine engagement. * A single `FULL JOIN` was not covered: the n-way rules only suppress `FULL` when it appears in a join stack, while `allowParallelReplicasForJoinTree` never lets any join kind other than an `ALL INNER`, `LEFT` or `RIGHT` join drive parallel replicas. The top-level check now mirrors that directly, which also covers `COMMA` and `PASTE` joins and subsumes the previous non-`ALL` `INNER JOIN` special case. Both are covered by new forced-mode cases in `04489_recursive_cte_join_index` (`full_join_local_pr`, `view_fn_left_dist_pr`). The same test was failing reproducibly in CI (38 jobs) on the previous head, and this commit fixes those failures too: * The `if (!engagement.any()) return;` shortcut added in the previous pass left `allow_experimental_parallel_reading_from_replicas = 2` in place for every step whose engagement is zero. A recursive CTE whose *non-recursive* part reads a table then really does use parallel replicas for that ordinary read, which needs a configured cluster - hence `CLUSTER_DOESNT_EXIST` in the 31 jobs that run with the default (empty) `cluster_for_parallel_replicas`. The preflight disables the setting again, and the two cases whose seed reads a table (`traverse_pr_cross_join`, `traverse_pr_any_inner_join`) name the cluster like the plan-based cases already do. * `view_pr` / `view_pr_throw` join the view on the *right* side, so the leftmost-view gate does not apply to them: their view inner query really can engage parallel replicas and the forced mode must keep failing closed. Their expectations are restored to `SUPPORT_IS_DISABLED` (they were flipped to success in the previous pass, which is what made 7 jobs fail), and the query-log assertion that they used parallel replicas is dropped with them. * The plan-based query-log assertion now covers only the forced mode. The best-effort mode (`= 1`) is free to decide from the plan that parallel replicas are not worth it for the tiny recursive steps of `plan_based_joined_pr`, which it does. * The `float_walk_auto` case is removed: with `join_algorithm = 'auto'` the switcher does not fall back to `partial_merge` in this shape at any `max_bytes_in_join` / `max_rows_in_join` value - it either throws `SET_SIZE_LIMIT_EXCEEDED` or stays on the hash path - so the case could not test what it intended. `float_walk_pm` and `float_walk_ppm` keep the value-comparing coverage.

alexey-milovidov · 2026-08-26T07:27:36Z

🕵 Updated the branch with current master (merge 478a834dc75f, the branch was 3310 commits behind and GitHub reported it as CONFLICTING) and pushed the review fixes in 93a525de362a.

The merge was a criss-cross one (two merge bases), so the artificial conflict set was 274 files while the PR touches 12: every conflicted file outside the PR set was taken from master verbatim, the two overlapping files were re-merged against the real base, and the final git diff --cached origin/master was verified byte-identical to the pre-merge <base>..HEAD diff (12 files), so the result is master plus this PR exactly.

Both open review threads are addressed (replies on discussion_r3860524415 and discussion_r3860525744, both resolved):

the leftmost-view escape now covers a StorageView behind a table function, mirroring parallelReplicasEnabledForStorage rather than escaping unconditionally, and
the top-level join-kind check now mirrors allowParallelReplicasForJoinTree directly, which covers a single FULL JOIN.

The same commit also fixes the CI red on the previous head, where all 45 failures were this PR's own tests (04489_recursive_cte_join_index in 38 jobs, 04926_recursive_cte_merge_view_pruning in 1):

31 jobs failed with CLUSTER_DOESNT_EXIST. The if (!engagement.any()) return; shortcut from the previous pass left allow_experimental_parallel_reading_from_replicas = 2 in place for steps with zero engagement, and the non-recursive part of traverse_pr_cross_join reads a table — that ordinary read really does use parallel replicas and needs a cluster, which the default (empty) cluster_for_parallel_replicas does not provide. The preflight disables the setting again, and the two cases whose seed reads a table now name the cluster like the plan-based cases already do.
7 jobs failed with SUPPORT_IS_DISABLED on view_pr. That query joins the view on the right side, so the leftmost-view gate never applied to it: its inner view query can engage parallel replicas and the forced mode must keep failing closed. view_pr / view_pr_throw are back to expecting SUPPORT_IS_DISABLED.
The plan-based query-log assertion now covers only the forced mode — the best-effort mode is free to decide from the plan that parallel replicas are not worth it for those tiny steps, and it does. The float_walk_auto case is removed: with join_algorithm = 'auto' the switcher never reaches its partial_merge fallback in that shape at any limit value, so the case could not test its intent; float_walk_pm and float_walk_ppm keep that coverage.

Verified locally on the merged branch: 04489_recursive_cte_join_index and 04926_recursive_cte_merge_view_pruning pass with fixed settings and across randomized runs, both with a configured cluster_for_parallel_replicas and with the default empty one (the configuration that was failing in CI).

…index-optimization

…6.9 block The Style check test `settings_changes_history` failed because the entry sat in the already-released 26.8 block while master is on 26.9. Report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=97254&sha=93a525de362a69e55cfe874485b1d2f99b779446&name_0=PR&name_1=Style%20check Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

alexey-milovidov · 2026-08-28T08:30:30Z

🕵 Updated the branch (was 917 commits behind, GitHub showed UNKNOWN mergeability):

Merged master (8b1aad536d80). The plain git merge exploded into ~90 spurious conflicts in files this PR never touched — git merge-base --all returned two bases (criss-cross history, 2026-08-26 + 2026-05-29), which poisons the ort virtual-ancestor merge. Redid the merge single-base via git merge-tree --write-tree --merge-base=887b93e16db9 — fully clean, and the net diff versus master is exactly the PR's own 12 files.
Fixed the Style check failure settings_changes_history (the only real red on 93a525de; everything else was DROPPED): master moved on to 26.9, so the recursive_cte_max_in_filter_cardinality history entry sat in the already-released 26.8 block. Moved it to the 26.9 block (2d2d38b09138).

RecursiveCTESource.cpp, Settings.cpp, SettingsChangesHistory.cpp, and findParallelReplicasQuery.cpp all compile clean against the merged tree. 0 unresolved review threads. Waiting for a fresh CI run on 2d2d38b09138.

clickhouse-gh · 2026-08-28T09:02:20Z

+            if (storage)
+                engagement.merge(mayEngageParallelReplicasForRemoteStorage(*storage, scope_context));
+
+            if (const auto * view = typeid_cast<const StorageView *>(storage.get()))


view(...) over an eligible MergeTree is still treated as if it could not engage parallel replicas. The TableNode arm handles ordinary VIEWs by combining canUseTableForParallelReplicas with mayEngageParallelReplicasForView, but the TableFunctionNode arm only does the latter.

For FROM view(SELECT * FROM mt) AS e LEFT JOIN rec AS t ... with parallel_replicas_allow_view_over_mergetree = 1, StorageView::getViewContext disables parallel replicas in the inner view context precisely because the outer planner can unwrap the leftmost table function directly to the MergeTree. mayEngageParallelReplicasForView therefore reports no engagement here, so mode 2 is silently downgraded instead of throwing even though allowParallelReplicasForJoinTree would accept the recursive step. That breaks the documented force-or-throw contract.

Please mirror the TableNode branch here: if the StorageView-backed table function unwraps to an eligible MergeTree, mark engagement.local_merge_tree and the read count from the outer storage-level rule, then merge the inner-view engagement only for the cases where the outer join tree is ineligible but the view's inner query can still use parallel replicas.

clickhouse-gh Bot added the pr-performance Pull request with some performance improvements label Feb 18, 2026

Fix style check: use SYSTEM FLUSH LOGS query_log instead of bare `S…

8321f6f

…YSTEM FLUSH LOGS` Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

alexey-milovidov marked this pull request as draft February 19, 2026 08:21

alexey-milovidov added 2 commits February 21, 2026 05:09

Merge remote-tracking branch 'origin/master' into HEAD

029b11e

Merge remote-tracking branch 'origin/master' into recursive-cte-join-…

2d71074

…index-optimization

geldot reviewed Feb 24, 2026

View reviewed changes

Comment thread src/Processors/Sources/RecursiveCTESource.cpp Outdated

alexey-milovidov added 3 commits March 3, 2026 00:42

Merge master into recursive-cte-join-index-optimization

15ff2b6

Merge master into recursive-cte-join-index-optimization

94a7cd3

Merge master into recursive-cte-join-index-optimization

37f2618

alexey-milovidov marked this pull request as ready for review March 5, 2026 22:19

alexey-milovidov requested a review from novikd March 5, 2026 22:22

cursor Bot reviewed Mar 5, 2026

View reviewed changes

Comment thread src/Processors/Sources/RecursiveCTESource.cpp

Merge branch 'master' into recursive-cte-join-index-optimization

547bc5c

clickhouse-gh Bot reviewed Mar 28, 2026

View reviewed changes

Comment thread src/Processors/Sources/RecursiveCTESource.cpp Outdated

alexey-milovidov and others added 2 commits March 30, 2026 07:51

Merge remote-tracking branch 'origin/master' into recursive-cte-join-…

efc9a0c

…index-optimization

clickhouse-gh Bot reviewed Mar 30, 2026

View reviewed changes

Comment thread src/Processors/Sources/RecursiveCTESource.cpp Outdated

alexey-milovidov and others added 2 commits March 30, 2026 11:54

Merge remote-tracking branch 'origin/master' into recursive-cte-join-…

c52d958

…index-optimization

clickhouse-gh Bot reviewed Mar 30, 2026

View reviewed changes

Comment thread src/Processors/Sources/RecursiveCTESource.cpp Outdated

Merge branch 'master' into recursive-cte-join-index-optimization

cc6eba5

Address review: honor view query parallel-replica settings

9445276

clickhouse-gh Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread src/Planner/findParallelReplicasQuery.cpp

clickhouse-gh Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread src/Processors/Sources/RecursiveCTESource.cpp

alexey-milovidov added 5 commits August 16, 2026 17:07

Merge remote-tracking branch 'origin/master' into continue-pr-97254-auto

1676046

Address review: align parallel-replica min-row fallback

051ec9f

Merge remote-tracking branch 'origin/master' into recursive-cte-join-…

86b3811

…index-optimization

Address review: honor FINAL on views for parallel replicas

047d8e3

Fix FINAL view regression expectation

b7d5991

clickhouse-gh Bot reviewed Aug 17, 2026

View reviewed changes

Comment thread src/Processors/Sources/RecursiveCTESource.cpp Outdated

alexey-milovidov added 3 commits August 17, 2026 22:05

Merge remote-tracking branch 'origin/master' into recursive-cte-join-…

3bbdf46

…index-optimization

Address review: honor Merge child pruning in parallel-replica prefl…

b11071b

…ight

Merge remote-tracking branch 'origin/master' into HEAD

9fd0afb

clickhouse-gh Bot reviewed Aug 18, 2026

View reviewed changes

Comment thread src/Processors/Sources/RecursiveCTESource.cpp

alexey-milovidov added 2 commits August 19, 2026 00:40

Merge remote-tracking branch 'origin/master' into HEAD

6b21f8a

Address review: mirror allowParallelReplicasForJoinTree

1a4ca13

clickhouse-gh Bot reviewed Aug 19, 2026

View reviewed changes

Address review: preserve view recursive joins

5fb7366

clickhouse-gh Bot reviewed Aug 19, 2026

View reviewed changes

Comment thread tests/queries/0_stateless/04489_recursive_cte_join_index.sql

Comment thread tests/queries/0_stateless/04489_recursive_cte_join_index.sql

Comment thread tests/queries/0_stateless/04489_recursive_cte_join_index.sql

alexey-milovidov added 2 commits August 19, 2026 22:15

Merge remote-tracking branch 'origin/master' into HEAD

9c4cb82

Address review: preserve safe view parallel replicas

a0b5c77

clickhouse-gh Bot reviewed Aug 19, 2026

View reviewed changes

Comment thread src/Processors/Sources/RecursiveCTESource.cpp

Comment thread src/Processors/Sources/RecursiveCTESource.cpp

iskakaushik mentioned this pull request Aug 20, 2026

Fix AI SQL generation failing on prompts that need several schema lookups #115509

Closed

alexey-milovidov mentioned this pull request Aug 25, 2026

Compute the transformed header once per step instead of per stream #115677

Merged

alexey-milovidov added 2 commits August 26, 2026 05:32

alexey-milovidov and others added 2 commits August 28, 2026 08:24

Merge remote-tracking branch 'origin/master' into recursive-cte-join-…

8b1aad5

…index-optimization

clickhouse-gh Bot reviewed Aug 28, 2026

View reviewed changes

Uh oh!

Conversation

alexey-milovidov commented Feb 18, 2026 • edited by clickhouse-gh Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Documentation entry for user-facing changes

Uh oh!

clickhouse-gh Bot commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI Review

Summary

Findings

Tests

Final Verdict

LLVM Coverage Report

Uh oh!

alexey-milovidov commented Feb 19, 2026

Fix for ParallelReplicas CI failure

Uh oh!

Uh oh!

asad-awadia commented Feb 28, 2026

Uh oh!

alexey-milovidov commented Mar 5, 2026

Uh oh!

alexey-milovidov commented Mar 5, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

geldot commented Mar 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clickhouse-gh Bot Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

clickhouse-gh Bot Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

clickhouse-gh Bot Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexey-milovidov commented Aug 26, 2026

Uh oh!

alexey-milovidov commented Aug 28, 2026

Uh oh!

clickhouse-gh Bot Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alexey-milovidov commented Feb 18, 2026 •

edited by clickhouse-gh Bot

Loading

clickhouse-gh Bot commented Feb 18, 2026 •

edited

Loading