[feature](cache) Add session variable to keep INSERT source scans in the normal file cache queue - #66804
Open
raghav-reglobe wants to merge 1 commit into
Open
Conversation
…normal file cache queue INSERT ... SELECT / CTAS source scans are forced onto the disposable file cache queue to protect interactive queries from one-shot load-scan pollution. For recurring ETL over external tables whose source working set is ALSO the interactive working set (the same files served to queries), that protection inverts: every cycle re-reads the same data from remote storage and the cache can never retain it. Add enable_file_cache_for_insert_source (default false — existing behavior preserved). When set on the loading account/session, insert source scans use the normal cache queue and recurring builds hit warm data. Broker load (no session context, genuinely one-shot) keeps the forced disposable classification.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
NereidsCoordinator.setForInsert()unconditionally setsdisableFileCache(true), so the source scan of everyINSERT ... SELECT/ CTAS is classified into the disposable file cache queue (the comment explains the intent: keep one-shot load scans from polluting the normal LRU).That one-shot assumption doesn't hold for a common warehouse shape: recurring ETL. Hourly dbt-style builds re-read the same source tables (in our case Iceberg external tables) every cycle — but because the disposable queue is capped at 5% of the cache and hits never promote, those scans re-fetch from remote storage every hour, forever, no matter how large the file cache is.
This PR adds an opt-in session variable,
enable_file_cache_for_insert_source(default false — existing behavior is unchanged byte-for-byte). When enabled, the insert source scan uses the normal cache queue. The broker-load constructor path is deliberately untouched (no ConnectContext, genuinely one-shot).Design notes: we considered conditioning on the target catalog instead of a variable, but the pollution question is a property of the source scan's reuse pattern, not the target — a variable lets the operator state that intent directly. Automatic frequency-based promotion between queues would be the ideal long-term answer; that's out of scope here.
Measured in production (recurring hourly ETL over Iceberg external tables, 280GB file cache per BE): after enabling, the ETL account's remote-read rate dropped ~8x within hours (23.5 GB/h -> 3 GB/h), byte-level cache hit rate rose from 64% to 78% and climbing, and fleet block hit-ratio gained ~5pp.
Release note
New session variable
enable_file_cache_for_insert_source(default false): when enabled, source scans of INSERT ... SELECT / CTAS use the normal file cache queue instead of the disposable queue.Check List (For Author)
Test
Behavior is opt-in and default-off (existing behavior byte-identical). Manually verified in a production deployment: with the variable enabled, insert source scans populate the normal queue (
normal_queue_cache_sizegrowth, disposable evict flatline in BE bvars) and remote reads drop as described above.Behavior changed:
Does this need documentation?
Will add the session variable to the docs site once the approach is confirmed by reviewers.
Check List (For Reviewer who merge this PR)