Support LowCardinality identifiers in the TimeSeries table engine - #117033
Open
nikitamikhaylov wants to merge 6 commits into
Open
Support LowCardinality identifiers in the TimeSeries table engine#117033nikitamikhaylov wants to merge 6 commits into
nikitamikhaylov wants to merge 6 commits into
Conversation
Support LowCardinality components in the TimeSeries id type (e.g. Tuple(UInt64, LowCardinality(UUID))) and optimize for the dictionary encoding instead of stripping it: - TimeSeriesIDGenerator supports LowCardinality-wrapped component types and dictionary-encodes the hash right in the generated DEFAULT expression (toLowCardinality(reinterpretAsUUID(sipHash128(tags)))). - The whole-metric primary-key range conditions of timeSeriesSelector are emitted for LowCardinality id components too. - timeSeriesSelector narrows the id to its second component for two-component ids with a LowCardinality second component and the canonical id generator: the second component (a hash of all the tags) alone identifies a time series, and reading only it keeps the identifiers dictionary-encoded end-to-end, so the IN <ids> filter over the samples table is executed per dictionary key instead of per row. For a whole-metric selector the primary-key range conditions keep handling the index analysis; for other selectors the tags subquery reassembles full-shaped ids around the narrowed component, keeping the set usable for primary-key index analysis. - ContextTimeSeriesTagsCollector::getGroupByID/getTagsByID process LowCardinality id columns per dictionary key, resolving lazily only the keys referenced by some row: a shared dictionary can contain identifiers whose rows were all filtered out and which are therefore unknown to the collector. timeSeriesIdToGroup and timeSeriesIdToTags opt out of the default LowCardinality implementation for the same reason. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ric selectors With id narrowing the narrowed id set filters the rows exactly, so the whole-metric primary-key range conditions are index-analysis-only: wrap them in indexHint, and the runtime filter then reads only the second-component subcolumn of the id (the first component is not read at all). For other narrowed selectors the full-shaped id set filters the raw id column, which is therefore read anyway: disable the tupleElement-to-subcolumn rewrite there so the second component is extracted from the read column instead of being read from disk twice. Measured on 128M samples (8 metrics x 800 series x 20k samples, single part, warm): whole-metric scan 63 ms -> 44 ms, range rate queries 113-117 ms -> 93-97 ms vs the plain Tuple(UInt64, UUID) id. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Configuration::id_data_type is now simply the type of the id column returned by timeSeriesSelector (the second component's type when the id is narrowed), so the table function returns it as is; the type of the TimeSeries table's own id column moved to table_id_data_type. - Replace the narrow_id_to_second_component/return_first_component_too parameter pair of makeSelectQueryFromTagsTable with the TagsSubqueryIDForm enum describing what the tags subquery returns as the series identifier. - makeSelectQueryFromDataTable takes the ready-made id SELECT-list expression instead of a narrowing-specific flag. - Explain why the narrowed id set is used only together with the whole-metric range conditions: a set of second components cannot constrain a prefix of the sorting key (id, timestamp), and giving every selector a second, index-analysis-only set would cost a second tags-table scan per selector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ke plain ones Per review: remove the id-narrowing machinery from timeSeriesSelector (the second-component SELECT rewrite, TagsSubqueryIDForm, the indexHint wrapping of the range conditions and the subcolumns-rewrite tweak). A LowCardinality id now goes through exactly the same query shapes as a plain one; what remains of the feature is: - the default id generator supports LowCardinality-wrapped component types (the hash is dictionary-encoded in the generated expression); - the whole-metric primary-key range conditions are emitted for LowCardinality id components too; - ContextTimeSeriesTagsCollector::getGroupByID/getTagsByID process LowCardinality id columns per dictionary key, resolving lazily only the keys referenced by some row (a shared dictionary can contain identifiers whose rows were all filtered out and which are therefore unknown to the collector), and timeSeriesIdToGroup/timeSeriesIdToTags opt out of the default LowCardinality implementation for the same reason. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New tables get the dictionary-encoded id layout by default; existing tables keep the id type declared in their inner tables. The plain layout stays available by declaring the type explicitly, e.g. TAGS INNER COLUMNS (id Tuple(UInt64, UUID)). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nality-id # Conflicts: # src/Interpreters/ContextTimeSeriesTagsCollector.cpp # src/Interpreters/ContextTimeSeriesTagsCollector.h # tests/queries/0_stateless/04600_timeseries_column_validation.reference
Contributor
|
Workflow [PR], commit [6b0fcd2] Summary: ❌
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In the
TimeSeriessamples table the series identifier repeats in every row (sorted by(id, timestamp)), so theidcolumn dominates the materialized bytes of every scan: 24 of ~40 bytes per row with the defaultTuple(UInt64, UUID). ALowCardinalityid, e.g.Tuple(UInt64, LowCardinality(UUID)), stores small per-block dictionaries instead, cutting the materializedidbytes by ~58%, and now works exactly like a plain one:DEFAULTexpression:tuple(sipHash64(metric_name), toLowCardinality(reinterpretAsUUID(sipHash128(tags)))).timeSeriesSelectorare emitted forLowCardinalityid components too.timeSeriesIdToGroup/timeSeriesIdToTagsresolveLowCardinalityids per dictionary key, lazily for only the keys referenced by some row. This also fixes an exception: a shared dictionary can contain identifiers whose rows were all filtered out and which are unknown to the per-query tags collector, so these functions must not run over the whole dictionary.Tuple(UInt64, LowCardinality(UUID))is the default id type for new tables now. Existing tables keep the id type declared in their inner tables, and the plain layout stays available by declaring the type explicitly, e.g.TAGS INNER COLUMNS (id Tuple(UInt64, UUID)).On the time-series benchmark at scale 64 (71.4B samples, ~413k series, 30 days, same binary and codecs in both legs): 1.16x geometric-mean speedup over the 38-query PromQL suite, 1.26x on the 30d high-cardinality class (e.g. 24.4s → 19.8s, −23% CPU), unchanged ingest rate, identical query results.
Test:
04891_timeseries_lowcardinality_id.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
The
TimeSeriestable engine supports dictionary-encoded series identifiers: theidcolumn can be declared with aLowCardinalitytype, and newly created tables useTuple(UInt64, LowCardinality(UUID))by default (existing tables keep their declared id type; the previous layout stays available asTAGS INNER COLUMNS (id Tuple(UInt64, UUID))). This layout stores per-block dictionaries instead of repeating the full identifier in every row of the samples table, giving a 1.16x geometric-mean PromQL speedup (up to 1.29x per query class) in our benchmark.Workflow [PR]
Sync PR [sync-upstream/pr/117033]