Skip to content

Support LowCardinality identifiers in the TimeSeries table engine - #117033

Open
nikitamikhaylov wants to merge 6 commits into
masterfrom
timeseries-lowcardinality-id
Open

Support LowCardinality identifiers in the TimeSeries table engine#117033
nikitamikhaylov wants to merge 6 commits into
masterfrom
timeseries-lowcardinality-id

Conversation

@nikitamikhaylov

@nikitamikhaylov nikitamikhaylov commented Aug 29, 2026

Copy link
Copy Markdown
Member

In the TimeSeries samples table the series identifier repeats in every row (sorted by (id, timestamp)), so the id column dominates the materialized bytes of every scan: 24 of ~40 bytes per row with the default Tuple(UInt64, UUID). A LowCardinality id, e.g. Tuple(UInt64, LowCardinality(UUID)), stores small per-block dictionaries instead, cutting the materialized id bytes by ~58%, and now works exactly like a plain one:

  • The default id generator dictionary-encodes the hash in the generated DEFAULT expression: tuple(sipHash64(metric_name), toLowCardinality(reinterpretAsUUID(sipHash128(tags)))).
  • The whole-metric primary-key range conditions of timeSeriesSelector are emitted for LowCardinality id components too.
  • timeSeriesIdToGroup/timeSeriesIdToTags resolve LowCardinality ids per dictionary key, lazily for only the keys referenced by some row. This also fixes an exception: a shared dictionary can contain identifiers whose rows were all filtered out and which are unknown to the per-query tags collector, so these functions must not run over the whole dictionary.

Tuple(UInt64, LowCardinality(UUID)) is the default id type for new tables now. Existing tables keep the id type declared in their inner tables, and the plain layout stays available by declaring the type explicitly, e.g. TAGS INNER COLUMNS (id Tuple(UInt64, UUID)).

On the time-series benchmark at scale 64 (71.4B samples, ~413k series, 30 days, same binary and codecs in both legs): 1.16x geometric-mean speedup over the 38-query PromQL suite, 1.26x on the 30d high-cardinality class (e.g. 24.4s → 19.8s, −23% CPU), unchanged ingest rate, identical query results.

Test: 04891_timeseries_lowcardinality_id.

Changelog category (leave one):

  • Experimental Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

The TimeSeries table engine supports dictionary-encoded series identifiers: the id column can be declared with a LowCardinality type, and newly created tables use Tuple(UInt64, LowCardinality(UUID)) by default (existing tables keep their declared id type; the previous layout stays available as TAGS INNER COLUMNS (id Tuple(UInt64, UUID))). This layout stores per-block dictionaries instead of repeating the full identifier in every row of the samples table, giving a 1.16x geometric-mean PromQL speedup (up to 1.29x per query class) in our benchmark.


Workflow [PR]
Sync PR [sync-upstream/pr/117033]

nikitamikhaylov and others added 4 commits August 28, 2026 16:32
Support LowCardinality components in the TimeSeries id type (e.g.
Tuple(UInt64, LowCardinality(UUID))) and optimize for the dictionary
encoding instead of stripping it:

- TimeSeriesIDGenerator supports LowCardinality-wrapped component types
  and dictionary-encodes the hash right in the generated DEFAULT
  expression (toLowCardinality(reinterpretAsUUID(sipHash128(tags)))).
- The whole-metric primary-key range conditions of timeSeriesSelector
  are emitted for LowCardinality id components too.
- timeSeriesSelector narrows the id to its second component for
  two-component ids with a LowCardinality second component and the
  canonical id generator: the second component (a hash of all the tags)
  alone identifies a time series, and reading only it keeps the
  identifiers dictionary-encoded end-to-end, so the IN <ids> filter
  over the samples table is executed per dictionary key instead of
  per row. For a whole-metric selector the primary-key range conditions
  keep handling the index analysis; for other selectors the tags
  subquery reassembles full-shaped ids around the narrowed component,
  keeping the set usable for primary-key index analysis.
- ContextTimeSeriesTagsCollector::getGroupByID/getTagsByID process
  LowCardinality id columns per dictionary key, resolving lazily only
  the keys referenced by some row: a shared dictionary can contain
  identifiers whose rows were all filtered out and which are therefore
  unknown to the collector. timeSeriesIdToGroup and timeSeriesIdToTags
  opt out of the default LowCardinality implementation for the same
  reason.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ric selectors

With id narrowing the narrowed id set filters the rows exactly, so the
whole-metric primary-key range conditions are index-analysis-only:
wrap them in indexHint, and the runtime filter then reads only the
second-component subcolumn of the id (the first component is not read
at all). For other narrowed selectors the full-shaped id set filters
the raw id column, which is therefore read anyway: disable the
tupleElement-to-subcolumn rewrite there so the second component is
extracted from the read column instead of being read from disk twice.

Measured on 128M samples (8 metrics x 800 series x 20k samples,
single part, warm): whole-metric scan 63 ms -> 44 ms, range rate
queries 113-117 ms -> 93-97 ms vs the plain Tuple(UInt64, UUID) id.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Configuration::id_data_type is now simply the type of the id column
  returned by timeSeriesSelector (the second component's type when the
  id is narrowed), so the table function returns it as is; the type of
  the TimeSeries table's own id column moved to table_id_data_type.
- Replace the narrow_id_to_second_component/return_first_component_too
  parameter pair of makeSelectQueryFromTagsTable with the
  TagsSubqueryIDForm enum describing what the tags subquery returns as
  the series identifier.
- makeSelectQueryFromDataTable takes the ready-made id SELECT-list
  expression instead of a narrowing-specific flag.
- Explain why the narrowed id set is used only together with the
  whole-metric range conditions: a set of second components cannot
  constrain a prefix of the sorting key (id, timestamp), and giving
  every selector a second, index-analysis-only set would cost a second
  tags-table scan per selector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ke plain ones

Per review: remove the id-narrowing machinery from timeSeriesSelector
(the second-component SELECT rewrite, TagsSubqueryIDForm, the indexHint
wrapping of the range conditions and the subcolumns-rewrite tweak).
A LowCardinality id now goes through exactly the same query shapes as a
plain one; what remains of the feature is:

- the default id generator supports LowCardinality-wrapped component
  types (the hash is dictionary-encoded in the generated expression);
- the whole-metric primary-key range conditions are emitted for
  LowCardinality id components too;
- ContextTimeSeriesTagsCollector::getGroupByID/getTagsByID process
  LowCardinality id columns per dictionary key, resolving lazily only
  the keys referenced by some row (a shared dictionary can contain
  identifiers whose rows were all filtered out and which are therefore
  unknown to the collector), and timeSeriesIdToGroup/timeSeriesIdToTags
  opt out of the default LowCardinality implementation for the same
  reason.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@nikitamikhaylov nikitamikhaylov added the comp-promql PromQL / time-series subsystem: TimeSeries storage engine, PromQL parser, PromQL-to-SQL converter... label Aug 29, 2026
nikitamikhaylov and others added 2 commits August 29, 2026 10:28
New tables get the dictionary-encoded id layout by default; existing
tables keep the id type declared in their inner tables. The plain
layout stays available by declaring the type explicitly, e.g.
TAGS INNER COLUMNS (id Tuple(UInt64, UUID)).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nality-id

# Conflicts:
#	src/Interpreters/ContextTimeSeriesTagsCollector.cpp
#	src/Interpreters/ContextTimeSeriesTagsCollector.h
#	tests/queries/0_stateless/04600_timeseries_column_validation.reference
@clickhouse-gh

clickhouse-gh Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Workflow [PR], commit [6b0fcd2]

Summary:

job_name test_name status info comment
Fast test FAIL
04409_time_series_referential_dependencies FAIL cidb
Code Review DROPPED
Fast test (arm_darwin) DROPPED
Build (amd_debug) DROPPED
Build (amd_asan_ubsan) DROPPED
Build (amd_tsan) DROPPED
Build (amd_msan) DROPPED
Build (amd_binary) DROPPED
Build (arm_debug) DROPPED
Build (arm_asan_ubsan) DROPPED

@clickhouse-gh clickhouse-gh Bot added the pr-experimental Experimental Feature label Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp-promql PromQL / time-series subsystem: TimeSeries storage engine, PromQL parser, PromQL-to-SQL converter... pr-experimental Experimental Feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant