DataTerminology.comNEW
A data attribute with many distinct values relative to total rows, impacting indexing, storage, and query performance.
DESCRIPTION
High cardinality indicates a column or metric with many unique values. In databases, it affects index selectivity, clustering, and compression. In telemetry, high-cardinality metrics can explode time-series series counts, increasing cost and latency. Modeling high-cardinality dimensions requires careful partitioning, summarization, and aggregation strategies. Systems may use hashing, bloom filters, or rollups to mitigate overhead. Identifying and managing high-cardinality features is crucial for scalability and operability in analytical and monitoring workloads.
ALTERNATIVE DEFINITIONS
InfluxDB
Cardinality is the number of unique series in a database.
Defines cardinality in time-series context.
Datadog
High cardinality refers to metrics with a very large number of distinct tag combinations.
Focuses on telemetry tag dimensions driving cardinality.
Snowflake
Columns with high cardinality (many distinct values) may reduce clustering effectiveness.
Highlights impact on clustering and performance.
RELATED TERMS
Cardinality Distinct Count Index Selectivity Dimension Time-Series Cardinality Partitioning Sharding Bloom Filter
COMPETING TERMS
Low Cardinality
RELATIONS
- Broader: Cardinality
KNOWLEDGE AREAS
Data Modeling Data Storage & Operations Observability & Telemetry Data Quality
TARGET AUDIENCE
Data Engineer Data Architect DevOps Engineer MLOps Engineer BI Specialist
EXTERNAL SOURCES
- Wikipedia - Cardinality
- InfluxDB - Series cardinality internals
- Datadog - Explainer on metrics cardinality
- Snowflake - Clustering keys and cardinality impacts
TOP QUESTIONS
What does High Cardinality mean in databases?
It means a column has many distinct values. This affects index selectivity, compression, and query performance.
Category: Overview
Why is High Cardinality challenging in observability systems?
It creates many metric series due to tag combinations, increasing storage, ingestion costs, and query latency.
Category: Risk
How do you detect High Cardinality?
Compute distinct counts, monitor series growth, and track index selectivity over time.
Category: Validation
What strategies mitigate High Cardinality in telemetry?
Limit tag values, aggregate or roll up, hash sensitive IDs, and filter unnecessary tags.
Category: Practices
How does High Cardinality affect clustering keys in warehouses?
It may reduce clustering effectiveness, leading to more partitions and slower scan performance.
Category: Performance
What are common modeling mistakes with High Cardinality dimensions?
Creating unbounded identifiers as dimensions, over-normalization, and insufficient indexing are typical errors.
Category: Mistakes
When is High Cardinality acceptable?
When necessary for uniqueness or security; careful engineering and cost controls are required.
Category: Opinion
What’s a competing concept to High Cardinality?
Low Cardinality, where a column has few distinct values, enabling efficient compression and indexing.
Category: Alternatives
How to monitor High Cardinality growth?
Track distinct counts, series cardinality dashboards, and alert on thresholds or sudden spikes.
Category: Management