🗝️Shard Key

NEW

Also known as: Partition Key

A field or compound key that determines how data is partitioned and routed to shards in a distributed database.

DESCRIPTION

The shard key defines the distribution of records across shards, enabling horizontal scalability. Good shard key design balances load, minimizes cross-shard queries, and supports growth with predictable routing. Common strategies include hash, range, and compound keys incorporating high-cardinality fields. Poorly chosen shard keys cause hotspots, skew, and operational pain during rebalancing. Shard key selection should reflect query patterns, data growth, and resharding strategies.

ALTERNATIVE DEFINITIONS

MongoDB

The shard key is either a single indexed field or compound indexed fields that determines the distribution of the collection’s documents across the shards.

Ties shard key to indexing and distribution in MongoDB sharding.

Source

Microsoft Azure

A partition key is the value by which Azure Cosmos DB logically partitions data to scale and distribute requests and storage.

Highlights logical partitioning for scalability and request distribution.

Source

RELATED TERMS

Sharding Range Partitioning Hash Partitioning Consistent Hashing Data Distribution Skew Rebalancing Routing

COMPETING TERMS

Auto-Sharding

RELATIONS

• Broader: Key

KNOWLEDGE AREAS

Data Storage & Operations Data Architecture Infrastructure Cloud

TARGET AUDIENCE

Data Engineer Data Architect Cloud Architect Software Architect DevOps Engineer

EXTERNAL SOURCES

• Wikipedia - Shard (database architecture)
• MongoDB - Shard key selection guidance and trade-offs.
• Microsoft Azure - Partition key design considerations.
• Apache Cassandra - Partitioning and consistent hashing architecture.

TOP QUESTIONS

What is a Shard Key?

It is an attribute or set of attributes used to distribute data across shards. A good shard key balances load and supports efficient queries.

Category: Definition

How do I choose a Shard Key?

Favor high-cardinality, evenly distributed fields aligned with query filters. Avoid monotonically increasing values that cause hotspots.

Category: Requirements

What are the risks of a poor Shard Key?

Hot shards, uneven storage, cross-shard queries, and expensive rebalancing. It can degrade performance and complicate operations.

Category: Risks

Can I change the Shard Key later?

Some systems support resharding, but it may require full data migration and downtime risk. Plan shard key evolution early.

Category: Management

What is a hot topic around Shard Keys?

Adaptive and auto-sharding strategies that rebalance based on observed traffic and storage patterns are gaining traction.

Category: Topic

Hash vs Range Shard Keys: which to choose?

Hash keys evenly distribute writes but hinder range queries. Range keys favor ordered scans but risk hotspots unless carefully chosen.

Category: Alternatives

How does a Shard Key affect queries?

Queries with predicates on the shard key can route directly to relevant shards, reducing fan-out and latency.

Category: Performance

Should the Shard Key be indexed?

Yes, shard keys are typically indexed to support routing and efficient lookups. Some systems require it.

Category: Requirements

What is shard key skew and how to detect it?

Skew means disproportionate load on certain shards. Monitor shard-level throughput, storage, and latency to identify hotspots.

Category: Management