🔄Change Data Capture

NEW

Also known as: CDC

Method for capturing database inserts, updates, and deletes to propagate incremental changes downstream.

DESCRIPTION

Change Data Capture (CDC) identifies and streams row-level inserts, updates, and deletes from operational databases to downstream systems with minimal latency. It reduces processing costs by avoiding full refreshes and supports near-real-time analytics and microservices. Implementations include log-based, trigger-based, and timestamp-based capture, with log-based CDC favored for low overhead and fidelity. CDC powers event-driven architectures, data lakehouse ingestion, and materialized views. Popular ecosystems use Debezium with Kafka Connect, cloud-native CDC services, or vendor tools like Oracle GoldenGate. Robust CDC requires idempotent consumers, ordering guarantees, and exactly-once semantics when possible. Schema evolution and backfill strategies are essential to maintain data consistency. Security and privacy controls are necessary because change logs may expose sensitive values.

ALTERNATIVE DEFINITIONS

Microsoft Learn

Change data capture records insert, update, and delete activity applied to a SQL Server table.

Emphasizes SQL Server implementation and the specific focus on table-level DML activity.

Source

Wikipedia

In databases, change data capture (CDC) is a set of software design patterns used to determine and track the data that has changed so that action can be taken using the changed data.

Frames CDC as a design pattern and highlights actionability on changed data.

Source

RELATED TERMS

Log-Based Replication Kafka Connect Debezium Stream Processing Event Sourcing Incremental Refresh Extract-Transform-Load Pipelines Audit Logging Change Tracking

COMPETING TERMS

Full Load ETL

KNOWLEDGE AREAS

Data Engineering Real-time & Streaming Data Data Integration & Interoperability Data Storage & Operations

TARGET AUDIENCE

Data Engineer Data Architect MLOps Engineer Cloud Architect Data Management Consultant

EXTERNAL SOURCES

• Wikipedia - Change data capture
• Debezium - Open-source CDC documentation
• Confluent - Kafka Connect CDC connectors
• Microsoft - SQL Server CDC overview
• Oracle - Oracle GoldenGate CDC docs

TOP QUESTIONS

Why use Change Data Capture instead of full-load ETL?

CDC reduces compute and time by only moving deltas. It enables near-real-time pipelines and minimizes source impact. This makes analytics fresher and operations more efficient.

Category: Relevancy

What are common CDC implementation approaches?

Log-based CDC reads transaction logs, trigger-based CDC uses database triggers, and query-based CDC compares snapshots or timestamps. Log-based approaches typically have the lowest overhead and best fidelity.

Category: Example

How does Change Data Capture integrate with Kafka?

CDC tools like Debezium publish change events to Kafka topics. Downstream consumers process these events for analytics, materialized views, or microservices in near real time.

Category: Application

What are the risks of applying Change Data Capture?

Incorrect ordering or duplicate handling can cause inconsistent states. Sensitive fields may be exposed, and insufficient backpressure control can overwhelm consumers. Robust monitoring is essential.

Category: Risks

How is Change Data Capture managed in production?

Teams monitor lag, throughput, and error rates, manage schema evolution, and ensure idempotence. They maintain replay capabilities and implement access controls for sensitive change logs.

Category: Management

What is a hot topic around Change Data Capture?

Exactly-once processing semantics and lakehouse ingestion via CDC are hot. Cloud-managed CDC services and change tables for time travel also spark attention.

Category: Topic

How do schema changes impact Change Data Capture?

Schema evolution requires updating event schemas, connectors, and consumers. Using schema registries and versioning strategies helps maintain compatibility and minimize breakage.

Category: Requirements

What’s a competing approach to Change Data Capture?

Full-load ETL periodically reloads entire tables. It’s simpler but more resource-intensive and has higher latency, making it less suitable for real-time needs.

Category: Alternatives

What is the biggest misconception about Change Data Capture?

Many assume CDC always guarantees exactly-once delivery; in practice, idempotent consumers and deduplication are required to achieve effective exactly-once behavior end-to-end.

Category: Misunderstanding