DataTerminology.comNEW
Also known as: Feature Engineering Pipeline
A data pipeline that computes, validates, and publishes features to offline and online stores.
DESCRIPTION
Feature pipelines transform raw events and records into reliable, reusable features for machine learning. They implement business logic, aggregations, window computations, and validation checks. Pipelines run in batch, streaming, or hybrid modes to meet freshness SLAs. Outputs are materialized to offline stores for training and to online stores for serving. Orchestration coordinates dependencies, backfills, and retries. Contracts and schema evolution protect downstream users. Testing, monitoring, and lineage ensure correctness and explainability over time.
ALTERNATIVE DEFINITIONS
Databricks
A feature engineering pipeline transforms raw data into features for model training and serving.
Connects transformation with both training and serving destinations.
Tecton
A feature pipeline defines how to compute a feature from raw data and materialize it to stores.
Highlights definition, computation, and materialization responsibilities.
Feast
Feature pipelines compute feature values from sources and publish them to offline and online stores.
Emphasizes dual publishing to offline and online stores.
RELATED TERMS
Feature Engineering Feature Store Online Feature Store Offline Feature Store Pipelines Extract-Transform-Load Streaming Pipeline Data Validation Orchestration Model Training Pipeline
COMPETING TERMS
On-demand Feature Computation
KNOWLEDGE AREAS
MLOps Data Engineering Real-time & Streaming Data Testing & Validation AI
TARGET AUDIENCE
MLOps Engineer Data Engineer AI Engineer Data Scientist Analytics Engineer Data Architect Software Developer
EXTERNAL SOURCES
- Wikipedia - Feature engineering
- Databricks - Feature engineering pipelines overview
- Google Cloud - Feature engineering on GCP
- Feast - Open-source feature pipelines
TOP QUESTIONS
What problems do Feature Pipelines solve?
They standardize feature computation and delivery, ensuring consistent definitions across training and serving, with governance, validation, and lineage built in.
Category: Relevancy
How do Feature Pipelines prevent data leakage?
They enforce point-in-time correctness, proper windowing, and avoid joins with future data, often validated through unit and integration tests.
Category: Quality
When should Feature Pipelines be streaming vs. batch?
Choose streaming for freshness-sensitive features and batch for heavy aggregations and backfills. Hybrid designs can combine both for cost and latency balance.
Category: Architecture
How are Feature Pipelines orchestrated?
Workflows use Airflow, Dagster, or similar tools with dependency graphs, retries, SLAs, and alerts to guarantee timely feature materialization.
Category: Management
What is a common mistake in Feature Pipelines?
Hard-coding business logic without tests or metadata leads to silent drift, broken schemas, and inconsistent feature semantics across teams.
Category: Mistakes
How do Feature Pipelines interact with feature registries?
Pipelines reference registry definitions for schemas, owners, and serving keys, and update lineage and version metadata upon materialization.
Category: Application
What’s required to productionize Feature Pipelines?
Version-controlled code, CI/CD, data validations, backfill strategy, observability, rollback plans, and documentation for ownership and SLAs.
Category: Requirements
How do Feature Pipelines support experimentation?
By generating reproducible training datasets and experiment-ready feature sets, enabling fair comparisons across model versions and A/B tests.
Category: Experimentation
What are risks in Feature Pipeline maintenance?
Undocumented changes, schema evolution without compatibility, and missing alerts can cause service regressions and degrade model performance.
Category: Risks