DataTerminology.comNEW
An operation or function that converts data from one format, structure, or representation into another.
DESCRIPTION
Transformations reshape, clean, and enrich data for analytics, operations, and machine learning. They include filtering, joining, aggregating, normalizing, and mapping columns or entities. In ETL, transformations occur before loading; in ELT, they execute within the target system. Robust transformations are declarative, tested, and lineage-tracked. In modeling, feature transformations improve model performance. Effective governance documents assumptions, quality rules, and reproducibility.
ALTERNATIVE DEFINITIONS
IBM
Data transformation is the process of converting data from one format or structure into another.
Concise definition centered on format and structure changes.
Microsoft Learn
Transformations are operations that cleanse, aggregate, and convert data into a format suitable for analysis.
Emphasizes analytics-ready outcomes and common operations.
dbt Labs
In dbt, transformations are SQL statements that build models by selecting and transforming raw data into analytics-ready tables or views.
Frames transformations as declarative SQL within warehouses.
RELATED TERMS
Extract-Transform-Load ELT Data Wrangling Feature Engineering Feature Scaling Aggregation Mapping SQL Schema Evolution
COMPETING TERMS
Inverse Transformation
RELATIONS
KNOWLEDGE AREAS
Data Engineering Data Integration & Interoperability Data Management
TARGET AUDIENCE
Data Engineer Analytics Engineer Data Scientist Data Architect Data Modeler
EXTERNAL SOURCES
- Wikipedia - Transformation (function)
- Microsoft - ETL guidance including transformations
- dbt Labs - Transformation modeling in dbt
- Google Cloud - Patterns for ETL and ELT
TOP QUESTIONS
Why is Transformation critical in data pipelines?
It turns raw data into clean, consistent, and analytics-ready assets, enabling reliable insights and downstream interoperability.
Category: Relevancy
How do ETL and ELT differ in Transformation?
ETL transforms before loading, often off-warehouse; ELT loads first and transforms in the warehouse, leveraging scalable compute.
Category: Opinion
What are common mistakes in Transformation design?
Undocumented assumptions, no tests, and brittle logic cause downstream issues and hidden data debt.
Category: Mistakes
How is Transformation governed?
Use data contracts, lineage, quality checks, and change management with version control and reviews.
Category: Management
What is the risk of excessive Transformation?
Complexity increases maintenance, obscures lineage, and can degrade performance or accuracy.
Category: Risks
How do you validate a Transformation?
Apply unit tests, schema checks, sampling, and compare aggregates and business rules against source-of-truth.
Category: Testing
When to use Transformation vs. Inverse Transformation?
Transform for analysis; use inverse to recover original scales for interpretation or reporting.
Category: Requirements
Where to start with Transformation tooling?
Begin with SQL-based tools like dbt, add orchestration, and integrate testing frameworks.
Category: Example
What’s a hot topic around Transformation?
Declarative dataframes and incremental, tested transformations integrated with catalogs and contracts are rising.
Category: Topic