🔧Data Transformation

NEW

The process of converting data from one format or structure to another to make it usable for analysis or applications.

DESCRIPTION

Data transformation reshapes and enriches raw inputs into analysis-ready or application-ready outputs. Common operations include cleaning, type conversion, joins, aggregations, normalization, and feature creation. It can occur before loading (ETL) or after loading (ELT) using scalable compute. Declarative frameworks and pipelines improve maintainability and reuse. Robust testing and lineage ensure trust and traceability.

ALTERNATIVE DEFINITIONS

AWS Glue

AWS Glue jobs can perform data transformations to convert, clean, and enrich data as it moves through ETL pipelines.

Positions transformation within ETL jobs on managed service.

Source

Databricks

Data transformation applies operations like filtering, joining, aggregating, and cleaning to prepare datasets for analytics and machine learning.

Enumerates typical operations for analytic readiness.

Source

Oracle

Transformation in ETL changes data into a desired format, structure, or value set before loading into a target.

Describes transformation as the T in ETL pipeline context.

Source

RELATED TERMS

Extract-Transform-Load ELT Data Wrangling Schema Mapping Feature Scaling Cleansing Feature Engineering Aggregation Join Model Data

COMPETING TERMS

Data Virtualization

RELATIONS

• Broader: Transformation

KNOWLEDGE AREAS

Data Engineering Data Integration & Interoperability Data Modeling Data Management

TARGET AUDIENCE

Data Engineer Analytics Engineer Data Modeler Data Architect MLOps Engineer BI Specialist

EXTERNAL SOURCES

• Wikipedia - Extract, transform, load
• dbt Labs - Declarative SQL transformations and testing
• Apache Spark - Distributed transformations using Spark SQL/DataFrames
• Google BigQuery - SQL transformations at warehouse scale

TOP QUESTIONS

Why is Data Transformation necessary?

Raw data rarely fits analytical models or application schemas. Transformation makes data consistent, useful, and performant.

Category: Relevancy

ETL or ELT for Data Transformation?

Use ETL when pre-load cleansing is required; ELT leverages modern warehouses for scalable post-load transforms. Context determines choice.

Category: Opinion

What mistakes occur in Data Transformation?

Hidden business logic, poor testing, and undocumented changes cause breakage and confusion.

Category: Mistakes

How to ensure quality in Data Transformation?

Add unit tests, data contracts, schema validation, and lineage tracking. Automate CI for transformations.

Category: BestPractice

What is the risk of over-transforming data?

You lose raw fidelity and complicate governance. Maintain raw zones and document derivations.

Category: Risk

Which tools perform Data Transformation?

dbt, Spark, SQL engines, Glue jobs, and ETL platforms are common choices.

Category: Platform

How to start with Data Transformation design?

Define target models, write declarative transforms, test incrementally, and version changes.

Category: GettingStarted

How does Data Transformation relate to Data Modeling?

Transforms implement model logic, mapping source structures to modeled schemas and metrics.

Category: Relation

What is currently hot in Data Transformation?

Declarative SQL pipelines, data contracts, and metadata-driven transformation orchestration.

Category: Topic