🏗️Azure Data Factory

NEW

Also known as: ADF

A cloud-based ETL and data integration service that builds, schedules, and orchestrates data pipelines on Azure.

DESCRIPTION

Azure Data Factory enables code-free and code-centric data integration across on-premises and cloud sources. It orchestrates ingestion, transformation, and movement using pipelines, activities, and integration runtimes. Mapping Data Flows provide scalable transformations without managing Spark clusters. ADF integrates with Synapse, Databricks, and Azure Data Lake for end-to-end analytics. It supports hybrid scenarios via self-hosted integration runtime and offers monitoring, parameterization, and CI/CD.

ALTERNATIVE DEFINITIONS

Microsoft Learn

Azure Data Factory is a cloud-based ETL and data integration service that allows you to create data-driven workflows for orchestrating data movement and transforming data at scale.

Highlights cloud-based orchestration and scalable transformation.

Source

Microsoft Azure

Build, orchestrate, and monitor your ETL/ELT workflows code-free and at scale with Azure Data Factory.

Emphasizes code-free development and both ETL/ELT patterns.

Source

RELATED TERMS

Extract-Transform-Load ELT Pipelines Azure Synapse Analytics Azure Databricks Azure Data Lake Storage Integration Runtime Workflow Orchestration Dataflow Mapping Data Flows

COMPETING TERMS

AWS Glue Google Cloud Data Fusion Apache Airflow

RELATIONS

• Broader: Azure Extract-Transform-Load

KNOWLEDGE AREAS

Data Engineering Cloud Data Services Data Integration & Interoperability Platform & Tooling

TARGET AUDIENCE

Data Engineer Data Architect Cloud Architect MLOps Engineer Data Management Consultant

EXTERNAL SOURCES

• Microsoft - Official ADF introduction
• Microsoft - Azure data integration technology choices
• Microsoft - Mapping Data Flows in ADF

TOP QUESTIONS

When should I use Azure Data Factory?

Use ADF for orchestrating hybrid data pipelines, especially when you need managed connectivity, code-free transformations, and scheduling on Azure.

Category: Relevancy

How does Azure Data Factory compare to AWS Glue?

Both provide managed integration. ADF emphasizes code-free pipelines and hybrid connectivity; Glue focuses on Spark-based ETL with Glue Data Catalog integration.

Category: Alternatives

Can Azure Data Factory perform ELT?

Yes. ADF can orchestrate ELT by pushing transformations into data warehouses or Spark engines like Synapse and Databricks.

Category: Concept

How is Azure Data Factory deployed and versioned?

Use ARM/Bicep or Terraform for infrastructure, Git integration for pipeline JSON, and Azure DevOps or GitHub Actions for CI/CD.

Category: Management

What is a self-hosted Integration Runtime in ADF?

It is an on-premises or VM runtime enabling secure data movement between on-premises networks and Azure services.

Category: Infrastructure

What are Mapping Data Flows in ADF?

A visual, Spark-based transformation capability that scales without managing clusters, suitable for many ETL patterns.

Category: Tools

How do I monitor pipelines in Azure Data Factory?

Use the ADF monitoring UI, Azure Monitor, and Log Analytics for alerts, metrics, and diagnostics.

Category: Observability

What is a common mistake when using Azure Data Factory?

Overusing copy activities for transformations instead of leveraging ELT or Data Flows can increase cost and maintenance.

Category: Mistakes

How does Azure Data Factory integrate with Databricks and Synapse?

ADF orchestrates notebooks, SQL scripts, and Spark jobs, managing dependencies and parameters across services.

Category: Integration