DataTerminology.comNEW
Also known as: Airflow
An open-source platform to programmatically author, schedule, and monitor workflows as DAGs.
DESCRIPTION
Apache Airflow orchestrates workflows defined as Directed Acyclic Graphs (DAGs) of tasks. It provides a scheduler, web UI, and pluggable executors to run tasks on various backends. Operators and hooks integrate with databases, cloud services, and APIs. Airflow supports retries, SLAs, backfills, and dependencies, enabling robust data pipeline automation. It suits batch ETL, ML pipelines, and complex multi-step workflows.
ALTERNATIVE DEFINITIONS
Apache Software Foundation
Apache Airflow is a platform to programmatically author, schedule and monitor workflows.
Concise statement emphasizing programmatic workflow management.
Wikipedia
Apache Airflow is an open-source workflow management platform.
Identifies Airflow as an open-source workflow platform.
Astronomer
Apache Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows.
Highlights batch orientation and core lifecycle actions.
RELATED TERMS
Workflow Orchestration Directed Acyclic Graph Extract-Transform-Load Operator Scheduler Sensor Task Executor Orchestration
COMPETING TERMS
RELATIONS
- Broader: Workflow Orchestration
KNOWLEDGE AREAS
Data Engineering Platform & Tooling MLOps Software Engineering
TARGET AUDIENCE
Data Engineer MLOps Engineer DevOps Engineer Analytics Engineer
EXTERNAL SOURCES
- Wikipedia - Apache Airflow
- Apache Airflow - Official Airflow documentation
- Astronomer - Airflow concepts and best practices
- Google Cloud - Managed Airflow (Cloud Composer)
- AWS - Managed Workflows for Apache Airflow
TOP QUESTIONS
Why use Apache Airflow for data pipelines?
Airflow provides robust scheduling, dependency management, and a rich ecosystem of integrations. DAGs offer transparency and versionable definitions. It suits complex, batch-oriented workflows.
Category: Relevancy
What is a DAG in Apache Airflow?
A DAG is a Directed Acyclic Graph representing tasks and dependencies. It defines execution order without cycles. Airflow uses DAGs to orchestrate workflows.
Category: Overview
When should I consider alternatives to Airflow?
For simpler jobs or primarily event-driven pipelines, consider Prefect or cloud-native orchestrators. For heavy streaming, use stream processors instead of job schedulers.
Category: Alternatives
What are common mistakes with Airflow?
Putting heavy logic inside operators instead of calling external services causes bloat. Overly complex DAGs reduce maintainability. Neglecting retries and idempotency harms reliability.
Category: Mistakes
How does Airflow handle retries and failures?
Tasks have retry policies, exponential backoff, and alerts. Airflow supports resuming from failed tasks and backfilling missed schedules.
Category: Management
Can Airflow orchestrate ML workflows?
Yes. It triggers training, evaluation, and deployment steps and integrates with ML platforms. It complements, not replaces, model training frameworks.
Category: Application
How do executors affect Airflow scalability?
Executors (Local, Celery, Kubernetes) determine how tasks run. KubernetesExecutor and Celery scale horizontally for larger workloads.
Category: Requirements
How is Airflow different from stream processing engines?
Airflow schedules discrete tasks in batch-oriented workflows. Stream processors continuously handle events with low latency. They serve different workloads.
Category: Distinction
What is the role of XComs in Airflow?
XComs let tasks share small pieces of metadata. For large data, use external storage; XComs are not meant for bulk transfers.
Category: BestPractice