DataTerminology.comNEW
A parameter carried across workflow steps to control configuration, branching, and execution dynamically at runtime.
DESCRIPTION
Flow variables parameterize workflows and pipelines, enabling dynamic configuration, branching, and reuse of nodes across environments. They can be set at global, workflow, or node scope, passed downstream, and bound to properties. Tools like KNIME, Airflow, NiFi, and Prefect implement variants (Variables, XCom, Parameters) with storage in memory, metadata, or key-value backends. Best practices include scoping, consistent naming, secrets handling, and validation to avoid brittle dependencies.
ALTERNATIVE DEFINITIONS
Apache Airflow
Airflow has a concept of Variables that are a generic way to store and retrieve arbitrary content or settings as a simple key value store.
Describes runtime configuration via key-value storage in orchestration.
KNIME
Flow variables represent parameters that can be used to control node settings and the execution of workflows.
Explicitly ties variables to node configuration and execution control.
Prefect
Parameters provide a way to pass inputs to flows and tasks at run time.
Focuses on runtime inputs passed into orchestration tasks.
RELATED TERMS
Workflow Orchestration Parameterization Airflow Variables XCom Pipeline Configuration Templating Metadata Directed Acyclic Graph
COMPETING TERMS
Environment Variable Configuration Parameter
KNOWLEDGE AREAS
Platform & Tooling Data Engineering MLOps Data Integration & Interoperability
TARGET AUDIENCE
Data Engineer MLOps Engineer DevOps Engineer Analytics Engineer
EXTERNAL SOURCES
- Wikipedia - Workflow
- Apache Airflow - Variables concept in Airflow
- KNIME - Flow variables in KNIME
- Apache NiFi - Variables in NiFi expression language
TOP QUESTIONS
How do Flow Variables differ from environment variables?
Flow variables are scoped to workflows and passed between nodes, while environment variables are system-level and accessed by processes globally.
Category: Misunderstanding
How are Flow Variables stored in orchestration tools?
They may be held in memory, metadata stores, or key-value backends. Airflow uses its metadata DB; Prefect uses runtime parameter passing.
Category: Requirements
What are common mistakes with Flow Variables?
Overusing globals, leaking secrets in logs, inconsistent naming, and implicit dependencies across nodes that reduce reproducibility.
Category: Mistakes
How do Flow Variables enable portability?
They parameterize paths, credentials, and thresholds, allowing workflows to move between dev, test, and prod with minimal changes.
Category: Best practices
How should secrets be handled as Flow Variables?
Use secret managers, never plain-text logs, and restrict variable scope. Rotate keys and audit access.
Category: Risk
Can Flow Variables control branching?
Yes. Variables guide conditional paths, loop iterations, and dynamic task selection based on upstream outcomes.
Category: Example
How do you test workflows using Flow Variables?
Mock variables, define defaults, and validate types. Create test DAGs or flows that simulate scenarios and edge cases.
Category: Testing
Are Flow Variables persisted across runs?
Persistence depends on the tool: some variables are stored; others exist only per run unless explicitly saved.
Category: Opinion
What naming conventions help with Flow Variables?
Use prefixes by scope or domain, lowercase with underscores, and document expected types and defaults.
Category: Best practices