🔗Query Federation

NEW

Technique enabling a single query to access multiple heterogeneous sources and combine results without moving data.

DESCRIPTION

Query federation uses an engine or connectors to access diverse sources—databases, object stores, and APIs—and execute a unified query. It applies pushdown where possible and combines partial results, enabling agility without centralizing data. Common platforms include Trino/Presto, BigQuery external tables, Athena federation, and PolyBase. Trade-offs include performance, security, and governance complexity.

ALTERNATIVE DEFINITIONS

Google Cloud

With federated queries, you can run SQL queries on data in external sources without having to load the data into BigQuery.

Emphasizes querying external sources without ingestion.

Source

AWS Athena

Athena federation allows you to run SQL queries across multiple data sources using data source connectors.

Highlights connector-based federation across diverse sources.

Source

Trino Documentation

Trino is a distributed SQL query engine for running fast analytic queries across multiple sources of data of varying sizes.

Positions Trino as a distributed engine spanning heterogeneous sources.

Source

RELATED TERMS

Data Virtualization Federated Query Presto Trino BigQuery External Tables Athena Federation PolyBase Dremio Logical Data Warehouse Data Lakehouse

COMPETING TERMS

Extract-Transform-Load

RELATIONS

• Broader: Federation

KNOWLEDGE AREAS

Data Integration & Interoperability Data Engineering Data Services Data Architecture Data Warehousing & BI

TARGET AUDIENCE

Data Engineer Data Architect Analytics Engineer BI Specialist Cloud Architect Solution Architect

EXTERNAL SOURCES

• Wikipedia - Federated database system
• Google Cloud - BigQuery federated queries
• AWS - Athena federation connectors
• Trino - Distributed SQL engine overview
• Microsoft - PolyBase external data guide

TOP QUESTIONS

When should Query Federation be used instead of ETL?

Use federation for exploratory analytics, low-latency access to changing data, or when copying is impractical. ETL suits stable, performance-critical workloads.

Category: Comparison

What are performance considerations in Query Federation?

Pushdown, partition pruning, and caching improve performance. Network latency and source limits can bottleneck distributed queries.

Category: Performance

How does Query Federation affect governance?

It requires consistent security, auditing, and metadata across sources. Data virtualization can centralize policies.

Category: Governance

What are common mistakes with Query Federation?

Ignoring source constraints, poor join strategies, and lack of caching result in slow queries and high costs.

Category: Mistakes

Can Query Federation support joins across databases?

Yes, engines like Trino, Athena, and PolyBase can join across connectors, subject to pushdown and type compatibility.

Category: Capability

How do security controls work in Query Federation?

Use centralized identity, least privilege, and per-connector access policies; audit federated query plans and data paths.

Category: Security

What is the cost impact of Query Federation?

Costs arise from scanning remote data and connector operations; optimize with filters, caching, and sampled exploration.

Category: Risks

What metadata challenges exist in Query Federation?

Heterogeneous schemas and types require harmonization; catalogs and schema mapping help standardize.

Category: Requirements

How do external tables enable Query Federation?

They reference data stored outside the warehouse, allowing queries without loading, often with limited DML support.

Category: Concept