🕸️Apache Tez

NEW

A distributed execution framework that runs complex DAGs of tasks to accelerate big data processing on Hadoop.

DESCRIPTION

Apache Tez provides a DAG-based execution engine for building high-performance batch and interactive data processing applications. By optimizing data movement and task scheduling on YARN, Tez improves on MapReduce’s rigid phases. Hive-on-Tez and Pig-on-Tez significantly reduced query latencies in Hadoop ecosystems. Tez’s flexible APIs let engines express complex workflows as DAGs, minimizing disk IO and shuffles. Although Spark has become dominant for many workloads, Tez remains relevant for certain Hadoop deployments.

ALTERNATIVE DEFINITIONS

Apache Tez

Apache Tez is an application framework which allows for a complex directed-acyclic-graph of tasks for processing data.

Emphasizes DAG execution model for data processing.

Source

Wikipedia

Apache Tez is a framework for building high performance batch and interactive data processing applications, coordinated as directed acyclic graphs.

Highlights performance improvements and DAG coordination in Hadoop.

Source

RELATED TERMS

Apache Hive Apache Pig YARN Directed Acyclic Graph Hadoop

COMPETING TERMS

MapReduce Apache Spark

KNOWLEDGE AREAS

Data Engineering Real-time & Streaming Data Platform & Tooling Infrastructure

TARGET AUDIENCE

Data Engineer Data Architect Software Developer DevOps Engineer

EXTERNAL SOURCES

• Wikipedia - Apache Tez
• Apache Tez - Tez Application Master architecture
• Apache Hive - Hive on Tez configuration and benefits
• Hortonworks (archive) - Background and vision for Tez

TOP QUESTIONS

Why was Apache Tez created?

Tez was created to overcome MapReduce’s fixed phases by expressing data processing as DAGs, reducing IO and improving performance.

Category: Origin

How does Tez improve Hive performance?

Hive-on-Tez uses DAG optimization to minimize shuffles and disk writes, significantly reducing query latency over MapReduce.

Category: Performance

What is the role of YARN with Tez?

Tez leverages YARN for cluster resource management, while Tez coordinates DAG execution within allocated containers.

Category: Architecture

Is Tez still relevant in modern data stacks?

It remains relevant in Hadoop-centric environments, though Spark is preferred for many new workloads.

Category: Relevancy

How do I migrate from MapReduce to Tez?

Enable Tez in Hive or Pig, test query plans, and tune configurations to exploit DAG execution benefits.

Category: Migration

What are common Tez tuning knobs?

Adjust parallelism, container sizes, and I/O sort buffers; use vectorized execution in Hive for further gains.

Category: BestPractices

Can Tez be used outside Hive and Pig?

Yes, Tez provides APIs to build custom DAG-based engines, though most adoption comes via Hive and Pig.

Category: Capabilities

What risks exist when adopting Tez?

Operational complexity and dependency on YARN and Hadoop versions can complicate upgrades and maintenance.

Category: Risk

How is fault tolerance handled in Tez?

Failed tasks are retried within the DAG; YARN container failures are handled via reallocation and re-execution.

Category: Resilience