DataTerminology.comNEW
An open-source framework enabling distributed deep learning across GPUs and nodes using efficient ring-allreduce.
DESCRIPTION
Horovod is a distributed training framework originally developed at Uber that supports TensorFlow, PyTorch, and MXNet. It uses ring-allreduce to aggregate gradients efficiently across workers. Horovod simplifies scaling by abstracting communication primitives with MPI or NCCL backends. It integrates with Kubernetes, Spark, and common experiment tooling. Typical usage achieves near-linear scaling for many models. Horovod reduces boilerplate and improves portability of distributed training code. It remains a practical choice for multi-GPU and multi-node training workloads.
ALTERNATIVE DEFINITIONS
Uber Engineering
Horovod is a distributed training framework for TensorFlow, Keras, and PyTorch, built to make distributed deep learning fast and easy to use.
Emphasizes speed and ease of distributed training.
Horovod
Horovod is a distributed deep learning training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.
Lists supported frameworks and scope.
TensorFlow
Horovod is a distributed training framework that uses the ring-allreduce algorithm to average gradients over multiple processes.
Explains the ring-allreduce mechanism for gradient averaging.
RELATED TERMS
Distributed Training Ring-Allreduce TensorFlow PyTorch MXNet NCCL MPI Gradient Aggregation Parameter Server
COMPETING TERMS
DeepSpeed PyTorch DistributedDataParallel TensorFlow MirroredStrategy
KNOWLEDGE AREAS
MLOps AI Infrastructure Platform & Tooling
TARGET AUDIENCE
MLOps Engineer AI Engineer Data Scientist DevOps Engineer Cloud Architect
EXTERNAL SOURCES
- Wikipedia - Ring algorithm
- Uber Engineering - Foundational blog introducing Horovod
- Horovod - Source code and documentation
- NVIDIA NCCL - GPU communication library underpinning Horovod
- TensorFlow - Horovod tutorial for distributed training
TOP QUESTIONS
What is Horovod and why use it?
Horovod is a framework for distributed deep learning that simplifies scaling training across GPUs and nodes with efficient gradient aggregation.
Category: Overview
Which deep learning frameworks does Horovod support?
It supports TensorFlow, Keras, PyTorch, and MXNet via common communication backends like MPI and NCCL.
Category: Requirements
How does Horovod achieve scalability?
It uses ring-allreduce to average gradients across workers, minimizing bandwidth hotspots and enabling near-linear scaling.
Category: Performance
How does Horovod compare to PyTorch DistributedDataParallel?
Horovod abstracts cross-framework distribution with similar allreduce semantics, while DDP is PyTorch-native and tightly integrated.
Category: Difference
What are common mistakes when using Horovod?
Incorrectly setting batch sizes, not tuning fusion buffers, and misconfigured NCCL or MPI can hinder performance.
Category: Mistakes
What are the risks of distributed training with Horovod?
Network saturation, non-determinism, and failure recovery complexity can pose risks; monitoring and checkpointing are essential.
Category: Risk
How do you deploy Horovod on Kubernetes?
Use MPI operators or Kubernetes job templates, configure GPU resources, and ensure proper NCCL and network settings.
Category: Implementation
Can Horovod be combined with Spark?
Yes, HorovodRunner and integrations allow Spark clusters to launch distributed training jobs efficiently.
Category: UseCases
How to monitor Horovod performance?
Track throughput, step time, allreduce timing, network utilization, and GPU metrics; adjust fusion thresholds accordingly.
Category: Management