DataTerminology.comNEW
A standardized benchmark suite for evaluating machine learning training and inference performance across hardware, software, and services.
DESCRIPTION
MLPerf is a community-driven benchmark maintained by MLCommons, covering standardized tests for ML training and inference. It includes divisions such as Training, Inference, Tiny, and HPC, spanning vision, NLP, recommendation, and more. Submissions follow strict rules for comparability, reproducibility, and fairness. Results guide hardware and system selection and inform optimization strategies. It fosters transparency in ML system performance.
ALTERNATIVE DEFINITIONS
MLCommons
MLPerf is a benchmark suite for machine learning that aims to provide unbiased evaluations of training and inference performance for hardware, software, and services.
Emphasizes unbiased, comparable performance across systems.
arXiv
MLPerf Training is a suite of benchmarks for evaluating training performance of machine learning hardware, software, and services.
Focuses on the Training division and performance evaluation objectives.
RELATED TERMS
MLCommons Benchmark Training Performance Inference Performance Throughput Delays Hardware Accelerator Reproduction Submission Rules
COMPETING TERMS
DAWNBench TPCx-AI
RELATIONS
- Implemented by: AMD MI300X
KNOWLEDGE AREAS
AI MLOps Infrastructure Observability & Telemetry
TARGET AUDIENCE
MLOps Engineer AI Engineer Hardware Engineer Cloud Architect
EXTERNAL SOURCES
- Wikipedia - MLPerf
- MLCommons - Official MLPerf overview
- MLCommons - Benchmarks and rules repositories
TOP QUESTIONS
What does MLPerf measure?
It measures standardized training and inference performance across representative ML workloads. Metrics include time-to-train, throughput, and latency under specified accuracy targets.
Category: Overview
Who maintains MLPerf?
MLCommons, an open engineering consortium, maintains the MLPerf benchmarks, rules, and submission processes in collaboration with industry and academia.
Category: Organization
How are MLPerf results comparable?
Strict rules define datasets, models, accuracy thresholds, and run conditions. Submissions are audited to ensure fairness and reproducibility.
Category: Quality
What are MLPerf divisions?
Divisions include Training, Inference, Tiny, and HPC, each targeting distinct deployment scenarios and device classes from datacenter to edge.
Category: Scope
How should practitioners use MLPerf results?
Use them to inform hardware and framework choices for target workloads. Always consider your specific data, constraints, and cost profiles alongside benchmark scores.
Category: Guidance
What are common mistakes when interpreting MLPerf?
Overgeneralizing from one workload to all tasks and ignoring power efficiency or cost can mislead decisions. Context matters beyond headline numbers.
Category: Mistakes
How often are MLPerf results published?
Results are typically released in cycles throughout the year for different divisions, with public leaderboards and detailed submission reports.
Category: Process
What benchmarks compete with MLPerf?
DAWNBench and TPCx-AI provide alternative perspectives. However, MLPerf is the most widely adopted community benchmark suite for ML systems.
Category: Alternatives
Does MLPerf include generative AI workloads?
Coverage is expanding. Some divisions include transformer-based tasks, and ongoing updates address emerging generative and large language model scenarios.
Category: Topic