DataTerminology.comNEW
Also known as: Split Test
A controlled experiment comparing two variants to measure causal effects.
DESCRIPTION
An A/B test randomly assigns users to two variants to estimate the causal impact of a change on a predefined metric. It relies on randomization, sufficient sample size, and pre-registered hypotheses. Proper guardrails include SRM checks, power analysis, and invariant metrics. Sequential monitoring risks inflate false positives without correction. A/B testing underpins product optimization, marketing experiments, and model rollout validation.
RELATED TERMS
Hypothesis Testing Randomized Controlled Trial Power Analysis Statistical Significance Sequential Testing Sample Ratio Mismatch Experiment Design Uplift Modeling Metric Definition
COMPETING TERMS
Multivariate Test Multi-armed Bandit
RELATIONS
KNOWLEDGE AREAS
Data Science Testing & Validation Data Products
TARGET AUDIENCE
Data Scientist Data Analyst BI Specialist AI Product Manager Product Owner MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - A/B testing
- Microsoft Learn - Azure architecture guide on A/B testing
- Optimizely - Vendor guide to A/B testing
- Nielsen Norman Group - Usability perspective on A/B testing
TOP QUESTIONS
When should an A/B Test be preferred over other methods?
Use A/B tests for binary choices and when clean randomization is feasible. They are simple, interpretable, and robust for isolated changes. Avoid when interactions dominate.
Category: Application
How many users are needed for an A/B Test?
Use power analysis based on baseline rate, desired effect size, alpha, and power. This quantifies minimum detectable effect and sample size requirements.
Category: Requirements
What is Sample Ratio Mismatch in an A/B Test?
SRM occurs when observed allocation deviates from planned split, often due to implementation bugs. It invalidates inference and must be resolved before analysis.
Category: Mistakes
How do you choose metrics for an A/B Test?
Define a primary outcome aligned to goals, guardrail metrics for safety, and diagnostics for mechanism insight. Pre-register definitions to avoid p-hacking.
Category: Best practices
What are risks of peeking in A/B Tests?
Repeated looks inflate false positive rates unless corrected with sequential methods. Use alpha spending or group sequential designs to maintain validity.
Category: Risks
Can A/B Tests handle seasonality or external shocks?
Randomization helps, but large shocks can bias results. Stratify, run longer, or use CUPED to reduce variance. Monitor for concurrent changes.
Category: Challenges
How to run A/B Tests for ML model rollouts?
Randomly assign traffic to model versions, define guardrails, and monitor latency. Use canary then A/B for confidence, with rollback plans.
Category: Deployment
What is the biggest misunderstanding about A/B Tests?
Many think any observed difference is meaningful; without power and correct analysis, results can mislead. Statistical rigor is essential.
Category: Misunderstanding
How do Multi-armed Bandits relate to A/B Tests?
Bandits adaptively allocate traffic to better variants, optimizing reward during the experiment. A/B tests focus on inference under fixed allocation.
Category: Comparison