DataTerminology.comNEW
Also known as: Randomization Test
A nonparametric hypothesis test that assesses significance by computing a statistic over many label permutations to approximate the null distribution.
DESCRIPTION
Permutation tests evaluate hypotheses without distributional assumptions by generating a null distribution through systematic or random relabeling of observations. They are widely used in A/B testing, biomedical studies, and small-sample scenarios. Exact tests enumerate all permutations; approximate tests use Monte Carlo sampling to estimate p-values efficiently. They are robust to non-normality and handle complex statistics. Careful selection of the test statistic and sufficient permutations are critical for reliability. Multiple testing corrections may be needed when many hypotheses are assessed.
ALTERNATIVE DEFINITIONS
Wikipedia
In statistics, a permutation test (also called a randomization test) is a type of significance test in which the null distribution of a test statistic is obtained by calculating all possible values of the test statistic under rearrangements of the labels on the observed data points.
Emphasizes construction of the null distribution by all possible label rearrangements and the nonparametric nature.
SciPy
Perform a permutation test to determine the p-value for a null hypothesis using a statistic function evaluated under many permutations of sample labels.
Focuses on practical computation via user-supplied statistic and Monte Carlo permutations.
Penn State Eberly College of Science
Randomization (permutation) tests assess significance by repeatedly shuffling treatment labels among observations to build a reference distribution for the test statistic without distributional assumptions.
Highlights applicability when parametric assumptions are questionable.
NIST/SEMATECH e-Handbook of Statistical Methods
Randomization tests use permutations of the data to evaluate the statistical significance of an observed result, providing exact or approximate inference without relying on parametric distributions.
Underlines exact inference and independence from parametric forms.
RELATED TERMS
Hypothesis Testing p-value Null Distribution Monte Carlo Simulation A/B Testing Nonparametric Statistics Resampling Multiple Testing
COMPETING TERMS
KNOWLEDGE AREAS
Mathematics Data Science Testing & Validation
TARGET AUDIENCE
Data Scientist Data Analyst AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Permutation test
- SciPy - Python API for permutation tests
- Penn State - Course notes on randomization tests
- NIST - Handbook section on randomization tests
TOP QUESTIONS
When should a Permutation Test be preferred over a t-test?
Use it when normality and equal variance assumptions are doubtful, sample sizes are small, or the test statistic is complex. It offers robust, distribution-free inference.
Category: Requirements
How does a Permutation Test compute the p-value?
It builds the null distribution by permuting labels, then measures the proportion of permuted statistics at least as extreme as the observed statistic.
Category: Example
What is a common mistake in Permutation Tests?
Using too few permutations can yield noisy p-values. Always set sufficient iterations or compute exact permutations when feasible.
Category: Mistakes
How is a Permutation Test applied in A/B testing?
Shuffle assignment labels between variants to estimate the null distribution of your chosen metric and derive a p-value without parametric assumptions.
Category: HowTo
What are the risks of using a Permutation Test incorrectly?
Improper label shuffling, dependence violations, or leakage can bias results. Inadequate permutations inflate Type I/II errors.
Category: Risks
Is the Permutation Test computationally expensive?
Exact tests can be expensive for large datasets, but Monte Carlo approximations are efficient and often sufficient.
Category: Opinion
How many permutations are enough in a Permutation Test?
Thousands are typical; ensure stability of p-value estimates. Use stopping rules or confidence intervals for the p-value if uncertain.
Category: Requirements
What is a hot topic around Permutation Tests today?
Efficient permutation strategies and multiple testing corrections for high-dimensional data and complex metrics are active areas.
Category: Topic
What is the biggest misunderstanding about a Permutation Test?
Some think it’s always exact; in practice, Monte Carlo estimates are common and reliable with adequate sample size.
Category: Misunderstanding