DataTerminology.comNEW
Also known as: Bandit problem, MAB
A sequential decision problem balancing exploration and exploitation to maximize cumulative reward across options with uncertain payoffs.
DESCRIPTION
The multi-armed bandit problem models repeated choices among uncertain options (arms) to maximize long-term reward. Algorithms trade exploration of less-known arms with exploitation of high-performing ones. Common strategies include epsilon-greedy, Thompson sampling, and upper confidence bounds. Bandits power real-time experimentation and personalization, offering faster learning than static A/B tests. Contextual variants use features to tailor decisions per user or situation.
ALTERNATIVE DEFINITIONS
Google Developers
multi-armed bandit: A problem in which a decision-maker must choose among multiple options (arms) with uncertain rewards, balancing exploration and exploitation to maximize cumulative reward.
Explicitly states the exploration–exploitation balance in sequential decisions.
Sutton & Barto (Reinforcement Learning, 2nd ed.)
The k-armed bandit problem is a learning task that exemplifies the fundamental trade-off between exploration and exploitation.
Canonical textbook framing of the exploration–exploitation dilemma.
RELATED TERMS
Exploration-Exploitation Tradeoff Epsilon-Greedy Thompson Sampling Upper Confidence Bound Contextual Bandits Online Learning Reinforcement Learning Regret Reward
COMPETING TERMS
A/B Testing
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer AI Product Manager Data Analyst
EXTERNAL SOURCES
- Wikipedia - Multi-armed bandit
- Sutton & Barto - Authoritative RL textbook
- Lilian Weng - Comprehensive bandit tutorial
- Google Developers - Concise bandit definition
TOP QUESTIONS
Why choose Multi-Armed Bandit over A/B Testing?
Bandits adapt allocation to better-performing options in real time, reducing regret. They are efficient when effects vary over time or multiple variants compete.
Category: Alternatives
What are common Multi-Armed Bandit algorithms?
Epsilon-greedy, Thompson sampling, and Upper Confidence Bound are widely used due to simplicity and strong empirical/regret guarantees.
Category: Example
What is the biggest mistake when using Multi-Armed Bandit in production?
Ignoring delayed or censored rewards leads to biased updates. Logging and counterfactual evaluation are essential for reliable learning.
Category: Mistakes
How does a Contextual Multi-Armed Bandit differ?
It conditions decisions on features (context) such as user or time, tailoring arm selection to the situation and improving performance.
Category: Topic
How do you evaluate Multi-Armed Bandit performance?
Regret curves, cumulative reward, and off-policy evaluation methods assess algorithm efficiency and stability.
Category: Requirements
What risks exist with Multi-Armed Bandit algorithms?
Non-stationarity, reward delays, and feedback loops can mislead learning. Guardrails and monitoring are critical.
Category: Risks
When did the Multi-Armed Bandit problem originate?
The bandit problem dates to early sequential design research, with formal treatments from the 1930s onward and extensive reinforcement learning work later.
Category: Origin
Can Multi-Armed Bandits handle constraints?
Yes. Constrained bandit formulations and contextual policies can manage budgets, fairness, or safety limits with appropriate algorithms.
Category: Opinion
How does exploration strategy affect Multi-Armed Bandit outcomes?
Aggressive exploration can discover better options faster but increases short-term regret; conservative strategies may converge prematurely. Balance depends on business goals.
Category: Relevancy