DataTerminology.comNEW
On-policy temporal-difference control updating Q-values from state–action–reward–state–action transitions.
DESCRIPTION
SARSA is a temporal-difference control algorithm that learns action values while following the current policy. It updates using quintuple transitions (s,a,r,s′,a′), hence its name. Being on-policy, it evaluates and improves the same behavior policy, typically ε-greedy. SARSA converges under standard conditions with decreasing exploration and appropriate step sizes. Eligibility traces yield SARSA(λ) for multi-step backups. Compared to Q-learning, SARSA tends to learn safer policies under exploration. It is effective in tabular settings and can be extended with function approximation.
ALTERNATIVE DEFINITIONS
Wikipedia
State–action–reward–state–action (SARSA) is an on-policy reinforcement learning algorithm for Markov decision processes.
Stresses on-policy nature and applicability to MDPs.
Sutton & Barto
SARSA is an on-policy TD control algorithm.
Highlights temporal-difference control while following the behavior policy.
RELATED TERMS
Temporal-Difference Learning On-Policy Learning Markov Decision Process Epsilon-Greedy Policy Evaluation Control Eligibility Traces TD(λ) Exploration-Exploitation
COMPETING TERMS
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - State–action–reward–state–action
- Sutton & Barto - SARSA algorithm details
- UCL (David Silver) - TD control including SARSA
- Stanford CS234 - RL course with TD methods
TOP QUESTIONS
How does SARSA differ from Q-Learning?
SARSA is on-policy and updates using the next action actually taken, whereas Q-learning is off-policy and uses the greedy next action. SARSA often learns safer policies under exploration.
Category: Alternatives
When should you use SARSA?
Use SARSA when on-policy behavior and safety under exploration matter. In stochastic environments, SARSA reflects the performance of the exploratory policy itself.
Category: Opinion
What is SARSA(λ)?
SARSA(λ) augments SARSA with eligibility traces for multi-step credit assignment. It blends Monte Carlo and TD updates for improved bias–variance trade-offs.
Category: Example
What exploration strategies fit SARSA?
ε-greedy and softmax (Boltzmann) exploration are common. Annealing ε over time ensures convergence under standard conditions.
Category: HowTo
What are common mistakes when implementing SARSA?
Using non-annealed high ε prevents convergence, and off-policy evaluation breaks theoretical guarantees. Incorrect step sizes can cause divergence.
Category: Mistakes
How does function approximation affect SARSA?
Function approximation scales SARSA but can introduce instability. Target networks and careful features can mitigate divergence.
Category: Risks
What is the origin of the name SARSA?
It derives from the quintuple transition used for updates: state, action, reward, state, action. This sequence defines the learning target.
Category: Origin
How is SARSA updated at each step?
It performs a TD update toward r + γ q(s′,a′), using the current policy’s chosen a′. The Q-table or approximator is then adjusted accordingly.
Category: HowTo
What environments highlight SARSA’s behavior?
Gridworlds with stochastic wind or slippage show SARSA’s safer, less risky paths during exploration compared to Q-learning’s optimistic updates.
Category: Example