DataTerminology.comNEW
Also known as: RNN
A neural network architecture with recurrent connections that process sequences by maintaining a hidden state over time.
DESCRIPTION
Recurrent Neural Networks model sequential data by feeding previous hidden states into the next step, capturing temporal dependencies. Variants like LSTM and GRU mitigate vanishing gradients and learn long-range patterns. RNNs powered earlier breakthroughs in speech recognition and machine translation. Transformers have largely supplanted vanilla RNNs for many NLP tasks, but RNNs remain useful for lightweight or streaming scenarios. Training uses backpropagation through time with sequence-aware regularization.
ALTERNATIVE DEFINITIONS
Wikipedia
A recurrent neural network (RNN) is a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence.
Highlights temporal directed cycles enabling sequence processing.
Goodfellow, Bengio, Courville
Recurrent networks are neural networks with loops, allowing information to persist.
Intuitive definition emphasizing memory via loops in the network.
RELATED TERMS
LSTM GRU Sequence Modeling Backpropagation Through Time Attention Mechanism Language Model Time Series Forecasting Vanishing Gradient Sequence-to-Sequence
COMPETING TERMS
RELATIONS
- Broader: Neural Network
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Recurrent neural network
- TensorFlow - Guide to RNNs in Keras
- PyTorch - API docs for RNN module
TOP QUESTIONS
What are RNNs good at?
Modeling sequences like text, audio, and time series where temporal dependencies matter.
Category: Overview
Why use LSTM or GRU instead of vanilla RNN?
They mitigate vanishing gradients and capture longer-term dependencies with gating mechanisms.
Category: Advice
How are RNNs trained?
Using backpropagation through time, unrolling the sequence and accumulating gradients across steps.
Category: Concept
What replaced RNNs in many NLP tasks?
Transformers, due to parallelization and superior long-range context handling.
Category: Trends
What is a common mistake with RNNs?
Feeding extremely long sequences without truncation or attention, leading to gradient issues and inefficiency.
Category: Mistakes
How do you regularize RNNs?
Use dropout on recurrent and input connections, weight decay, and early stopping.
Category: Practical
Are RNNs good for streaming inference?
Yes, RNNs can process step-by-step with a persistent state, enabling low-latency streaming.
Category: Application
What is teacher forcing in RNN training?
A technique that feeds the ground-truth token as the next input during training to stabilize learning.
Category: Concept
How do bidirectional RNNs help?
They process sequences in both directions, improving context for each timestep in offline settings.
Category: Benefits