DataTerminology.comNEW
A routing component that selects which expert networks process each token or example in a Mixture-of-Experts architecture.
DESCRIPTION
Router LMs use a learned or heuristic router to dispatch tokens to a subset of expert layers, enabling sparse activation for efficiency at scale. Designs include top-1 or top-2 routing with auxiliary losses to balance load. They can improve throughput and capacity while controlling compute cost. Notable implementations include GShard and Switch Transformer. Production use requires careful routing stability, expert capacity limits, and failure handling.
ALTERNATIVE DEFINITIONS
Switch Transformers (Google Research)
A routing layer selects a single expert for each token, which vastly simplifies routing and allows us to scale to very large sparse models.
Defines top-1 routing design and scalability benefits.
Sparsely-Gated Mixture-of-Experts (Google Brain)
A gating network routes examples to a sparse mixture of experts, enabling conditional computation and dramatically increasing model capacity.
Introduces gating as conditional computation for capacity expansion.
Google AI Blog
Mixture-of-Experts layers use a router to send each token to one or a few experts, activating only part of the model per example.
Explains token-level routing and sparse activation to broad audience.
RELATED TERMS
Sparse Activation Gating Network Switch Transformer Top-k Routing Expert Parallelism Load Balancing Transformer Model
COMPETING TERMS
Dense Transformer Static Ensemble
KNOWLEDGE AREAS
AI Data Science MLOps Infrastructure
TARGET AUDIENCE
AI Engineer MLOps Engineer Data Scientist Software Architect Hardware Engineer
EXTERNAL SOURCES
- Wikipedia - Mixture of experts
- Switch Transformers - Seminal paper on top-1 routing
- Google AI Blog - Overview of MoE with routing
- GShard - Scalable mixture-of-experts with routing
TOP QUESTIONS
Why use a Router LM in large models?
Routing enables sparse activation, increasing capacity without proportional compute. It’s crucial for scaling with manageable costs.
Category: Benefits
How does top-1 routing differ from top-2 routing in Router LMs?
Top-1 sends tokens to a single expert, simplifying compute; top-2 increases robustness and expressivity at slightly higher cost.
Category: Comparison
What are common pitfalls with Router LMs?
Load imbalance, expert collapse, and routing instability are common. Auxiliary losses and capacity constraints help mitigate them.
Category: Mistakes
How do Router LMs affect latency?
They can reduce average compute per token but introduce communication overhead. Efficient sharding and batching are key.
Category: Performance
Can Router LMs be fine-tuned?
Yes, both router and experts can be adapted. Freezing routers while fine-tuning experts can stabilize training.
Category: Application
What hardware patterns suit Router LMs?
Expert parallelism with high-bandwidth interconnects (NVLink, Infiniband) and pipeline scheduling helps manage communication.
Category: Infrastructure
Do Router LMs generalize better?
They can, due to increased capacity and specialization, but require careful regularization to avoid overfitting specialists.
Category: Opinion
How are experts chosen in Router LMs?
A learned gating function computes scores per expert; top-k with softmax and auxiliary losses guides dispatch and balance.
Category: Mechanism
How to monitor Router LM health in production?
Track expert utilization, load imbalance, token drop rates, and latency. Alert on drift in routing distributions.
Category: Observability