📐Evidence Lower Bound

NEW

Also known as: ELBO

A variational objective that lower-bounds the log evidence, optimized to approximate the posterior distribution.

DESCRIPTION

The evidence lower bound is the objective optimized in variational inference to approximate intractable posteriors. Maximizing the ELBO is equivalent to minimizing the Kullback–Leibler divergence between the variational distribution and the true posterior. It decomposes into an expected log-likelihood term and a complexity penalty (negative KL to the prior). ELBO underpins variational autoencoders and scalable stochastic variational inference. While fast, ELBO-based approximations can be biased and underdispersed versus MCMC. Practical success depends on flexible variational families, reparameterization gradients, and robust optimization.

ALTERNATIVE DEFINITIONS

Wikipedia

In variational inference, the evidence lower bound (ELBO) is a lower bound on the logarithm of the marginal likelihood (evidence).

Defines ELBO directly as a bound on log evidence within VI.

Source

Blei, Kucukelbir, McAuliffe (JASA Tutorial)

We derive the evidence lower bound (ELBO), an objective whose maximization corresponds to minimizing the Kullback–Leibler divergence to the posterior while maximizing a lower bound to the marginal likelihood.

Links ELBO optimization to KL minimization and model evidence.

Source

Pyro

The ELBO is the fundamental objective in variational inference; it provides a lower bound on the log marginal likelihood that is maximized to fit variational parameters.

Developer documentation framing ELBO as core VI objective.

Source

TensorFlow Probability

Variational inference optimizes the evidence lower bound (ELBO) to approximate posterior distributions in probabilistic models.

Shows ELBO’s role in practical VI with TFP examples.

Source

RELATED TERMS

Variational Inference Bayesian Inference Kullback–Leibler Divergence Log Marginal Likelihood Amortized Inference Autoencoding Variational Bayes Stochastic Variational Inference

COMPETING TERMS

Markov Chain Monte Carlo

KNOWLEDGE AREAS

Mathematics Data Science AI MLOps

TARGET AUDIENCE

Data Scientist AI Engineer MLOps Engineer

EXTERNAL SOURCES

• Wikipedia - Evidence lower bound
• Blei et al. - Foundational VI tutorial
• Pyro - Pyro ELBO docs
• TensorFlow Probability - TFP VI example
• Stan - VI in Stan overview

TOP QUESTIONS

Why is the Evidence Lower Bound central to variational inference?

Optimizing the ELBO yields the closest variational distribution to the posterior under KL divergence and provides a tractable training objective.

Category: Concept

How does the Evidence Lower Bound decompose?

It equals expected log-likelihood minus KL divergence to the prior, balancing data fit and model complexity.

Category: Concept

What are practical challenges optimizing the Evidence Lower Bound?

Variance in gradient estimates, poor variational families, and local optima can hinder training. Use reparameterization and richer families.

Category: Mistakes

How does the Evidence Lower Bound compare to MCMC approaches?

ELBO-based VI is faster and scalable but approximate; MCMC is asymptotically exact but computationally heavier.

Category: Comparison

What is the role of the reparameterization trick for the Evidence Lower Bound?

It reduces gradient variance by expressing sampling as deterministic transforms, enabling efficient stochastic optimization.

Category: Implementation

Can the Evidence Lower Bound be tightened?

Yes, objectives like the importance-weighted bound tighten the ELBO at added computational cost.

Category: Advances

How do I monitor convergence when maximizing the Evidence Lower Bound?

Track ELBO across iterations, validate with held-out likelihood estimates, and inspect posterior predictive performance.

Category: Evaluation

What models commonly use the Evidence Lower Bound?

Variational autoencoders, topic models, Bayesian neural networks, and probabilistic matrix factorization rely on ELBO optimization.

Category: Example

How does choice of variational family impact the Evidence Lower Bound?

Expressive families (e.g., normalizing flows) can achieve higher ELBOs and better posterior approximations.

Category: Best Practices