DataTerminology.comNEW
Also known as: Self-Information
Measure of an event’s information equal to the negative logarithm of its probability.
DESCRIPTION
Shannon information, also called self-information, quantifies how surprising an event is based on its probability. Rare events have higher information content, measured as the negative log of probability, typically in bits. It underpins entropy, coding theory, and optimal compression, guiding how data is encoded to minimize redundancy. In machine learning, it relates to loss functions and model evaluation via cross-entropy and KL divergence. It supports anomaly detection, decision-making, and information-theoretic regularization across statistical modeling.
ALTERNATIVE DEFINITIONS
Wikipedia
In information theory, self-information is a measure of the information content associated with the outcome of a random variable, defined as the negative logarithm of the probability of the event.
Emphasizes the negative logarithm of probability and ties to event outcomes.
Wolfram MathWorld
Self-information, also called information content or surprisal, is defined as −log p for an event with probability p.
Highlights synonyms and the core formula using logarithms.
RELATED TERMS
Shannon Entropy Mutual Information Log Loss Kullback–Leibler Divergence Surprisal Information Content Log-Likelihood Coding Theory
COMPETING TERMS
Kolmogorov Complexity Fisher Information
RELATIONS
- Broader: Information
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer Data Architect Data Modeler
EXTERNAL SOURCES
- Wikipedia - Self-information
- David J. C. MacKay - Open book on information theory
- MIT OpenCourseWare - Lecture notes on information theory
TOP QUESTIONS
What is Shannon Information in simple terms?
Shannon Information measures how surprising an event is; rarer events carry more information, computed as the negative log of their probability.
Category: Definition
How is Shannon Information used in machine learning?
It underlies cross-entropy and KL divergence, which quantify model fit and guide optimization in classification and representation learning.
Category: Application
Why is logarithm used in Shannon Information?
Logarithms make information additive across independent events and connect to bits for practical coding and compression.
Category: Explanation
What units express Shannon Information?
Typically bits when using base-2 logarithms, but nats with natural logs and bans with base-10 also occur.
Category: Basics
How does Shannon Information relate to entropy?
Entropy is the expected self-information across a distribution, representing average uncertainty.
Category: Relation
Is Shannon Information the same as Fisher Information?
No. Fisher Information concerns parameter estimation sensitivity; Shannon Information measures event surprisal.
Category: Distinction
When was Shannon Information introduced?
Claude Shannon introduced the concept in 1948 in “A Mathematical Theory of Communication.”
Category: Origin
What is a common misunderstanding about Shannon Information?
People often confuse self-information with entropy; entropy is an average of self-information over possible outcomes.
Category: Misunderstanding
How does Shannon Information help with anomaly detection?
Low-probability events have high information; high surprisal can flag anomalies in data streams.
Category: Application