Shannon entropy

E1168
entropy measure information theory concept random variable functional uncertainty measure

Shannon entropy is a fundamental measure in information theory that quantifies the average uncertainty or information content in a random variable or message source.

Try in SPARQL Jump to: Surface forms Statements Referenced by

All labels observed (5)

Label Occurrences
Shannon entropy canonical 9
Shannon additivity axiom 1
Shannon information 1

Statements (50)

Predicate Object
instanceOf entropy measure ⓘ
information theory concept ⓘ
random variable functional ⓘ
uncertainty measure ⓘ
appliesTo discrete probability distributions ⓘ
discrete random variables ⓘ
captures expected codeword length lower bound in lossless compression ⓘ
dependsOn probability distribution of a random variable ⓘ
field information theory ⓘ
generalizes Hartley entropy ⓘ
hasFormula H(X) = -\sum_x p(x) \log p(x) ⓘ
introducedBy Claude Shannon ⓘ
introducedInWork A Mathematical Theory of Communication ⓘ
introducedInYear 1948 ⓘ
invariantUnder relabeling of outcomes ⓘ
isAdditiveFor independent random variables ⓘ
isConcaveIn probability distribution ⓘ
isMaximumWhen distribution is uniform ⓘ
isMinimumWhen distribution is degenerate ⓘ
isNonNegative true ⓘ
isSpecialCaseOf Rényi entropy ⓘ
Tsallis entropy ⓘ
logarithmBaseDetermines unit of information ⓘ
measuredIn bits ⓘ
hartleys ⓘ
nats ⓘ
minimumValue 0 ⓘ
namedAfter Claude Shannon ⓘ
quantifies average information content of a random variable ⓘ
average uncertainty of a random variable ⓘ
relatedConcept Kullback–Leibler divergence ⓘ
conditional entropy ⓘ
differential entropy ⓘ
mutual information ⓘ
relative entropy ⓘ
satisfies Shannon–Khinchin axioms ⓘ
chain rule for entropy ⓘ
symbol H ⓘ
usedIn bioinformatics ⓘ
channel coding theory ⓘ
cryptography ⓘ
data compression theory ⓘ
ecology diversity indices ⓘ
machine learning ⓘ
neuroscience ⓘ
signal processing ⓘ
statistical mechanics ⓘ
thermodynamics analogies ⓘ
usedToDefine A Mathematical Theory of Communication ⓘ
surface form: Shannon capacity of a channel

entropy rate of a stochastic process ⓘ

Referenced by (13)

Full triples — surface form annotated when it differs from this entity's canonical label.

Claude Shannon → knownFor → Shannon entropy ⓘ
Claude → notableConcept → Shannon entropy ⓘ
subject surface form: Claude Shannon
A Mathematical Theory of Communication → associatedWithConcept → Shannon entropy ⓘ
this entity surface form: Shannon limit
Kullback–Leibler divergence → relatedTo → Shannon entropy ⓘ
Rényi entropy → generalizes → Shannon entropy ⓘ
Shannon–Khinchin axioms → characterizes → Shannon entropy ⓘ
Shannon–Khinchin axioms → hasAxiom → Shannon entropy ⓘ
this entity surface form: Shannon additivity axiom
Shannon–Khinchin axioms → relatedTo → Shannon entropy ⓘ
Maxwell's demon thought experiment → relatedConcept → Shannon entropy ⓘ
this entity surface form: Shannon information
Boltzmann–Gibbs entropy in statistical mechanics → relatedTo → Shannon entropy ⓘ
subject surface form: Boltzmann–Gibbs entropy
Faddeev’s axioms → characterizes → Shannon entropy ⓘ
information theory → usesUnit → Shannon entropy ⓘ
this entity surface form: shannon
Mathematical Foundations of Information Theory → centralConcept → Shannon entropy ⓘ