Kullback–Leibler divergence
quantifies the information difference when describing P in terms of Q.
symbols, variables and units
P,Q: discrete probability; D_KL: nat.
applicable conditions and boundaries
Where P(x)>0 requires Q(x)>0 to be finite; asymmetric, not a distance measure.
formula source code
The following is a copyable LaTeX expression.
D_{KL}(P\Vert Q)=\sum_x P(x)\ln\frac{P(x)}{Q(x)}Reference and Extended Learning
MIT OpenCourseWare · Information Theory ↗is organized according to model definition and assumptions. Please check actual conditions and original literature before engineering, research and clinical use.
KL divergencerelative entropy