In mathematical statistics, the Kullback–Leibler divergence, {\displaystyle D_{\text{KL}}}, is a measure of how one probability distribution is different from a second, reference probability distribution.

What is KL divergence used for?

To measure the difference between two probability distributions over the same variable x, a measure, called the Kullback-Leibler divergence, or simply, the KL divergence, has been popularly used in the data mining literature. The concept was originated in probability theory and information theory.

What is KL divergence in machine learning?

The Kullback-Leibler divergence (hereafter written as KL divergence) is a measure of how a probability distribution differs from another probability distribution. ... In this context, the KL divergence measures the distance from the approximate distribution Q to the true distribution P .