Maximum Likelihood Learning
Table of Contents
Suppose we a general generative model \( \mathcal{M} \). We want to learn the model given a dataset \( \mathcal{D} \) of \( m \) samples from \( p_{data} \). The standard assumption is that the data instances are independent and identically distributed.
1. Motivation
The goal of learning is to return a model \( p_{\theta} \) that best approximate the underlying distribution \( p_{data} \).
How can we define the “best”? It depends on our task. If our task is image generation, then the “best” means: we want to construct \( p_{\theta} \) as “close” as possible to \( p_{data} \).
2. Derivation
How can we evaluate “closeness” between 2 distributions? One common approach is the KL-divergence, which is defined as
\[ D(p \parallel q) = \sum_{x} p(x) \log \frac{p(x)}{q(x)} := \mathbb{E}_{x\sim p}\left[ \log\frac{p(x)}{q(x)} \right] \]
Thus, in the context of image generation, if the learned distribution is close enough to \( p_{data} \), then we can simply sample images from our model’s distribution. So, we can try to minimize the KL-divergence between model’s distribution and \( p_{data} \)
\[ D(p_{data}\parallel p_{\theta}) = \mathbb{E}_{x\sim p_{data}}\left[ \log\frac{p_{data}(x)}{p_{\theta}(x)} \right] \]
The key here is that, KL-divergence measures the “compression loss” of using \( p_{\theta} \) instead of \( p_{data} \), because we can only modify \( p_{\theta} \) through training parameters.
2.1. From Minimizing KL Divergence to Maximizing Expected Log-Likelihood
We can simplify the KL divergence
\[ D(p_{data}\parallel p_{\theta}) = \mathbb{E}_{x\sim p_{data}}[\log p_{data(x)}] - \mathbb{E}_{x\sim p_{data}}\left[ \log p_{\theta}(x) \right] \]
Since the first term is independent on \( \theta \), so minimizing KL divergence is equivalent to maximizing expected log-likelihood.
However, the problem nowhere is: we don’t know \( p_{data} \), how can we evaluate \( p_{data}(x) \)?
2.2. Applying Monte Carlo Estimation
We don’t know \( p_{data} \), but we have dataset, which is assumed to be an unbiased sample from \( p_{data} \).
We can apply Monte Carlo Estimation on dataset
\[ \mathbb{E}_{x\sim p_{data}}\left[ \log p_{\theta}(x) \right] \to \mathbb{E}_{x\in \mathcal{D}}\left[ \log p_{\theta}(x) \right] = \frac{1}{ |\mathcal{D}| } \sum_{x\in\mathcal{D}} \log p_{\theta}(x) \]
This can be regarded as the loss for training, we denote it as \( \log L(\theta, \mathcal{D}) \). Then the rest is to figure out a expression of \( \log p_{\theta}(x) \) to train the model.
A typical example is autoregressive LLM.
2.3. Gradient Estimation
We may apply gradient descent with the loss, since \( \ell(\theta) = \log L(\theta, \mathcal{D}) \). Then the gradient can be computed by \( \nabla_{\theta}\ell(\theta) = \sum_{x\in\mathcal{D}} \nabla_{\theta} \log p_{\theta}(x) \).
3. Take-Away
Under the framework of maximum likelihood estimation (MLE), the most important part is to figure out a feasible form of \( p_{\theta}(x) \), as well as gradient \( \nabla_{\theta} \log p_{\theta}(x) \).