DDPM: Denoising Diffusion Probabilistic Models

DDPM parameterizes the reverse-process mean through a network that predicts the noise added to \(\mathbf{x}_{0}\). Predicting noise is not the definition of a diffusion probabilistic model—one could instead predict \(\mathbf{x}_{0}\) or the reverse mean directly—but it gives the particularly simple and effective DDPM training objective.

The schedule \(\{\beta_t\}_{t=1}^{T}\) controls how quickly information is destroyed. It must satisfy \(0<\beta_t<1\), and is chosen so that \(\bar{\alpha}_{T}\) is close to zero. The original DDPM used a linear \(\beta_t\) schedule; Improved DDPM introduced a cosine schedule for \(\bar{\alpha}_t\).

A useful interpretation is the timestep signal-to-noise ratio

\[ \operatorname{SNR}(t)=\frac{\bar{\alpha}_t}{1-\bar{\alpha}_t}. \]

Early timesteps have high SNR and mostly preserve the data; late timesteps have low SNR and are dominated by noise.

One neural network is shared across all timesteps. It must therefore receive both the noisy sample \(\mathbf{x}_{t}\) and a representation of \(t\) (or equivalently the noise level). The original image DDPM uses a U-Net-like architecture with sinusoidal timestep embeddings injected into its residual blocks, along with attention at selected resolutions.

Common prediction targets include:

• \(\boldsymbol{\epsilon}\)-prediction: predict the added noise, as in the original DDPM objective;
• \(\mathbf{x}_{0}\)-prediction: predict the clean sample directly;
• \(\mathbf{v}\)-prediction: predict a linear combination of \(\mathbf{x}_{0}\) and \(\boldsymbol{\epsilon}\), often used in later diffusion systems.

These parameterizations contain equivalent information in the idealized setting but can behave differently during optimization.

The score of the noisy marginal is \(\nabla_{\mathbf{x}_{t}}\log q_t(\mathbf{x}_{t})\). For the conditional Gaussian corruption,

\[ \nabla_{\mathbf{x}_{t}}\log q(\mathbf{x}_{t}\mid\mathbf{x}_{0}) =-\frac{\boldsymbol{\epsilon}}{\sqrt{1-\bar{\alpha}_{t}}}. \]

Consequently, a noise predictor defines a score estimator

\[ \mathbf{s}_{\theta}(\mathbf{x}_{t},t) :=-\frac{\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_{t},t)} {\sqrt{1-\bar{\alpha}_{t}}}. \]

Minimizing the noise-prediction MSE is therefore equivalent, up to timestep-dependent weighting, to denoising score matching. This explains the close relationship between DDPMs and score-based generative models.

• Image values are usually scaled to a fixed range such as \([-1,1]\).
• \(L_0=-\log p_{\theta}(\mathbf{x}_{0}\mid\mathbf{x}_{1})\) is handled as a decoder likelihood; for discrete image values, the DDPM paper uses a discretized Gaussian likelihood.
• An exponential moving average of model parameters is commonly used for sampling.
• Diffusion training is stable and easily parallelized across examples, but ancestral sampling is slow because the reverse steps are sequential.
• Sample quality and likelihood are different objectives: the unweighted \(L_{\mathrm{simple}}\) improves sample quality but is not the tightest likelihood objective.

• Improved DDPM learns the reverse variance, introduces a cosine schedule, and improves likelihood and sampling efficiency.
• DDIM constructs a non-Markovian reverse process and can sample deterministically with fewer steps.
• Conditional diffusion incorporates labels, text, or other context; classifier guidance and classifier-free guidance strengthen adherence to the condition.
• Continuous-time formulations express diffusion using stochastic differential equations, with corresponding reverse-time SDE and probability-flow ODE samplers.
• Latent diffusion performs denoising in a learned latent space rather than directly in pixel space, greatly reducing computation.

• Jonathan Ho, Ajay Jain, Pieter Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS 2020. https://arxiv.org/abs/2006.11239
• Alexander Quinn Nichol, Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Models. ICML 2021. https://arxiv.org/abs/2102.09672
• Jiaming Song, Chenlin Meng, Stefano Ermon. Denoising Diffusion Implicit Models. ICLR 2021. https://arxiv.org/abs/2010.02502
• Yang Song et al. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021. https://arxiv.org/abs/2011.13456