Abstract:Implicit generative modeling (IGM) aims to produce samples of synthetic data matching the characteristics of a target data distribution. Recent work (e.g. score-matching networks, diffusion models) has approached the IGM problem from the perspective of pushing synthetic source data toward the target distribution via dynamical perturbations or flows in the ambient space. In this direction, we present the score difference (SD) between arbitrary target and source distributions as a flow that optimally reduces the Kullback-Leibler divergence between them. We apply the SD flow to convenient proxy distributions, which are aligned if and only if the original distributions are aligned. We demonstrate the formal equivalence of this formulation to denoising diffusion models under certain conditions. We also show that the training of generative adversarial networks includes a hidden data-optimization sub-problem, which induces the SD flow under certain choices of loss function when the discriminator is optimal. As a result, the SD flow provides a theoretical link between model classes that individually address the three challenges of the "generative modeling trilemma" -- high sample quality, mode coverage, and fast sampling -- thereby setting the stage for a unified approach.
| Comments: | 25 pages, 5 figures, 4 tables. Updated version of a paper originally published in Transactions on Machine Learning Research (TMLR), including post-publication commentary connecting the SD flow to drifting models |
| Subjects: | Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2304.12906 [cs.LG] |
| (or arXiv:2304.12906v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2304.12906 arXiv-issued DOI via DataCite |
|
| Journal reference: | Transactions on Machine Learning Research (7/2023) |
Submission history
From: Romann Weber [view email]
[v1]
Tue, 25 Apr 2023 15:21:12 UTC (2,184 KB)
[v2]
Tue, 18 Jul 2023 15:31:25 UTC (707 KB)
[v3]
Fri, 6 Dec 2024 16:02:25 UTC (854 KB)
[v4]
Wed, 13 May 2026 12:40:29 UTC (702 KB)