Abstract:Discrete diffusion models offer a promising alternative to autoregressive generation through parallel decoding, but they suffer from a sampling wall: once categorical sampling occurs, rich distributional information collapses into one-hot vectors and cannot be propagated across steps, forcing subsequent steps to operate with limited information. To mitigate this problem, we introduce Loopholing, a novel and simple mechanism that preserves this information via a deterministic latent pathway, leading to Loopholing Discrete Diffusion Models (LDDMs). Trained efficiently with a self-conditioning strategy that avoids unrolling the full denoising trajectory, LDDMs achieve substantial gains-reducing generative perplexity by up to 61% over prior baselines, thereby closing (and in some cases surpassing) the gap with autoregressive models, and producing more coherent text. Applied to reasoning tasks, LDDMs also improve performance on arithmetic benchmarks such as Countdown and Game of 24. These results also indicate that loopholing mitigates idle steps and oscillations, providing a general and effective path toward high-quality non-autoregressive text generation.
| Comments: | Accepted at ICLR 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2510.19304 [cs.LG] |
| (or arXiv:2510.19304v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2510.19304 arXiv-issued DOI via DataCite |
Submission history
From: Mingyu Jo [view email]
[v1]
Wed, 22 Oct 2025 07:08:47 UTC (706 KB)
[v2]
Sun, 1 Mar 2026 11:23:39 UTC (792 KB)
[v3]
Wed, 13 May 2026 06:53:46 UTC (804 KB)