Abstract:Autoregressive language models suffer from high inference latency due to their sequential decoding nature. Speculative decoding (SD) mitigates this by employing a lightweight draft model to propose candidate tokens, which are selectively verified by a larger target model. While existing methods either adopt multi-draft strategies to increase acceptance rates or block verification techniques to jointly verify multiple tokens, they remain limited by treating these improvements in isolation. In this work, we propose SpecTr-GBV, a novel SD method that unifies multi-draft and greedy block verification (GBV) into a single framework. By formulating the verification step as an optimal transport problem over draft and target token blocks, SpecTr-GBV improves both theoretical efficiency and empirical performance. We theoretically prove that SpecTr-GBV achieves the optimal expected acceptance length physically attainable within the framework of i.i.d. draft generation, and this bound improves as the number of drafts increases. Empirically, we evaluate SpecTr-GBV across five datasets and four baselines. Our method achieves superior speedup and significantly higher block efficiency while preserving output quality. In addition, we perform comprehensive ablation studies to evaluate the impact of various hyperparameters in the model.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2604.25925 [cs.CL] |
| (or arXiv:2604.25925v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.25925 arXiv-issued DOI via DataCite |
Submission history
From: Yijun Lin [view email]
[v1]
Wed, 1 Apr 2026 11:52:46 UTC (444 KB)