Abstract:Recent advancements in large language models demonstrate that injecting perturbations can substantially enhance extrapolation performance. However, current approaches often rely on discrete perturbations with fixed designs, which limits their flexibility. In this work, we propose a framework where token prefixes are perturbed by a learnable transformation of a continuous latent vector within an embedding space. To overcome the challenge of an intractable marginal likelihood, we derive unbiased estimating equations for model parameters and optimize them via stochastic gradient descent. We establish the statistical properties of the resulting estimator in over-parameterized regimes. Empirical evaluations on both synthetic and real-world datasets demonstrate that our proposal yields significant gains in out-of-domain settings over a range of state-of-the-art baseline methods.
| Comments: | 35 pages |
| Subjects: | Machine Learning (stat.ML); Machine Learning (cs.LG); Statistics Theory (math.ST) |
| Cite as: | arXiv:2605.13284 [stat.ML] |
| (or arXiv:2605.13284v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2605.13284 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zetai Cen [view email]
[v1]
Wed, 13 May 2026 10:00:58 UTC (71 KB)