Abstract:Test-Time Adaptation (TTA) via entropy minimization (EM) has proven effective for classification tasks, yet its application to generative autoregressive models remains theoretically fragmented. Existing approaches typically rely on distinct heuristics, such as teacher forcing with pseudo labels or policy-gradient-based reinforcement learning, without a unified mathematical foundation. In this work, we resolve this discrepancy by deriving a rigorous formulation of EM tailored to autoregressive models. We show that the exact objective naturally decomposes into a token-level policy gradient loss and a token-level entropy loss, and we reinterpret prior methods as partial realizations of this unified formulation. Using Whisper ASR as a testbed, we demonstrate that our approach consistently improves performance across more than 20 diverse domains, including acoustic noise, accents, and multilingual settings.
| Comments: | Submitted to INTERSPEECH 2026 |
| Subjects: | Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.08186 [eess.AS] |
| (or arXiv:2605.08186v1 [eess.AS] for this version) | |
| https://doi.org/10.48550/arXiv.2605.08186 arXiv-issued DOI via DataCite |
Submission history
From: Wei Ping Huang [view email]
[v1]
Tue, 5 May 2026 12:00:06 UTC (2,158 KB)