Abstract:We present CALM, a joint Contextual Acoustic-Linguistic Modeling framework for multi-speaker automatic speech recognition (ASR). In personalized AI scenarios, the joint availability of acoustic and linguistic cues naturally motivates the integration of target-speaker conditioning with contextual biasing in overlapping conversations. CALM implements this integration in an end-to-end framework through speaker embedding-driven target-speaker extraction and dynamic vocabulary-based contextual biasing. We evaluate CALM on simulated English (LibriSpeechMix) and Japanese (Corpus of Spontaneous Japanese mixtures, CSJMix). On two-speaker mixtures, CALM reduces biased word error rate (B-WER) from 12.7 to 4.7 on LibriSpeech2Mix and biased character error rate (B-CER) from 16.6 to 8.4 on CSJMix2 (eval3), demonstrating the effectiveness of joint acoustic-linguistic modeling across languages. We additionally report results on the AMI corpus (IHM-mix condition) to validate performance on standardized speech mixtures.
| Comments: | Accepted to IEEE ICASSP 2026 |
| Subjects: | Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Sound (cs.SD) |
| Cite as: | arXiv:2601.22792 [eess.AS] |
| (or arXiv:2601.22792v2 [eess.AS] for this version) | |
| https://doi.org/10.48550/arXiv.2601.22792 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.1109/ICASSP55912.2026.11463102
DOI(s) linking to related resources |
Submission history
From: Muhammad Shakeel [view email]
[v1]
Fri, 30 Jan 2026 10:12:16 UTC (200 KB)
[v2]
Wed, 13 May 2026 09:53:49 UTC (200 KB)