Abstract:Visual Automatic Speech Recognition (V-ASR) is a challenging task that involves interpreting spoken language solely from visual information, such as lip movements and facial expressions. This task is notably challenging due to the absence of auditory cues and the visual ambiguity of phonemes that exhibit similar visemes-distinct sounds that appear identical in lip motions. Existing methods often aim to predict words or characters directly from visual cues, but they commonly suffer from high error rates due to viseme ambiguity and require large amounts of pre-training data. We propose a novel phoneme-based two-stage framework that fuses visual and landmark motion features, followed by an LLM model for word reconstruction to address these challenges. Stage 1 consists of V-ASR, which outputs the predicted phonemes, thereby reducing training complexity. Meanwhile, the facial landmark features address speaker-specific facial characteristics. Stage 2 comprises an encoder-decoder LLM model, NLLB, that reconstructs the output phonemes back to words. Besides using a large visual dataset for deep learning fine-tuning, our PV-ASR method demonstrates superior performance by achieving 17.4% WER on the LRS2 and 21.0% WER on the LRS3 dataset.
| Comments: | Accepted at ICASSP 2026. This version corresponds to the camera-ready manuscript |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2507.18863 [cs.CV] |
| (or arXiv:2507.18863v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2507.18863 arXiv-issued DOI via DataCite |
Submission history
From: Matthew Kit Khinn Teng Mr [view email]
[v1]
Fri, 25 Jul 2025 00:38:39 UTC (801 KB)
[v2]
Sat, 30 May 2026 15:03:01 UTC (1,607 KB)