Abstract:We introduce MiRAGE, an evaluation framework for retrieval-augmented generation (RAG) from multimodal sources. As audiovisual media becomes a prevalent source of information online, it is essential for RAG systems to integrate information from these sources into generation. However, existing evaluations for RAG are text-centric, limiting their applicability to multimodal settings. MiRAGE is a claim-centric approach to multimodal RAG evaluation, consisting of InfoF1, which assesses factuality and information coverage, and CiteF1, which assesses citation support and completeness. We show that, when applied by humans, MiRAGE strongly aligns with extrinsic judgments of output quality. We additionally introduce an automatic implementation of MiRAGE as well as multimodal variants of three prominent text-based RAG metrics -- ALCE, ARGUE, and RAGAS -- demonstrating the limitations of text-centric work and laying the groundwork for automatic evaluation. We release open-source implementations and outline evaluation methods for multimodal RAG.
| Comments: | this https URL |
| Subjects: | Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2510.24870 [cs.CL] |
| (or arXiv:2510.24870v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2510.24870 arXiv-issued DOI via DataCite |
Submission history
From: Alexander Martin [view email]
[v1]
Tue, 28 Oct 2025 18:21:19 UTC (9,688 KB)
[v2]
Mon, 1 Jun 2026 15:43:17 UTC (9,796 KB)