Abstract:Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using \textit{meta-correlation}, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data will become publicly available upon paper acceptance.
| Comments: | 16 pages, 1 figure, 14 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.09403 [cs.CL] |
| (or arXiv:2603.09403v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.09403 arXiv-issued DOI via DataCite |
Submission history
From: Lukáš Eigler [view email]
[v1]
Tue, 10 Mar 2026 09:15:19 UTC (121 KB)
[v2]
Sat, 30 May 2026 14:09:07 UTC (124 KB)