Abstract:This paper presents the first systematic comparison investigating whether Large Reasoning Models (LRMs) are superior judges to non-reasoning LLMs. Our empirical analysis yields four key findings: 1) LRMs outperform non-reasoning LLMs in terms of judgment accuracy, particularly on reasoning-intensive tasks; 2) LRMs demonstrate superior evaluation instruction-following capabilities; 3) LRMs exhibit enhanced robustness against adversarial attacks targeting judgment tasks; 4) However, LRMs still exhibit strong evaluation biases. To mitigate this bias vulnerability, we propose PlanJudge, a lightweight evaluation strategy that prompts the model to generate an explicit evaluation plan before executing the judgment. Despite its simplicity, our experiments demonstrate that PlanJudge significantly mitigates biases in LLM-as-a-Judge while preserving overall judgment accuracy.
| Comments: | Accepted by ACL 2026 Workshop EvalEval |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2601.03630 [cs.CL] |
| (or arXiv:2601.03630v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2601.03630 arXiv-issued DOI via DataCite |
Submission history
From: Hui Huang Mr. [view email]
[v1]
Wed, 7 Jan 2026 06:19:26 UTC (336 KB)
[v2]
Thu, 14 May 2026 03:13:06 UTC (339 KB)