Abstract:Recent advances in large language models (LLMs) have prompted a growing body of work that questions the methodology of prevailing evaluation practices. However, many such critiques have already been extensively debated in natural language processing (NLP): a field with a long history of methodological reflection on evaluation. We conduct a scoping review of research on evaluation concerns in NLP and develop a taxonomy, synthesizing recurring positions and trade-offs within each area. We also discuss practical implications of the taxonomy, including a structured checklist to support more deliberate evaluation design and interpretation. By situating contemporary debates within their historical context, this work provides a consolidated reference for reasoning about evaluation practices.
| Comments: | Under review |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2604.25923 [cs.CL] |
| (or arXiv:2604.25923v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.25923 arXiv-issued DOI via DataCite |
Submission history
From: Ruchira Dhar [view email]
[v1]
Wed, 1 Apr 2026 10:29:43 UTC (911 KB)