Abstract:We investigate the calibration of large language models' (LLMs') confidence across diverse tasks. The results of our preregistered study show that the current crop of LLMs are, like people, too sure they are right: confidence exceeds accuracy, on average. Importantly, however, this tendency is moderated by a powerful hard-easy effect, wherein overconfidence is greatest on difficult tests; by contrast, easy tests actually show substantial underconfidence. We develop LifeEval, a test for evaluating model calibration across levels of difficulty.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.23909 [cs.AI] |
| (or arXiv:2605.23909v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.23909 arXiv-issued DOI via DataCite |
Submission history
From: Noam Michael [view email]
[v1]
Fri, 3 Apr 2026 19:43:24 UTC (3,200 KB)