Abstract:Deploying language models as autonomous agents requires more than per-task accuracy: when an agent faces a queue of problems under a finite token budget, it must decide which to attempt, in what order, and how much compute to commit to each, all before any execution feedback is available. This is the prospective form of metacognitive control studied for decades in human cognition, yet whether language models possess it remains untested. We introduce TRIAGE, an evaluation framework in which a model receives a task pool and a token budget calibrated to its own baseline cost, and commits to a single ordered plan that jointly encodes selection, sequencing, and per-problem allocation. Plans are scored against an oracle with full knowledge of the model's solvability and cost on each problem, yielding a triage efficiency ratio on a common scale. We evaluate frontier and open-source models, with and without reasoning enabled, across competition mathematics, graduate-level science, code generation, and expert multidisciplinary knowledge, and find that current language models exhibit substantial gaps in prospective metacognitive control, revealing a previously unmeasured capability dimension with direct implications for resource-efficient agent deployment.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.13414 [cs.AI] |
| (or arXiv:2605.13414v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.13414 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zabir Al Nazi [view email]
[v1]
Wed, 13 May 2026 12:10:05 UTC (250 KB)