Abstract:Estimating how often an ML model will fail at deployment scale is central to pre-deployment safety assessment, but a feasible evaluation set is rarely large enough to observe the failures that matter. Jones et al. (2025) address this by extrapolating from the largest k failure scores in an evaluation set to predict deployment-scale failure rates. We give a finite-k decomposition of this estimator's forecast error and show that it has a built-in bias toward over-prediction in the typical case, which is the safety-favorable direction. This bias is offset when the evaluation set misses a rare high-failure mode that the deployment set contains, leaving the forecast to under-predict at deployment scale. We propose a fine-tuning objective, the forecastability loss, that addresses this failure mode. In two proof-of-concept experiments, a language-model password game and an RL gridworld, fine-tuning substantially reduces held-out forecast error while preserving primary-task capability and achieving safety similar to that of supervised baselines.
| Comments: | 32 pages, 9 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.15134 [cs.LG] |
| (or arXiv:2605.15134v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15134 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Will Schwarzer [view email]
[v1]
Thu, 14 May 2026 17:41:52 UTC (308 KB)