Abstract:While large language models (LLMs) are trained to align with human values, their generations may still violate safety constraints. A growing line of work addresses this problem by modifying the model's sampling policy at decoding time using a safety reward. However, existing decoding-time steering methods often intervene unnecessarily, modifying generations that would have been safe under the base model. Such unnecessary interventions are undesirable, as they can distort key properties of the base model such as helpfulness, fluency, style, and coherence. We propose a new test-time steering method designed to reduce such unnecessary interventions while improving the safety of unsafe responses. Our approach filters tokens using a value-based safety criterion and provides an explicit bound on the probability of false interventions. A single threshold hyperparameter controls this bound, allowing practitioners to trade off higher rates of unnecessary intervention for better output safety. Across multiple datasets and experiments, we show that our value-filtered decoding method outperforms existing baselines, achieving better trade-offs between safety, helpfulness, and similarity to the base model.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.14746 [cs.LG] |
| (or arXiv:2605.14746v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.14746 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bat-Sheva Einbinder [view email]
[v1]
Thu, 14 May 2026 12:13:08 UTC (2,963 KB)