Abstract:Recent interpretability methods have proposed to translate LLM internal representations into natural language descriptions using a second verbalizer LLM. This is intended to illuminate how the target model represents and operates on inputs. But do such activation verbalization approaches actually provide privileged knowledge about the internal workings of the target model, or do they merely convey information about the inputs provided to it? We critically evaluate popular verbalization methods and datasets used in prior work and find that one can perform well on such benchmarks without access to target model internals, suggesting that these datasets are not ideal for evaluating verbalization methods. We then run controlled experiments which reveal that verbalizations often reflect the parametric knowledge of the verbalizer LLM that generated them, rather than the knowledge of the target LLM whose activations are decoded. Taken together, our results indicate a need for targeted benchmarks and experimental controls to rigorously assess whether verbalization methods provide meaningful insights into the operations of LLMs.
| Comments: | ICML 2026. 41 pages, 23 tables, 6 figures |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2509.13316 [cs.CL] |
| (or arXiv:2509.13316v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2509.13316 arXiv-issued DOI via DataCite |
Submission history
From: Millicent Li [view email]
[v1]
Tue, 16 Sep 2025 17:59:04 UTC (397 KB)
[v2]
Mon, 29 Sep 2025 16:57:58 UTC (403 KB)
[v3]
Tue, 9 Dec 2025 18:35:28 UTC (408 KB)
[v4]
Wed, 13 May 2026 17:55:29 UTC (403 KB)