Abstract:KV cache compression promises increased throughput and efficiency with negligible loss in performance. While the gains in throughput are indisputable and recent literature has indeed shown minimal degradation on particular benchmarks, in general the consequences of compression in realistic scenarios such as multi-instruction prompting have been insufficiently studied. In this paper, we identify several pitfalls that practitioners should be aware of when deploying KV cache compressed LLMs. We evaluate five KV cache compression methods (StreamingLLM, SnapKV, TOVA, H2O, and K-Norm) on Llama3.1 8B and Qwen2.5 14B under multi-instruction prompting with IFEval. Importantly, we show that certain instructions degrade much more rapidly with compression, effectively causing them to be completely ignored by the LLM. As a practical example, we highlight system prompt leakage as a case study, empirically demonstrating the impact of compression on leakage and general instruction-following. We identify several factors that contribute to system prompt leakage: compression method, instruction order, and KV eviction bias. We then propose simple changes to KV cache eviction policies that can reduce the impact of these factors and improve the overall performance in multi-instruction tasks.
| Comments: | In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, ACL 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2510.00231 [cs.LG] |
| (or arXiv:2510.00231v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2510.00231 arXiv-issued DOI via DataCite |
Submission history
From: Alex Chen [view email]
[v1]
Tue, 30 Sep 2025 19:55:26 UTC (164 KB)
[v2]
Wed, 13 May 2026 19:46:01 UTC (1,608 KB)