Abstract:Large Reasoning Models (LRMs) are becoming integral to many AI inference systems, enhancing their capabilities with advanced reasoning. However, deploying these models in production environments presents a significant QoS challenge: the substantial memory overhead from their long, auto-regressive inference processes severely limits throughput and increases latency, thereby affecting the quality of service for concurrent users. We observe that LRMs frequently generate highly similar intermediate reasoning steps, which, in turn, correspond to highly similar KV cache states across layers. Building on this insight, we propose ReasonCache, a novel KV cache management approach designed to improve the QoS of AI inference systems. ReasonCache utilizes a Collaborative Filtering Algorithm to efficiently identify reusable KV cache blocks and enables zero-copy cache reuse. Experimental evaluation demonstrates that ReasonCache achieves a peak throughput improvement of 89.2% and an average gain of 40-60%, leading to more responsive and cost-effective AI inference services. Notably, this performance is achieved while maintaining higher accuracy compared to existing KV cache management techniques.
| Comments: | 10 pages, 7 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2507.21433 [cs.LG] |
| (or arXiv:2507.21433v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2507.21433 arXiv-issued DOI via DataCite |
Submission history
From: Kaiwen Chen [view email]
[v1]
Tue, 29 Jul 2025 02:05:51 UTC (2,455 KB)
[v2]
Thu, 31 Jul 2025 07:53:53 UTC (2,459 KB)
[v3]
Thu, 14 May 2026 02:27:46 UTC (683 KB)