Abstract:The increasing deployment of Large Language Model (LLM) inference on edge AI systems demands efficient execution under tight memory budgets. A key challenge arises from Key-Value (KV) caches, which often exceed available device memory. Although NVMe-based offloading offers scalable capacity, existing file-based designs rely heavily on the kernel page cache, leading to cache thrashing, unpredictable latency, and high software overhead under memory pressure. We present DUAL-BLADE, a dual-path KV residency framework that dynamically assigns KV tensors to either a page-cache path or an NVMe-direct path based on runtime memory availability. The NVMe-direct path bypasses the filesystem by mapping KV tensors to contiguous logical block address (LBA) regions, enabling low-overhead direct storage access. DUAL-BLADE further incorporates adaptive pipeline parallelism to overlap storage I/O with GPU DMA, improving inference throughput. Our evaluation shows that DUAL-BLADE substantially mitigates I/O bottlenecks, reducing prefill and decode latency by up to 33.1% and 42.4%, respectively, while improving SSD utilization by 2.2x across diverse memory budgets.
| Comments: | To appear in IEEE International Conference on Distributed Computing Systems (ICDCS) 2026 |
| Subjects: | Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Performance (cs.PF) |
| Cite as: | arXiv:2604.26557 [cs.DC] |
| (or arXiv:2604.26557v1 [cs.DC] for this version) | |
| https://doi.org/10.48550/arXiv.2604.26557 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hongsu Byun [view email]
[v1]
Wed, 29 Apr 2026 11:44:35 UTC (690 KB)