Abstract:SVD-based Low-rank compression reduces transformer parameters and nominal FLOPs, but these savings often translate poorly into real LLM serving speedups. We show that this gap is largely a runtime problem: factorized checkpoints fragment execution paths, and the resulting overhead differs substantially between prefill and autoregressive decode. We present FlashSVD v1.5, a unified inference runtime for serving SVD-compressed transformers. FlashSVD v1.5 maps diverse public SVD compression families to a common factorized representation and combines phase-specific kernels with dense-KV decode, packed MLP execution, and per-layer CUDA-graph replay to reorganize the low-rank serving path into a thin runtime. Across representative decoder-serving settings, FlashSVD v1.5 achieves up to 2.55x decode and 2.39x end-to-end speedup, and it attains 1.48x average decode and 1.44x average end-to-end speedup across multiple popular SVD compression families. These results suggest that practical low-rank acceleration requires runtime co-design, not compression algorithms alone. Our code is available at: this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Performance (cs.PF) |
| Cite as: | arXiv:2605.08314 [cs.LG] |
| (or arXiv:2605.08314v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.08314 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zishan Shao [view email]
[v1]
Fri, 8 May 2026 14:20:42 UTC (886 KB)