Abstract:Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at this https URL.
| Comments: | Accepted by ACL 2026 Demo |
| Subjects: | Distributed, Parallel, and Cluster Computing (cs.DC); Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.07770 [cs.DC] |
| (or arXiv:2603.07770v2 [cs.DC] for this version) | |
| https://doi.org/10.48550/arXiv.2603.07770 arXiv-issued DOI via DataCite |
Submission history
From: Yuzhuang Xu [view email]
[v1]
Sun, 8 Mar 2026 19:20:25 UTC (693 KB)
[v2]
Wed, 13 May 2026 15:47:37 UTC (694 KB)