Abstract:Large Language Models (LLMs) show remarkable semantic understanding but often struggle with structural understanding when processing graph topologies in a serialized format. Existing solutions rely on training external graph-based adapters or fine-tuning, which incur high costs and lost generalizability. In this work, we investigate the internal mechanisms of LLMs and present a critical finding: LLMs spontaneously reconstruct the graph's topology internally, evidenced by a distinct "sawtooth" pattern in their attention maps that structurally aligns with the "token-level adjacency matrix". However, this intrinsic structural understanding is diluted by the attention sink. We theoretically formalize this dilution as a representation bottleneck, stemming from a fundamental conflict: the model's anisotropic bias, essential for language tasks, suppresses the topology-aware local aggregation required for graph reasoning. To address this, we propose a training-free solution, named StructuraL Attention SHarpening (Slash), which amplifies this internal structural understanding via a plug-and-play attention redistribution. Experiments on pure graph tasks and molecular prediction validate Slash delivers significant and consistent performance gains across diverse LLMs.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.10503 [cs.AI] |
| (or arXiv:2605.10503v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.10503 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yiming Liu [view email]
[v1]
Mon, 11 May 2026 12:59:07 UTC (11,186 KB)
