Abstract:Large language models (LLMs) require constant updates to remain aligned with evolving real-world knowledge. Model editing offers a lightweight alternative to retraining, but sequential editing often destabilizes representations and induces catastrophic forgetting. In this work, we seek to better understand and mitigate performance degradation caused by sequential editing. We hypothesize that hyperspherical uniformity, a property that maintains uniform distribution of neuron weights on a hypersphere, helps the model remain stable, retain prior knowledge, while still accommodate new updates. We use Hyperspherical Energy (HE) to quantify neuron uniformity during editing, and examine its correlation with editing performance. Empirical studies across widely used editing methods reveals a strong correlation between HE dynamics and editing performance, with editing failures consistently coinciding with high HE fluctuations. We further theoretically prove that HE dynamics impose a lower bound on the degradation of pretrained knowledge, highlighting why HE stability is crucial for knowledge retention. Motivated by these insights, we propose SPHERE (Sparse Projection for Hyperspherical Energy-Regularized Editing), an HE-driven regularization strategy that stabilizes neuron weight distributions, ultimately preserving prior knowledge while enabling reliable sequential updates. Specifically, SPHERE identifies a sparse space complementary to the principal hyperspherical directions of the pretrained weight matrices and projects new knowledge onto it, attenuating perturbations on the principal directions. Extensive experiments on LLaMA3 (8B) and Qwen2.5 (7B) show that SPHERE outperforms the best baseline in editing capability by an average of 16.41%, while most faithfully preserving general model performance, thereby offering a principled path toward reliable large-scale knowledge editing.
| Comments: | Accepted by ICLR 2026. The code is available at this https URL. Project page: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2510.01172 [cs.CL] |
| (or arXiv:2510.01172v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2510.01172 arXiv-issued DOI via DataCite |
Submission history
From: Jia-Chen Gu [view email]
[v1]
Wed, 1 Oct 2025 17:55:43 UTC (33,940 KB)
[v2]
Mon, 2 Mar 2026 09:55:05 UTC (33,960 KB)
[v3]
Thu, 14 May 2026 05:50:33 UTC (33,960 KB)