Abstract:We propose Adam-SHANG, a Lyapunov-guided Adam-type method that couples momentum, adaptive preconditioning, and a curvature-aware correction through a more stable lagged-preconditioner update. For stochastic smooth convex optimization, we prove convergence in expectation under an admissible stepsize condition that can always be satisfied by a conservative spectral bound, without imposing global monotonicity on the second-moment sequence. To obtain a less conservative practical rule, we introduce a computable trace-ratio stepsize, motivated by a local coordinatewise alignment condition. The same structural update is also tested beyond the convex setting with simplified parameters. Experiments validate the predicted stochastic decay and show competitive training performance against Adam and AdamW on deep learning tasks.
| Comments: | 25 pages, 13 figures |
| Subjects: | Optimization and Control (math.OC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.12878 [math.OC] |
| (or arXiv:2605.12878v1 [math.OC] for this version) | |
| https://doi.org/10.48550/arXiv.2605.12878 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yaxin Yu [view email]
[v1]
Wed, 13 May 2026 01:46:23 UTC (7,316 KB)