Abstract:We propose Representational Effective Theory (RET), a framework for describing large language model computation in terms of learned macrostates rather than microscopic details. RET learns these macrostates from hidden-state trajectories using a BYOL/JEPA-style self-supervised objective, coarse-graining activations into macrovariables that preserve higher-level structure relevant for prediction and interpretation. We evaluate whether these macrovariables are practically relevant for interpretability: RET yields temporally consistent states that reveal "mental-state" trajectories of reasoning, capture high-level semantic structure, support early prediction of behavioral outcomes such as sycophancy, and provide causal handles for steering generations toward interpretable computational phases. Together, these results suggest that LLM computation admits useful effective descriptions via RET: high-level, dynamically meaningful variables that support interpretation, prediction, and intervention.
| Comments: | Project webpage: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.09294 [cs.LG] |
| (or arXiv:2605.09294v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.09294 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Muhammed Ustaomeroglu [view email]
[v1]
Sun, 10 May 2026 03:42:37 UTC (15,987 KB)