Abstract:Adapting to latent confounded shift remains a core challenge in modern AI. This setting is driven by hidden variables that induce spurious correlations between inputs and outputs during training, leading models to rely on non-causal shortcuts. For example, a model may learn to treat metadata (e.g., data source like "Amazon") as a proxy for positive sentiment, causing failure when the source becomes predominantly negative during deployment. To address this latent confounded shift, we introduce Causal Fine-Tuning(CFT). Using a structural causal model as an inductive bias, we derive sufficient identification conditions that motivate a fine-tuning objective for decomposing representations into high-level stable and low-level shift-sensitive components. Instantiating this framework in BERT, we show that learning such causal/spurious representations and adjusting them accordingly yield a more robust predictor. Experiments on spurious correlation injection attacks in text demonstrate that our method outperforms black-box domain generalization baselines, highlighting the benefits of explicitly modeling causal structure.
| Comments: | ICML 2026 Camera Ready Version |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2410.14375 [cs.LG] |
| (or arXiv:2410.14375v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2410.14375 arXiv-issued DOI via DataCite |
Submission history
From: Jialin Yu [view email]
[v1]
Fri, 18 Oct 2024 11:06:23 UTC (791 KB)
[v2]
Thu, 12 Jun 2025 20:01:43 UTC (759 KB)
[v3]
Tue, 12 May 2026 21:39:41 UTC (845 KB)