Abstract:Whether in agentic workflows, social studies, or chat settings, large language models (LLMs) are increasingly being asked to replace humans in choosing which goals to pursue, rather than completing predefined tasks. However, the assumption that LLMs accurately reflect human preferences for goal setting remains largely untested. We assess the validity of LLMs as proxies for human goal selection in a controlled, self-directed learning task borrowed from cognitive science. Across five models (GPT-5, Gemini 2.5 Pro, Claude Sonnet 4.5, Qwen3 32B, and Centaur), we find substantial divergence from human behavior. While people gradually explore and learn to achieve goals with diversity across individuals, most models exploit a single identified solution or show surprisingly low performance, with distinct patterns across models and little variability across instances of the same model. Chain-of-thought reasoning and persona steering provide limited improvements, and our conclusions hold across experimental settings. While they await confirmation in applied settings, these findings highlight the uniqueness of human goal selection and caution against its replacement with current models.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2603.03295 [cs.CL] |
| (or arXiv:2603.03295v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.03295 arXiv-issued DOI via DataCite |
Submission history
From: Gaia Molinaro [view email]
[v1]
Fri, 6 Feb 2026 15:39:54 UTC (4,022 KB)
[v2]
Wed, 13 May 2026 10:48:07 UTC (14,886 KB)