Abstract:We introduce CARTE 1 (Culturally Anchored Regional-Territorial Evaluation), a multiplechoice benchmark for evaluating the ability of large language models (LLMs) to perform fine-grained reasoning over geographically grounded and regionally differentiated knowledge within France. While prior benchmarks focus on national-level cultural understanding, they largely overlook intra-country variation and the need to distinguish between closely related regional contexts. CARTE addresses this gap by introducing 2,431 questions spanning the 13 metropolitan regions of France and covering 14 thematic domains, including culture, language, demographics, economy, environment, and mobility. We further introduce CARTE-LV, a subset targeting Linguistic Variation across French regions, enabling focused evaluation of language-related differences. We evaluate 27 LLMs ranging from 1B to 12B parameters under few-shot settings. Our experiments reveal performance disparities across regions and model scales, suggesting systematic gaps in pretraining coverage and limited robustness to intra-national variation.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.01995 [cs.CL] |
| (or arXiv:2606.01995v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.01995 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sarah Almeida Carneiro [view email] [via CCSD proxy]
[v1]
Mon, 1 Jun 2026 09:50:50 UTC (191 KB)