Abstract:This study proposes a quantitative framework for profiling LLM dispositions as stable, model-specific regularities in output under repeated, controlled elicitation. Using a structured narrative constraint-selection task administered across six frontier models and three instruction types, we operationalize disposition through two dimensions: "consistency", measured as cross-replication selection overlap via Jaccard similarity, and "diversity", measured as dispersion across options via the inverse Simpson index. We further introduce Narrative Landscape, a PCA-based visualization that maps each model's selection profile into a shared space for direct comparison. Results reveal a clear rigidity-exploration spectrum across model families and show that instruction types shift the geometry of selection spaces even when scalar metrics appear similar, indicating that comparable scores can mask qualitatively distinct selection topologies.
| Comments: | Accepted to NLP4DH 2026, camera-ready version |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.08742 [cs.CL] |
| (or arXiv:2605.08742v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.08742 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Donghoon Jung [view email]
[v1]
Sat, 9 May 2026 07:06:57 UTC (1,180 KB)