Abstract:Foundation models tokenize Ukrainian legal text with vastly different efficiency, yet no systematic comparison exists for this domain. We benchmark seven models from five providers on 273 validated court decisions from Ukraine's state registry (EDRSR), measuring tokenizer fertility and zero-shot performance on three tasks. Three findings emerge. (1) Tokenizer fertility varies 1.6x: Qwen3 models consume 60% more tokens than Llama-family models on identical input, directly reducing API cost. (2) NVIDIA Nemotron Super 3 (120B) achieves the highest composite score (83.1), outperforming Mistral Large 3 (675B total, 41B active) -- a model with 5.6x more total parameters and 3.4x more active parameters per token -- at one-third the API cost. (3) Few-shot prompting degrades performance by up to 26 percentage points; stratified and prompt-sensitivity ablations confirm this is intrinsic to Ukrainian-language demonstrations, not an artifact of example selection. For practitioners: tokenizer analysis should precede model selection, and zero-shot is a more reliable default than few-shot for morphologically rich languages.
| Comments: | 22 pages, 21 tables, 3 figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.14890 [cs.CL] |
| (or arXiv:2605.14890v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.14890 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Volodymyr Ovcharov [view email]
[v1]
Thu, 14 May 2026 14:35:05 UTC (26 KB)