From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetan

View PDF HTML (experimental)

Abstract:Large language models (LLMs) have achieved remarkable success across a wide range of natural language processing tasks, yet their performance remains heavily biased toward high-resource languages. Tibetan, despite its cultural significance and large speaker population, is still substantially underrepresented. In this work, we present a comprehensive pipeline for advancing Tibetan language modeling through large-scale data curation and continual pre-training. We construct a 72 GB high-quality Tibetan corpus, the largest to date, and adapt Qwen2.5-7B through balanced multilingual continual pre-training with Tibetan, Chinese, and English, followed by multilingual instruction tuning. To further scale capacity efficiently, we extend the dense model to a 50B-A10B Mixture-of-Experts architecture. Due to the absence of standardized Tibetan benchmarks, we build multiple evaluation datasets via high-quality translation and human verification. Experimental results show that both dense and MoE models consistently outperform existing open-source and Tibetan-focused models of similar scale across diverse tasks. Our work advances Tibetan-centric LLM research and provides transferable insights for extending LLMs to other low-resource languages. We will release the model weights, evaluation benchmarks, and detailed data processing documentation in the follow-up.

Submission history

From: Leiyu Pan [view email]
[v1] Sat, 12 Jul 2025 08:54:05 UTC (225 KB)
[v2] Tue, 22 Jul 2025 14:15:52 UTC (1 KB) (withdrawn)
[v3] Wed, 23 Jul 2025 13:30:04 UTC (96 KB)
[v4] Mon, 28 Jul 2025 04:56:37 UTC (96 KB)
[v5] Wed, 13 May 2026 12:42:40 UTC (278 KB)