Abstract:Advances in transformer-based language models have highlighted the benefits of language-specific pre-training on high-quality corpora. In this context, German NLP stands to gain from updated architectures and modern datasets tailored to the linguistic characteristics of the German language. GeistBERT seeks to improve German language processing by incrementally training on a diverse corpus and optimizing model performance across various NLP tasks. We pre-trained GeistBERT using fairseq, following the RoBERTa base configuration with Whole Word Masking (WWM), and initialized from GottBERT weights. The model was trained on a 1.3 TB German corpus with dynamic masking and a fixed sequence length of 512 tokens. For evaluation, we fine-tuned the model on standard downstream tasks, including NER (CoNLL 2003, GermEval 2014), text classification (GermEval 2018 coarse/fine, 10kGNAD), and NLI (German XNLI), using $F_1$ score and accuracy as evaluation metrics. GeistBERT achieved strong results across all tasks, leading among base models and setting a new state-of-the-art (SOTA) in GermEval 2018 fine text classification. It also outperformed several larger models, particularly in classification benchmarks. To support research in German NLP, we release GeistBERT under the MIT license.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2506.11903 [cs.CL] |
| (or arXiv:2506.11903v5 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2506.11903 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models, 2025, pp. 42-50 |
| Related DOI: | https://doi.org/10.26615/978-954-452-105-9-006
DOI(s) linking to related resources |
Submission history
From: Raphael Schmitt [view email]
[v1]
Fri, 13 Jun 2025 15:53:17 UTC (131 KB)
[v2]
Wed, 18 Jun 2025 23:06:09 UTC (131 KB)
[v3]
Tue, 24 Jun 2025 12:31:06 UTC (131 KB)
[v4]
Thu, 10 Jul 2025 20:40:55 UTC (5,303 KB)
[v5]
Mon, 1 Jun 2026 11:28:04 UTC (5,305 KB)