Abstract:Transformer-based models have advanced NLP, yet Hebrew still lacks a RoBERTa encoder that is trained at scale and released in both base and large variants. We present HalleluBERT, a RoBERTa-based encoder family trained from scratch on 49.1~GB of deduplicated Hebrew web text and Wikipedia using a Hebrew-specific byte-level BPE vocabulary. On native Hebrew benchmarks for named entity recognition (BMC, NEMO) and sentiment classification (SMCD), HalleluBERT outperforms monolingual and multilingual baselines, and yields the highest unweighted mean score across the three benchmarks. We release model weights and tokenizer under the MIT license to support reproducible Hebrew NLP research.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2510.21372 [cs.CL] |
| (or arXiv:2510.21372v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2510.21372 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), pp. 3022-3030, 2026 |
| Related DOI: | https://doi.org/10.63317/3qdqexx4e9i2
DOI(s) linking to related resources |
Submission history
From: Raphael Schmitt [view email]
[v1]
Fri, 24 Oct 2025 11:52:29 UTC (140 KB)
[v2]
Mon, 1 Jun 2026 11:52:23 UTC (137 KB)