LexiBERT

LexiBERT is mmBERT-small (140M parameters) whose pretraining we continued on 15 Uralic languages: Erzya, Moksha, Komi-Zyrian, Komi-Permyak, Udmurt, North Sámi, Skolt Sámi, Inari Sámi, Livvi, Karelian, Veps, Livonian, Meadow Mari, Hill Mari and Tundra Nenets. Besides masked language modelling, a quarter of the Uralic training samples are 20 words of a sentence followed by their word-by-word English gloss, which we built with the FSTs and the Ve'rdd dictionaries, and the masked words of both parts are predicted.

The model is a standard ModernBERT model and is used like mmBERT-small:

from transformers import AutoModel, AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained("khalidmokha/lexibert")
model = AutoModel.from_pretrained("khalidmokha/lexibert")

fill = pipeline("fill-mask", model="khalidmokha/lexibert")

Training

We continued the training for 6,000 steps (batch size 128, maximum length 128 tokens, learning rate 1e-4) with the temperature and masking schedules of mmBERT. The training data were our corpora for 7 of the languages and FineWeb2 for all 15, deduplicated and without the sentences of the UD development and test sets, together with English, Finnish and Russian web text.

Results

Fine-tuned for UD parsing (gold tokenization, LAS):

	mmBERT-small	LexiBERT
Erzya (trained on Erzya, 3 seeds)	58.7	66.0
North Sámi (trained on North Sámi, 3 seeds)	67.8	73.0
Average over 11 Uralic test sets (one multilingual model)	46.4	59.5

With a sentiment classifier trained on English SST-2 only, the average accuracy on the Erzya, Moksha, Komi-Zyrian and Udmurt test data of Alnajjar et al. (2023) is 63.1% (3 seeds), against 54.3% for mmBERT-small. See the paper for all results.

Task models built on LexiBERT: UD parser, unknown word model, sentiment and the MorphEmb word embeddings.

Limitations

For several of the languages, the training data are small and mostly web text, and the model has seen less than one epoch of most corpora. Its masked word predictions are therefore not reliable as such; the model is meant to be fine-tuned.

Citation

@inproceedings{alnajjar-etal-2026-grounding,
  title     = {Grounding Encoders and Word Embeddings in {FST}s and Dictionaries for {U}ralic Languages},
  author    = {Alnajjar, Khalid and H{\"a}m{\"a}l{\"a}inen, Mika and Rueter, Jack and Partanen, Niko},
  booktitle = {Proceedings of the International Workshop on Computational Linguistics for Uralic Languages (IWCLUL 2026)},
  year      = {2026}
}

Code: https://github.com/mokha/lexibert · All models and seeds: https://doi.org/10.5281/zenodo.23545487.