MorphEmb: word embeddings for Uralic languages

Static word vectors (384 dimensions) for 15 Uralic languages, English, Finnish and Russian, all in one vector space. The vector of a word is the average of its character n-gram vectors (n = 3–6) plus the average over its FST analyses of the lemma vector and the average tag vector, so all inflected forms of a word share its lemma vector. The vectors were distilled from the contextual vectors of LexiBERT (layers 8–16), and because the same encoder is used for every language, the languages share one space.

Precomputed vectors

vectors/<lang>.vec.gz has vectors for the most frequent words of our corpora and the words that the FST can analyse (fastText text format). For words that the FST does not know, the analysis of the unknown word model was used when its probability was at least 0.5.

from gensim.models import KeyedVectors

myv = KeyedVectors.load_word2vec_format("vectors/myv.vec.gz")
myv.most_similar("кудо")

Vectors for any word form

config.json, shared.safetensors (character n-gram and tag vectors) and lemmas/<lang>.{safetensors,txt} (the lemma vectors of one language) build the vector of any word that the FST can analyse. The function load_morphemb that reads them, with the FSTs of UralicNLP, is being added to MLTools and is not released there yet:

from MLTools.LM import word_embeddings

model = word_embeddings.load_morphemb("morphemb", languages=["myv", "fin"])
model.embed("кудосо", "myv")
model.similarity("кудо", "talo", "myv", "fin")
model.nearest_neighbors("кудо", "myv", n=5, target_lang="fin")

The tables are stored in float16; the vectors are the same as those of the trained model in float32 up to rounding.

Language	Words in vectors/	Lemmas
Erzya (myv)	262,778	58,317
Moksha (mdf)	184,682	54,874
Komi-Zyrian (kpv)	217,983	47,547
Komi-Permyak (koi)	226,284	42,569
Udmurt (udm)	305,257	51,780
North Sámi (sme)	283,508	62,755
Skolt Sámi (sms)	103,972	30,830
Inari Sámi (smn)	150,108	43,782
Livvi (olo)	220,881	45,074
Karelian (krl)	203,161	29,351
Veps (vep)	187,383	36,439
Livonian (liv)	17,902	7,716
Meadow Mari (mhr)	281,678	69,009
Hill Mari (mrj)	167,356	54,811
Nenets (yrk)	50,224	13,235
English (eng)	170,110	137,192
Finnish (fin)	200,000	175,526
Russian (rus)	200,000	174,212

Results

In a BiLSTM parser with frozen word vectors (3 seeds), MorphEmb reaches 60.6 LAS on Erzya and 67.7 on North Sámi, close to fastText vectors trained on the same corpora (60.7 and 68.3). When one parser is trained for all of the Uralic treebanks, the average LAS is 54.1 with MorphEmb and 32.5 with fastText, as the fastText vectors of different languages are not in the same space. As candidates for translation prediction in Ve'rdd, the 20 closest words of MorphEmb raise Hits@1 from 19.2% to 23.6% on average.

Citation

@inproceedings{alnajjar-etal-2026-grounding,
  title     = {Grounding Encoders and Word Embeddings in {FST}s and Dictionaries for {U}ralic Languages},
  author    = {Alnajjar, Khalid and H{\"a}m{\"a}l{\"a}inen, Mika and Rueter, Jack and Partanen, Niko},
  booktitle = {Proceedings of the International Workshop on Computational Linguistics for Uralic Languages (IWCLUL 2026)},
  year      = {2026}
}

Code: https://github.com/mokha/lexibert · All models and seeds: https://doi.org/10.5281/zenodo.23545487.