khalidmokha/lexibert-unknown-words

LexiBERT unknown word model

For a word form that the FST does not know, the model suggests its lemma, part of speech, tags and inflection class (contlex), which is what is needed for adding the word to the FST. It was trained on corpus words that the FSTs can analyse, with the contlex of their lemmas from Ve'rdd. The lemma is predicted as an edit rule on the end of the word.

pip install "lexibert @ git+https://github.com/mokha/lexibert"
lexibert-unknown --model khalidmokha/lexibert-unknown-words --lang myv кудосо

from lexibert import UnknownWordModel

model = UnknownWordModel.from_pretrained("khalidmokha/lexibert-unknown-words")
model.analyse("myv", "кудосо")
# {'lemma': [('кудо', 0.9855), ...], 'pos': [('N', 0.9941), ...], 'tags': [('SP+Ine+Indef', 0.9892), ...],
#  'contlex': [('N_KUDO', 0.9406), ...]}

Results

Words of held-out lemmas: the correct lemma among the first (L@1) or five best (L@5) suggestions, and the correct inflection class among the first (C@1) or three best (C@3). Karelian has no inflection classes in Ve'rdd and Komi-Permyak too few to test.

	L@1	L@5	C@1	C@3
Erzya (myv)	77.3	94.5	63.4	82.0
Moksha (mdf)	83.0	96.7	69.2	87.6
Komi-Zyrian (kpv)	88.3	98.4	59.1	80.4
Komi-Permyak (koi)	90.7	98.1	–	–
Udmurt (udm)	90.4	99.2	83.2	92.2
Meadow Mari (mhr)	81.9	96.6	69.8	88.0
Hill Mari (mrj)	69.1	85.2	69.8	89.2
North Sámi (sme)	70.9	87.7	58.2	73.7
Inari Sámi (smn)	59.3	75.4	56.0	73.0
Skolt Sámi (sms)	58.4	72.6	49.5	61.6
Livvi (olo)	64.9	84.5	58.7	79.9
Karelian (krl)	47.4	72.3	–	–
Veps (vep)	57.5	84.9	56.2	76.3
Livonian (liv)	47.5	73.6	21.6	38.8
Nenets (yrk)	35.6	64.0	20.8	38.5

Rare inflection classes are much harder: the class is correct for 76.1% of the words in classes with more than 100 training lemmas, but for 8.2% in classes with ten lemmas or fewer. Many real unknown words are names and loanwords whose lemma is the word itself.

Citation

@inproceedings{alnajjar-etal-2026-grounding,
  title     = {Grounding Encoders and Word Embeddings in {FST}s and Dictionaries for {U}ralic Languages},
  author    = {Alnajjar, Khalid and H{\"a}m{\"a}l{\"a}inen, Mika and Rueter, Jack and Partanen, Niko},
  booktitle = {Proceedings of the International Workshop on Computational Linguistics for Uralic Languages (IWCLUL 2026)},
  year      = {2026}
}

Code: https://github.com/mokha/lexibert · All models and seeds: https://doi.org/10.5281/zenodo.23545487.