sharpsy/slovene-word-formation-segmentation-sloberta

LLMSegm SloBERTa: Slovene Word Formation Segmentation

Model Description

SloBERTa-based model for word formation segmentation of Slovene words, fine-tuned using the LLMSegm binary classification approach. The model predicts derivational boundaries at each character position, identifying how words are constructed from base words through derivational processes. This model focuses on derivational morphology - it identifies word-formation segments that reflect how new words are created from base words through prefixation, suffixation, and other derivational processes, grouping affixes that were added in the same derivational step.

Training Data

Fine-tuned on a manually annotated dataset of 1,935 Slovene words with multidimensional morphological annotations: - Word formation segments: Derivational formants added during word formation (e.g., nepozidan → ne-po-zida-n) - Morphological segments: All morphemes including inflectional endings (e.g., nepozidan → ne-po-zid-a-n-Ø) - Simplex annotations: Base words from which derivatives are formed - Zero-morpheme (Ø): Grammatical markers without phonetic form

Dataset: Slovene Morphological and Word Formation Segmentation Dataset (CLARIN.SI repository) The dataset achieves inter-annotator agreement of 85.16% Krippendorff's Alpha for word formation segments.

Performance

Metric	Score
BPR F1 (boundary precision-recall)	83.05%
Accuracy (exact word match)	52.58%
Evaluated using a holdout test set (10% split) with systematic hyperparameter optimization.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
# Load model
tokenizer = AutoTokenizer.from_pretrained("sharpsy/slovene-word-formation-segmentation-sloberta")
model = AutoModelForSequenceClassification.from_pretrained("sharpsy/slovene-word-formation-segmentation-sloberta")
model.eval()
# Segment a word
word = "nepozidan"  # "not built-up"
candidates = [f"{word} <__word-separator> {word[:i]}<__morph-boundary>{word[i:]}" 
              for i in range(1, len(word))]
inputs = tokenizer(candidates, return_tensors="pt", padding=True)
predictions = model(**inputs).logits.argmax(1).tolist()
# Reconstruct segmentation
segmented = word[0]
for i, boundary in enumerate(predictions):
    segmented += ("-" if boundary else "") + word[i+1]
print(segmented)  # Output: ne-po-zida-n

Model Architecture

• Base model: SloBERTa (Slovenian BERT)
• Method: LLMSegm binary classification
• Task: For each character position, predict whether a word-formation boundary exists
• Training: Binary cross-entropy loss with class weighting for imbalanced data

Hyperparameters

• Batch size: 128
• Learning rate: 3.05 × 10⁻⁵
• Weight decay: 2.77 × 10⁻³
• Label smoothing: 0.086
• Optimizer: AdamW

Word Formation vs Morphological Segmentation

This model predicts word-formation segments which:

• Group affixes added in the same derivational step
• Ignore inflectional endings (e.g., ignores -a in nepozidana)
• Focus on derivational formants (e.g., identifies ne-, po-, -zid-, -n)
• Reveal how complex words are built from simpler base words Example derivational chain: korakati (to step) → pozidati (to build up) → pozidan (built-up) → nepozidan (not built-up) For inflectional analysis (grammatical endings, case, number, aspect), use our companion model for morphological segmentation.

Limitations

• Training dataset is relatively small (1,935 words)
• Model may struggle with rare derivational patterns
• Trained on standard Slovene; may not handle dialectal or archaic forms well
• Assumes a single valid segmentation per word during training
• Does not explicitly predict zero-morphemes (Ø) - these are implicit in word-formation analysis
• Does not predict simplex forms (base words) - requires separate annotation

Citation

If you use this model, please cite:

@inproceedings{pranjic-etal-2026-dataset,
  author    = {Pranjić, Marko and Kern, Boris and Voršič, Ines and Pollak, Senja},
  title     = {Slovene Morphological and Word Formation Segmentation: A Novel Dataset and Evaluation},
  booktitle = {Proceedings of the 15th Language Resources and Evaluation Conference (LREC 2026)},
  month     = {May},
  year      = {2026},
  address   = {Palma de Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
}

Acknowledgments

This work was supported by the Slovenian Research and Innovation Agency (ARIS) through:

• Core research program Knowledge Technologies (P2-0103)
• Project Formant combinatorics in Slovenian (J6-3131)
• Project Large Language Models for Digital Humanities (GC-0002)

---

Companion model: For morphological (inflectional) segmentation, see: sharpsy/slovene-morphological-segmentation-sloberta (https://huggingface.co/sharpsy/slovene-morphological-segmentation-sloberta)