Sandhi-1.0-8M

A dictionary has 2% of Malayalam word forms, but 81% of their roots. This 8M model bridges that gap, offline

Try it in your browser: nithinmanoj10.github.io/ml-Sandhi. It runs an ONNX export of these weights on your device and shows every field the model returns. Nothing you type is sent anywhere.

Given one word, it returns the sandhi split, the morphemes, the root lemma, grammatical features (pos, case, number, tense, mood, aspect, voice, gender, polarity, root type), IPA and syllables, and a confidence level.

Why it exists

A dictionary is not competing here; the model is what makes one usable. Measured on a 2.74M-token Malayalam corpus:

	Share of analysable word forms
Surface form already in the dictionary	2.0%
Surface form out, root in the dictionary	81.0%
Both out	17.0%

കേരളത്തിന്റെ is in no dictionary. കേരള + ത്തിന്റെ → കേരളം + genitive is. The dictionary had the answer and could not reach it.

What is in this repo

File	What
sandhi_fp32.safetensors	The trained fp32 weights (sandhi_v3, epoch 19), weights only, 34.7 MB
config.json	The architecture and training configuration needed to rebuild the model
code/transduction_model/	The model definition (PyTorch) that these weights load into
code/normalize.py	The one normalize() function. Input must be normalized with this before the model sees it. SHA-256 71135f869d1827ebae02f02f098406f98bacc1461f16cc10776e38e6f37210fc, identical to the copy in the int8 repo
reports/	The holdout evaluation, as measured
LICENSE, NOTICE	Apache-2.0, and attribution for the training sources

How it works

An encoder (RoPE) with task heads, plus a small shared decoder:

• Boundary tagger gives the split, and a syllable tagger gives syllables, in one forward pass.
• 11 classifier heads give the grammatical features and the confidence level.
• Decoder (3 layers) generates only the short fields: root.lemma and ipa.

These are not transformers models, so from_pretrained does not apply. To use the fp32 weights in PyTorch:

import json
from safetensors.torch import load_file
from transduction_model.sandhi import SandhiModel   # from code/

cfg = json.load(open("config.json"))["configs"]
model = SandhiModel(cfg, cfg["head_sizes"])
model.load_state_dict(load_file("sandhi_fp32.safetensors"))
model.eval()

This gives the raw network (encoder, taggers, heads, decoder). The full pipeline around it (input normalization, greedy decoding, thresholds, assembling the analysis) is src/analyse.py in the project repository. The fp32 weights reproduce the int8 golden answers on 393 checked words: split, root, IPA, syllables and confidence match 100%, and features 99.7% (the difference is int8 rounding).

Text must be run through code/normalize.py first: Malayalam has several byte sequences that render identically and NFC does not merge them.

Example

The code that runs this model (normalization, decoding, the word-list step) is the package ml-Sandhi.

pip install git+https://github.com/nithinmanoj10/ml-Sandhi

import json
from ml_sandhi import Sandhi

sandhi = Sandhi.from_pretrained()          # downloads these weights on first use
result = sandhi.analyse("കമ്പ്യൂട്ടറുകളിലൂടെ")
print(json.dumps(result, ensure_ascii=False, indent=2))

Output (fields shown; the rest is omitted):

{
  "input": "കമ്പ്യൂട്ടറുകളിലൂടെ",
  "source": "model",
  "split": "കമ്പ്യൂട്ടറ + ുകള + ിലൂടെ",
  "root": {"lemma": "കമ്പ്യൂട്ടർ", "pos": "n", "type": "known"},
  "features": {"case": "perlative"},
  "ipa": "kampjuːʈʈarukaɭiluːʈe",
  "syllables": ["ക", "മ്പ്യൂ", "ട്ട", "റു", "ക", "ളി", "ലൂ", "ടെ"],
  "confidence_text": "Fairly sure"
}

കമ്പ്യൂട്ടറുകളിലൂടെ ("through the computers") is a loanword plus two endings, and is not in the dictionary, so source is model. The lemma കമ്പ്യൂട്ടർ is not a prefix of the surface: sandhi rewrote the stem. The full result also carries morphemes, display and alternatives.

Evaluation

Scored on held-out words that were reserved and hash-verified before any training data was generated, and are not training words. Set-match, with strict match in brackets (Malayalam reuses one form for several functions, so a word can have more than one valid analysis).

Test set	What it asks	Words	Split	Lemma	All features
Loanword stems	Unseen loanwords	5,000	97.0 (97.0)	95.4 (95.4)	91.6 (91.6)
Roots	Unseen dictionary roots	2,761	95.7 (93.7)	96.2 (93.4)	87.1 (78.1)
Depth stacks	Real 4–6-morpheme words	1,549	64.9 (63.7)	88.6 (85.1)	74.2 (70.9)
Compound pairs	Unseen real compounds	414	80.9 (75.6)	78.7 (76.1)	82.6 (71.3)
Tag stacks	Unseen ending combinations	324	77.8 (75.6)	85.5 (84.0)	75.3 (70.4)
Dev (words like training)		14,747	98.3 (97.9)	98.6 (98.2)	96.9 (95.6)

Limitations

• Deep words are weak. 89.5% of training rows are two-morpheme. On real 4–6-morpheme words the split score is 64.9%. Many of the gold labels there come from mlmorph and are themselves suspect, so the true figure is unresolved until a sample is hand-checked. Treat it as unmeasured, not as a pass.
• Real compounds are the main quality gap (split 80.9%). The model often leaves an attested compound whole.
• Proper nouns degrade rather than refuse. The ending is analysed and the root is marked unanalysable. Only 0.3% of training data teaches abstention, and it was deliberately not padded.
• No meanings. The model does not produce glosses. Meanings are a lookup after the model, not part of these weights.
• IPA exists only in the weights. There is no lookup fallback for pronunciation.
• Words up to 39 characters. Single words only, not sentences.

Training

From scratch, with no pretrained backbone. All labels were generated automatically by deterministic tools; no human annotation and no frontier-model labels. Sources are listed in NOTICE.

License

Apache-2.0. Training sources are attributed in NOTICE. mlmorph and Mlphon are dev-time label generators and are not distributed here.