jakequist/kanjiland-annotation

Kanjiland — Annotation model (Ja→full format)

A from-scratch 52.3M Transformer that maps raw Japanese to the full Kanjiland reading-comprehension format — segmentation, furigana, glosses, word groupings, translation, and grammar labels (⟨T⟩/⟨W⟩/⟨S⟩/⟨G⟩), with no MeCab at inference (it learned segmentation itself). Part of Kanjiland.

• Training: 6,445 teacher-supervised silver annotations (the M7 dataset).
• Format-validity eval: parse-rate 77%, fully-valid (linter) 38%.
• On-device: runs on CPU at 521 tok/s; int8 is 2.4× smaller.

⚠ Numeric fragility

Generation must run under bf16 autocast (cpu + cuda). In fp32 the long autoregressive decode diverges to 0% parseable output. The loader handles this.

Use

git clone https://github.com/jakequist/kanjiland && cd kanjiland
uv run python scripts/annotate.py --config config.yaml --checkpoint model.pt \
    --text "彼は古い寺を訪れた。" --device cpu

License & limitations

Weights: MIT. This is an early de-risk baseline trained on only 6.8k examples — valid structure, rough content (glosses/translations loop). Improve via more silver data + constrained decoding, not architecture. Grammar inventory: docs/GRAMMAR_RULES.md.