kotoshu grammar-cjk v0.2.0

ONE local grammar-correction model over Simplified Chinese, Traditional Chinese, and Korean (mdeberta-v3-base encoder, union sentencepiece vocab 93,526 pieces, 210.5MB int8 ONNX, iterative 4-round decode in pure Ruby, ~6ms-class per sentence on CPU). Japanese joins as v5 after its base rebuild + teacher pass (2026-10-21 in flight).

Measured results — typed-corruption gold, held-out pairs

(per-slice corruption packs, seed 103, pairs never seen in training; scorer: token-level P/R/F0.5 vs the pack's own ground truth, 2026-10-21)

slice	n	P	R	F0.5
zh-Hans	254	0.996	0.996	0.996
zh-Hant	253	1.000	1.000	1.000
ko	1582/643 gold edits	(v3: 0.413 / 0.403 / 0.411)		v4 pending

v1 context: v3 (trained on a whitespace-tokenized union corpus) scored R 0.05 / 0.02 on zh — the corpus builder split raw Chinese on spaces, collapsing each sentence into one unrepresentable token. v4 retrains on word-level rows from our DAG segmenter (166,554 rebuilt rows); ko, being natively space-delimited, was unaffected and remains the honest-weakest slice pending the v6 breadth pack.

Training corpus (248,115 GECToR-labeled rows, v4)

• zh-Hans + zh-Hant: 166,554 word-level rows — teacher (GLM-5.3-Flash, z.ai PAAS) exact-restore-verified pairs over OUR typed corruption packs (的/得/地, measure words, aspect, ba-construction, homophone lex) on DAG-segmented native-script corpora (opencc t2s/s2t identity + Han-majority purity gates)
• ko: 81,561 rows — josa/띄어쓰기/lex packs on whitespace tokens, teacher-verified (43.7% yield)

Recipe: mdeberta bf16, rounds=1 (4 ep differential LR + 2 ep KEEP-weighted, keep 0.6), fresh repo gec-ckpt-cjk-v4, 2026-10-21. Artifact parity gate: ORT-vs-torch row agreement 1.00/1.00 (max abs diff 0.00144).

License

CC-BY-NC-SA-4.0 (research line — mirrors grammar-de/fr/es): base corpora wikipedia/mC4, our corruption packs and teacher outputs are our property; the model card numbers are reproducible from the framework (factory/gold_score.py + published eval slices in kotoshu/kotoshu-code).