kotoshu grammar tagger (English), GECToR-style — v0.3.0-fusion-k09
A 122.8 MB int8 ONNX token tagger that detects and corrects grammatical errors fully locally — no data leaves the machine. GECToR architecture (Omelianchuk et al. 2020): roberta-base encoder, 5,000-label closed vocabulary ($KEEP / $REPLACE / $APPEND / $DELETE / $TRANSFORM / $MERGE), iterative apply-and-retag decoding (4 rounds).
Official results — CoNLL-2014 test (1,312 sentences)
Scored with the official M2 scorer, noalt gold, --max_unchanged_words 2,
4-round iterative decode:
| system | Precision | Recall | F0.5 |
|---|---|---|---|
| this model (v0.3.0, fusion mix, KEEP-weight 0.9) | 0.607 | 0.334 | 0.522 |
| previous (v0.2.0, KEEP 0.8) | 0.591 | 0.344 | 0.517 |
| previous (v0.1.0, mega-6L) | 0.615 | 0.280 | 0.496 |
| LanguageTool 6.6 (free) | 0.305 | 0.060 | 0.168 |
| GECToR (published, roberta-large) | ~0.61 | ~0.51 | ~0.56 |
v0.2.0 ablation (same data mix, same decode; only the stage-2 $KEEP loss weight varies):
| KEEP weight | P | R | F0.5 |
|---|---|---|---|
| 0.6 | 0.554 | 0.362 | 0.501 |
| 0.7 | 0.573 | 0.342 | 0.505 |
| 0.8 | 0.591 | 0.344 | 0.517 |
| 0.9 | 0.607 | 0.334 | 0.522 |
The KEEP axis saturates here (precision rises, recall falls; F0.5 gains shrink). The remaining gap to GECToR ≈0.56 is recall-shaped and is the target of the next data work, not of this knob.
3.1x LanguageTool; runs in ~6 ms/sentence on a laptop CPU through onnxruntime.
Runtime
gem install kotoshu
# (grammar model setup: `kotoshu setup :en --model` — HF fetch path)
Model directory contents: gec-tagger.int8.onnx (int8 quantized),
labels.json (id -> label map — do not rebuild), vocab.json +
merges.txt (distilroberta byte-level BPE).
Training data (provenance)
| corpus | sentences | license |
|---|---|---|
| cLang-8 (cleaned Lang-8, google pipeline) | 2,345,088 | research-only (NAIST terms) |
| W&I + LOCNESS (BEA-2019) | 68,616 | research-only |
| NUCLE | 57,131 | research-only (NUS) |
| FCE v2.1 | 28,337 | research-only |
| CoEdIT (quad share) | 19,484 | Apache-2.0 |
| c4_200m-gec (filtered 93,193) | 93,193 | research use |
| total | 2,611,849 | mixed — see limitations |
Recipe: 8 epochs, batch 16, LR 5e-5, 10% warmup + cosine decay, label vocab = top 5,000 from the training mix.
Where this model stands
State-of-the-art in its product class: fully local (no network), CPU
real-time (6 ms/sentence), 122.8 MB int8 ONNX, embeddable in Ruby/
Rust/TS. No competing grammar checker occupies this class —
LanguageTool 6.6 (free) measures F0.5 0.168 on the same official
benchmark vs this model's 0.517 (3.1x). This is NOT an absolute
leaderboard claim: T5-11B (0.72) and LLM-API systems score higher at
45 GB+/cloud-only; our metric is accuracy-per-MB, GECToR-class at
~1/30th the size. Measurement: official CoNLL-2014 M2 scorer, noalt
gold, --max_unchanged_words 2, 4-round iterative decode (v0.2.0 measured 2026-10-12).
Limitations
- Research use only. The training mix includes research-only
corpora (Lang-8/NAIST, NUCLE/NUS, W&I); the weights inherit those
terms. A commercial-use line (Apache-2.0, clean-corpus training) is
planned as
kotoshu/grammar-en-cc. - English, sentence-level, learner-text domain.
- Precision-heavy operating point: strong on corrections it makes (P 0.591), conservative on recall (R 0.344) — tuned for real-world false-positive avoidance rather than leaderboard recall.
Provenance
Trained 2026-10-03 (H100, Modal), 8 epochs, final loss 0.2192. Evaluated with nusnlp/m2scorer (py3 build). Campaign log: kotoshu TODO.grammar/5-comparison-matrix.md.
License
- Code (kotoshu gem): BSD-2-Clause
- Weights: CC-BY-NC-SA-4.0 (research-only, inherited from the training-data terms)