romaji2ja

Romaji → Japanese conversion (ローマ字→日本語変換). A compact Qwen3-style causal LM plus a hybrid IME-style fast path that resolves most inputs from dictionaries/segmentation in well under a millisecond and only falls back to the neural model for genuinely novel input.

Code in the companion GitHub repository is MIT-licensed. Model weights and model artifacts are published under CC BY-SA 4.0 because the training corpus is derived from Japanese Wikipedia.

What this is

• Task: transduce romaji (incl. wapuro/kunrei/Hepburn, typos, IME small-tsu, case/space/hyphen noise) into Japanese (kanji/kana).
• Model: Qwen3ForCausalLM, hidden 1024 · 20 layers · 16 heads / 4 KV (GQA) · intermediate 3584 · vocab 8000 · context 1024 · tied embeddings (~0.3B, fp32). Trained on rule-generated romaji↔Japanese pairs from a Japanese corpus.
• Published checkpoint: accepted A75 interpolation checkpoint, SHA-256 97cd23ebb00582d461c04c0e8ed3ebd140395b298aa80f98f73cdfb5aa67b332.
• Special tokens: U+EE00 romaji-input start · U+EE01 Japanese-output start · U+EE02 reserved context · <|endoftext|> EOS/PAD. The model is prompted as U+EE00 {romaji} U+EE01 and generates the Japanese continuation.

Hybrid fast path (recommended for product use)

infer_fast.py routes each input through, in order:

1. input normalization (NFKC, lowercase, strip spaces / soft separators / zero-width)
2. learned choice feedback → exact lexicon → SQLite cache
3. exact lexicon segmentation
4. generic romaji/kana phrase fallback
5. whole-input fuzzy + fuzzy/weighted/dense segmentation routes
6. general-phrase rescue — a generic noisy-romaji recovery stage: bidirectional romaji style canonicalization (wapuro syudan ↔ Hepburn shudan both match the dictionary), IME small-tsu (moxtute→もって) and run-collapse (syyyuu→syuu) canonicalization, then an anchor-and-fill beam Viterbi over a general reading lexicon + reusable colloquial layer with a confidence gate. It runs only when every earlier route declines and strictly before the neural model, abstaining (→ neural model) on low-confidence / ambiguous input rather than emitting a wrong answer.
7. neural model fallback

Verified accuracy (acceptance gate)

Practical Windows hybrid fast path — python src/check_acceptance_98.py, 12/12 required gates pass, evidence SHA-256 verified:

Gate	Result
fixed gold	99/99 (100%)
typo stress	1980/1980 (100%)
realworld noise	594/594 (100%)
expanded realworld case/space/hyphen	1188/1188 (100%)
full-v1 expanded case/space/hyphen	9432/9432 (100%)
large realworld case/space/hyphen (95 gate)	5675/5675 (100%)
large full-v1 case/space/hyphen (95 gate)	45598/45598 (100%)
long composition	800/800 (100%)
fresh case/space/hyphen	2358/2358 (100%)
strict neural full-v1 (normalized)	771/786 (98.09%)
strict neural fresh case/space/hyphen (normalized)	2313/2358 (98.09%)
post-shared-normalization fresh rerun	2358/2358 (100%)

Caveat (honest boundary): the 98% claim is for the practical hybrid path and for normalized strict-neural input. Raw, un-normalized strict-neural case/space/hyphen input is not a 95% claim (measured ~93.4%). Product input should go through the hybrid path, which normalizes before routing. Full evidence and per-file hashes are in acceptance/summary.json.

Usage

Neural model directly (normalized input recommended):

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("limoXD/romaji2ja")
model = AutoModelForCausalLM.from_pretrained("limoXD/romaji2ja")

bos_in, bos_out = chr(0xEE00), chr(0xEE01)   # romaji-start, japanese-start
prompt = bos_in + "kyouhaiitenkidesu" + bos_out
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
ids.pop("token_type_ids", None)  # Qwen-style model does not consume this field
out = model.generate(**ids, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
# 今日はいい天気です

Hybrid fast path (clone the repo for code/ + lexicon/):

python code/infer_fast.py --model . --lexicon lexicon/romaji2ja_typo_95.json \
  --general-lexicon lexicon/general_reading_lexicon.json "syudan no hani nara"

Repository layout

config.json, model.safetensors, tokenizer.json, ...   # the model (load at repo root)
code/        infer_fast.py, general_phrase.py, romaji_kana.py, normalization.py, infer.py, eval_general_phrase.py
lexicon/     romaji2ja*.json, general_reading_lexicon.json, candidate_feedback.jsonl, adversarial_piece_aliases.json
acceptance/  summary.json, gate_report.json   # 12/12 gate evidence with SHA-256 manifest
release_manifest.json                         # exact file hashes and checkpoint identity

Companion projects: source code · interactive Space.

Limitations

• Tuned for Windows-local use (CPU / DirectML); fp32 weights.
• Severe multi-error corruption or genuine homophone ambiguity is intentionally left to the neural fallback (or abstained) rather than guessed.
• Context window 1024 tokens.