IME Context Reranker v1
Japanese IME cross-encoder that reranks dictionary-generated conversion candidates from the preceding context, reading, and candidate surface form. The reference runtime scores the first 64 algorithmic candidates and blends the model score with calibrated dictionary cost.
Input format
The model receives a sentence pair:
文脈：{context}
読み：{reading} [SEP] 候補：{candidate}
Higher regression logits indicate better candidates. The tokenizer uses
custom code inherited from the base model, so load it with
trust_remote_code=True.
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model_id = "fa0311/ime-context-reranker-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
query = "文脈：家に\n読み：かえる"
candidate = "候補：帰る"
inputs = tokenizer(query, candidate, return_tensors="pt")
score = model(**inputs).logits.item()
Runtime configuration
- Maximum sequence length: 192
- Context window: 160 characters
- Model-scored candidates: 64
- Dictionary cost weight: 0.19
- Reference IME runtime: CUDA only
Evaluation
| Evaluation set | Top-1 | Top-5 |
|---|---|---|
| Development | 94.83% | 100.00% |
| Locked 276 | 80.80% | 99.28% |
| Curated 1,416 | 82.20% | 99.44% |
| Difficult full-input 805 | 42.11% | 62.48% |
The difficult full-input set has 72.80% candidate recall, so its reranking scores include failures where the target was absent from the candidate set. Metrics depend on the associated dictionary candidate generator and cost blend; they are not standalone sentence-classification accuracy.
Related repositories
License
Apache License 2.0. The model is fine-tuned from
line-corporation/line-distilbert-base-japanese,
which is also distributed under Apache-2.0.
- Downloads last month
- 71