Bengali LLM prompt router
An XLM-RoBERTa classifier that selects between answering a Bengali multiple-choice
question directly (direct) and answering a validated English machine translation
(mt). It scores the complete Bengali MCQ together with its actual English
translation; it is not a general-purpose language classifier.
This is policy version 3. It replaces the earlier question-router checkpoint, whose tie handling was incompatible with this policy.
Intended use
Translate and validate the MCQ first, then provide the Bengali MCQ and English
translation as a pair to the classifier. Route to MT only if the model's mt
probability is at least 0.6395421028137207; otherwise use the direct Bengali
path. If translation is blank, invalid, or still contains Bengali script, bypass
the classifier and use the direct path.
The routing_policy.json file is the source of truth for the threshold, input
contract, and evaluation details.
Training target and data
The model was trained only on clean paired executions where one route was correct
and the other wrong: direct means direct-only correct, and mt means
MT-only correct. Both-correct and both-wrong examples were excluded from the
training loss. The source contained 27,269 clean paired rows, including 7,106
decisive route-preference candidates; 4,974 rows entered the training loss.
Evaluation
On an untouched 4,090-row test partition, direct answering achieved 60.073%, always-MT achieved 55.892%, and the routed policy achieved 60.171%, selecting MT for 11.39% of questions. The gain over direct is 0.098 percentage points (four net answers), so this checkpoint should be treated as experimental rather than evidence of a production accuracy improvement.
Limitations
This policy was evaluated on Bengali MCQs and a particular validated English translation workflow. It should not be assumed to transfer to other languages, tasks, translators, or answer models without new evaluation.
- Downloads last month
- 18