lenabarretta/sharada-multilingual-small

A small encoder that makes typed decisions about text in one forward pass. The options come with the request and what comes back is a probability for each of them: nothing is generated and nothing is parsed. Because the labels are part of the input rather than part of the weights, a label set it has never been trained on still gets an answer.

Code, examples and the training run: https://github.com/LenaBarretta/sharada. The design and the experiments behind it: https://lenatriestounderstand.com/notes/llm/024-rlcr/.

pip install sharada

from sharada import DecisionModel

model = DecisionModel.from_pretrained("lenabarretta/sharada-multilingual-small")

d = model.decide(
    text="My card still hasn't arrived and I ordered it two weeks ago.",
    question="Which team should handle this?",
    options=["billing", "card delivery", "technical support", "account closure"],
)
d.answer, d.confidence, d.probabilities

Fine-tuning on a few hundred of your own labelled examples is the intended use:

report = model.fit(examples)      # holds 20% out, early-stops, then fits a temperature per task
model.save("my-router")

What it is

encoder	jhu-clsp/mmBERT-small
parameters	141M
text	up to 256 tokens, plus 48 for the question and 12 per option
kinds of question	choice (unordered labels), scale (ordered steps), binary (yes or no)
output	one probability per option, from a single forward pass
weights	float16 on disk, float32 in memory — a storage format, not quantisation
one decision	21.8 ms per question

Three properties hold by construction rather than by training: the order of the options cannot change the answer (every option branch starts at the same position id), an option's score does not depend on which other options are offered (an option reads the text, the question and itself), and the text is read once however many questions are asked of it. The tests in the repository check all three on an untrained model.

How it was trained

663,357 examples from 35 public label sets — intents, topics, review scores, emotion, toxicity, spam and entailment — for 15,000 steps of 32, AdamW at 3e-05 with a cosine schedule, cross entropy loss. The published weights are the ones that measured best on held-out data, at step 15,000 of 15,000 — past that the model stops answering better and only grows more certain. 7 further label sets were held out of training entirely and only measured: arxiv-category, claim-veracity, massive-scenario, medical-pair, poem-tone, subjective, topic-unseen-languages.

In training the options were shuffled, long label sets were often shown as a sampled handful, and each label set was asked through several wordings of its question — so the model reads the options and the question rather than their positions.

What it scores

Trained on is what that label set actually contributed to this run — not its cap, since a small dataset runs out first and a pooled multilingual one is split between its languages; a dash means the model never saw it. Measured on is how many held-out examples the accuracy beside it rests on, and it is worth reading first: a row measured on ninety-six examples moves by a full point when one answer changes.

Measured on held-out examples, with every label offered at once — all 151 intents of clinc, all 77 of banking — because that is what a request actually looks like. One temperature per label set was fitted on the same held-out examples. ECE is the expected calibration error over 15 equal bands: how far the stated probability is from how often it turns out right. Rows marked unseen are label sets kept out of training entirely, never trained on, only measured.

label set	answer options	kind	trained on	measured on	accuracy	log loss	ECE	T
clinc-intent	151	choice	—	480	0.844	0.551	0.047	0.944
banking-intent	77	choice	—	480	0.835	0.583	0.029	1.059
massive-intent	59	choice	—	480	0.865	0.488	0.026	1.26
question-type-fine	50	choice	—	240	0.883	0.387	0.030	1.122
massive-intent-multi	35	choice	—	1,440	0.834	0.617	0.023	1.414
fine-emotion	28	choice	—	360	0.603	1.345	0.095	1.122
newsgroup	20	choice	—	292	0.644	1.028	0.062	1.26
entity-type	14	choice	—	360	0.997	0.012	0.003	0.794
forum-topic	10	choice	—	360	0.728	0.797	0.061	0.944
topic-multi	7	choice	—	660	0.756	0.706	0.054	2.119
question-type	6	choice	—	240	0.967	0.100	0.016	1.059
emotion	6	choice	—	300	0.887	0.275	0.043	1.414
app-stars	5	scale	—	300	0.687	0.863	0.052	1.0
review-stars	5	scale	—	480	0.640	0.807	0.060	0.944
sentence-tone	5	scale	—	300	0.550	1.048	0.054	1.122
news-section	4	choice	—	360	0.928	0.214	0.022	0.944
tweet-emotion	4	choice	—	240	0.808	0.475	0.068	1.059
entailment-short	3	scale	—	360	0.850	0.449	0.037	1.0
entailment	3	scale	—	480	0.779	0.546	0.046	1.189
entailment-multi	3	scale	—	1,428	0.691	0.685	0.032	1.335
tweet-sentiment	3	scale	—	300	0.673	0.738	0.077	1.122
spam	2	binary	—	240	0.992	0.019	0.010	1.0
product-tone	2	scale	—	300	0.927	0.176	0.018	1.059
toxic-comment	2	binary	—	300	0.923	0.200	0.028	1.122
movie-verdict	2	scale	—	300	0.920	0.183	0.021	1.059
short-verdict	2	scale	—	240	0.892	0.220	0.054	1.059
paraphrase	2	binary	—	300	0.883	0.286	0.051	1.26
offensive	2	binary	—	300	0.863	0.323	0.039	1.0
answers-question	2	binary	—	360	0.853	0.345	0.032	1.0
hateful	2	binary	—	300	0.817	0.396	0.023	1.122
same-question	2	binary	—	360	0.811	0.431	0.041	1.122
same-meaning	2	binary	—	240	0.792	0.463	0.068	1.122
irony	2	binary	—	240	0.704	0.562	0.056	0.749
grammatical	2	binary	—	300	0.700	0.589	0.025	1.414
follows	2	binary	—	240	0.696	0.556	0.043	1.498
massive-scenario (unseen)	18	choice	—	240	0.696	0.864	0.083	1.26
arxiv-category (unseen)	11	choice	—	180	0.339	1.828	0.108	1.682
topic-unseen-languages (unseen)	7	choice	—	240	0.596	1.157	0.096	1.888
poem-tone (unseen)	4	scale	—	96	0.521	1.053	0.150	1.122
claim-veracity (unseen)	4	choice	—	180	0.350	1.370	0.046	4.49
medical-pair (unseen)	2	binary	—	180	0.667	0.600	0.039	2.245
subjective (unseen)	2	choice	—	180	0.383	0.704	0.176	8.0

Overall: accuracy 0.773, log loss 0.586, Brier 0.304, calibration error 0.013 (95% interval [0.011, 0.020]).

The family

checkpoint	encoder	download
lenabarretta/sharada-base	ModernBERT-base, 150M	300 MB	the default, and the one to fine-tune
lenabarretta/sharada-large	ModernBERT-large, 397M	794 MB	a few points better, about a third slower
lenabarretta/sharada-multilingual-base	mmBERT-base, 308M	616 MB	the same model over many more languages
lenabarretta/sharada-multilingual-small — you are here	mmBERT-small, 141M	282 MB	narrower body, for throughput rather than for one fast answer

Everything that makes a model what it is lives in the checkpoint: which encoder it wants, how long a text it reads, the temperature fitted for each task. The library reads all of that out of config.json, so a bigger model or a multilingual one is another repository rather than another version of sharada — and from_pretrained loads any of them with the same line.

That is also why a multilingual model cannot be a flag on this one. This encoder is English down to its tokenizer; reading another language means other weights, not another setting.

The probabilities expire

A temperature is fitted on a distribution, not on a model, so it goes stale when the traffic moves. The run's passport.json records what it was fitted on and what should make you fit it again:

from sharada import check_passport
import json

check_passport(json.load(open("passport.json")), recent_examples, model)
# [] when it finds nothing wrong

Fit it again on your own labelled examples before trusting the numbers on your own traffic — model.calibrate(examples) does it in one call, and model.fit(examples) does it for you.

What it will not do

It does not generate: options or nothing. It reads 256 tokens of text, so longer documents need chunking. It has seen the forty-odd languages of its mix and has not been measured on any outside them. And there is no medicine, law or code in its training mix: the two medical label sets are measured only, and verifying public-health claims lands at the majority-class baseline, which is to say it does not work. Those domains need fine-tuning on your own labelled examples. Finally it is small — where an answer needs a fact that is not in the text in front of it, a frontier model wins.

Apache 2.0.