ollaya-dev/coba-400m

Coba 400m

Coba 400m is a small English decision model by Ollaya. Give it a state (a message, a support ticket, a JSON object, an agent's next tool call) and questions, and it answers every question in one forward pass with calibrated probabilities:

• noul: yes or no, as the probability of yes;
• choice: one of up to 255 options, each described in plain words;
• score: a level on a scale of 2 to 10 described levels.

Close to the 1b's accuracy at 40% of its size, for Ollaya on a CPU or GPU. It trained zero-shot with respect to both benchmarks below: neither its data nor its teacher's labels include any of their datasets.

Run it

With Ollaya (Linux, macOS, Windows; CPU, CUDA, Metal):

curl -fsSL https://ollaya.dev/install.sh | sh
ollaya run coba:400m --ask "Is the customer asking for a refund?" < ticket.txt

or POST http://localhost:11435/api/decide with {"model": "coba:400m", "state": …, "questions": …} (API). Ollaya also speaks TypeSafe's /v1/systemone, so Jev clients work unchanged.

In Python with laya (pip install laya):

import laya

agent = laya.load("ollaya-dev/coba-400m")
agent.system_one("I was charged twice for order A-104.", {
    "refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"},
})

Ollaya and laya give the same answers: Ollaya checks its runtime against this checkpoint's PyTorch outputs on 2,383 questions (every decision the same, probabilities within 1.4e-5).

Results

Model	Weights	typed-decisions	Public benchmark
coba:1b	fp32	0.608	0.561 zero-shot
coba:400m (this model)	fp32	0.591	0.561 zero-shot
coba:400m-ternary	ternary	0.559	0.654 in-domain
coba:150m-ternary	ternary	0.535	0.622 in-domain
coba:68m-ternary	ternary	0.483	0.603 in-domain
pplx-decider:flash (the teacher, 4B decoder)	bf16	0.726	0.824 zero-shot
decider:0.8b (0.8B decoder)	bf16	0.506	0.669 zero-shot
laya:en (Laya's English model)	fp32	0.358	0.532

• typed-decisions: accuracy on the 400 rows of LocalLLaMA/typed-decisions' test split, which no model here trained on.
• Public benchmark: macro accuracy over the 3,880 records of the eleven datasets Ollaya's bench uses (MNLI, BoolQ, PAWS, VitaminC, civil_comments, Aegis 2.0, HelpSteer2, PubMedQA, SQuAD 2.0, MASSIVE, SummEval). In-domain: the model also trained on the train splits of ten of them (never their test records; rows that share any text with a test record were dropped). Zero-shot: trained without them.
• For scale: pplx-decider:flash, the 4B decoder this model learned from, scores 0.726 and 0.824 zero-shot; 0.8B decoders score 0.46–0.51 and 0.61–0.67.

Training

• Network: Laya's (laya 0.3.7): the question, a [MASK] before every option and the state in one sequence of up to 512 tokens; a two-layer transformer head scores each option at its marker. Encoder: Ettin 400m (ModernBERT architecture, MIT).
• Data: about 400,000 English questions in 344,000 states from permissively licensed decision datasets (Apache-2.0, MIT, CC BY, CC BY-SA, CC0, BSD; every source and its license is in DATA_SOURCES.tsv). Rows from datasets with a non-commercial or unknown license were dropped.
• Labels: each question's gold answer where the data has one, mixed with the full probability distribution of pplx-decider:flash (Apache-2.0), computed through Ollaya.
• Recipe: soft cross-entropy plus the ranked probability score on score questions; options shuffled half the time; one temperature per question type fitted on held-out data.

Limits

• English only. Other languages are not refused but are not measured.
• 512 tokens for the state and the question together: messages, tickets and small JSON states, not long documents or agent traces.
• score questions are the weakest kind for every model of this size; read the probabilities, not only the expected level.
• Probabilities are calibrated on average, not per domain: check them on your own data before acting on them automatically, and keep a person in the loop for decisions that matter.

License

Apache-2.0. The encoder started from Ettin (MIT; see NOTICE). The training data's sources and their licenses are in DATA_SOURCES.tsv; CC BY and CC BY-SA sources are credited there.