ouro-2.6b-decision-lora

A calibrated decision model: state in, typed probability distributions out, one forward pass, no text generation. Three question types, the same shape as TypeSafe's Jev API: choice (distribution over options), score (distribution over ordered levels, plus the expected level) and noul (P(true)).

It is a LoRA adapter (r=16, 1.2 % of parameters) plus a 2-layer scoring head on top of ByteDance/Ouro-2.6B, a looped language model (48 layers, 4 recurrent steps). The head reads the last-token hidden state of state + question + listed options + "Answer: <letter>" and adds a learned multiple of the letter's next-token log-probability, so at initialisation the model scores exactly like prompting the base model and training only learns a residual.

Results

TypeSafe public eval (evals.typesafe.ai, rebuilt with the open harness from system-one-open: 20 cases, 372 reference pairs, strict common subset of 343 pairs answered by every published model). None of these four workflows (security incidents, agent-trace observability, invoice processing, customer service) was in the training data.

model	accuracy (343 common pairs)
Opus / Sol (frontier models the references were built from)	89.5 / 90.4
Jev (TypeSafe)	86.6
this model (trained on ARC-Easy + BoolQ only)	76.7 (ECE 0.078, NLL 0.653)
system-one-open, Gemma 4 E2B (trained on 70 tasks + 59k synthetic)	76.7
Qwen2.5-7B prompting (jev-on-a-laptop)	73.8
Ouro-2.6B prompting, no training (same template, letter read-out)	63.3
per-question majority answer (knows the test)	81.0

Training on two unrelated QA datasets moved this benchmark from 63.3 to 76.7. All 372 pairs: 76.6 %. By workflow: invoice 82.6 (184), customer service 78.3 (92), agent trace 70.8 (48), security 56.2 (48). By type: noul 80.9 (236), choice 76.1 (109), score 40.7 (27, a type absent from the training data). With 372 pairs the per-workflow numbers carry roughly ±7–14 points of sampling error.

In-domain (500 held-out items per task, temperature fitted on the same validation set):

	before training (prompting the base model)	after 1 epoch
ARC-Easy accuracy	95.1	97.2
BoolQ accuracy	87.4	91.4
NLL (both, calibrated)	0.292	0.165

Held-out tasks (never trained, temperature not refitted):

	after 1 epoch	Qwen2.5-7B prompting
OpenBookQA (500)	90.4	83.2
CommonsenseQA (1,221)	79.3	81.2
HellaSwag (2,000)	70.5	77.0

Base-model dependence: with the recurrence reduced to 2 steps the prompting baseline drops to ARC 77.5 / BoolQ 77.3, and with 1 step to 57.5 / 37.8. Keep total_ut_steps = 4.

Training

• Data: 2,000 ARC-Easy (choice, 3–5 options) + 2,000 BoolQ (noul) training records, 1 epoch, lr 5e-5, batch 4 × 4 accumulation, gradient checkpointing, ~45 minutes on one RTX 4090.
• Loss: cross-entropy + Brier on the option distribution. Checkpoint selected by calibrated NLL on the validation set.
• Calibration: one global temperature T = 1.3155 fitted on the ARC+BoolQ validation set (ECE 0.027 → 0.013). On out-of-domain data the model is somewhat over-confident (TypeSafe eval ECE 0.078); refit T on a small labelled sample of your own task if you rely on the probabilities.
• Licenses: Ouro-2.6B Apache-2.0; ARC (CC BY-SA 4.0); BoolQ (CC BY-SA 3.0). This adapter: Apache-2.0.

Usage

Requires transformers<4.56 (the Ouro remote code breaks on 5.x), peft>=0.17,<0.18, torch. The base model is downloaded from the Hub with trust_remote_code=True.

pip install "torch" "transformers==4.54.1" "peft==0.17.1" "huggingface_hub"

from inference import DecisionModel

model = DecisionModel.from_pretrained("noscienthoon/ouro-2.6b-decision-lora", device="cuda")   # ~6 GB in bf16
answers = model.decide(
    state={"channel": "email", "subject": "Charged twice", "body": "Two charges for one order. Fix this today."},
    questions={
        "queue":    {"type": "choice", "prompt": "Which support queue should handle this ticket?",
                     "options": {"billing": "Payments, refunds, duplicate charges", "shipping": "Delivery", "technical": "Bugs", "general": "Other"}},
        "priority": {"type": "score", "prompt": "How should this ticket be prioritized?", "levels": ["Low", "Normal", "High", "Critical"]},
        "angry":    {"type": "noul", "prompt": "The customer sounds angry."},
    },
)
print(answers)
# {'queue': {'type': 'choice', 'choice': 'billing', 'probabilities': {...}, 'confidence': ...},
#  'priority': {'type': 'score', 'score': 2.3, 'probabilities': {...}, 'legend': [...], 'confidence': ...},
#  'angry': {'type': 'noul', 'noul': 0.87, 'confidence': ...}}

decide(...) accepts any JSON-serialisable state (a string is used as is, anything else is pretty-printed as JSON), scores each question's options in one forward pass (K sequences for K options), and applies the stored temperature. Pass temperature= to override it. max_query_len (default 4096 here; 512 was used in training, the backbone supports 65k) controls left-side truncation of long states: the end of the state and the question are always kept.

Cost: every option is scored as its own sequence (state + question + options + "Answer: A"), so a question with K options costs K forward passes of the prefix. Ouro's recurrent cache does not support prefix sharing.

Files

file	what
adapter/	PEFT LoRA adapter (safetensors, 121 MB)
head.pt	scoring head state dict (4 MB): mlp.* and lm_scale
decision_config.json	prompt/scoring configuration and the fitted temperature
calibration.json	validation metrics before/after temperature fitting
inference.py	the whole inference path (load, templates, scoring, calibration), ~250 lines, no other dependency

Limitations

• Trained on two public QA datasets only; the TypeSafe-eval workflows, score questions and non-English inputs were never seen in training. score is the weakest type (40.7 % on the eval's 27 score pairs).
• Calibration was fitted in-domain and does not transfer perfectly (see above).
• At most 26 options per choice question through the letter readout; larger option sets are scored without the options listed in the prefix.
• The reference answers of the TypeSafe eval are a frontier-model consensus, not human labels.

Related

• jev-ood-calibration: our independent measurement of Jev's calibration in and out of domain (same metrics code as the calibration numbers above).
• system-one-open: the open harness used to rebuild the TypeSafe public eval; its Gemma 4 E2B replica is the 76.7 row in the table.