qwen3.5-0.8B-decision-model

A decision model fine-tuned from Qwen/Qwen3.5-0.8B-Base. It answers typed questions about a state and returns calibrated probability distributions, not generated text:

type	question	answer
Choice	pick one option from a set you define	choice, probabilities, confidence
Score	pick a position on ordered levels you describe	score (= Σ i·pᵢ), legend, probabilities, confidence
Noul	yes/no	noul = P(yes)

The interface follows the publicly documented request/response shape of TypeSafe's System One API (docs). This is an independent open model. It is not affiliated with or endorsed by TypeSafe, and it is not Jev.

Head to head: TypeSafe Jev vs this model on unseen scenarios

We sent the same 150 held-out scenarios (714 questions) to both models, as identical POST /v1/systemone requests. None of these scenarios was used to train, select or calibrate this model.

• Jev: jev-1.13.0 through TypeSafe's hosted API.
• This model (0.8B parameters): ran locally on a MacBook Pro (Apple M2 Max, 64 GB), not on a GPU server.

	TypeSafe Jev (jev-1.13.0)	this model (0.8B, MacBook Pro (Apple M2 Max, 64 GB))
Accuracy (top answer = label's top answer)	93.1%	81.1%
Accuracy: Choice / Noul / Score	93.9% / 94.1% / 90.3%	81.2% / 81.6% / 80.0%
Brier (lower is better)	0.042	0.158
Log-loss vs the soft labels (lower is better)¹	0.957	0.615
Calibration error, ECE: Choice / Noul / Score¹	0.088 / 0.018 / 0.118	0.037 / 0.026 / 0.059
Choice answers with confidence ≥ 0.7: share automated, accuracy	91% at 97%	55% at 93%
Choice answers with confidence ≥ 0.9: share automated, accuracy	83% at 100%	4% at 100%
Latency per request, p50 / p95	1.3 s / 2.3 s (hosted API, incl. network)	1.7 s / 5.2 s (local, no network)
Cost for the whole test set	$0.005 (128k input tokens)	$0, offline, on your own machine

What this means:

• Head to head: on questions where only one model was right, Jev was right 103 times and this model 17. The two models agree on 82.5% of top answers.
• Jev is the stronger model on accuracy, Brier and confidence-gated automation.
• This model is a small open alternative that runs fully offline. On a GPU it answers in about 50 ms per request (see the evaluation section below).
• This model is under-confident. Its average top probability is 0.75, while the labels average 0.87, so it automates fewer decisions at high confidence thresholds.
• ¹ Read log-loss and ECE with care. The labels are soft probabilities written with this project's rubric, and this model was trained on the same labelling style. Jev often answers with near-certainty (42% of its top probabilities are ≥ 0.99), which log-loss penalises whenever the labels spread some probability to other options. Accuracy and Brier are the fairer comparison.

The comparison script and a guide to reproduce it with your own TypeSafe API key are in the project repository (scripts/compare_jev.py, docs/compare_jev.md).

How it works

• Logit readout. Each question is rendered after the state and ends in Answer:. The probabilities are a softmax of the next-token logits restricted to single-token labels: A… Z/ a… z for options and levels, and yes/ no for Noul. Every answer is a valid distribution over exactly the options you declared, from one forward pass per question, with no JSON parsing.
• Confidence is (K·p_max − 1)/(K − 1), the definition published at docs.typesafe.ai/confidence.
• Training objective (RLCD-inspired). Because the output is a distribution, the expected log-score reward equals negative cross-entropy. We therefore minimise CE(target, p) + 0.5·Brier(target, p), two strictly proper scoring rules, against soft expert-agreement targets. TypeSafe has not published RLCD's actual objective, so this is an approximation, not a reproduction.
• Post-hoc temperature per primitive, fitted on the validation set: choice T=1.3, noul T=1.5, score T=1.2. It is stored in calibration.json and applied by the bundled jevlite package.

Usage

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("mghafiri/qwen3.5-0.8B-decision-model")
sys.path.insert(0, path)              # bundled jevlite/ package
from jevlite.model import SystemOne

engine = SystemOne(path)              # applies calibration.json automatically
print(engine.system_one(
    state="Help! My payouts have been failing for 3 days.",
    questions={
        "is_urgent": {"type": "noul", "instructions": "Does this convey urgency?"},
        "department": {"type": "choice", "instructions": "Which team should handle this?",
                        "criteria": {"billing": "Payments, invoicing, refunds",
                                      "technical": "Bugs, outages, integrations",
                                      "sales": "Pricing, upgrades, new accounts"}},
        "frustration": {"type": "score", "instructions": "How frustrated is the customer?",
                         "criteria": ["Calm", "Frustrated", "Very angry"]},
    },
))

Training data

The dataset has 2000 synthetic English scenarios (9716 questions), each a state plus 3–6 independent questions. It covers 32 domains: agent_tool_routing, airline_travel_ops, aml_kyc_compliance, app_marketplace_policy, banking_fintech, citation_verification, clinical_literature_screening, customer_support, cybersecurity_soc, ecommerce, education_admissions, energy_utilities, entity_matching, government_services, healthcare_admin, hr_recruiting, insurance_claims, it_helpdesk, legal_compliance, llm_guardrails, logistics_sales_crm, manufacturing_qa, news_claim_verification, payroll_benefits, procurement_vendor_risk, property_management, rag_passage_relevance, smart_home_iot, software_engineering, span_selection_extraction, telecom_support, trust_safety_moderation.

• Question mix: noul 41%, choice 37%, score 22%.
• Difficulty mix: clear 39%, moderate 31%, borderline 16%, insufficient 9%, adversarial 6%.
• How targets were made: Claude authored them as the fraction of careful experts who would pick each option. A blind second annotator relabelled every question without seeing the originals.
  • Agreement on the top answer: 9434/9718 = 97.1%.
  • Where the two agreed, the target is 0.6·author + 0.4·reviewer.
  • Borderline disagreements were averaged 50/50.
  • The remaining disagreements were adjudicated one by one.
• Splits by scenario: train 1700 / val 150 / test 150.

Training

• Full fine-tune with frozen token embeddings, fp32 weights and bf16 autocast.
• AdamW, lr 1e-5, cosine schedule. Option-order shuffles were used as augmentation.
• Epochs run: 10 of 10 requested (completed all epochs), with 2720 optimizer steps in 48 min.
• The published weights are the checkpoint with the lowest validation cross-entropy: epoch 9.

epoch	train CE	val CE	val Brier	val acc
0	–	0.966	0.375	0.562
1	0.777	0.743	0.227	0.738
2	0.578	0.714	0.211	0.762
3	0.483	0.737	0.210	0.783
4	0.457	0.658	0.180	0.786
5	0.423	0.669	0.177	0.797
6	0.404	0.673	0.174	0.808
7	0.390	0.662	0.174	0.797
8	0.381	0.664	0.171	0.803
9 (best, published)	0.376	0.652	0.166	0.803
10	0.374	0.657	0.168	0.801

Evaluation (held-out test split)

Accuracy compares the predicted argmax with the target argmax. Log-loss and Brier are measured against the soft targets. ECE uses 10 bins over the top-option probability (P(yes) for Noul).

model	type	n	acc	log-loss	Brier	ECE	conf≥0.7 coverage / acc
base Qwen3.5-0.8B-Base	choice	261	0.559	1.113	0.406	0.077	0.09 / 0.92
base Qwen3.5-0.8B-Base	noul	288	0.552	0.723	0.362	0.165	–
base Qwen3.5-0.8B-Base	score	165	0.479	1.188	0.399	0.117	0.15 / 0.71
this model	choice	261	0.808	0.702	0.191	0.034	0.55 / 0.93
this model	noul	288	0.812	0.468	0.141	0.028	–
this model	score	165	0.800	0.734	0.134	0.064	0.30 / 0.96

model	option-permutation TVD (choice)	score MAE	latency p50 / p95 (ms, 1 request)
base Qwen3.5-0.8B-Base	0.2010	0.562	53 / 729
this model	0.0574	0.255	52 / 87

Limitations

• The targets are expert-agreement estimates written by an LLM, not real-world outcomes. Calibration here means calibration to those targets. Validate on your own labelled data before automating decisions.
• The test set is small (about 714 questions), so ECE is noisy.
• The model is English-only and text-only.
• Keep arithmetic, counting and date comparison in code. Ask atomic, literal questions.
• Use confidence to route uncertain cases to a human or a larger model.