mvbalaji/od1-base

OD-1 Base (4B)

OD-1 is an open System One decision model: give it a state (text or JSON) and typed questions (choice, noul = yes/no, score = ordinal) and it returns every answer with full probability distributions (plus an explicit NOT_ANSWERABLE outcome), in the Jev / TypeSafe /v1/systemone shape.

Qwen3.5-4B, fully fine-tuned with the OD-1 recipe; the checkpoint was selected on out-of-domain validation accuracy (later checkpoints scored higher in-domain but lost zero-shot ability).

When to use it

The most accurate zero-shot OD-1. Use the built-in hand-off to answer most requests at Nano speed.

Results on held-out test sets

Same fixed samples for every system (≤ 2,000 decisions per set, seed 0; typed-decisions: all 2,000 test decisions). Baselines answer each question as one choice over its options (2–24 options; n/a = more options than their interface allows). Jev 1.13 = jev-1.13.0, measured 2026-09-25. Accuracy:

test set	this model	Jev 1.13	Laya	Laya typed-decisions	Tev1-4B	CLM-8B
typed-decisions	0.582 (n=2000)	0.741	0.353	0.737	0.690	0.393
AG News	0.843 (n=2000)	0.882	0.924	0.922	0.886	0.354
Emotion	0.559 (n=2000)	0.595	0.597	0.603	0.583	0.281
SST-5	0.474 (n=2000)	0.579	0.341	0.463	0.533	0.273
Banking77	0.482 (n=2000)	n/a	n/a	n/a	n/a	n/a
MASSIVE-en	0.694 (n=2000)	n/a	n/a	n/a	n/a	n/a
BFCL native	0.942 (n=1252)	0.975	0.679	0.847	0.954	0.694
BFCL 24 options	0.938 (n=1909)	0.969	0.600	0.741	0.950	0.625
BFCL 100 options	0.851 (n=1909)	n/a	n/a	n/a	n/a	n/a
BFCL 1,000 options	0.439 (n=1909)	n/a	n/a	n/a	n/a	n/a
BFCL irrelevance	0.586 (n=1101)	0.702	0.788	0.390	0.701	0.661
Adversarial (ours)	0.887 (n=2000)	0.832	0.619	0.628	0.793	0.449

On the typed-decisions test split, scored with the benchmark card's formulas (KL from the soft gold and Brier verified against the card's reference rows): accuracy 0.582, KL 0.496, Brier 0.241 (generalist, zero-shot). For reference the card lists TypeSafe Jev 1.13 at 0.727 / 1.442 / 0.148 and meraGPT Decider 1 at 0.768 / 0.096 / 0.052 (both generalists).

Built-in hand-off (self-contained)

This repo bundles the matching Nano model in nano/ and a fitted hand-off in cascade.json:

• two-model: the Nano answers first; the request goes to this model when the Nano's lowest per-question confidence < 0.5.
• self-exit: this model answers at its layer-8 exit first and runs its full depth when that exit's lowest confidence < 0.99.
• Default: two-model (recommended on validation data; thresholds fitted on validation data only).
• Validation: nano alone 0.737, this model alone 0.761, two-model hand-off 0.766 with 0.283 of requests escalated.
• Validation, self-exit: exit alone 0.589; at tau 0.99 accuracy 0.760 with 0.997 escalated (an early exit this shallow rarely clears the threshold, so self-exit mostly costs time).
• Test (two-model): typed-decisions 0.582, AG News 0.802, Emotion 0.560, SST-5 0.482, Banking77 0.541, MASSIVE-en 0.679, BFCL native 0.945, BFCL 24 options 0.936, BFCL 100 options 0.868, BFCL 1,000 options 0.474, BFCL irrelevance 0.579, Adversarial (ours) 0.892.
• Test (self-exit): typed-decisions 0.582, AG News 0.843, Emotion 0.559, SST-5 0.474, Banking77 0.482, MASSIVE-en 0.694, BFCL native 0.942, BFCL 24 options 0.938, BFCL 100 options 0.851, BFCL 1,000 options 0.439, BFCL irrelevance 0.586, Adversarial (ours) 0.887.

from od1.cascade import OD1Cascade
m = OD1Cascade.load(path)                  # default mode; or mode="two-model" / "self-exit"
m.enable_cuda_graphs()
answers = m.answer(state, questions)   # each answer says which model answered

Speed (one H100, bf16, batch-1 requests, CUDA graphs)

state / questions	this model (p50 ms)	Laya typed-decisions, all questions in one call (p50 ms)
short (≤64 tokens) / 1	10.7	15.9
short (≤64 tokens) / 5	26.3	17.9
short (≤64 tokens) / 20	95.3	21.4
medium (200–400 tokens) / 1	15.6	16.9
medium (200–400 tokens) / 5	73.4	17.7
medium (200–400 tokens) / 20	126.1	41.4

Limitations

• On unfamiliar tasks accuracy is well below the best closed systems (see the table); NOT_ANSWERABLE is only returned above the fitted threshold in serving.json.

• English only. Baselines were called through a choice-wrapped interface (Laya's typed-decisions checkpoint also natively), which may understate them. No frontier-LLM reference was run.

• Scores on typed-decisions measure agreement with its labelling teacher (see the dataset card).

Usage

pip install torch "transformers>=5.17.0" huggingface_hub

from huggingface_hub import snapshot_download
path = snapshot_download('mvbalaji/od1-base')
import sys; sys.path.insert(0, path)
from od1.model import OD1Model
m = OD1Model.load(path, dtype='bf16', device='cuda')   # applies the fitted serving.json
m.enable_cuda_graphs()
answers = m.answer(state, {'route': {'type': 'choice', 'instructions': 'Which team?', 'criteria': {'billing': 'payments', 'tech': 'bugs'}}})

Smoke test of example.py for this checkpoint: {"route": {"answer": "billing", "confidence": 0.994}, "refund": {"answer": 0.9936571213287037, "confidence": 0.99}, "urgency": {"answer": 2, "confidence": 0.461}}

Requirements and speed

The Qwen3.5 backbone mixes full attention with gated delta-rule (linear attention) layers. Install the fast kernels, otherwise transformers silently falls back to a slow pure-PyTorch implementation of those layers and a one-question request takes hundreds of milliseconds instead of a few:

pip install "transformers>=5.17.0" flash-linear-attention==0.5.2
pip install causal-conv1d==1.7.0 --no-build-isolation   # compiles against your torch/CUDA (needs nvcc)

Check: import fla, causal_conv1d must succeed; when they are missing, transformers logs "falling back to its reference PyTorch implementation". Then enable CUDA graphs and warm up before timing:

m.enable_cuda_graphs()
for _ in range(5):                       # warm-up: kernel autotuning + graph capture per input shape
    m.answer(state, questions)
torch.cuda.synchronize(); t0 = time.perf_counter()
answers = m.answer(state, questions)
torch.cuda.synchronize(); print((time.perf_counter() - t0) * 1000, "ms")

Tested with torch 2.14.0+cu130, transformers 5.17.0, flash-linear-attention 0.5.2, causal-conv1d 1.7.0 on one NVIDIA H100 80GB HBM3 (bf16). Expected on that setup for a short one-question request: about 35 ms without CUDA graphs and about 6 ms with them (see the speed table); other GPUs differ. CPU inference works but is slow.

Training data and licenses

Openly licensed datasets (see sources.csv: dataset, revision, license, split, counts), project-generated synthetic policy checks and teacher-verified structured JSON workflows (written by Qwen3.5-27B-FP8, which is never a compared system, without access to any benchmark data), with teacher distillation labels. Held-out evaluation sets were never trained on (13-gram deduplication against every test set).

Citation

Paper in preparation (OpenDecide-1).