Open Decision Model 150M: calibrated decisions for LLM agents

A small open decision model of the Jev / Laya type. Give it a state and a list of options. It returns a calibrated probability for each option in about 50 ms on a CPU. It does not generate text.

Try it with no download: add the free MCP endpoint to your agent.

Try it with no download: add the free MCP endpoint https://decision-model.vercel.app/mcp to your agent.

Use it where an agent must pick, not write:

LLM routing: send a request to a small, medium or large model. Save cost.
Tool risk: decide if a tool call needs human approval.
Yes/no checks: "Can this task delete user data?" gives p_yes.
Any multiple-choice: support intents, shopping options, difficulty levels. Any option count.


Size	149M parameters (ModernBERT-base encoder plus one linear head)
Speed	52 ms median on CPU for one question with 4 options
Calibration	ECE 0.047 after temperature scaling
Licence	Apache 2.0 (commercial use is permitted)

Quickstart (no extra package)

pip install torch transformers huggingface_hub safetensors, then:

import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoModel, AutoTokenizer

path = snapshot_download("gparitosh007/Open-Decision-Model-150M")
tok = AutoTokenizer.from_pretrained(path)
enc = AutoModel.from_pretrained(path, dtype=torch.float32).eval()
w = load_file(f"{path}/model.safetensors")
W, b, T = w["decision_head.weight"].float(), w["decision_head.bias"].float(), w["decision_head.temperature"].item()

@torch.inference_mode()
def decide(state, instructions, options, preferences=None):
    """options: dict label -> description. Returns dict label -> probability."""
    e = lambda t: tok.encode(t, add_special_tokens=False)
    ids = [tok.cls_token_id]
    if preferences:
        ids += e("\n".join(f"{k}: {v}" for k, v in preferences.items())) + [tok.sep_token_id]
    ids += e(state) + [tok.sep_token_id] + e(instructions) + [tok.sep_token_id]
    spans = []
    for label, desc in options.items():
        o = e(f"{label}: {desc}" if desc else label)
        spans.append((len(ids), len(ids) + len(o)))
        ids += o + [tok.sep_token_id]
    h = enc(input_ids=torch.tensor([ids])).last_hidden_state[0]
    logits = torch.stack([h[s:t].mean(0) for s, t in spans]) @ W.T + b
    return dict(zip(options, torch.softmax(logits.squeeze(-1) / T, -1).tolist()))

print(decide(
    "Refactor the auth module and add tests. The repo has 40 files.",
    "Which model must handle this request?",
    {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"},
    preferences={"cost": "low cost before speed"},
))

Output: {'haiku': 0.16, 'sonnet': 0.50, 'opus': 0.34}. This code gives the same probabilities as the official package.

Use it now: free public MCP endpoint

No download and no API key. A public, stateless instance runs on Vercel with the ONNX model: https://decision-model.vercel.app/mcp.

Claude Code:

claude mcp add --transport http decision https://decision-model.vercel.app/mcp

Other MCP clients (Claude Desktop, Cursor and others): add a remote HTTP MCP server with the same URL.

Tool	What it does	Required inputs	Returns
`choice`	Picks between options	`state`, `instructions`, `criteria` (one description per option)	A probability per option and a confidence
`score`	Rates on an ordered scale (for example trivial to very hard)	`state`, `instructions`, `levels`	A probability per level, the expected level and a legend
`noul`	Answers a yes/no question	`state`, `instructions`, optional yes/no definitions in `criteria`	`p_yes`
`decide`	Runs several questions against one state (Jev / Laya format)	`state`, `questions` (each with `id`, `type`, `instructions`, optional `criteria`)	One result per question
`build_preferences`	Asks a preference interview for one domain	`domain`, optional `answers`	A preference file for the domain
`feedback`	Records what you did with a decision	`decision_id`, `action` (accept, reject or other), optional `chosen`	A log of the outcome

All decision tools also accept domain (a saved preference file) or inline preferences, and question_id. The usual flow is: build_preferences, then choice, score or noul, then feedback.

The public instance is stateless: it saves no files and logs no request bodies. On it, send preferences inline in each call. To keep preference files and feedback, run your own server (see "How to use").

HTTP, without MCP:

curl -s https://decision-model.vercel.app/v1/decide -H 'Content-Type: application/json' -d '{
  "state": "Refactor the auth module and add tests. The repo has 40 files.",
  "questions": [{"id": "model", "type": "choice",
    "instructions": "Which model must handle this request?",
    "criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"}}]}'

The first call after an idle period takes about 50 s, because the function downloads the model. Later calls take less than 1 s.

For AI agents

MCP endpoint: https://decision-model.vercel.app/mcp. Use the choice tool to route or to pick an option.
Model id: gparitosh007/Open-Decision-Model-150M. Call decide(state, instructions, options) above and pick the option with the highest probability.
When the top probability is low, use a fallback. Read "Limitations and bias".

Former name: decision-model. The old URL redirects here.

Details

Model version: decision-model 0.1.0. Export format version: 1. The encoder is tasksource/ModernBERT-base-nli, a ModernBERT-base fine-tuned on NLI, with 149,015,041 (149.0M) parameters. This card lives in README.md of the export folder.

Summary

An open decision model of the Jev / Laya type. The model reads a state, optional user preferences and one or more questions. The model returns calibrated probabilities for every option and does not generate text. A linear head maps the mean-pooled vector of each option span to one logit, so one model works with any option count. The attribute temperature calibrates the probabilities. An MCP server serves the model. The source code is not public yet.

Intended use

The model serves decisions with calibrated probabilities over explicit options:

Model routing: pick haiku, sonnet or opus for one request.
Tool risk: decide if a tool call needs approval.
Shopping and support: pick one option, or score difficulty levels.
Score answers return the expected level; noul answers return p_yes.

The consumers are routers, MCP agents and policy checks. The probabilities are calibrated on the training domains with temperature scaling.

NOT intended use

Do not use the model to generate text. The model returns probabilities only.
Do not expect high accuracy in domains that are not in the training data. In a held-out domain test, the accuracy was 0.57; see Evaluation.
Do not use the model as the only check in a safety critical flow. The model is a probability estimate, not an approval.
Do not use the model for medical, legal or financial decisions with no human in the loop.
Do not trust the confidence value as a correctness guarantee. It measures the concentration of the distribution only.

Input and output format

A request holds one state, optional preferences, and a list of questions. Each question has id, type (choice, score or noul), instructions, and optional criteria. This shape is the one format for the whole repo.

{
  "state": "Refactor the auth module and add tests. The repo has 40 files.",
  "preferences": {"cost": "low cost before speed"},
  "questions": [
    {"id": "model", "type": "choice", "instructions": "Which model must handle this request?",
     "criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"}},
    {"id": "difficulty", "type": "score", "instructions": "How difficult is this task?",
     "criteria": ["trivial", "easy", "medium", "hard", "very hard"]},
    {"id": "risky", "type": "noul", "instructions": "Can this task delete user data?"}
  ]
}

The response holds answers with one entry per question. probs holds one probability per option label. confidence is 1 minus the normalized entropy of the probability row. A score answer adds expected_level (0-based) and a legend. A noul answer adds p_yes.

{
  "decision_id": "3f9d2b1e-4c8a-4b6f-9e21-a7d5c0b8e1f2",
  "model_version": "decision-model 0.1.0",
  "answers": [
    {"id": "model", "type": "choice",
     "probs": {"haiku": 0.62, "sonnet": 0.31, "opus": 0.07}, "confidence": 0.51},
    {"id": "difficulty", "type": "score",
     "probs": {"trivial": 0.05, "easy": 0.35, "medium": 0.40, "hard": 0.15, "very hard": 0.05},
     "confidence": 0.42, "expected_level": 1.8, "legend": {"0": "trivial", "1": "easy", "2": "medium", "3": "hard", "4": "very hard"}},
    {"id": "risky", "type": "noul",
     "probs": {"yes": 0.09, "no": 0.91}, "confidence": 0.82, "p_yes": 0.09}
  ]
}

Questions in one request are evaluated independently. The encoder reads one sequence: [preferences] [SEP] [state] [SEP] [instructions] [SEP] [option 1] [SEP] .... For the exact option order rules read decision_config.json.

How to use

The Quickstart above needs no extra package. The decision_model package below is not public yet.

Load the export folder and predict:

from decision_model.model import DecisionModel

model = DecisionModel.load("export/NAME")
result = model.predict({
    "state": "Refactor the auth module and add tests.",
    "questions": [{
        "id": "model",
        "type": "choice",
        "instructions": "Which model must handle this request?",
        "criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"},
    }],
})
print(result["answers"][0]["probs"])

Run example.py in this folder for the same flow.

Serve the model with the MCP server:

DECISION_MODEL_PATH=/absolute/path/to/this/folder claude mcp add decision -- uv run --directory <repo> decision-mcp

Serve the model over HTTP (decision-http):

DECISION_MODEL_PATH=/absolute/path/to/this/folder python -m decision_model.http_server

Training data

Teacher-labelled synthetic tasks (v2, 185,131 records). Template generators for model routing, tool call risk, shopping, support and travel, plus about 30,000 generic tasks, paraphrases and turned-around questions. The template domains are balanced.
The teacher is Qwen3-235B FP8 (run q235fp8) served through the vLLM OpenAI API. Soft labels come from the top-20 log-probabilities of the first answer token. Each task is labelled with two option orders, and the mean removes position bias. Pairs that disagree by more than 0.5 total variation distance are removed.
Open data with human labels (109,767 records). Only datasets with a clear commercial-use license are used:

Source	Records	License	Use
tasksource/bigbench	44,427	Apache 2.0	choice questions
tasksource/mmlu	15,573	MIT	choice questions
nvidia/HelpSteer2	21,362	CC BY 4.0	rating questions
PolyAI/banking77	16,430	CC BY 4.0	support intents
clinc/clinc_oos	11,975	CC BY 3.0	support intents

Training setup

Item	Value
Base encoder	tasksource/ModernBERT-base-nli
Loss	KL divergence to the label probabilities, plus 0.1 x cross-entropy to the top option
Optimizer	AdamW, LR 3e-5, batch 16, 3 epochs, 48,174 steps
Max length	1024 tokens; a long state is trimmed from the start, the options stay
Split	by template: validation templates are not in the training set
Weights	class weights; domain weights generic=3, open_tasksource=2
Seed	42 (one run)
Hardware	one B300 GPU, about 47 minutes
Calibration	temperature scaling on the validation split; temperature 3.9863

Evaluation

All numbers are agreement with the label of the top option (teacher or human label), not human-verified correctness. The validation templates are not in the training set.

This checkpoint (seed 42)

Metric	Value
Validation accuracy	0.468 (majority baseline 0.404)
Validation balanced accuracy	0.455
ECE raw / after temperature scaling	0.229 / 0.047
CPU latency, one question with 4 options	52 ms median (6 threads)

Domain	Accuracy	Balanced accuracy	ECE
generic	0.717	0.654	0.191
shopping	0.677	0.650	0.204
tool_risk	0.504	0.398	0.058
open_tasksource	0.407	0.470	0.063
routing	0.367	0.460	0.152
support	0.215	0.254	0.276

Question type	Accuracy
noul	0.683
choice	0.436
score	0.312

Held-out domain test (same backbone, without open data)

This checkpoint has no held-out domain. A run with the same backbone and settings, but without the open data and with travel held out, gave travel accuracy 0.566, balanced accuracy 0.588 and ECE 0.098. That run had routing accuracy 0.450 and shopping accuracy 0.733. With the open data, routing and shopping are lower (one seed each).

Limitations and bias

Routing and support are weak (0.37 and 0.22 accuracy). Use the probabilities and a fallback when the top probability is low.
Score questions are weak (0.31 accuracy).
The open data lowered the accuracy on routing and shopping compared with a run without it.
One seed only; differences of a few points can be noise.
The confidence value measures concentration. It does not guarantee a correct answer.

Licenses

Part	License
This model	Apache 2.0
Encoder tasksource/ModernBERT-base-nli (from answerdotai/ModernBERT-base)	Apache 2.0
Teacher Qwen3-235B	Apache 2.0
Open training data	Apache 2.0, MIT, CC BY 4.0, CC BY 3.0 (see Training data; CC BY sources are attributed above)

Citation

Cite the blog post that defines the decision model type: Jev, Laya and the emerging role of decision models, https://runware.ai/blog/jev-laya-and-the-emerging-role-of-decision-models.