Valrico-400M-v2: typed decisions with stated policies

By Timothy Epping.

Valrico models answer a decision question about a record, under a policy that the question states. They return a probability for each option. They are small text encoders, 400M parameters, built on Ettin, and run on a single modest GPU.

Valrico-400M-v2 supersedes Valrico-400M. What changed from v1: a new training lineage (the v6 and v7 rounds on many more public task datasets, then a final annealing stage on soft-labelled judgement data). The result is about as accurate as v1 on our holistic-judgement benchmark and clearly better on a held-out daily-use panel: about 6 points more accurate, and much better at choosing "none of these" when no option fits. On zero-shot typed decisions it is a little below v1 (0.642 vs 0.662); the gap is in score questions. The input format, valrico.py and the checkpoint layout are unchanged.

Model	Use it when	Typed-decisions accuracy
Valrico-400M-Fitted-v2	Your decisions resemble the typed-decisions benchmark's workflows. On daily use the two models tie; the fitted one abstains correctly a little more often.	0.744 (fitted)
Valrico-400M-v2 (zero-shot)	You need a model that has never seen the typed-decisions benchmark, for example to report a zero-shot number.	0.642

The fitted model was trained on a slice of the typed-decisions benchmark's train split: it has seen the benchmark's four workflows and their question wording, though not its test records. Its 0.744 is a fitted result, like v1 Fitted's 0.749, not a zero-shot one. The zero-shot model was trained without that slice. On zero-shot typed decisions it is a little below v1's zero-shot model (0.642 vs 0.662); the gap is in score questions (see Limitations).

What a decision looks like

A request is one record (a ticket, a claim, a log excerpt, a chat) and one or more questions about it. Each question:

• starts with the policy that decides it ("Policy: refuse a claim sent in after the time limit ...");
• has a fixed set of options, of one of three kinds:
  • a choice among labels;
  • yes / no;
  • an ordered score.

The model scores every option and returns a probability distribution over the options. Facts that can be counted or computed belong in the record's header as computed fields (for example "Days beyond the time limit: 12"). The models are trained to rely on them; they are not trained to do the arithmetic themselves.

How to use

Code written for v1 works unchanged with the new repo id.

from valrico import Valrico

m = Valrico.load("eppingt/Valrico-400M-v2")       # or "eppingt/Valrico-400M-Fitted-v2", or a local directory

record = """Ticket: MEA-75460
Customers affected: 85
Account tier: free
Hours since received: 86

Hey, my meal kit box never arrived this week and I was still charged."""

m.decide(record,
         "Policy: P1 when 20 or more customers are affected; P2 when two or more are affected or the message "
         "has waited over 24 hours; P3 when one sender is blocked with no workaround; P4 otherwise. "
         "Which SLA level applies?",
         ["P1", "P2", "P3", "P4"],
         record_type="Service desk ticket")
# -> {"P1": ..., "P2": ..., "P3": ..., "P4": ...}, probabilities summing to 1

m.decide(record, "Policy: refund when the customer was charged for a box that never arrived. Refund?",
         ["false", "true"], record_type="Service desk ticket")

Plain transformers, no extra file

The model is a standard ModernBertForTokenClassification checkpoint with one output. Each option's score is the logit at its [MASK] token. Write the input as one string, with no spaces around [SEP] and [MASK]:

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("eppingt/Valrico-400M-v2")
model = AutoModelForTokenClassification.from_pretrained("eppingt/Valrico-400M-v2").eval()

def decide(record, question, options, record_type=None):
    q = f"{record_type}: {question}" if record_type else question
    text = q + "[SEP]" + "".join("[MASK]" + o for o in options) + "[SEP]" + record
    enc = tok(text, return_tensors="pt", truncation=True, max_length=2048)
    with torch.no_grad():
        scores = model(**enc).logits[0, :, 0]
    masks = enc.input_ids[0] == tok.mask_token_id
    return dict(zip(options, torch.softmax(scores[masks], -1).tolist()))

This gives the same tokens as valrico.py. Keep the question and options well under 2,048 tokens: valrico.py always keeps them whole and cuts only the record, while plain truncation cuts from the end of the string.

Notes

• Yes/no questions use the options ["false", "true"], in that order.
• record_type is optional: a short name for the kind of record ("Service desk ticket", "Payment review record"). The models were trained with one.
• Many questions at once: m.decide_many([{"record": ..., "question": ..., "options": [...]}, ...]) batches them and returns one probability list per question.
• Records as JSON work as well as "Field: value" text; key order and JSON layout barely matter (see Robustness).
• On a GPU the model runs in bfloat16; on a CPU, in float32.

Results

Accuracy is top-1. ECE is expected calibration error (15 bins, by mass): lower is better. All probabilities are the raw softmax (temperature 1); no temperatures ship with the model.

Typed-decisions benchmark (public test split)

	Valrico-400M-v2 (zero-shot)	Valrico-400M-Fitted-v2	v1 Valrico-400M	v1 Valrico-400M-Fitted
Overall	0.642	0.744	0.662	0.749
Choice: accuracy / ECE	0.597 / 0.151	0.702 / 0.111	0.568 / 0.128	0.717 / 0.091
Yes / no: accuracy / ECE	0.788 / 0.089	0.833 / 0.075	0.812 / 0.086	0.822 / 0.083
Score: accuracy / ECE	0.566 / 0.167	0.709 / 0.097	0.621 / 0.100	0.719 / 0.081

The fitted model saw a slice of the benchmark's train split (relabelled by our teachers); the zero-shot model never saw it. Neither saw the test split.

Holistic judgement and daily use

Set	Zero-shot: accuracy / ECE	Fitted: accuracy / ECE	v1 zero-shot: accuracy / ECE
Synthbench (2,000 judgement questions; gold from a labeller never used in training)	0.676 / 0.121	0.672 / 0.129	0.670 / 0.127
Daily-use panel (12 held-out public sources), accuracy	0.706 / 0.080	0.707 / 0.083	0.643 / 0.152
Daily-use panel: picks "none of these" when the right answer was removed	0.514	0.552	0.358
Daily-use panel: accuracy on reworded and noisy questions	0.701	0.701	0.637

The panel's sources (intents, tool use, NLI, retrieval, contracts, maths, commonsense, argument quality, sarcasm) are not read by any training family. The panel gains over v1 are significant in a paired bootstrap; the synthbench differences are not.

Decision Index 0.3 (public suite)

Run once per model as a final measurement, with no tuning on the suite. The index is the chance-corrected, coverage-adjusted weighted mean over 37 benchmarks in five areas (0 = random, 100 = perfect).

	Valrico-400M-v2 (zero-shot)	Valrico-400M-Fitted-v2	v1 Valrico-400M
Index (balanced skill)	19.90	19.58	15.53
Breadth skill	18.40	18.17	13.88
Knowledge & reasoning	8.23	8.27	3.92
Language understanding	20.98	20.93	18.36
Retrieval & classification	28.11	28.12	20.01
Tools	31.79	30.19	28.30
Arts	9.07	8.87	5.94

Multi-step reasoning, maths and code-execution benchmarks (e.g. GPQA, HLE, GSM8K, CRUXEval) stay near zero: these need generation or long chains of reasoning that a 400M decision encoder does not do.

Our own evaluation sets (held-out workflows, written by us)

Every set uses workflows, wording and questions that never appear in training.

Set	Zero-shot: accuracy / ECE	Fitted: accuracy / ECE
Tool routing	0.810 / 0.065	0.803 / 0.080
Inbox triage	0.819 / 0.049	0.823 / 0.053
Retrieval relevance	0.854 / 0.024	0.848 / 0.039
Policy compliance	0.855 / 0.057	0.858 / 0.058
Moderation and assistant scope	0.772 / 0.055	0.764 / 0.059
Judging model outputs	0.728 / 0.090	0.723 / 0.087
Security alert triage	0.765 / 0.053	0.764 / 0.063
Code decisions	0.757 / 0.093	0.753 / 0.102
Synthetic workflows	0.854 / 0.065	0.853 / 0.068
Real-record workflows	0.704 / 0.098	0.700 / 0.110
Instruction flips	0.824 / 0.054	0.821 / 0.058

Plain zero-shot classification (public sets, 100 examples each)

Set	Zero-shot: accuracy / macro-F1	Fitted: accuracy / macro-F1
AG News	0.890 / 0.882	0.880 / 0.872
Banking77 (its train split is in v2 training)	0.850 / 0.808	0.830 / 0.788
Emotion	0.500 / 0.475	0.510 / 0.487

Following computed facts: the escalation check

Before release, each model must follow header facts on two expense-escalation sets. On the computed-facts variant, false approvals and false refusals must each stay at or below 0.200.

	Zero-shot	Fitted
Set 1, manager review: false approvals / false refusals	0.004 / 0.016	0.000 / 0.016
Set 1, reimburse as submitted	0.083 / 0.043	0.065 / 0.050
Set 2, manager review	0.007 / 0.017	0.002 / 0.016
Set 2, reimburse as submitted	0.107 / 0.050	0.088 / 0.054
Result	pass	pass

v1 Fitted's false-approval rate on reimburse-as-submitted was 0.201; v2 cuts it by more than half.

Robustness

Check	Valrico-400M-v2	Valrico-400M-Fitted-v2	v1 Valrico-400M
Option order: same answer after reordering the options (daily-use panel, 3,476 choice questions)	0.901	0.899	0.882
Field order: same answer after shuffling the JSON record's keys (synthbench, 2,000)	0.901	0.904	0.922
Field order: same answer after shuffling the keys (typed-decisions train-state records, 1,534)	0.871	0.892	0.892
Instruction flips: both questions of a flipped pair right (131 pairs)	0.664	0.679	0.611
JSON layout: accuracy spread over 4 layouts (compact, spaced, indent 1, indent 2; synthbench)	0.008	0.014	not run

The instruction-flip pairs are near-identical questions whose stated policy reverses the answer; "both right" checks that the model follows the policy rather than surface cues. Reordering options or shuffling a record's keys changes the top answer on about one question in ten, mostly on questions the model is unsure of; accuracy barely moves (synthbench 0.672 -> 0.676 fitted, 0.677 -> 0.671 zero-shot). The exception: the fitted model, trained on typed-decisions records in their usual key order, loses 0.024 accuracy on those records when their keys are shuffled (0.902 -> 0.879); the zero-shot model loses none. Keep a stable field order.

Speed

The architecture and input layout are the same as v1's, so the v1 latency figures apply (one NVIDIA L4, bf16: about 12 ms median for one question on a typical record, about 32 questions per second). They were not re-measured for v2.

Fitting your own head

fit_head.py is unchanged: it runs the model once over your labelled questions and trains only a small scoring head. Its benchmark figure for v1 (0.718 on typed decisions) was not re-measured for v2.

Choosing a cutoff

For yes/no questions the models rank cases well, but the default cutoff of 0.5 is not always where you want it. If false positives and misses cost you different amounts, label a few hundred examples of each yes/no question and pick the cutoff per question that meets your target false-positive or miss rate.

Sending uncertain questions to a larger model

Each answer's margin, the gap between the top two probabilities, is a good signal of when the model is unsure. Escalate the least certain questions to a larger model or a person.

Limitations

• English only.
• State the policy in the question. Unstated or implied policies are outside what the models were trained for.
• Score questions are the weak spot, most of all for the zero-shot model: on the typed-decisions benchmark its score accuracy is 0.566 (v1 zero-shot 0.621). Treat an ordered score as a rough band and check the probability spread.
• "None of these" is only partly learned. When the right option is missing and a "none" option is offered, the models pick it about half the time (0.51 zero-shot, 0.55 fitted). They rarely pick it wrongly.
• Probabilities are not temperature-calibrated. They are the raw softmax. ECE is about 0.12–0.13 on holistic judgement and higher on some typed question kinds (up to 0.17). Temperatures fitted on our calibration set did not carry over to other workflows, so none ship. If you need calibrated probabilities, fit a temperature per question kind on a few hundred of your own labelled examples.
• Free-text reasoning is weak. The models do poorly when the answer needs inference from loosely worded text, hedged evidence or deliberate distractors.
• Labels come from open teacher models. They are not human annotations, so the teachers' systematic biases can carry over.
• Not for consequential decisions without human review. The models are not for decisions about people's access to credit, housing, employment, healthcare or legal outcomes unless a person reviews each decision.

Training data and lineage

• Base: Ettin encoder, 400M parameters (Johns Hopkins University, MIT license).
• How the models were built: v2 was trained from the Ettin base in successive rounds (the v6 rounds, then a v7 round on public task data and code-built families), then a final annealing stage:
  • final stage: about 585k questions of soft-labelled judgement data (our synthetic decision workflows, rubric and instruction-flip workflows), plus an 18% replay of all earlier data, at 2,048 tokens, learning rate 1e-5 decaying to 0;
  • the fitted model's final stage also includes a typed-decisions train-split slice (2.8% of the stage); the zero-shot model's final stage is identical except that this slice is left out.
• Training data: our own synthetic decision workflows, plus public datasets reformatted as typed decisions: intents, NLI and reading comprehension, emotion and sentiment, public-record triage (CFPB, HMDA, NHTSA, NVD, USDA FoodData Central, openFDA, NOAA), news topics, product search relevance, tool use, maths and commonsense reasoning, arguments, court opinions and more. NOTICE lists every source with its license. No non-commercial or share-alike data was used.
• Labels: gpt-oss-120b (OpenAI) and Gemma 4 (Google) answered each judgement question several times, with the options reordered; the label is the average (soft label). Where the header facts or the source dataset settle an answer, the label was computed or taken from the dataset instead.
• Fitted model only: it was also trained on a slice of the typed-decisions benchmark's train split, relabelled by those teachers.
• Model origin: the base, teacher and writer models are US-origin (gpt-oss-120b and Gemma 4; NVIDIA Nemotron 3 Super wrote 130 emotion training posts, and Nemotron 3.5 Super was used only as a judge to filter records). Third-party datasets keep their own provenance, listed in NOTICE.

License

Copyright (c) 2026 Timothy Epping. The weights and code in this repository are released under the Apache License 2.0. If you redistribute the models or a derivative, keep LICENSE and pass on NOTICE: it carries the Ettin base's MIT notice and the attributions our training data requires. The license grants no rights to the Valrico name.