eppingt/Valrico-400M

Valrico 400M: typed decisions with stated policies

Superseded: this model is replaced by eppingt/Valrico-400M-v2 (zero-shot, released 2026-10-25), which is stronger on daily use (held-out panel accuracy 0.643 → 0.706) and on the Decision Index 0.3 (15.53 → 19.90). This version stays available for reproducibility.

By Timothy Epping.

Valrico models answer a decision question about a record, under a policy that the question states. They return a calibrated probability for each option. They are small text encoders, 400M parameters, built on Ettin, and run on a single modest GPU.

Model	Use it when	Typed-decisions accuracy
Valrico-400M (zero-shot)	You have no labelled examples of your decision.	0.662
Valrico-400M-Fitted	Your decisions resemble the typed-decisions benchmark's workflows.	0.749
Zero-shot + your own head (fit_head.py)	You have labelled examples of your own decision (this figure: about 7,000).	0.718

What a decision looks like

A request is one record (a ticket, a claim, a log excerpt, a chat) and one or more questions about it. Each question:

• starts with the policy that decides it ("Policy: refuse a claim sent in after the time limit ...");
• has a fixed set of options, of one of three kinds:
  • a choice among labels;
  • yes / no;
  • an ordered score.

The model scores every option and returns a probability distribution over the options. Facts that can be counted or computed belong in the record's header as computed fields (for example "Days beyond the time limit: 12"). The models are trained to rely on them; they are not trained to do the arithmetic themselves.

How to use

The repository ships valrico.py, a small optional helper that needs only torch and transformers. Download it from the repository's file list (or clone the repository), then:

from valrico import Valrico

m = Valrico.load("eppingt/Valrico-400M")       # or "eppingt/Valrico-400M-Fitted", or a local directory

record = """Ticket: MEA-75460
Customers affected: 85
Account tier: free
Hours since received: 86

Hey, my meal kit box never arrived this week and I was still charged."""

m.decide(record,
         "Policy: P1 when 20 or more customers are affected; P2 when two or more are affected or the message "
         "has waited over 24 hours; P3 when one sender is blocked with no workaround; P4 otherwise. "
         "Which SLA level applies?",
         ["P1", "P2", "P3", "P4"],
         record_type="Service desk ticket")
# -> {"P1": ..., "P2": ..., "P3": ..., "P4": ...}, probabilities summing to 1

m.decide(record, "Policy: refund when the customer was charged for a box that never arrived. Refund?",
         ["false", "true"], record_type="Service desk ticket")

Plain transformers, no extra file

The model is a standard ModernBertForTokenClassification checkpoint with one output. Each option's score is the logit at its [MASK] token. Write the input as one string, with no spaces around [SEP] and [MASK]:

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("eppingt/Valrico-400M")
model = AutoModelForTokenClassification.from_pretrained("eppingt/Valrico-400M").eval()

def decide(record, question, options, record_type=None):
    q = f"{record_type}: {question}" if record_type else question
    text = q + "[SEP]" + "".join("[MASK]" + o for o in options) + "[SEP]" + record
    enc = tok(text, return_tensors="pt", truncation=True, max_length=2048)
    with torch.no_grad():
        scores = model(**enc).logits[0, :, 0]
    masks = enc.input_ids[0] == tok.mask_token_id
    return dict(zip(options, torch.softmax(scores[masks], -1).tolist()))

This gives the same tokens as valrico.py. Keep the question and options well under 2,048 tokens: valrico.py always keeps them whole and cuts only the record, while plain truncation cuts from the end of the string.

Notes

• Yes/no questions use the options ["false", "true"], in that order.
• record_type is optional: a short name for the kind of record ("Service desk ticket", "Payment review record"). The models were trained with one.
• Many questions at once: m.decide_many([{"record": ..., "question": ..., "options": [...]}, ...]) batches them and returns one probability list per question.
• On a GPU the model runs in bfloat16; on a CPU, in float32.

Results

All numbers are generated from results.json. Accuracy is top-1. ECE is expected calibration error (15 bins, by mass): lower is better.

Typed-decisions benchmark (public test split)

	Zero-shot	Fitted
Overall	0.662	0.749
Choice: accuracy / ECE	0.568 / 0.128	0.717 / 0.091
Yes / no: accuracy / ECE	0.812 / 0.086	0.822 / 0.083
Score: accuracy / ECE	0.621 / 0.100	0.719 / 0.081

The fitted model was trained on the benchmark's train split; the zero-shot model never saw it. Neither model saw the test split.

Our own evaluation sets (held-out workflows, written by us)

Every set uses workflows, wording and questions that never appear in training.

Set	Zero-shot: accuracy / ECE	Fitted: accuracy / ECE
Tool routing	0.788 / 0.063	0.785 / 0.068
Inbox triage	0.825 / 0.061	0.820 / 0.060
Retrieval relevance	0.788 / 0.036	0.794 / 0.032
Policy compliance	0.850 / 0.037	0.843 / 0.030
Moderation and assistant scope	0.749 / 0.049	0.736 / 0.030
Judging model outputs	0.662 / 0.095	0.659 / 0.085
Security alert triage	0.774 / 0.040	0.767 / 0.041
Code decisions	0.747 / 0.090	0.742 / 0.087
Synthetic workflows	0.856 / 0.031	0.853 / 0.024
Real-record workflows	0.673 / 0.093	0.668 / 0.098
Instruction flips	0.786 / 0.069	0.794 / 0.058

The instruction-flip set pairs near-identical questions whose stated policy reverses the answer. It checks that the model follows the policy rather than surface cues.

Plain zero-shot classification (public sets, 100 examples each)

Set	Zero-shot: accuracy / macro-F1	Fitted: accuracy / macro-F1
AG News	0.880 / 0.875	0.890 / 0.886
Banking77	0.780 / 0.754	0.770 / 0.740
Emotion	0.550 / 0.517	0.520 / 0.482

Following computed facts: the escalation check

Before release, each model must follow header facts on an expense-escalation set. On the set's computed-facts variant, false approvals and false refusals must each stay at or below 0.200.

	Zero-shot	Fitted
Manager review: false approvals / false refusals	0.054 / 0.012	0.049 / 0.028
Reimburse as submitted: false approvals / false refusals	0.195 / 0.073	0.201 / 0.074
Result	pass	fail (within measurement noise of the limit; accepted)

Speed

Measured on one NVIDIA L4 (an entry-level inference GPU) in bf16 with PyTorch's standard attention, no further optimisation. The test used real records from our evaluation sets: a median presentation is 361 tokens. All the questions about one record go in one batch.

Questions per record	Median	95th percentile	99th percentile
1	12 ms	16 ms	18 ms
2	21 ms	35 ms	42 ms
5	50 ms	76 ms	97 ms
10	106 ms	172 ms	217 ms
20	247 ms	439 ms	600 ms
50	684 ms	1058 ms	1486 ms

Each question is scored as its own sequence, record included, so time grows roughly linearly with the number of questions per record; throughput is about 32 questions per second. The tail comes from records with long questions. The L4 is power-capped at 72 W and runs below its peak clock under sustained load; these figures include that.

Sensitivity to option order

We reshuffled the options on 4113 choice questions from our evaluation sets two ways: rotated by one, and reversed. The top answer changed on 0.054 of questions for the zero-shot model and 0.061 for the fitted model. Averaging the three orders adds little: zero-shot accuracy went from 0.752 to 0.754. One order per question is enough.

Fitting your own head

fit_head.py runs the zero-shot model once over your labelled questions. It then trains only a small scoring head; the encoder is unchanged. On the typed-decisions benchmark, a head fitted on its train split (about 7,000 labelled questions) scores 0.718. That recovers about two thirds of the full fine-tune's gain, at a small fraction of the cost. Use it when your decisions differ from the benchmark's.

Choosing a cutoff

For yes/no questions the models rank cases well, but the default cutoff of 0.5 is not always where you want it. On some questions in our tests the model acted too often at 0.5; a stricter cutoff halved the false positives for a small rise in misses. If false positives and misses cost you different amounts, label a few hundred examples of each yes/no question and pick the cutoff per question that meets your target false-positive or miss rate. The encoder and head stay as they are; only the cutoff changes.

Sending uncertain questions to a larger model

Each answer's margin, the gap between the top two probabilities, is a good signal of when the model is unsure. In our tests, sending the least certain 20% of questions to a large open model recovered most of the gap to that model. A learned router did no better than the margin alone.

Limitations

• English only.
• State the policy in the question. Unstated or implied policies are outside what the models were trained for.
• Free-text reasoning is weak. The models do poorly when the answer needs inference from loosely worded text, hedged evidence or deliberate distractors. They also do poorly on "none of the above" policies.
• Labels come from open teacher models. They are not human annotations, so the teachers' systematic biases can carry over (see Training data and lineage).
• Not for consequential decisions without human review. The models are not for decisions about people's access to credit, housing, employment, healthcare or legal outcomes unless a person reviews each decision.

Training data and lineage

• Base: Ettin encoder, 400M parameters (Johns Hopkins University, MIT license).

• How the models were built: each released model was trained in successive rounds from that base:

  • Valrico-400M took five rounds.
  • Valrico-400M-Fitted shares the first four rounds, then had two rounds of its own.

• Training data: our own synthetic decision workflows, plus public datasets reformatted as typed decisions.

  • Intents: BANKING77, ABCD.
  • Inference and reading: MultiNLI non-fiction, Cosmos QA.
  • Emotion: GoEmotions, BRIGHTER.
  • Public-record triage: CFPB complaints, HMDA, NHTSA, NVD, USDA FoodData Central.
  • News topics: Wikinews.

  NOTICE lists every source with its license. No non-commercial or share-alike data was used.

• Labels: the synthetic workflows were written and labelled by open-weight US models. gpt-oss-120b (OpenAI) and Gemma 4 (Google) wrote the records and answered each question several times, with the options reordered; the label is the average. Where the header facts settle an answer, the label was computed in code instead.

• Fitted model only: it was also trained on the typed-decisions benchmark's train split, relabelled by those teachers.

• Model origin: the base, teacher and writer models are all US-origin, with one exception: the benchmark's train-split records (fitted model only) were written by a model its publisher does not name.

License

Copyright (c) 2026 Timothy Epping. The weights and code in this repository are released under the Apache License 2.0. If you redistribute the models or a derivative, keep LICENSE and pass on NOTICE: it carries the Ettin base's MIT notice and the attributions our training data requires. The license grants no rights to the Valrico name.