ballot-jev-0.5b

A small typed-decision model whose answer does not change when you reorder the options.

Decision models take some context and return a structured answer from a fixed set of candidates — route this ticket, is this claim supported, rate this response 1–5. Almost all of them share one quiet bug: feed the same options in a different order and the answer changes. Measured here on real weights, kev flips on 28.1% of reversed-order decisions, laya on 25.0%.

ballot-jev flips on 0 of 32. Not because it was trained not to — because it structurally cannot. The state and question are encoded once into a KV cache, and every option is then decoded from its own private clone of that cache. No option is ever an input to another option's computation, so permuting the option list permutes the output distribution and does nothing else. Verified on the trained weights: max |p(forward) − p(reversed)| = 3e-8, which is float32 rounding noise.

---

At a glance

Parameters	506M total — Qwen2.5-0.5B base (494M) + LoRA (8.8M) + decision head (3.3M)
Trainable	12.1M (2.4%) — the base is frozen, LoRA rank 16
Checkpoint size	2.02 GB (fp32 safetensors). ~1.0 GB if you cast to fp16
Runs on	CPU — that is the intended target, not a fallback. CUDA and MPS also work
Speed (CPU)	0.19 s at 2 options, 0.30 s at 4, 1.44 s at 20 — see below
RAM	~3 GB resident for inference in fp32
Precision	fp32 as shipped
License	Apache-2.0 (base model Apache-2.0)

It is a 0.5B-class model: small enough to sit next to your application on an ordinary CPU box, with no GPU, no server, and no per-call API bill.

Latency scales with the number of options

Each option is decoded from its own branch off the shared cache, so cost grows roughly linearly with K — about +65 ms per option on an Apple Silicon CPU at 4 threads, single request, measured with realistic option text:

question	options	p50
yes/no policy check	2	0.19 s
ticket intent	4	0.30 s
star rating	5	0.36 s
client routing	20	1.44 s

Plan for the K you actually use. A wide roster (20+ candidates) is roughly a second per decision on CPU, not a fifth of one — budget for it, or move that workload to a GPU.

---

Install

pip install torch transformers peft safetensors huggingface_hub

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("vigneshlabs/ballot-jev-0.5b")   # ~2 GB
sys.path.insert(0, path)

from inference import BallotJev
jev = BallotJev.from_pretrained(path, threads=4)     # CPU by default

The Qwen2.5-0.5B base is fetched from the Hub automatically the first time (another ~1 GB). To pin your own copy: BallotJev.from_pretrained(path, base_model="/your/Qwen2.5-0.5B"). For GPU: device="cuda".

---

The three question types

# 1. choice — one of K
jev.choice(
    state="Customer: 'Where is my package? Ordered last week, still nothing.'",
    question="Which intent does this express?",
    options=["track order", "cancel order", "change address", "billing question"],
)
# {'answer': 'track order', 'index': 0, 'confidence': 0.556,
#  'probabilities': {'track order': 0.556, 'cancel order': 0.197,
#                    'change address': 0.040, 'billing question': 0.207}}

# 2. noul — true/false under evidence
jev.noul(
    state="Policy: refunds need a receipt and purchase within 30 days. "
          "Customer bought 12 days ago but has no receipt.",
    proposition="Under the stated policy, is a refund permitted?",
)
# {'answer': False, 'probability_true': 0.352, 'confidence': 0.648}

# 3. score — ordinal rating
jev.score(
    state="Shipping took nine days, but the product itself is flawless.",
    question="How many stars does this review give?",
    levels=["1 star", "2 stars", "3 stars", "4 stars", "5 stars"],
)
# {'answer': '5 stars', 'index': 4, 'expected_level': 2.437, 'confidence': 0.291,
#  'probabilities': {'1 star': 0.099, '2 stars': 0.177, '3 stars': 0.202,
#                    '4 stars': 0.230, '5 stars': 0.291}}

That last one is worth reading closely, because it is this model being honest rather than impressive: the argmax is 5 stars at only 0.291, but the distribution is spread across 3–5 and the expected level is 2.44. A mixed review ("nine days, but flawless") genuinely is ambiguous, and the model reports that instead of faking certainty. For ordinal questions, expected_level is usually the number you want.

Every call returns the full distribution, not just a label. That is the point — see Calibration below.

The invariance, if you want to check it yourself

opts = ["track order", "cancel order", "change address", "billing question"]
a = jev.choice(state, question, opts)["probabilities"]
b = jev.choice(state, question, list(reversed(opts)))["probabilities"]
print(max(abs(a[k] - b[k]) for k in a))   # 2.98e-08  (float32 noise)

Shuffle, reverse, rotate — the number stays at float-noise. No other open model in this class makes that claim, and several measurably fail it.

---

Using it for real work

Where it fits. A cheap, local, well-calibrated first pass on high-volume typed decisions, in front of something bigger:

• Ticket and email routing — classify intent, hand the low-confidence tail to an LLM.
• Content moderation triage — return P(violates policy) and threshold it, rather than forcing a yes/no on a genuinely ambiguous post.
• Retrieval / rerank gating — is this passage actually responsive?
• Claim checking against evidence — supports / refutes / not enough info.
• Form and field extraction — pick one of K known values for a slot.

Not this, yet: grading or severity scoring. Measured, not guessed — asked to rate a helpful answer and a deliberately useless one, it returned "5 - excellent" for both; asked to rate six very different news stories for newsworthiness, it returned 1.95–2.01 out of 4 for all of them. The ordinal head does not currently separate quality. Use choice and noul; treat score as experimental.

The pattern that makes it pay for itself — route on confidence, not on the label:

r = jev.choice(state=ticket, question="Which team handles this?", options=TEAMS)

if r["confidence"] >= 0.75:
    assign(r["answer"])            # ~0.3s at K=4, no API call
else:
    escalate_to_llm(ticket)        # only the genuinely hard ones

This works because the probabilities are trustworthy — hard-tier ECE is 0.054, meaning when it says 70% it is right about 70% of the time. A miscalibrated model makes this pattern actively dangerous.

Where order-invariance stops being academic: any time the candidate list is built dynamically — teams pulled from a database, retrieved documents, tools available to an agent, A/B'd label sets. With an order-sensitive model, an unrelated change to list ordering silently changes production decisions and the diff looks like nothing happened.

Do not use it for multi-hop reasoning over long policies, arithmetic or temporal deduction, anything safety-critical without a human, or as a general chat model. It has no generative head — it scores candidates, it does not write.

---

Evaluation

All numbers below were produced on this machine against real downloaded weights and each project's own inference code — never a reimplementation, never a figure copied from someone's README.

Head-to-head — identical held-out set

140 records, stratified 10 from each of 14 sources. Flip rate is measured the way kev's own evaluation defines it: reverse the option list, check whether the same option text still wins.

model	accuracy	flip rate	params
ballot-jev-0.5b	64.4%	0.0% (0/32)	0.5B
laya	50.4%	25.0% (8/32)	0.4B
kev-0.5b	50.0%	28.1% (9/32)	0.5B
openJev-verdict-2.0	25.4%	9.1% (1/11)	0.15B

Read the accuracy column with care. ballot-jev trained on 11 of these 14 sources and kev on 6, so the aggregate partly reflects training overlap rather than pure capability. On kev's own sources kev often wins (banking77 0.80 vs 0.50, mnli 0.90 vs 0.50). Most of ballot-jev's margin comes from ordinal/score tasks kev handles poorly (summeval 0.73 vs 0.23, helpsteer2 0.63 vs 0.13). The per-source table is in results/final_comparison_results.json — read it before quoting the headline.

The flip-rate column carries no such caveat. It is architectural, it is not a sampling artifact, and it held after training.

JevBench (public subset)

Run through JevBench's own harness and scoring code on its 231 MIT-licensed public tasks, with CPU-class peers quoted from their published v1.2 results:

axis	ballot-jev	laya	openJev-verdict-2.0
Calibration	89.2	62.5	51.3
Speed	82.9	71.1	76.7
Cost	86.2	86.2	83.1

Calibration is the standout: hard-tier ECE 0.054, the best of any CPU-class system on that board. Soft-label training plus a Brier term in the loss is doing exactly what it was added to do.

What is deliberately not claimed here. JevBench's fourth axis is Intelligence. It was measured, and it is not being claimed at this checkpoint — this is a cross-entropy-only model after a single epoch, with the RLCD stage still to come, and it is weakest on the multi-hop and long-policy reasoning that axis is built around. It would be dishonest to show three favourable axes and let you assume the fourth is comparable. It is not. The raw figures are in results/jevbench_public_subset.json if you want them; they will be claimed when there is a trained model behind them.

Three further limits on any JevBench number here: only 231 of 534 tasks are public, the entire judge tier is held out (146 items), and speed was measured on Apple Silicon rather than the project's Ryzen 5 3600 reference box. This is a public-subset measurement, not an official leaderboard score.

Reproducing all of it

git clone https://github.com/fstandhartinger/jevbench       # public tasks, MIT
python -m jevbench.cli run \
  --tasks datasets/public/easy.jsonl,datasets/public/original.jsonl,datasets/public/hard.jsonl \
  --adapter ballot_local --endpoint <path-to-this-checkpoint> \
  --results out.jsonl

To check the invariance claim against any model you like, the method is four lines: score the options, score them reversed, map each prediction back to its option text, and count how often the text changes. That is the whole test, and it is what produced every flip-rate number in the table above.

---

How it was built

Architecture. Qwen2.5-0.5B with LoRA (rank 16, α 32, on q/k/v/o_proj + gate/up/down_proj), plus a listwise set-attention head over the option branches. state + question is encoded once with use_cache=True; each option then runs a short forward pass over a .clone() of that cache. The clone is load-bearing in two directions — it keeps options isolated from each other, and because it is a clone rather than a detached copy, gradients from every branch still flow back into the shared prefix.

Training. One epoch, cross-entropy stage, on CPU. Loss is generalized CE against the full gold probability distribution (not the argmax) plus a Brier term — which is where the calibration comes from. Gradients are normalized per-question rather than per-record, so a record carrying five questions does not give each of them a fifth of the weight.

Data. 3,000 training records over 11 public sources: agnews, mnli, yelp, emotion, clinc_oos, civil_comments, typed-decisions, vitaminc, Aegis, HelpSteer2, summeval. banking77, sst5, boolq and pubmedqa were excluded from training as benchmark-contamination risks.

Final validation accuracy: 65.14% (full 750-record held-out set).

---

Limitations

• One epoch, CE only. The RLCD stage is not done. This is an early checkpoint.
• Weak at reasoning. Shallow classification is strong (JevBench easy tier 93.8%); multi-hop (16.7%) and long-policy (26.3%) are poor. On hard-tier binary questions it sits at chance. Do not point it at problems that need inference chains.
• fp32 only, 2.02 GB. No quantized build yet.
• English only.
• Scores candidates, does not generate. You must supply the option set.
• Accuracy comparisons are shaped by training-mix overlap — see the per-source table.

Citation

@software{ballot_jev_0_5b_2026,
  title  = {ballot-jev-0.5b: order-invariant typed decisions},
  author = {Varadharajan, Vignesh},
  year   = {2026},
  url    = {https://huggingface.co/vigneshlabs/ballot-jev-0.5b}
}

Built on Qwen2.5-0.5B (Apache-2.0). Comparisons against kev, laya, and openJev-verdict-2.0 use those projects' own weights and inference code.