hiteshluke/arbiter-v4-12b

Arbiter v4 · 12B

A typed-decision model by Codekins · Zyot Lab. Arbiter answers the way software needs it — yes/no, multiple-choice, and ordinal ratings — from a single calibrated forward pass, not free text. It is Gemma-4 12B IT + LoRA (r 32, α 64) with a fixed 28-slot decision head read at the last token.

One model, three decision types. No sampling, no parsing — a probability over the valid options, every time.

---

Benchmarks

Benchmark	Score	Notes
BoolQ (validation, 3,270)	89.2%	Near-human (~90%) boolean reasoning; majority-class baseline is 62.2%
Ollaya typed-decisions	0.640	Head-scored on ollaya.dev; the metric reads the head's probabilities directly (no text generation)
↳ Yes / No (noul)	0.803	True/false + "unanswerable" detection — Arbiter's strength
↳ Multiple-choice	0.663	pick-one decisions
↳ Ordinal score	0.501	fine-grained ratings — an industry-wide ceiling (gold labels have ~0.61 annotator agreement)

Why this is strong. BoolQ at 89.2% puts a compact classifier head within a point of human performance and of much larger frontier models — confirming the boolean/answerable judgment that also shows as 0.803 on Ollaya's noul type. The ordinal score type is weak for every model on the board because its gold labels are noisy by construction; Arbiter leads where the signal is clean.

What makes it different

• Explicit typed decision head. Most models read a decision off next-token logits; Arbiter has a trained 28-slot head (T/F · A…P · 0…9) over the final hidden state — boolean, choice, and ordinal in one place.
• Calibrated & deterministic. A fitted temperature; one forward pass; the same answer every time. No sampling, no "please output JSON", no parse failures.
• Compact to deploy. The decision is a classifier, not a generation loop.

How it answers

Slots	Tokens	Used for
0, 1	T, F	yes/no — option logits [F, T]
2…17	A…P	choice — up to 16 options
18…27	0…9	score — up to 10 ordinal levels

The prompt is State: … / Question: … / Options: … / Answer:; the head is read at the last position. One causal row answers one question.

Files

• adapter_model.safetensors — the LoRA adapter (on q/k/v/o and gate/up/down), F32
• head.pt — the 28-slot decision head (nn.Linear(3840, 28, bias=False), BF16)
• head_meta.json — verbalizer + calibration temperature

Load the base google/gemma-4-12b-it, attach the adapter, and read the head over the last-token hidden state.

On Ollaya

Arbiter's 4B predecessor is live on ollaya.dev as the arbiter family; the 12B is being added as arbiter12b (a 28-slot head). Ollaya scores decision models by reading the head directly, so the leaderboard number reflects the real classifier — not a generated string.

Citation

Arbiter v4 (12B) · Codekins Pvt Ltd · Zyot Lab. Base model: Gemma-4 12B IT (© Google, Gemma license).