tkm-decision-v0.1 — a Gemma-4-31B-it decision model for JevBench

A Jev-compatible decision model built on google/gemma-4-31B-it (revision 842da3794eaa0b77d5f08bae87a17459d91ff475, Apache-2.0 weights by Google): a light LoRA fine-tune (a rank-64 adapter applied unmerged on the bf16 base) read as a single-pass option-letter classifier with per-type calibration. It answers JevBench's three request types (choice, noul, score) with native probability distributions and generates no tokens.

This repository holds both the adapter and the serving code:

Path	What
adapter_config.json, adapter_model.safetensors	the LoRA adapter (PEFT, rank 64, 1.96 GB); sha256 of the safetensors 327f7bc3b229d3d7837da22c5200253015d11a82c962ad3685f3618fdcd1627c
SOUP.json	the four training runs averaged into the adapter, with their weights
jevdec/	prompt, readout, calibration and the /v1/systemone HTTP shim (jevdec/shim/)
Dockerfile	offline serving image; base weights mounted at /model, this repository at /adapter
scripts/, tests/, docs/results.md	development tooling, tests and the full results log

How a decision is made

1. Prompt. The state (strings as-is, structured states as compact JSON) is followed by the question's instructions and its options, lettered A., B., … in the order the request lists them, and the line "Answer with the letter only." It goes through the Gemma chat template with thinking off. Choice options show label: description; noul is posed as A. Yes / B. No (with the request's true/false criteria text); score levels are lettered in level order.

2. Truncation. States longer than 5,000 tokens keep their first 2/3 and last 1/3 of that budget.

3. Readout. One forward pass. For each option letter the next-token probability of X and X is summed; these are renormalised over the options. Output tokens: 0. usage.input_tokens is the exact prompt length.

4. Calibration (jevdec/shim/calib.json, fixed in the code):

   • choice: softmax(log p / 0.74)
   • noul: P(yes) = σ((log p_yes − log p_no − 1.5) / 0.38)
   • score: softmax(log p) (identity); the reported score is the expected level.

   The calibration code also supports a temperature that grows when the model puts little total probability on the option letters (a_* slopes, computed only from the model's own next-token distribution). It helped the untrained base model; for the shipped fine-tuned model the slopes are 0.

Local results (not board scores)

Scored with a local replica of the JevBench v1.6.1 scorer (jevdec/jevscore.py). The board measures 1,500 decisions, most of them on sealed items, so expect different numbers there.

Set	Model	Capability	Notes
231 public JevBench items	untrained base, same readout	73.96	5-fold out-of-fold calibration
231 public JevBench items	this model	84.92	5-fold out-of-fold calibration
231 public JevBench items	this model	86.08	calibration fitted on these items (shipped constants)
Open-Jev-v1.1 test + ood (6,000 items)	untrained base	63.74
Open-Jev-v1.1 test + ood (6,000 items)	this model	75.32	shipped constants

Serving (RTX PRO 6000, bf16, serial, after warm-up, 231 public items): p50 0.113 s, p95 0.98 s; mean 635 input tokens per decision; zero output tokens. See docs/results.md for every run, checkpoint and caveat.

Data and disclosures

• Training. Four LoRA runs (rank 16, all language-model linear layers; KL weight 1/3/1/2; 150/150/150/300 steps of 16 examples, i.e. 2,400–4,800 examples each, drawn from a pool of 12,000 Open-Jev-v1.1 train items stratified by type; B1 and B2 share a data order) were trained with a soft-target loss over the option letters, a KL term to the frozen base model's own letter distribution, and an ordinal loss on score items, with option order shuffled per example. Open-Jev-v1.1 (ZefanCai/Open-Jev-v1.1) is procedurally generated typed decisions (generated content CC0-1.0; WANLI-derived records CC-BY-4.0). The shipped adapter is the exact average of the four runs' final LoRA deltas (stacked into one rank-64 adapter, safetensors), applied unmerged: merging into bf16 weights lost accuracy to rounding. A fifth run that mixed in natural-language sources (BoolQ, WANLI, CommonsenseQA, ARC, MMLU auxiliary-train, SST-5, prompt-injections, Banking77) was trained, evaluated and dropped. An 8-gram overlap check of the Open-Jev pool against the 231 public JevBench items found no overlap. No JevBench item was used for training.
• Use of the public JevBench items. The 231 public items from the jevbench repository (datasets/public/*.jsonl) were used for development decisions: comparing the untrained model with the fine-tuned runs (the ship decision), dropping the mixed-data run, leaving the score calibration at identity, choosing the unmerged adapter over merged weights, and fitting the three calibration constants above (choice temperature on hard-tier choice items; noul temperature and offset), with 5-fold cross-validation to check stability. Open-Jev-v1.1 calibration, test and ood splits were used for development checks. No rule in the code refers to any item.
• Prompt wording was fixed before evaluation; two answer-prefix variants ("Answer: ", "The answer is ") were tried on the public hard items and rejected.
• The calibration file records the sha256 of the adapter it was fitted for; the shim refuses to start with a different adapter or without one.
• No item tables, no evaluation detection, no telemetry, no network access at run time.

Running

BASE is a local copy of google/gemma-4-31B-it at revision 842da3794eaa0b77d5f08bae87a17459d91ff475; ADAPTER is a local copy of this repository (the adapter files sit at its root).

hf download google/gemma-4-31B-it --revision 842da3794eaa0b77d5f08bae87a17459d91ff475 --local-dir gemma-4-31B-it
hf download tkm-research/tkm-decision-v0.1 --local-dir tkm-decision-v0.1      # add --revision <commit> to pin a release
BASE=$PWD/gemma-4-31B-it ADAPTER=$PWD/tkm-decision-v0.1

# docker (offline; weights mounted read-only)
docker build -t jevdec "$ADAPTER"
docker run --gpus '"device=0"' --network none -v "$BASE":/model:ro -v "$ADAPTER":/adapter:ro -p 8000:8000 jevdec
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \
  -d '{"state": "Order #1182 shipped.", "questions": {"q": {"type": "noul", "instructions": "Shipped?"}}}'

# HTTP without docker (POST /v1/systemone, GET /health), from inside $ADAPTER
python -m jevdec.shim.server --model-dir "$BASE" --adapter-dir "$ADAPTER" --port 8000

# in-process, from inside $ADAPTER
python -c "from jevdec.shim import loader; m = loader('$BASE', adapter_dir='$ADAPTER'); \
  print(m.predict('Order #1182 shipped.', {'decision': {'type': 'noul', 'instructions': 'Shipped?'}}))"

Requirements: one GPU with ≥ 80 GB (bf16 base ≈ 62 GB + adapter ≈ 2 GB). The Docker image installs transformers==5.19.0, accelerate==1.15.0 and peft==0.21.2 on a digest-pinned PyTorch 2.14.1 / CUDA 13.0 image; trust_remote_code is not used. Questions with more than 26 options get HTTP 422.

Repository contents and licences

• Serving code: jevdec/prompt.py, jevdec/readout.py, jevdec/postprocess.py, jevdec/shim/ — the only modules in the Docker image. Everything else is development tooling and is never imported at serve time: the v1.6.1 scorer replica (jevdec/jevscore.py), item loaders (jevdec/items.py), calibration fitting (jevdec/calib.py), training (jevdec/train/), and scripts/.
• The evaluation sets are not included. scripts/build_devsets.py rebuilds them from a clone of https://github.com/fstandhartinger/jevbench (MIT License, Copyright (c) 2026 Florian Standhartinger and contributors) and from ZefanCai/Open-Jev-v1.1 on the Hugging Face Hub.
• tests/fixtures/v161_rows.json holds four published aggregate rows of the JevBench v1.6.1 results.
• Code and adapter: Apache-2.0 (LICENSE). Base model: google/gemma-4-31B-it, Apache-2.0. Training data: Open-Jev-v1.1 by Zefan Cai — generated content CC0-1.0; records derived from WANLI (Liu et al., 2022, "WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation") CC-BY-4.0.

Development

jevdec/jevscore.py is a local replica of the JevBench v1.6.1 scorer (its aggregation reproduces published rows exactly; see tests/test_jevscore.py). Scripts in scripts/ build dev sets, cache readouts, fit/evaluate calibration and measure latency. Tests: python -m pytest -q.