jbrashear/jebadiah-4b-v1

Jebadiah 4B v1

Jebadiah (Jeb for short) is an open System One style decision model: it answers typed questions with a calibrated probability over the option labels instead of generating text. Three question types, the same three Jev uses: choice (pick one of N), noul (a yes or no statement, returned as P(yes)) and score (place the state on an ordered rubric). It is trained with AINode's trainer on public data only and served by AINode's /v1/decide and /v1/systemone routes on any NVIDIA GPU. The routes answer Jev's three question types; the request schema is AINode's own (documented on the route), not Jev's exact wire format.

This repository holds the v1 4B adapter: a LoRA on Qwen/Qwen3.5-4B-Base (pinned revision 1001bb4d), one epoch over 14,900 public training questions, trained on one H100 PCIe in 99 minutes. v1 changes two things against v0: score questions train towards an ordinal target around the human label with the temperature fitted on that target, and the pool carries two licensed human-rubric sources (HelpSteer2 train, SummEval). v1 is a fixed version; newer versions are published as new repositories and never replace this one.

Made in Texas.

Results

Measured by us with AINode's bench, one logit read per question, the same rendered prompt for every model. Accuracy is the share of questions whose top label is the human label. Decision Score is Jevals' metric: 100 = perfect, 0 = guessing the label base rates, below 0 = worse than that. These are our numbers on the public suites, not rows on the Jevals board. Headline (macro over the zero-shot public sets): 70.3 (v0: 70.9).

Set (questions)	Type	Accuracy	Floor (majority label)	Decision Score (Jevals)	ECE, temperatures applied	Flips over identical repeats
Jevals PubMedQA (300)	noul	86.7	62.0	59.0	0.039	0.3%
Jevals Banking77 (300, 77 options)	choice	68.3	1.3	55.6	0.063	0.0%
Jevals HelpSteer2 helpfulness (300, 5 levels)	score	37.0	41.7	9.2	0.044 (raw 0.073)	1.0%
Nimble held-out eval (324)	mixed	71.9	17.6	57.5	0.058	0.0%
Kev transfer-v4 test (764)	mixed	82.2	21.5	64.7	0.041	0.3%
Kev decision-v7 test (1,440)	mixed	78.8	20.3	67.9	0.042	n/a
typed-decisions test (2,000)	mixed	79.8	15.3	56.6	0.170	0.1%

Nimble's 13 public human-labelled subsets (3,880 questions), macro accuracy: 75.9 (per subset: aegis2 80.0, boolq 85.7, civil_comments 84.0, helpsteer2 38.2, massive-de-DE 83.7, massive-en-US 83.7, multinli 86.0, paws 82.8, pubmedqa 70.8, squad2 79.6, summeval-consistency 86.1, summeval-relevance 51.2, vitaminc-dev 75.3). For scale, Bespoke's published table puts Nimble-9B at 74.8 and Jev at 76.0 on the same subsets with their scorer.

Two things to read carefully. HelpSteer2 and SummEval are no longer zero-shot for Jeb: v1 trained on the HelpSteer2 train split (the Jevals and Nimble items come from validation, and no item overlaps) and on SummEval articles that Nimble does not score (its 15 evaluation articles are held out entirely), so those rows are held-out items of a seen rubric. And the in-distribution score set (typed-decisions) gives back a few points of Decision Score against v0, the cost of honest calibration on human rubrics.

One honest regression on the 4B: the Nimble 324 held-out set fell from 77.5 (v0) to 71.9 with the rubric data in the pool, while the 9B gained on it. The smaller model does not absorb 45 percent score questions without its choice and yes/no judgments slipping; v0 stays the better general 4B, v1 the better calibrated one. A lower rubric share or a higher rank is the next probe.

Every eval record (per question: option keys, probabilities, pick, label, repeat, option order) is in eval/, with eval/results.json carrying the full metric set.

Robustness to irrelevant content (nonce test)

Every question on the four head sets was scored once clean and three more times with a fresh UUID planted in the state (state) or appended to the instructions (instr). Records are in eval/nonce/.

Set	Pick agreement (state / instr)	Questions with any flip (state / instr)	Mean change of p_max, points (state / instr)
Jevals PubMedQA	98.2% / 98.7%	3.0% / 2.3%	1.1 / 1.0
Jevals Banking77	96.4% / 98.2%	4.3% / 3.0%	2.4 / 2.1
Jevals HelpSteer2	92.1% / 95.4%	10.7% / 7.3%	1.1 / 0.6
Nimble 324	100.0% / 99.0%	0.0% / 1.5%	1.0 / 1.1

Choice and yes/no answers are close to immune. On score questions v1 flips its top level more often than v0 while moving its probabilities far less: a calibrated rubric model sits honestly between adjacent levels, so a small nudge tips the argmax. Read the expected level and the probabilities (which is what the route returns as score), not the top pick, for rubric questions.

How it decides

The prompt is AINode's own decide rendering (ainode.api.decide.build_messages, source commit e5c08938, prompt_source_sha256 in prompt_contract.json), through the base chat template with thinking off. The option labels are single tokens; the model's answer is the distribution over those label tokens at the last prompt position, read in fp32 from the last hidden state and then temperature scaled per type (temperatures.json: choice 1.20, noul 1.33, score 1.16). Nothing is generated.

The supported way to run it is AINode, which renders the prompt exactly as trained (a worked /v1/systemone request and response is in the AINode docs issue tracker until the page lands):

curl -sS https://<your-ainode>/v1/systemone -H "Authorization: Bearer <key>" -H "Content-Type: application/json" -d '{
  "model": "jebadiah/jebadiah-4b-v1",
  "state": {"ticket": "Customer says the invoice total does not match the quote."},
  "questions": {
    "route": {"type": "choice", "instructions": "Which team should take this ticket?", "criteria": {"billing": "an invoice, a charge or a refund", "support": "a product question", "sales": "a quote or a renewal"}},
    "urgent": {"type": "noul", "instructions": "The customer is blocked from working.", "criteria": {"true": "work has stopped", "false": "it can wait"}}
  }
}'

Standalone with transformers and peft, load the base at the pinned revision, then the adapter from this repository, render the prompt per the contract, and read the label-token logits at the last position.

Thresholds belong to the caller: act on a high probability, confirm or escalate on a middle one, hand a low one to a person or a bigger model. The model never refuses; policy is built from decisions.

Training

• Objective: cross-entropy over the candidate option-label logits at the answer position. Choice and noul use the source's gold distribution where it provides one (typed-decisions) and the hard label otherwise; score questions use an ordinal kernel around the human label (adjacent level 0.2 of the label's weight), and the score temperature is fitted on that same target.
• Adapter: LoRA r=16, alpha=32, dropout 0.05 on every linear projection including the Gated DeltaNet projections, 32.5M trainable parameters. bf16, SDPA attention, micro-batch 8, learning rate 1e-4, one epoch, 1,862 steps, gradient checkpointing.
• Data (public only), 15,813 questions in the pool: LocalLLaMA/typed-decisions train (Apache-2.0, revision ea930645); the Kev v7 training sources whose licenses permit derived weights (BoolQ, MNLI, DBpedia14 and Kev's contrastive and composition sets, Kev suites revision a88f56db); nvidia/HelpSteer2 train split (CC BY 4.0, 2,573 score questions: helpfulness in the Jevals task's exact shape for 1,609 rows, plus correctness, coherence, complexity and verbosity on 241 of them); SummEval (MIT, via mteb/summeval, 1,664 score questions over four dimensions on 26 articles, with Nimble's 15 evaluation articles excluded). 95/5 family split into 14,892 training and 689 calibration questions. Nimble's train set is excluded (no license stated); the Jevals test items are excluded from training. No private data of any kind.
• Compute: one NVIDIA H100 PCIe 80 GB, 99 min training, 24 min evaluation, torch 2.11, PEFT 0.21.

Limitations

• Single-hop judgments only. Split a chain of inference into hops.
• AINode caps a choice question at 20 options; Banking77's 77 options were scored with an extended single-token alphabet for the benchmark only.
• English data, plus Massive's German subset scoring well by accident of the base model.
• Calibration was fitted on the training distribution. Refit the temperatures on your own data before trusting a threshold, and remember the served route applies none yet.

Versioning and license

v1 is frozen. Later versions land as jebadiah-<size>-v<N> repositories. The adapter is Apache-2.0, the base model is Apache-2.0. Evaluation data: Jevals suite 0.1.0 (CC-BY-4.0, "Jevals (jevals.com), release 2026-09-18"), Nimble public subsets (Bespoke Labs), Kev test sets and typed-decisions test, each under its own license.