frontier-infra/jebadiah-9b-v2-GGUF

Jebadiah 9B v2 GGUF

GGUF builds of Jebadiah 9B v2 for llama.cpp, which runs on NVIDIA, AMD and Apple GPUs and on plain CPUs. Jebadiah answers a typed question (choice, noul or score) with a probability for every option, read from one forward pass. Nothing is generated. Code, trainer and evals: getainode/jebadiah.

Files

Every file was checked on the 260 held-out questions the merged weights were checked on, and compared with the merged bf16 weights and with the training run's own eval records.

File	Size	Same answer as bf16	Same as the run	choice + noul	score	Prob. diff median / max
jebadiah-9b-v2-Q8_0.gguf	9.8 GB	257 / 260	256 / 260	171 / 173	85 / 87	0.003 / 0.051
jebadiah-9b-v2-Q5_K_M.gguf	6.6 GB	250 / 260	251 / 260	169 / 173	82 / 87	0.012 / 0.252
jebadiah-9b-v2-Q4_K_M.gguf	5.8 GB	240 / 260	239 / 260	164 / 173	75 / 87	0.022 / 0.415
bf16 weights			257 / 260	172 / 173	85 / 87	0.002 / 0.029

Which one: Q8_0 if it fits (it changed 3 answers here); Q4_K_M when memory is short. A file needs about its own size in GPU or unified memory, plus about 1 GB for a 4k context.

Q5_K_M changes 10 of 260 answers against bf16 (7 on score questions) and moves probabilities more (median 0.012, max 0.25). Use it only when a larger build does not fit. Q4_K_M changes 20 of 260 answers against bf16 (10 on score questions) and moves probabilities more (median 0.022, max 0.41). Use it only when a larger build does not fit.

Run it

The answer is the log probability of each option label ("A", "B", ...) at the answer position, which llama-server's /completion returns. The script renders the prompt exactly as AINode does, sends the raw text (so the server's own chat template is never used), renormalises over the labels and applies temperatures.json (choice 1.1863, noul 1.0903, score 1.2162). You need a llama.cpp that knows the qwen35 architecture: we checked v0.5.0 (older builds refuse the file).

hf download frontier-infra/jebadiah-9b-v2-GGUF --include "*Q8_0.gguf" "scripts/*" "*.json" "*.jinja" "*.txt" --local-dir jebadiah-9b-v2-GGUF
cd jebadiah-9b-v2-GGUF
llama-server -m jebadiah-9b-v2-Q8_0.gguf -c 4096 -np 1 --port 8080
pip install transformers        # the tokenizer only, no torch
python scripts/decide_gguf.py --server http://127.0.0.1:8080 --request scripts/example-request.json

--no-temperatures returns the raw probabilities. We checked llama-server only. LM Studio or Ollama will load the file, but a decision needs the log probability of every option label at one position; if your runtime cannot return those, use llama-server.

On example-request.json (jebadiah-9b-v2-Q8_0.gguf):

{
 "route": {"type": "choice", "choice": "billing", "confidence": 0.461713, "probabilities": {"billing": 0.641142, "support": 0.036095, "sales": 0.322763}},
 "urgent": {"type": "noul", "noul": 0.167016}
}

How it was measured

Jevals PubMedQA, Banking77 (77 options) and HelpSteer2, plus Nimble: the merge check's fixed sample (seed 20260925), the run's option order and temperatures. "Same answer" is the top option; "prob. diff" is the largest change on any option against the run's CUDA record. llama-server ran on Metal (M3 Ultra) with the same tokens as the Python renderer on every prompt. Records: eval/agreement-*.json.

License

Apache-2.0, as the base model. Made in Texas.