blackfoam-12b

A decision model: it answers yes/no, pick-one and rating questions about a piece of text or JSON with calibrated probabilities, in one forward pass, without generating text - so an answer outside your options is impossible and nothing needs parsing. It speaks the System One wire format (POST /v1/systemone), so existing clients work by changing the base URL.

What it is made of

• Weights: google/gemma-4-12B-it by Google (Apache-2.0), revision 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7, with the weights of every language-model Linear layer stored in 8 bits (float8 e4m3, one scale per output channel; embeddings, output layer, norms unchanged; activations are not quantized): 13.0 GB. Conversion recipe, tool versions and the source's checksums are in PROVENANCE.json; all 9 files re-hashed at packaging and equal to the conversion record, 1 weight file listed in SHA256SUMS.
• Ours: the decision engine (engine/, Apache-2.0) - one read of the input shared by every question, the answer read from the model's next-token distribution over your labels - and the calibration in launch.env, fitted on public data by stated rules (benchmarks/).
• Not affiliated with Google or TypeSafe AI. 'Jev' and 'System One' are TypeSafe's names, used only to describe the comparison and the wire format.

Run it

pip install ./engine                        # Python 3.10+, CUDA GPU with 40 GB or more
API_KEYS=$(openssl rand -hex 24) ./serve.sh   # listens on 127.0.0.1:8080

from typesafe_sdk import Choice, Noul, TypeSafeClient

client = TypeSafeClient(api_key="<your key>", base_url="http://127.0.0.1:8080")
r = client.system_one(
    state={"ticket": "I was charged twice for order A-104. Refund the duplicate today."},
    questions={
        "team": Choice(instructions="Which team should handle `ticket`?",
                       criteria={"billing": "Charges, refunds", "technical": "Bugs, outages", "sales": None}),
        "refund": Noul(instructions="Does the customer ask for money back?"),
    },
)
print(r.answers["team"].choice, r.answers["team"].probabilities, r.answers["refund"].noul)

The 8-bit weights are expanded to 16 bits once at load and every layer computes in bfloat16, so the engine runs at the speed of the 16-bit model; memory and throughput are measured below. The file is in the compressed-tensors format (8-bit weights, no activation quantization); other runtimes that read that format were not tested with it.

Shipped settings (launch.env): bias correction off, temperatures noul 4.71 / choice 4.71 / score 6.22, option orders averaged per choice: 1. Every number below was measured with exactly these settings on this exact file unless a row says otherwise.

Against Jev, on every public suite with a recorded Jev run

The yardstick is jaredpalmer/kev (Apache-2.0): frozen suites in the System One request format with gold labels, plus reports from calling the real Jev (typesafe-ai/jev, hosted). This model answered the same records; kev's own metric code scored both. Accuracy and expected calibration error (ECE, lower is better) on each suite's clean records:

Suite	n	This model	ECE	Jev (recorded)	Jev ECE	Difference	p
decision-v2	1200	0.765	0.025	0.833	0.074	-6.8 pts	0.00
decision-v4	1264	0.775	0.023	0.845	0.067	-7.0 pts	0.00
semif-v1	144	0.951	0.072	0.965	0.011	-1.4 pts	0.56
transfer-v2	560	0.825	0.037	0.855	0.055	-3.0 pts	0.17
transfer-v4	656	0.829	0.031	0.857	0.049	-2.7 pts	0.17
transfer-v9	1046	0.749	0.029	0.854	0.033	-10.5 pts	0.00
typesafe-v1	102	0.863	0.065	0.892	0.055	-2.9 pts	0.52
wanli-v1	256	0.719	0.111	0.758	0.108	-3.9 pts	0.31
All suites	5228	0.783		0.846		-6.3 pts	0.000

Pooled over 8 suites: -6.3 points against Jev, standard error 0.8: behind Jev at the 5% level. The comparison is unpaired because Jev's per-record answers are not public; kev recorded them on its development splits, which is what both columns use.

Task by task on kev transfer-v4 (11 tasks, none used for tuning)

Task	n	This model	Jev (recorded)
Policy logic: and / or	32	1.000	0.906
Policy logic: conditional	32	0.719	0.781
Policy logic: or / not	32	0.844	0.969
Refund authorisation	40	1.000	1.000
Deadline arithmetic	40	0.950	0.925
Emotion (6 labels)	80	0.637	0.588
MMLU knowledge	80	0.725	0.900
PAWS paraphrase	80	0.787	0.787
QNLI	80	0.875	0.925
SciQ	80	0.988	0.988
Offensive tweets	80	0.787	0.812

Option order

Asked with the options reordered, a single-order read changes its answer on 0.056 of questions; averaged over 4 orders, 0.056 (Jev 0.000). Accuracy 0.829 with 1 order, 0.828 with 4. Shipped: 1 (rule in eval/finalize_env.py: 4 only if it at least halves the flips and loses no answers, paired). These two runs are calibration diagnostics (temperature 1, 16 records per forward pass): they decide the rule; the suite table above is the shipped configuration.

Public test sets (zero-shot, 500 examples per task, seed 0)

Raw read-out is what ships unless bias correction is listed above. ECE with the fitted temperature is measured on the half of each sample that was not used to fit it.

Task	Type	Accuracy	With bias correction	ECE, temperature 1	ECE, fitted temperature
SST-2 sentiment	noul	0.904	0.786	0.091	0.027
BoolQ reading comprehension	noul	0.910	0.866	0.088	0.026
AG News topic (4)	choice	0.858	0.854	0.138	0.055
MNLI inference (3)	choice	0.872	0.824	0.125	0.027
SST-5 sentiment level (5)	score	0.558	0.556	0.420	0.086

Serving

Measured on one NVIDIA A100-SXM4-80GB. The whole HTTP server in this folder with launch.env (1 worker, requests arriving within 5 ms answered together, up to 16 per forward pass), 400 requests from the suites, 16 at a time: 31.01 requests/s; latency under that load p50 450 ms, p95 856 ms, p99 903 ms; HTTP statuses {'200': 400}. The official typesafe-sdk client works against it unchanged.

By configuration, each measured alone on the GPU with the same 400 requests (the whole server; benchmarks/throughput/):

Workers	Requests per forward pass	Clients at once	Requests/s	p50 latency	p95 latency	GPU memory
1	1 (off)	1	5.33	187 ms	191 ms	32 GB
1	1 (off)	16	5.38	2971 ms	2988 ms	32 GB
1	1 (off)	64	5.36	11906 ms	11996 ms	32 GB
1	up to 16	1	10.74	92 ms	99 ms	32 GB
1	up to 16	16	30.78	465 ms	869 ms	32 GB
1	up to 16	64	30.25	1842 ms	3472 ms	32 GB
1	up to 32	1	10.64	93 ms	100 ms	32 GB
1	up to 32	16	30.71	461 ms	873 ms	32 GB
1	up to 32	64	30.08	1840 ms	4203 ms	32 GB
2	1 (off)	1	5.29	186 ms	199 ms	64 GB
2	1 (off)	16	8.97	1850 ms	3169 ms	64 GB
2	1 (off)	64	8.46	5684 ms	11738 ms	64 GB
2	up to 16	1	10.07	98 ms	106 ms	63 GB
2	up to 16	16	26.95	537 ms	1212 ms	63 GB
2	up to 16	64	27.63	2028 ms	4345 ms	63 GB
2	up to 32	1	10.34	96 ms	104 ms	63 GB
2	up to 32	16	27.9	545 ms	1020 ms	63 GB
2	up to 32	64	27.19	2054 ms	3963 ms	63 GB

Batching does not change what the model answers

Before it serves, the engine proves on the loaded model that its two fast paths - several requests in one forward pass, and one read of a state shared by all of its questions - answer like one request at a time, and switches off a path that does not (/healthz reports which are on). On this file their label log-probabilities differ from one-at-a-time scoring by 0.15 and 0.12 on average (the bound is 0.25): bfloat16 rounding in a batch of another shape. Every suite above was also answered one request at a time: of 5228 questions, 99.6% got the same answer, accuracy 0.783 one at a time and 0.783 batched (exact McNemar p = 0.38), and the probabilities returned moved by 0.004 on average. The tables show the batched answers, as served.

Limits

• Zero-shot: accuracy is the backbone's reading of your question. Keep arithmetic, dates and control flow in code; ask narrow questions; act on confidence; send low-confidence cases to a person.
• Calibration is measured above on public data; re-fit the temperatures on your own labelled data before automating on confidence.
• A forced choice always answers: add a 'none of these' option where needed.
• Instructions inside the input are discouraged by the prompt, not eliminated. English-first.

Licence

Apache-2.0. The weights are Google's work, modified as stated above and in NOTICE; the engine, calibration and packaging are ours. See LICENSE, NOTICE and PROVENANCE.json.