dennislee928tw/jef-v0

JEF 0.2.0rc12 — decision head and calibrator

The trained part of JEF, an open System One decision engine: typed questions in, probability distributions with calibrated confidence out, no text generated.

This repository is not a standalone model. It holds a bilinear decision head and a fitted calibrator that sit on top of a frozen jhu-clsp/mmBERT-base. That split is why it is ~400KB instead of ~800MB, and why it trains on CPU. The backbone revision it was fitted on was not recorded.

pip install "jef[torch]"

from jef_sdk import Jef, choice

jef = Jef(
    "jhu-clsp/mmBERT-base",
    head="hf://dennislee928tw/jef-v0/head.npz",
    calibration="hf://dennislee928tw/jef-v0/calibration.json",
)

jef.evaluate(alert, {"team": choice("誰處理？", soc="監控", infra="基礎設施")})

Results

Out-of-sample, on a split grouped by evidence rather than by sample — one CVE yields several questions over an identical description, and splitting per sample puts that text on both sides of the line. Test set: 1397 samples.

bucket	n	accuracy	macro-F1	ECE	temperature	P(correct) ECE
choice:14	64	0.688	0.592	0.153	1.16	unavailable
choice:4	423	0.787	0.520	0.072	1.13	0.052
noul:2	487	0.813	0.808	0.035	1.15	0.041
score:5	423	0.570	0.460	0.094	1.01	0.099

• Majority-class baseline: 0.5870
• Overall accuracy: 0.7258, macro-F1 0.6055
• ECE: 0.0420 uncalibrated → 0.0359 calibrated
• Excluded from training: soar_zh

On third-party benchmarks

On CTI-Bench VSP, with every training CVE excluded, attack vector reaches 0.840 against a 0.250 random baseline and user interaction 0.805 against 0.500. Conformal coverage holds at 0.895 against a nominal 0.900 there.

On TMMLU+ it scores 0.254 against a 0.250 baseline — chance. JEF judges evidence you supply; it does not know things. That boundary is stated because finding it in production is worse.

confidence is not correctness

confidence is (n × peak − 1) / (n − 1): how peaked the distribution is. A model can be decisive and wrong, which is why most of this ecosystem ships the disclaimer "confidence measures concentration, not correctness".

calibration.json carries three things fitted on a held-out split: temperature scaling per calibration bucket (task plus answer shape since schema 2), split-conformal quantiles, and an isotonic map from confidence onto observed correctness. That last one is what makes p_correct meaningful — and it is absent for buckets with too few calibration points, which is deliberately distinguishable from a low value.

Conformal coverage shortfalls

• choice:4 at alpha=0.10: 0.861 against a nominal 0.90

Split conformal guarantees coverage under exchangeability. Grouping by evidence puts different concepts in calibration and test, which breaks that assumption. Do not gate automated actions on prediction sets from these buckets.

Provenance

field	value
model_id	jef-v0
version	0.2.0rc4
git_commit	ae77515b21ba7f8d44e2bd77294691b4fc2bd959
backbone	jhu-clsp/mmBERT-base (frozen, loaded from Hugging Face)
calibration_scope	legacy (calibration schema 1): buckets are keyed by answer shape, not by task. JEF loads it, reports calibration_status: legacy for every question, and no scene gate automates on it.

file	sha256	bytes
head.npz	sha256:202b98571b09de6540176c6ca3e0436e3e0093750fd2621d2c9cebc06bb53d2e	394484
calibration.json	sha256:dacf41ff5f07f26389a8386b922a9aa8f0c56e895414a895e5dba9f0af04ef33	4577
report.json	sha256:5e087e0d974ab1992c5e56161edf0f623b571ce06977b9a486a0bb267d95a3a0	33844

manifest.json lists these digests. JEF verifies them at load and serves without calibration on any mismatch; the release workflow refuses to publish files that disagree with it, and attests each file with Sigstore (gh attestation verify <file> --repo cyber-security-dev-dep-mitake-com-tw/jef).

Intended use and scope

• Intended use: decision support for security operations: ranking, routing and enrichment suggestions an analyst reviews, read-only shadow mode, and low-impact (T0/T1) automation where the task is calibrated.
• Out of scope: closing or suppressing alerts, isolating hosts, disabling accounts or blocking addresses without a person's signed authorization; general knowledge questions; any task the calibration does not name.
• Known failures: chance-level on general knowledge (TMMLU+); conformal coverage can fall short under evidence-grouped splits (listed below when it does); ground truth is taxonomy-derived (ATT&CK, CVSS), not analyst verdicts on real alerts.

Training data

Labels come from taxonomies, not from a language model: MITRE ATT&CK states which tactic each technique serves, and a CVSS vector states the attack vector, the privileges required and the severity band. Nothing here derives from any commercial model's output.

Limitations

• Not a knowledge model. It reads the evidence you give it.
• Context is bounded by the backbone (8,192 tokens for mmBERT-base).
• Without this calibrator loaded, JEF refuses to automate: p_correct is unavailable, the conformal set excludes nothing, and every scene gate routes to a human. That is the design, not a failure.

License

Apache-2.0. Full methodology, including a contaminated run that scored 1.000 and why it was wrong, in docs/RESULTS.md.