train-jev-4b-lora v0.4.0

Qwen3.5-4B + LoRA + a 24-slot FP32 decision head, returning probabilities for yes/no (noul), multiple-choice (choice) and rating (score) questions without generating text. Uses standard causal attention and the train-jev System One API.

The small model reaches 76.03% train-jev-pk accuracy and 42.00 full public Decision Index (previous release: 74.18% / 39.01). Calibration is not uniformly better: classification ECE rises from 0.0501 to 0.0622, and transfer ECE from 0.0539 to 0.0588.

For the larger model, see shibing624/train-jev-27b-lora.

Use

git clone https://github.com/shibing624/train-jev.git && cd train-jev
pip install -r requirements.txt
python -m trainjev.serve --base Qwen/Qwen3.5-4B \
  --adapter shibing624/train-jev-4b-lora --port 8020

The first run downloads the backbone, LoRA, head.pt and saved temperatures. This is a custom decision adapter, not a chat/generation model: plain PEFT loading does not apply the decision head or calibration. Pin this release with --adapter_revision v0.4.0. API examples are in the serving guide.

Evaluation

Ours are the current weights. Other models are historical references, not rerun or newly matched in this round.

Evaluation	train-jev-4b-lora	train-jev-27b-lora	Kev-4B (historical)	autotrust/JEV-27B-VL (historical)	TypeSafe Jev v1.13 (historical API)
train-jev-pk accuracy, 2,494 questions ↑	76.03%	79.52%	63.50%	73.47%	–
train-jev-pk ECE15 ↓	0.0930	0.0901	0.159	0.107	–
Classification PK accuracy, 1,300 questions ↑	0.6523	0.6938	0.640	0.690	0.700
Classification PK ECE10 ↓	0.0622	0.0377	0.083	0.094	0.130
Kev transfer-v4 accuracy, 656 scored questions ↑	0.8201	0.8476	0.817	0.852	0.857
Kev transfer-v4 ECE10 ↓	0.0588	0.0681	0.042	0.097	–
Decision Index historical sample, 6,000 requests ↑	43.84	57.54	38.4	54.1	59.7
Decision Index full public panel, 110,087 requests ↑	42.00	57.01	–	–	–

Exact metrics, prior-release comparison, benchmark regressions and evidence hashes are in metrics.json; protocols and context are in the results guide.

• PK uses 22-task macro accuracy and per-task upper-inclusive ECE15; classification uses pooled upper-inclusive ECE10. Transfer evaluated 764 original requests and scores 656 clean knowable questions, preserving Kev normalization and its ECE10 boundaries.
• The historical sample has 6,000 requests / 47,463 child answers, 41 input catalogs and 35 scoring benchmarks. Full public evaluation has 110,087 requests / 272,229 children over 37 scoring benchmarks. Six unindexed diagnostics were not requested: canonical suite43 complete=false is retained. These are composite DI scores, not accuracy or the private Full leaderboard.
• Prior-release controls are original stored predictions and CPU recompositions, not new same-stack inference. Old classification wide choices used tournament; current weights use native single-pass readout.
• Improvements are not universal. For 27B, transfer accuracy drops 85.8232% → 84.7561% and ECE rises 0.031566 → 0.068090. Negative DI benchmark deltas are retained, not hidden by the aggregate score.

Training

Rank16 / alpha32 LoRA and the 24-slot head are trained; the backbone is frozen. Initial Jev-distribution distillation is followed by gold/rule refinement and 12,947 admitted tasksource rows across nine sources, including 2,692 native soft targets. Sources include CommonsenseQA, CosmosQA, DynaSent rounds 1/2, WNUT17, LexGLUE CaseHOLD, Fig-QA, QuaRTz and Amazon Counterfactual English.

This release selects step 1,024 on a 2,048-step scheduler, seed17, batch128. The tasksource stage replaces 32 same-kind choice slots per batch; earlier gold mixing occurs before replacement and is not the final all-draw fraction. Actual phase exposures are recorded in training_recipe.json; calibration bytes and temperatures were not refitted for these evaluations.

The training guide explains the reusable public-data workflow. Processed training data, generators, optimizer/RNG and raw evaluation predictions are not bundled; exact one-command training reproduction is not provided.

Limitations and attribution

• PK includes training-source and shared-rule-family items with previous selection exposure; it is not a wholly unseen-source independent final test.
• After generation/verification, a schema probe displayed native labels in two PK provenance records. They were not used for generation, oracles, screening, selection or tuning, but strict zero-label-access was not fully preserved. No 100% contamination-free claim is made.
• New vision quality was not re-evaluated. Earlier 4B final/SDK/MemWrite8 evidence is archival; no new 27B SDK or MemWrite8 test is claimed. API support is not model-quality validation.
• Only choice rows A–P are trained; the extra 240 rows for wide choices are untrained base readouts. Confidence is not a universal correctness guarantee.

Adapter/head and code: Apache-2.0. Upstream data retains its own terms and attribution; see LICENSE and THIRD_PARTY_ATTRIBUTIONS.txt. Independent work, not affiliated with TypeSafe AI, autotrust or upstream authors.