jsaurabh/qwen3.5-9b-jev-data-mix-v2

Qwen3.5-9B Jev-like Data Mix v2

This repository contains a LoRA adapter trained on Qwen/Qwen3.5-9B for typed decision scoring with candidate logits. It is an experimental, open reproduction inspired by the public Nimble methodology and Jev-like typed judgments. It does not reproduce TypeSafe's proprietary Jev architecture or weights.

The model consumes application state plus a typed question/schema and scores the allowed candidates directly. It supports choice, boolean (noul), and ordinal/score decisions through the upstream Nimble encoding and candidate-logit readout.

Results

Evaluation	Result
Nimble frozen holdout	282/324 (87.04%)
Synthetic gap holdout	420/480 (87.50%)
JevBench public v1	184/231 (79.65%)
JevBench Brier score	0.2895
JevBench ECE	0.0892

JevBench tiers: easy 48/48, standard 68/72, hard 68/111. This run matched the original reproduced Nimble adapter on public JevBench accuracy; it improved calibration error but did not improve accuracy.

Training

• Base revision: c202236235762e1c871ad0ccb60c8ee5ba337b9a
• LoRA rank 16, alpha 32, dropout 0.05
• 3,822 examples: 2,676 original Nimble rows, 764 synthetic gap rows, and 382 hard replay rows
• One epoch, 478 optimizer steps
• Microbatch 2 with gradient accumulation 4
• Maximum length 2,048; bf16
• A100 40 GB; 20.33 GiB peak allocated memory
• Training time: about 26.8 minutes

The exact configuration and data audit are in schema_config.json and data_audit.json. run_report.json, after_original.json, after_gap.json, and nimble-data-mix-v2-jevbench-public.json contain the evaluation evidence.

Loading

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3.5-9B"
adapter_id = "jsaurabh/qwen3.5-9b-jev-data-mix-v2"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto")
model = PeftModel.from_pretrained(base, adapter_id).eval()

The adapter was trained and evaluated with the candidate-scoring implementation from Bespoke Labs' Nimble repository, frozen at commit f136b3f75721fda4ea961f73993cc50b08488835. Normal text generation does not reproduce the reported scores; use Nimble's schema encoding and candidate-logit readout.

For the official JevBench runner, see bench/SUBMISSION.md. The included in-process adapter performs the same frozen candidate-logit scoring path and generates no output tokens.

Evaluation provenance

• Nimble commit: f136b3f75721fda4ea961f73993cc50b08488835
• JevBench commit: 5e95f23cbb7be098a9061fea924c4421620ab1a5
• Adapter archive SHA-256: c43828fe7a6005682754878a21b03eebd167cb745e212af81277757a262b903b

Limitations

This is a research adapter, not a production banking or compliance model. Its weakest public benchmark area is temporal/numeric reasoning. The synthetic gap set is private project-generated evaluation data and should not be interpreted as an independent benchmark. Review decisions in high-impact settings and validate the adapter on the intended distribution.

Benchmark-awareness disclosure

The original Nimble training projection was public. The added synthetic curriculum was created after inspecting this project's results on the public JevBench tasks and targeted broad weak families including temporal/numeric, probability, and long-policy reasoning. Public JevBench performance was used to compare experiments. No private or held-out JevBench item was available or used, and the adapter was not calibrated on JevBench probabilities. This is therefore a benchmark-aware submission, not a clean blind evaluation.