skamalj/signpost-d-4b

Signpost-D 4B

A decision model. Give it a piece of state (a JSON record or text) and typed questions about it, and it answers each question with a probability for every option: noul (yes or no), choice (one option of several) and score (a level on a scale). It doesn't generate text: each question is a single forward pass, with the probabilities read from the answer tokens. It serves unchanged through vLLM's /v1/systemone.

Base	Qwen/Qwen3.5-4B (text and vision weights kept; the vision tower is unchanged)
Parameters	4.5B, fp16, merged (no adapter needed)
Method	LoRA fine-tune, three seeds averaged (weight soup)
Input	up to 8,192 tokens per question
Licence	Apache-2.0, as the base

Results: Typed Decisions (general, zero-shot)

LocalLLaMA/typed-decisions, test split: 400 cases and 2,000 decisions, one request per case, all answered with no errors. Served with vLLM nightly with the benchmark's flags (--max-model-len 8192 --max-num-seqs 8 --logprobs-mode raw_logprobs) and scored with the benchmark's scorer.

	Accuracy ↑	KL from gold ↓	Brier ↓	ECE ↓
Signpost-D 4B	0.712 [0.691, 0.734]	0.162	0.085	0.091
Prior (ignores the input)	0.470	0.347	0.189	0.088
Uniform	0.308	0.444	0.238	0.169

The 95% interval is a bootstrap over cases.

Type	n	Accuracy
noul	600	0.788
choice	600	0.692
score	800	0.670

• Latency: median 62 ms per case (5 questions), on one Colab G4 GPU (RTX PRO 6000) with vLLM at fp16.
• Predictions: all 2,000 are in typed_decisions_test_predictions.json (case index in the test split, question id, type, probabilities).

Gold is the mean of three samples from a teacher of roughly 4B-class capability, so these scores measure agreement with that teacher; the benchmark gives the teacher's self-agreement as 0.735.

How the model was chosen

Candidates were compared on the benchmark's train split, which was never trained on. Accuracy and KL on all 1,200 cases:

Candidate	train split
seed 42	0.712 / 0.174
seed 1042	0.714 / 0.171
seed 2042	0.706 / 0.177
soup of 3 seeds, α 1.0	0.715 / 0.168
soup of 3 seeds, α 0.75 (this model)	0.720 / 0.166

There were four test runs for this model and its candidates: two candidate data mixes (0.7055 and 0.7105), one data batch that was then dropped (0.702), and this model, scored once (0.712). Every choice was made on the train split first; test confirmed it and never reversed one. The published weights were rebuilt from the three adapters and scored again on test before upload, with identical results.

Training

• Data: only our own generated data: 2,575 records (12,274 questions) in 163 invented domains. Each record is a short JSON state with five questions.
  • LLM generators design the families of records and write the records.
  • The gold is the mean of three simulated judges: a careful operator, a by-the-book auditor, and a generalist.
  • Every record passes accept-or-drop checks.
  • No row, label or question of Typed Decisions is in the data. No record uses the benchmark's four workflows (agent traces, customer service, invoices, security incidents) or their question schemas. The generators never had access to the benchmark.
• Use of the train split: the model was never trained, fitted or calibrated on it: no weights, temperatures or priors. We analysed a sample of it to understand the decisions it asks for and the shape of their gold distributions. That analysis was used to prompt the LLM generators for the data-generation jobs, together with three simulated judges.
• Format: one question per sequence, in exactly the prompt vLLM's /v1/systemone builds for this architecture: the state as JSON, the question, the labelled options and a reply line; no system message, thinking off. It is trained on the logit of the first answer token over the option labels.
• Recipe:
  • LoRA rank 16 (α 16) on 200 modules, including the linear-attention projections;
  • learning rate 1e-4, 15% warm-up, dropout 0.05, 2 epochs, fp16;
  • loss: soft-label cross-entropy, plus 0.5 × the expected-level error on score questions;
  • about 37 minutes per seed on one Colab G4 GPU (RTX PRO 6000).
• Soup: the three seeds' LoRA weight changes (seeds 42, 1042 and 2042, epoch 2) are averaged, and 0.75 of that average is added to the base weights. The adapters are in adapters/, and merged.json records the merge.

Use

vllm serve skamalj/signpost-d-4b --max-model-len 8192 --max-num-seqs 8 --logprobs-mode raw_logprobs

curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "skamalj/signpost-d-4b",
  "state": {"order": {"id": "A-17", "paid": true, "shipped": false, "days_since_payment": 9}},
  "questions": {
    "late": {"type": "noul",   "instructions": "The order is late to ship.", "criteria": null},
    "next": {"type": "choice", "instructions": "What should happen next?",
             "criteria": {"ship": "Ship now", "refund": "Refund the payment", "wait": "Wait for stock"}},
    "risk": {"type": "score",  "instructions": "How likely is a complaint?",
             "criteria": ["Low", "Medium", "High"]}
  }}'

• vLLM version: measured with vLLM nightly 0.33.0rc1.dev451; /v1/systemone needs a current nightly.
• Temperature: the probabilities are used as served, with no temperature.

Limits

• Weakest type: score questions (0.670).
• Language: trained and measured in English only.
• Teacher, not truth: the benchmark measures agreement with its teacher. Where the teacher is wrong, a better answer can score lower.
• Not a chat model: it doesn't write text.