mohit67890/imajev-4b

Decisions for real-world cases.

Small open models that read the photos, records and text a business already has and answer in the options you set, with a probability on each and an explicit can't tell. Your system acts when it is sure and hands the rest to a person.

imajev-4b is the recommended default of the family · other sizes: imajev-2b · imajev-9b
Live demo · Website · Code and results · Technical report

imajev-4b is the recommended default: level with the 9B on ImajevBench (82.4% vs 82.1%) at under half the size.

What sets it apart

• A photo read against your record. Checks a photo against your own fields and names the one that is wrong. Trained on 72k photo-vs-record and two-photo decisions.
• Two photos, one decision. A reference and a target in the same request: shipped against returned, a known-good part against the one on the line.
• A trained can't tell. Every answer carries a probability for unknown, so the app can stop instead of guessing.
• Open, small and local. Apache-2.0, MLX on a Mac or PyTorch on one GPU; photos and customer data never leave your network.
• Jev's contract, now with images. TypeSafe's Jev request and response (POST /v1/systemone), plus images, unknown_probability and abstained. Jev itself is text-only and hosted; its state limit (32k tokens) is larger than imajev's (32 KB).

One request, every answer typed

The exact script we ran against imajev-4b and its output (rounded, usage shortened); 1.15 s on a Mac Studio (four option orders averaged, calibration file applied). Swap the adapter for this size and the request is unchanged.

import json, requests

URL = "http://127.0.0.1:8765/v1/systemone"

listing = {
    "title": "Men's suede boat shoes",
    "color": "red",
    "product_type": "shoe",
}

questions = {
    "contradicted_field": {
        "type": "choice",
        "instructions":
            "Which field of `listing` does this photo contradict?",
        "criteria": {
            "listing.color": None,
            "listing.product_type": None,
            "none of these": "the photo agrees with every field",
        },
    },
    "color_matches": {
        "type": "noul",
        "instructions":
            "The product in the photo matches `listing.color`.",
    },
    "type_matches": {
        "type": "noul",
        "instructions": "The photo shows the kind of product "
                        "given in `listing.product_type`.",
    },
}

request = {"state": {"listing": listing}, "questions": questions}
with open("listing.jpg", "rb") as photo:
    r = requests.post(URL, files={"image": photo},
                      data={"request": json.dumps(request)})
print(json.dumps(r.json(), indent=2))

Result

{
  "model": "imajev-4b",
  "answers": {
    "contradicted_field": {
      "type": "choice",
      "choice": "listing.color",
      "probabilities": {
        "listing.color": 0.95,
        "listing.product_type": 0.006,
        "none of these": 0.043
      },
      "confidence": 0.919,
      "unknown_probability": 0.007,
      "abstained": false
    },
    "color_matches": {
      "type": "noul",
      "noul": 0.082,
      "unknown_probability": 0.022,
      "abstained": false
    },
    "type_matches": {
      "type": "noul",
      "noul": 0.989,
      "unknown_probability": 0.004,
      "abstained": false
    }
  },
  "usage": {
    "total_ms": 1152.9,
    "input_tokens": 224
  }
}

Automate what is clear, route the rest

imajev-4b on the 279 ImajevBench test questions (photos, records and text; 21 whose honest answer is can't tell), raw probabilities, scored with the benchmark's own rule:

Act automatically when at least…	Decisions automated	Automatic decisions right
80% sure	64%	91.6%
90% sure	58%	94.5%
99% sure	42%	99.1%

The rest go to a person. The benchmark is built to be hard; measure on a few hundred of your own cases before choosing a threshold. Other sizes at 90%: 2B 38% automated at 95.3% right, 4B 58% at 94.5%, 9B 70% at 91.8%.

Checked demos. Every clickable combination in the five playground apps (business checks, text only, wardrobe, stylist, tracing pad) was run on imajev-4b (four option orders) and compared with the right answer: 130 of 145 pass without the calibration file, 118 with it. Only passing combinations are shown as demos; the misses are listed in reports/scenarios/.

How it was made

About a million training decisions across the family, in four stages, for about $676 of rented GPU time for the whole project. The 4B was trained on stages 1 and 2 in one run (867k decisions: the 504k human-labelled set, 296k labelled by our 9B and 66k photo-vs-record and two-photo decisions), then on about 23k hard questions kept only when open-weight teachers agreed, then a soft-target continuation on 39,515 rows carrying Qwen3.6-35B-A3B's full probability distributions (with the strict slice of the Eikos decisions set (caiovicentino1/eikos-decisions, CC-BY-4.0; attribution and per-source licences in docs/eikos-decisions-usage.md) and 5k replayed image decisions). The shipped adapter is the weight-space average of two adapters: the hard-question adapter and that continuation. Every teacher is open-weight; no Jev outputs, paid-API outputs or JevBench items were used.

This adapter

This repository holds the 4B adapter: the recommended default tier — the best accuracy per millisecond in the family. It is a LoRA (rank 16, alpha 32) on the language layers of Qwen3.5-4B (revision 851bf6e8) plus a 255-code decision readout, in PEFT format at the root and in MLX format under mlx/; the weights are the element-wise average (0.5 / 0.5, LoRA matrices and readout) of the hard-question adapter and its soft-target continuation. Code, server and evaluation harness: https://github.com/mohit67890/imajev. Other tiers: https://huggingface.co/mohit67890/imajev-2b (latency), https://huggingface.co/mohit67890/imajev-9b (quality).

Technical specification

Base model	Qwen/Qwen3.5-4B, revision 851bf6e8 (Apache-2.0)
LoRA	rank 16, alpha 32, dropout 0, no bias, on every language-model projection: q,k,v,o, gate,up,down and the DeltaNet in_proj_qkv, in_proj_z, out_proj; vision encoder frozen, no LoRA
Decision readout	one bias-free linear layer, 255 × 2560, float32
Trainable parameters	30,474,240 LoRA + 652,800 readout = 31,127,040
Files	adapter_model.safetensors 122.0 MB (F32); readout 2.6 MB
Precision	base weights bfloat16; LoRA and readout float32 (MLX copies under mlx/ converted from the same files)
Request limits	0–2 images (resized to at most 400,000 pixels), state up to 32 KB, 1–8 questions, 2–254 options per choice, 2–10 levels per score, at most 4,096 tokens (longer requests are refused, not truncated); English only
Calibration	one temperature, 1.717, fitted on 150 template-generated JevBench-style items (none from JevBench), also used to pick checkpoints

Training path. One trainer for every stage (PyTorch + PEFT): cross-entropy on the readout logits (soft targets where a record carries a distribution), AdamW with weight decay 0, linear warm-up then cosine decay to 10% of the peak rate, gradient clipping 1.0, seed 0, 4 GPUs. The soft-target stage adds a rationale loss (weight 0.3, at most 192 tokens) and permutes the options of every question.

Stage	Started from	Epochs	Peak LR	Steps (kept / total)	Hardware	Time
First run (stages 1 and 2 combined)	Qwen3.5-4B	0.5	1.5e-4	1,900 / 2,595	4×H200	1.9 h (2 h 06 min wall)
Stage 3, round 1	first run	2	3e-5	250 / 303	4×H200	12 min
Stage 3, round 2	round 1	2	2e-5	266 / 266	4×H100	22 min (27 min wall)
Stage 4, soft-target continuation	round 2	2	2e-5	260 / 747	4×H100	1.4 h (out of memory at step 625, resumed at the same budget)
Weight-space average	½ round 2 + ½ stage 4, element-wise (LoRA and readout)	–	–	–	–	–

Data this size saw.

• First run: 866,854 decisions: 504,000 stage-1 decisions with their original labels (36 licence-admitted sources); 296,482 stage-2 decisions labelled by the 9B, with unknown targets capped at 15%; 66,372 photo-vs-record and two-photo decisions. 17.15% of its training targets are unknown; no base-model blend.
• Stage 3, round 1: 14,112 training records: kept teacher questions (9,368 of 13,386 kept on two-answerer agreement) plus the training share of 8,532 human reasoning items from 10 licensed sets.
• Stage 3, round 2: 7,812 training records: 3,598 new (4,852 of 8,097 kept on three-answerer agreement) + 4,214 replayed from round 1.
• Stage 4: 39,515 records: the stage-3 teacher questions relabelled with Qwen3.6-35B-A3B's probability distributions (thinking mode), 9,880 new hard, judge and programmatic questions, the strict slice of the Eikos decisions set (10,570 rows, open-weight teachers only) and 5,000 replayed image decisions.

Compute. $499.07 of rented GPU time on RunPod through stage 3 plus about $177 for stage 4 (8×H100, 6 h 20 min, all three sizes): about $676 for the whole project, every run included.

Full specification: https://github.com/mohit67890/imajev/blob/main/docs/technical-specification.md

Results (2026-09-24; every number reproducible from the repo's reports/)

Benchmark	imajev-4b	Notes
JevBench public hard (111)	70.3% served (4 option rotations + calibration.json; ECE 0.116), 69.4% raw (ECE 0.164)	same protocol, our runs: JevK5 v0.2.0 73.9% / 0.073, Eikos-4B 73.9% / 0.054, Hopper 67.6% / 0.050, Qwen3.5-4B base (generation) 48.6%; the previous imajev-4b 67.6% (ECE 0.112)
JevBench public original / easy	98.6% / 100%
ImajevBench v2.0-lite test (279 items: text, photo, photo+state)	82.4% (230/279), 95% CI [0.77, 0.89]; tracks text 26/37 · visual 107/120 · joint 97/122; correct Unknown 14/21, false abstention 3/258; ECE 0.063	Qwen3.5-4B base 70.6%; imajev-9b 82.1%; the previous imajev-4b 82.4%; frontier APIs 91–99.6% by structured generation
ImajevBench private-1 hidden split (202; aggregates only)	84.2% (170/202); text 23/30 · visual 80/84 · joint 67/88; ECE 0.056	the previous imajev-4b 84.7%
MLX (Mac) vs PyTorch on ImajevBench	82.1% vs 82.4%, 98.2% argmax agreement	parity check of the mlx/ weights against the pod run
MMLU-1000, text-only / with an unrelated photo	74.5% / 72.9%	measured on the previous imajev-4b, not re-run
Irrelevance panel (2,823: MMLU with and without an unrelated photo, ABO, VizWiz)	80.2%	the previous imajev-4b 79.4% (false abstention 1.5%, correct abstention 91.9%)
Hard-question test (435): correct on Unknown-gold rows / false abstention	14/14 · 0.24%	ship gates
typed-decisions test (2,000)	67.0%	measured on the previous imajev-4b, not re-run
Held-out photo sources / two-image / state probe / pairs probe	56.7% / 41.8% / 71.5% / 98.3%	the first two measured on an earlier imajev-4b, before the hard-question stage; 10 of the state probe's photos were seen elsewhere in training
Reasoning dev (6,240 items; also used for checkpoint selection)	not measured for the shipped average; 67.2% for the soft-target checkpoint it averages, 66.6% for the previous imajev-4b	67.8% before the last part of the hard-question stage (its one measured trade-off)

On ImajevBench the shipped imajev-4b beats its untuned base by +11.8 points [+5.8, +18.0] (paired cluster sign-flip test over 89 evidence clusters, p = 0.0006; exploratory) and is statistically indistinguishable from the shipped 9B (−0.4 for the 9B, p = 1.0), which scores 82.1% to the 4B's 82.4%. The previous adapters gave +11.8 [+5.1, +18.5], p = 0.0008, and +0.4 for the 9B, p = 1.0. JevBench is text-only; imajev's image capability shows only on ImajevBench and in use. The official JevBench leaderboard adds 308 sealed items and a four-axis score that only its maintainers can run; a measurement will be requested at launch.

Calibration

calibration.json (schema 1.1) applies one temperature (1.717) to every question type × option-count bucket, fitted by negative log-likelihood on 150 template-generated JevBench-style items (none from JevBench), also used to pick checkpoints. Temperature scaling never changes an answer, only its probability. unknown offsets are 0: bounded offsets were tested and changed no panel by more than 0.1 points.

Checked through the released server (ECE, uncalibrated → with calibration.json); the off-distribution rows were measured on the previous imajev-4b with its own temperature (2.32) and have not been re-run for this version:

Panel	ECE
JevBench public hard (served with 4 rotations, 111)	0.164 raw → 0.116
MMLU-1000, text-only	0.150 → 0.035
typed-decisions test (2,000)	0.149 → 0.047
SST-5 (2,210)	0.220 → 0.020
Photo-only verification (ABO + VizWiz, 823)	0.038 → 0.062

Pooled ECE over all 231 public JevBench items in the served configuration: 0.026. An ECE-fit temperature (2.05) lowers the hard-tier ECE to 0.091 but raises the pooled ECE to 0.048, so the NLL fit is shipped.

On photo-only verification the raw probabilities are already calibrated and the temperature over-softens them; if your traffic is mostly photo-against-record checks, serve without --calibration or fit your own temperature on a held-out sample.

Serving

git clone https://github.com/mohit67890/imajev && cd imajev
python3.11 -m venv .venv && . .venv/bin/activate
pip install -e ".[serve,mlx]"            # Apple silicon;  elsewhere: pip install -e ".[serve,torch]"
python scripts/download_model.py --model 4b
hf download mohit67890/imajev-4b --local-dir adapters/imajev-4b
# Mac (MLX)
PYTHONPATH=src:scripts python scripts/playground/server.py --model-bundle artifacts/model-qwen4b.json \
  --adapter adapters/imajev-4b/mlx --calibration adapters/imajev-4b/calibration.json --model-name imajev-4b --port 8765
# Linux / CUDA (PyTorch + PEFT)
PYTHONPATH=src:scripts python scripts/playground/server.py --backend torch --model-bundle artifacts/model-qwen4b.json \
  --adapter adapters/imajev-4b --calibration adapters/imajev-4b/calibration.json --model-name imajev-4b --port 8765

Then POST /v1/systemone with a Jev-shaped request. One forward pass per question: p50 96 ms raw on one H100 for a JevBench hard item, serially; 350 ms with the --rotations 4 (four option orders averaged, +0.9 hard) and calibration.json used for the numbers above (shared pod, under load).

Training data and provenance

Synthetic documents and typed questions written by Qwen3.6-27B, answered independently by Qwen3.6-27B (thinking), gpt-oss-20b and, in the last part of the hard-question stage, Qwen3.6-35B-A3B (thinking); a question is kept only when every answerer agrees with the intended answer. In the soft-target stage the same questions were relabelled with Qwen3.6-35B-A3B's probability distributions (thinking mode), joined by 9,880 new hard, judge and programmatic questions, the strict slice of the Eikos decisions set (10,570 rows, open-weight teachers only) and 5,000 replayed image decisions; the shipped adapter is the weight-space average of the hard-question adapter and that continuation's best checkpoint. Plus licensed image datasets and human-written states from earlier training stages (see the repo's datasheets). No JevBench items (8-gram lint), no outputs from Jev or any paid API. All teachers are open-weight, Apache-2.0.

Limits

Single-pass: no reasoning at inference, so multi-step arithmetic and answer-quality judging trail reasoning models (a frozen Qwen3.6-35B-A3B with thinking scores 97% on JevBench hard at seconds per decision). Over-confident without calibration.json. Two-image comparisons are the weakest visual task (41.8% on real pairs, measured on an earlier imajev-4b). English only.