QEV-2B

A 2B open multimodal decision model: text and image in, calibrated option probabilities out, in one forward pass with no generated tokens.

QEV-2B is the full-size member of the QEV family. It is built on Qwen3.5-2B. You supply the evidence (text, one or more images) and the candidate options with their meanings. The model returns a probability for each option, a typed answer, and an abstain signal.

Architecture, training, evaluation protocol, calibration, speed, ablations and limitations are described in the technical report.

Headline (self-scored, pending official run): On the JEV Decision Index vision board (edition 0.3.2, public part), QEV-2B scores 61.99, the highest score on the board among models up to 2.3B parameters: 2.84 points above vLLM-SR d3-nano (2.21B, 59.15), and above Intern-Decision-4B (4.66B, 61.69) and Jev-Omni (11.96B, 60.10). It ranks first among models up to 2.3B on five of the eleven benchmarks: RealWorldQA, Mind2Web, KIE, Moderation and R-Bench-M.

Results

Leaderboard comparison

JEV Decision Index, vision, edition 0.3.2, public part. Board generated 2026-10-16.

Model	Params	Index (public)
d3-mini	4.54B	67.67
RSI-Jev v6.1 VL 4B	4.66B	63.01
RSI-Jev v5.0 VL 3B	4.66B	62.58
QEV-2B (self-scored)	2.21B	61.99
Intern-Decision-4B	4.66B	61.69
Jev-Omni	11.96B	60.10
d3-nano	2.21B	59.15
Intern-Decision-2B	2.27B	52.92
OneJev1.5-0.8B	1.11B	52.32

_{Board values are copied as published (JEV Decision Index vision, edition 0.3.2, public part, generated 2026-10-16). The QEV-2B value is self-scored, pending official run: public evaluation kit, native resolution up to 1.6 MP per image (no shared multi-image budget), single run, full depth, scored on the exact released files. It is not an official leaderboard entry and will be replaced when the official run is available.}

Where QEV-2B leads (self-scored)

Benchmark	QEV-2B	Best other model up to 2.3B (board, public)
Mind2Web	82.3	d3-nano (2.21B) 63.7
KIE	93.9	Intern-Decision-2B (2.27B) 84.4
RealWorldQA	56.2	d3-nano (2.21B) 53.1
Moderation	29.9	OneJev1.5-0.8B (1.11B) 24.7
R-Bench-M	16.8	d3-nano (2.21B) 15.5

Per-benchmark scores

Index = weighted mean of the benchmarks below. Weights: 1.0 unless noted. R-Bench-M 0.5, MMMU-Pro 0.25.

Benchmark	QEV-2B (self-scored)	Best other model up to 2.3B (board)	Weight
CV-Bench	70.6	71.9 (d3-nano)	1.0
BLINK	47.2	53.8 (OneJev1.5-0.8B)	1.0
RealWorldQA	56.2	53.1 (d3-nano)	1.0
CharXiv	63.4	65.2 (d3-nano)	1.0
InfographicVQA	83.1	83.6 (d3-nano)	1.0
Mind2Web	82.3	63.7 (d3-nano)	1.0
Winoground	65.5	69.0 (d3-nano)	1.0
KIE	93.9	84.4 (Intern-Decision-2B)	1.0
Moderation	29.9	24.7 (OneJev1.5-0.8B)	1.0
R-Bench-M	16.8	15.5 (d3-nano)	0.5
MMMU-Pro	15.6	17.8 (d3-nano)	0.25
Index	61.99	59.15 (d3-nano)

Evaluation: public evaluation kit, native resolution up to 1.6 MP per image (no shared multi-image budget), single run, full depth, on the released files (merged weights). 95% bootstrap interval of the index (items resampled within each benchmark, 2,000 replicates): 60.63 to 63.34. With rows that share an image resampled together: 60.59 to 63.43. Benchmarks were not used for checkpoint selection. The previous QEV-2B release (0.2.1) has no score on this board.

Speed and fast mode

Mode	Image cap	Depth	Worst-case p95	Median p50	Peak GPU memory
Accurate (default)	1.6 MP	24 layers	738.6 ms	135.3 ms	4.44 GiB
Fast	0.26 MP	16 layers	225.0 ms	91.6 ms	4.13 GiB

_{Release loader only (qev.QEV.load + predict, no CUDA graphs or custom kernels), RTX 4060 Ti 8 GB, batch 1, warm. 47 request profiles (text and image requests with up to 16 options; the largest is 3,229 input tokens in accurate mode), 3 warm-up and 30 timed runs each. Worst-case p95 is the highest per-profile p95; median p50 is the median over profiles of each profile's p50. Peak GPU memory is PyTorch's peak allocation. In both modes the slowest profile is a 16-option image request near the token limit.}

Fast mode exits at layer 16 of 24 with a distilled exit head (exit16.safetensors) and caps images at 0.26 MP. At the same 0.26 MP input, exiting at layer 16 matched full depth on our internal DEV set: 85.0% versus 84.9% on the coverage-mix selection split (1,859 items), 89.8% versus 90.6% on NLVR2 dev (128), 88.8% versus 89.2% on image diagnostics (400) and 93.0% versus 92.8% on text diagnostics (485). The lower resolution costs accuracy against accurate mode, most on document, chart and screen sources. No index is reported for fast mode, so the numbers above apply to the full-depth mode only. Calibration is fitted for accurate mode and has not been refitted for fast mode.

Calibration and abstention

Per-type temperatures and abstain thresholds are fitted on the DEV selection split (the split also used for checkpoint selection; benchmarks are never used), on group-disjoint halves: temperature on one half, threshold on the other. Policy: the lowest confidence threshold at which answered questions have at most 15% error, with at least 20 answered. Numbers are on the held-out policy half, accurate mode. A response always carries the answer; abstained is a separate flag, so benchmark scores are unaffected.

Question type	DEV questions	Temperature	Threshold	Answered	Error on answered	ECE (raw -> calibrated)
Choice	1,296	0.84	0.31	100%	10.5%	0.025 -> 0.013
Score	79	0.43	0.52	100%	7.6%	0.140 -> 0.036
Yes/no	80	0.49	0.58	100%	6.3%	0.196 -> 0.088

For all three question types the 15% target is met on all DEV questions, so abstention only triggers below the lowest confidence seen on DEV and is rare. The score and yes/no thresholds rest on about 80 DEV questions each (internal text diagnostics) and are less certain than the choice threshold.

Architecture

• Backbone: Qwen3.5-2B language model (hidden size 2048, 24 layers: 18 GatedDeltaNet + 6 full attention), 2.21B parameters.
• Vision: Qwen3.5 ViT encoder (frozen).
• Readout: option probabilities read from the hidden states.
• Condition-modulated binding head, per-type temperature calibration and an abstain decision (fitted on DEV; see Calibration and abstention).
• Single forward pass, no generated tokens.
• Fast mode: a distilled exit head reads the options at layer 16 of 24 (exit16.safetensors); images are capped at 0.26 MP.
• GPU inference uses a BF16 backbone and FP32 readouts; the CPU path uses FP32.
• Fine-tuning: LoRA (rank 8, alpha 16) on the language model plus trained readout and binding heads; the vision encoder and the other base weights are frozen (board category: LoRA + head).
• Weights are shipped merged: model.safetensors (4.43 GB, BF16, both LoRA stages folded in) plus head.safetensors (0.7 MB, readout and binding head).

Training data

Every source used in training, read from the training script (qev_mix.py) and its training log.

Coverage mix (train split, 18 sources). Each source contributes items from its train split. 5% of each source (up to 120 items) is held out as a selection set and is never trained on.

Source	Split
AI2D	train
A-OKVQA	train
ChartQA	train
CORD	train
DocVQA	train
FUNSD	train
Hateful Memes	train
IconQA	train
InfographicVQA	train
Mind2Web	train
Mind2Web (keyed variant, mind2web_k)	train
Options-in-image	train
ScienceQA	train
Screen2Words	train
Synthetic (internal)	train
TallyQA	train
Visual7W	train
VSR	train

Additional sources

• NLVR2 train split (teacher-filtered targets). The NLVR2 dev split is used only for checkpoint selection.
• Replay of internal decision units (image and text families, train split), with teacher outputs as targets.
• Teacher outputs: option-probability distillation from QEV 0.2.1, the earlier QEV-2B release (ken-jo/qev at revision 0d2d13f): a LoRA (rank 8, alpha 16) with a condition-bound binding head on Qwen3.5-2B. The same teacher was used for QEV-2B and QEV-0.8B; it is not the released QEV-2B checkpoint.

Deduplication and contamination audit: all sources use train splits only (evaluation uses test/validation splits). Ten document/KIE/perception sources were filtered at build time: any item whose image is within Hamming distance 4 (64-bit dHash) of an evaluation image was dropped. A post-hoc audit of every training image against every evaluation image (dHash candidate, confirmed by 64x64 grayscale mean absolute difference <= 6/255) found **37 evaluation items** with a near-identical training image: 30 Hateful Memes (shared base images across the dataset's splits), 6 BLINK, 1 Mind2Web. Mind2Web has no action_uid or annotation_id overlap with the evaluation rows. Rescoring without those 37 items changes the index from 61.99 to 62.09 (Moderation 29.93 -> 30.85, BLINK 47.16 -> 47.07, Mind2Web 82.32 -> 82.44).

Teacher cleanup: a teacher target is kept only where the teacher's top option matches the gold answer. Otherwise only the gold label is trained on.

Selection: checkpoint chosen on held-out DEV data only (coverage-mix holdout, NLVR2 DEV, internal diagnostics). Public benchmarks were not used for selection.

Training recipe: continued from QEV-2B 0.2.1 (its LoRA merged into the backbone) with a fresh LoRA (rank 8, alpha 16) on the coverage mix plus replay, 56k items; the released checkpoint is optimizer step 3,616 (end of the cosine schedule). Resolution-elastic batches (0.26 / 1.05 / 1.64 MP). Answer options are permuted per visit, except for sources whose options refer to letters or boxes in the image (Mind2Web, Mind2Web-keyed, options-in-image).

Quickstart

This checkpoint needs the qev source (the published qev==0.2.1 package loads QEV-2B 0.2.1 only).

python -m pip install "qev @ git+https://github.com/ken-jo/qev" huggingface_hub

from huggingface_hub import snapshot_download
import qev
from qev import DecisionRequest

path = snapshot_download("ken-jo/QEV-2B")
model = qev.QEV.load(path, device="cuda")               # accurate: full depth, up to 1.6 MP per image
# model = qev.QEV.load(path, device="cuda", mode="fast")  # fast: exit at layer 16, 0.26 MP
request = DecisionRequest.model_validate({
    "state": {
        "text": "I was charged twice. Please refund the duplicate payment.",
    },
    "questions": {
        "department": {
            "type": "choice",
            "instructions": "Which team should handle this request?",
            "criteria": {"billing": "Payments and refunds", "technical": "Software faults"},
        }
    },
})
print(model.predict(request)["answers"])

Note on the name: the PyPI package qev is also the import name used by OneJev (from qev import Client). Install one of them per environment.

Limitations

• Weak areas: Moderation (29.9), BLINK (47.2), MMMU-Pro (15.6). QEV-2B trails d3-nano on CV-Bench, CharXiv, InfographicVQA, Winoground and MMMU-Pro, and OneJev1.5-0.8B on BLINK.
• Scores are self-scored on the public part, single run, up to 1.6 MP per image. The private-part scores are unknown and may differ.
• Contamination: the training mix includes train splits of several benchmark families (Mind2Web, InfographicVQA, CORD, FUNSD, Hateful Memes). The audit above found 37 near-duplicate evaluation images (0.3% of 12,210 rows); removing them does not lower the score. If you find an overlapping item, please report it.
• The model returns probabilities, not guarantees. Calibration and abstention are fitted on internal DEV data and can be wrong on other domains; refit on your own data where the abstain decision matters.
• Visual reasoning is limited; see the evaluation notes in the QEV repository.
• Evaluation focuses on English.
• No larger model: training and evaluating a 4B QEV did not fit our hardware (one RTX 4060 Ti 8 GB), so we could not build or measure one. We would welcome compute support to do so.

License

Apache-2.0, the same license as the base model. The base model Qwen3.5-2B is subject to its upstream license.

The training data is not redistributed with the model. For the third-party sources, ken-jo/qev-data (vision-mix-2026-10/) publishes item ids, gold labels and hashes, plus the build scripts. It also publishes the synthetic set in full and the NLVR2 teacher targets. Each training source keeps its own terms; some, such as FUNSD and Hateful Memes, restrict how their data may be used. Check those terms before reusing the data or retraining on it.

Citation

@misc{jo2026qev2b,
  title        = {QEV-2B: A Calibrated Multimodal Decision Model},
  author       = {Jo, Ken},
  year         = {2026},
  howpublished = {Hugging Face model card},
  url          = {https://huggingface.co/ken-jo/QEV-2B}
}