QEV-0.8B

A 0.8B open multimodal decision model: text and image in, calibrated option probabilities out, in one forward pass with no generated tokens.

QEV-0.8B is the small member of the QEV family. It is built on Qwen3.5-0.8B and distilled, like QEV-2B, from the QEV 0.2.1 teacher (see Training data). You supply the evidence (text, one or more images) and the candidate options with their meanings. The model returns a probability for each option, a typed answer, and an abstain signal.

Architecture, training, evaluation protocol, calibration, speed, ablations and limitations are described in the technical report.

Headline (self-scored, pending official run): On the JEV Decision Index vision board (edition 0.3.2, public part), QEV-0.8B scores 53.02: second on the board and narrowly ahead of Intern-Decision-2B (2.27B, 52.92) with about a third of its parameters, and the highest score among the board's models at or below 1.2B parameters. It also leads three benchmarks against models up to 2B: Mind2Web, KIE and Moderation.

Results

Leaderboard comparison

JEV Decision Index, vision, edition 0.3.2, public part. Board generated 2026-10-16.

Model	Params	Index (public)
d3-nano	2.21B	59.15
QEV-0.8B (self-scored)	0.8B	53.02
Intern-Decision-2B	2.27B	52.92
OneJev1.5-0.8B	1.11B	52.32
d3-lite	0.85B	47.88
JPT-0.8B	0.87B	44.55
Intern-Decision-0.8B	0.87B	38.96
d3-edge	0.59B	36.61

_{Board values are copied as published (JEV Decision Index vision, edition 0.3.2, public part, generated 2026-10-16). The QEV-0.8B value is self-scored, pending official run: public evaluation kit, native resolution up to 1.6 MP per image (no shared multi-image budget), single run, full depth, scored on the exact released files. The 0.10 margin over Intern-Decision-2B is well inside the 95% bootstrap interval (51.51 to 54.59), so the two are tied. It is not an official leaderboard entry and will be replaced when the official run is available.}

Where QEV-0.8B leads (self-scored)

Benchmark	QEV-0.8B	Comparison (board, public)
Mind2Web	71.4	d3-nano (2.21B) 63.7
KIE	88.9	Intern-Decision-2B (2.27B) 84.4
Moderation	31.9	24.7 (2B-class model)
MMMU-Pro	9.6	highest among models at or below 1.2B

Per-benchmark scores

Index = weighted mean of the benchmarks below. Weights: 1.0 unless noted. R-Bench-M 0.5, MMMU-Pro 0.25.

Benchmark	QEV-0.8B (self-scored)	Previous release	Weight
CV-Bench	60.0	54.6	1.0
BLINK	39.1	28.3	1.0
RealWorldQA	39.2	35.1	1.0
CharXiv	55.1	49.4	1.0
InfographicVQA	75.4	44.8	1.0
Mind2Web	71.4	0.2	1.0
Winoground	47.5	46.0	1.0
KIE	88.9	70.2	1.0
Moderation	31.9	24.8	1.0
R-Bench-M	11.9	11.4	0.5
MMMU-Pro	9.6	7.1	0.25
Index	53.02	37.02

Evaluation: public evaluation kit, native resolution up to 1.6 MP per image (no shared multi-image budget), single run, full depth, on the released files (merged weights). 95% bootstrap interval of the index (items resampled within each benchmark, 2,000 replicates): 51.51 to 54.59. With rows that share an image resampled together: 51.36 to 54.56. QEV-2B minus QEV-0.8B on the same resampled items: 8.97 (7.39 to 10.48). Benchmarks were not used for checkpoint selection.

Speed and fast mode

Mode	Image cap	Depth	Worst-case p95	Median p50	Peak GPU memory
Accurate (default)	1.6 MP	24 layers	377.3 ms	66.0 ms	1.86 GiB
Fast	0.26 MP	16 layers	123.2 ms	44.6 ms	1.60 GiB

_{Release loader only (qev.QEV.load + predict, no CUDA graphs or custom kernels), RTX 4060 Ti 8 GB, batch 1, warm. 47 request profiles (text and image requests with up to 16 options; the largest is 3,229 input tokens in accurate mode), 3 warm-up and 30 timed runs each. Worst-case p95 is the highest per-profile p95; median p50 is the median over profiles of each profile's p50. Peak GPU memory is PyTorch's peak allocation. In both modes the slowest profile is a 16-option image request near the token limit.}

Fast mode exits at layer 16 of 24 with a distilled exit head and caps images at 0.26 MP. At the same 0.26 MP input, accuracy matched full depth on our internal DEV set and a mini benchmark subset. Against accurate mode (1.6 MP), DEV accuracy is 82.0% versus 86.0%; the gap comes from the lower resolution and is largest on document, chart and screen sources. No index is reported for fast mode, so the numbers above apply to the full-depth mode only.

Calibration and abstention

Per-type temperatures and abstain thresholds are fitted on the DEV selection split (the split also used for checkpoint selection; benchmarks are never used), on group-disjoint halves: temperature on one half, threshold on the other. Policy: the lowest confidence threshold at which answered questions have at most 15% error, with at least 20 answered. Numbers are on the held-out policy half, accurate mode. A response always carries the answer; abstained is a separate flag, so benchmark scores are unaffected.

Question type	DEV questions	Temperature	Threshold	Answered	Error on answered	ECE (raw -> calibrated)
Choice	1,296	1.00	0.30	100%	13.5%	0.042 -> 0.042
Score	79	0.79	0.39	100%	13.9%	0.125 -> 0.093
Yes/no	80	0.77	0.82	51%	14.6%	0.103 -> 0.068

For choice and score questions the 15% target is met on all DEV questions, so abstention only triggers below the lowest confidence seen on DEV and is rare. The score and yes/no thresholds rest on about 80 DEV questions each (internal text diagnostics) and are less certain than the choice threshold. The calibration is fitted for accurate mode. In fast mode the same thresholds answer 97% of choice questions at 16.0% error and 71% of yes/no questions at 19.3% error.

Architecture

• Backbone: Qwen3.5-0.8B language model (hidden size 1024, 24 layers: 18 GatedDeltaNet + 6 full attention).
• Vision: Qwen3.5 ViT encoder (frozen).
• Readout: option probabilities read from the hidden states of the option tokens.
• Reasoning slots: 4 slots plus a result marker.
• Binding head, per-type temperature calibration and an abstain decision (fitted on DEV; see Calibration and abstention).
• Single forward pass, no generated tokens.
• Fast mode: a distilled exit head reads the options at layer 16 of 24 (exit16.safetensors); images are capped at 0.26 MP.
• Fine-tuning: LoRA (rank 8, alpha 16) on the language model plus trained readout and binding heads; the vision encoder and the other base weights are frozen (board category: LoRA + head).
• Weights are shipped merged (LoRA folded into the base weights).

Training data

Every source used in training, read from the training script (qev_mix.py) and its training log.

Coverage mix (train split, 18 sources). Each source contributes items from its train split. 5% of each source (up to 120 items) is held out as a selection set and is never trained on.

Source	Split
AI2D	train
A-OKVQA	train
ChartQA	train
CORD	train
DocVQA	train
FUNSD	train
Hateful Memes	train
IconQA	train
InfographicVQA	train
Mind2Web	train
Mind2Web (keyed variant, mind2web_k)	train
Options-in-image	train
ScienceQA	train
Screen2Words	train
Synthetic (internal)	train
TallyQA	train
Visual7W	train
VSR	train

Additional sources

• NLVR2 train split (teacher-filtered targets). The NLVR2 dev split is used only for checkpoint selection.
• Replay of internal decision units (image and text families, train split), with teacher outputs as targets.
• Teacher outputs: option-probability distillation from QEV 0.2.1, the earlier QEV-2B release (ken-jo/qev at revision 0d2d13f): a LoRA (rank 8, alpha 16) with a condition-bound binding head on Qwen3.5-2B. The same teacher was used for QEV-2B and QEV-0.8B; it is not the released QEV-2B checkpoint.

Deduplication and contamination audit: all sources use train splits only (evaluation uses test/validation splits). Ten document/KIE/perception sources were filtered at build time: any item whose image is within Hamming distance 4 (64-bit dHash) of an evaluation image was dropped. A post-hoc audit of every training image against every evaluation image (dHash candidate, confirmed by 64x64 grayscale mean absolute difference <= 6/255) found **37 evaluation items** with a near-identical training image: 30 Hateful Memes (shared base images across the dataset's splits), 6 BLINK, 1 Mind2Web. Mind2Web has no action_uid or annotation_id overlap with the evaluation rows. Rescoring without those 37 items changes the index from 53.02 to 53.03 (Moderation 31.90 -> 32.21, BLINK 39.11 -> 38.97, Mind2Web 71.36 -> 71.33).

Teacher cleanup: a teacher target is kept only where the teacher's top option matches the gold answer. Otherwise only the gold label is trained on.

Selection: checkpoint chosen on held-out DEV data only (coverage-mix holdout, NLVR2 DEV, internal diagnostics). Public benchmarks were not used for selection.

Training recipe: coverage mix plus replay, 56k items; the released checkpoint is optimizer step 3,616 (end of the cosine schedule, about two thirds of one pass). Resolution-elastic batches (0.26 / 1.05 / 1.64 MP). Answer options are permuted per visit, except for sources whose options refer to letters or boxes in the image (Mind2Web, Mind2Web-keyed, options-in-image). The early-exit head is distilled from full depth (KL plus 0.3 CE).

Quickstart

The 0.8B checkpoint needs the qev source (the published qev==0.2.1 package loads QEV-2B 0.2.1 only).

python -m pip install "qev @ git+https://github.com/ken-jo/qev" huggingface_hub

from huggingface_hub import snapshot_download
import qev
from qev import DecisionRequest

path = snapshot_download("ken-jo/QEV-0.8B")
model = qev.QEV.load(path, device="cuda")               # accurate: full depth, up to 1.6 MP per image
# model = qev.QEV.load(path, device="cuda", mode="fast")  # fast: exit at layer 16, 0.26 MP
request = DecisionRequest.model_validate({
    "state": {
        "text": "Is this photo of a parked car blocking a driveway?",
        "images": [{"path": "photo.jpg"}],
    },
    "questions": {
        "blocking": {
            "type": "choice",
            "instructions": "Does the car block the driveway?",
            "criteria": {"yes": "The car blocks the driveway.", "no": "The driveway is clear."},
        }
    },
})
print(model.predict(request)["answers"])

Note on the name: the PyPI package qev is also the import name used by OneJev (from qev import Client). Install one of them per environment.

Limitations

• Weak areas: BLINK (39.1), Winoground (47.5), RealWorldQA (39.2). QEV-0.8B trails OneJev1.5-0.8B on BLINK, Winoground and CharXiv.
• Scores are self-scored on the public part, single run, up to 1.6 MP per image. The private-part scores are unknown and may differ.
• Contamination: the training mix includes train splits of several benchmark families (Mind2Web, InfographicVQA, CORD, FUNSD, Hateful Memes). The audit above found 37 near-duplicate evaluation images (0.3% of 12,210 rows); removing them does not lower the score. If you find an overlapping item, please report it.
• The model returns probabilities, not guarantees. Calibration and abstention are fitted on internal DEV data in accurate mode and can be wrong on other domains; refit on your own data where the abstain decision matters.
• Input is supplied per request. Candidate meanings are defined by the caller, and the model scores only the options you provide.
• Evaluation focuses on English.
• No larger model: training and evaluating a 4B QEV did not fit our hardware (one RTX 4060 Ti 8 GB), so we could not build or measure one. We would welcome compute support to do so.

License

Apache-2.0, the same license as the base model. The base model Qwen3.5-0.8B is subject to its upstream license.

The training data is not redistributed with the model. For the third-party sources, ken-jo/qev-data (vision-mix-2026-10/) publishes item ids, gold labels and hashes, plus the build scripts. It also publishes the synthetic set in full and the NLVR2 teacher targets. Each training source keeps its own terms; some, such as FUNSD and Hateful Memes, restrict how their data may be used. Check those terms before reusing the data or retraining on it.

Citation

@misc{jo2026qev08b,
  title        = {QEV-0.8B: A Calibrated Multimodal Decision Model},
  author       = {Jo, Ken},
  year         = {2026},
  howpublished = {Hugging Face model card},
  url          = {https://huggingface.co/ken-jo/QEV-0.8B}
}