dormice

dormice is a fine-tune of Qwen/Qwen3.8-27B that answers multiple-choice questions about images with a probability for each option. This repository holds a low-rank update to the base model's language-model weights (dormice_update.safetensors, 1.4 GB), a 26-way linear head (dormice_head.safetensors) whose output is the answer, and a Decision Index vision engine (dormice.py) that puts them together. The base model is not copied here: the engine loads it from the Hub at the pinned revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0.

Two parts of the model are not optional:

• Attention. In the base model's full-attention layers, every token of the prompt attends to every token of the prompt; the recurrent (Gated DeltaNet) layers stay causal. The model was trained this way, and stock causal attention gives different answers. The engine registers the attention function with transformers' attention interface and runs it on PyTorch SDPA.
• The head. The answer is the head's output on the hidden state of the last prompt token. Nothing is decoded, and the letter logits are not used.

Usage

The engine needs the Decision Index kit and the packages pinned in requirements.txt (which installs the kit at the commit used here):

REPO=$(python -c "from huggingface_hub import snapshot_download; print(snapshot_download('plantwaterer/dormice'))")
pip install -r "$REPO/requirements.txt"
PYTHONPATH="$REPO" python -m decision_index vision pipeline --dir work/vision --engine dormice:Engine --workers 1 --compact --out runs/dormice

work/vision is the public vision suite, built with the kit (decision-index suite rebuild --vision --work work; see the kit's docs/vision.md). The engine takes no options for a board run. --option base=/path/to/snapshot loads the base model from a local directory instead of the Hub, and --option device=cuda:1 picks another GPU.

In Python:

import sys
from huggingface_hub import snapshot_download

repo = snapshot_download("plantwaterer/dormice")
sys.path.insert(0, repo)
from dormice import Engine                      # needs the decision_index kit installed

engine = Engine()                               # downloads the pinned base model on first use; bf16 on "cuda:0"
response, _ = engine(
    {},
    {"answer": {"type": "choice", "instructions": "Which object is closer to the camera?",
                "criteria": {"A": "the bicycle", "B": "the bench"}}},
    images=["street.jpg"])
print(response["answers"]["answer"])            # {"type": "choice", "choice": ..., "probabilities": {"A": ..., "B": ...}}

The inference procedure is as follows:

• Prompt: one user turn with the model's chat template: the images first, then the question, the options lettered A., B., ... one per line, and the line Answer with the option's letter from the given choices directly. Thinking is switched off (enable_thinking=False). When a row says its options are drawn in the image, the option lines are left out; when a row gives a read-out prefix, it is appended after the assistant header.
• Images: native resolution up to 1,638,400 pixels per image (the board's cap). The base processor's own lower cap is lifted to that value; larger images are downscaled by the kit to fit it.
• Read-out: one forward pass in bf16. The head (fp32) reads the last token's hidden state and gives one logit per option letter; the answer is the softmax over the question's options at temperature 1.
• One question per request, with up to 12 options lettered A to L (the kit's vision format; the head has 26 outputs). No truncation and no option filtering; nothing in the engine looks at the benchmark a row comes from.
• Warm-up at load. The recurrent layers' GPU kernels (flash-linear-attention, Triton) are compiled and tuned the first time a range of prompt lengths is seen: about 35 s for the first request on a new machine and about 9 s for the first request in each further range of 1,365 tokens, cached under ~/.triton afterwards. The engine runs synthetic requests of increasing length when it loads, so this one-off work is part of model loading for prompts of up to 8,192 tokens (the longest prompt in the public suite has 3,405). Longer prompts work; the first one in each further range is slower. --option warmup_tokens=N moves the limit and warmup_tokens=0 skips the warm-up.

Requirements: one GPU with room for the bf16 weights (54 GB) and the request; the public suite ran in about 58 GB (as reported by nvidia-smi during the run). Tested on 1 x RTX PRO 6000 (96 GB) and 1 x H200 with torch 2.14.1 (CUDA 13.0), transformers 5.12.1 and flash-linear-attention 0.5.2. bf16 results differ slightly between GPU types.

Training

dormice is a weighted average (weights 0.5, 0.25 and 0.25, applied to the low-rank updates and to the heads) of three fine-tunes of Qwen/Qwen3.8-27B. Each is a rank-16 LoRA on the linear layers of the language model, with the vision tower frozen, plus the 26-way head, trained on multiple-choice questions about images built from the public sources listed under Data attribution, in updates of 64 questions:

• the first for 3,000 updates on 192,000 questions;
• the other two for 1,000 updates each on two different mixes of 64,000 questions, both starting from the same earlier fine-tune (1,000 updates on 60,632 questions).

The repository stores the three updates separately, already multiplied by their weights (<layer>.A and <layer>.B, three rank-16 terms per matrix, 496 matrices); the engine adds them to the base weights in fp32 when it loads.

Evaluation

Decision Index vision 0.3, public suite: all eleven benchmarks (12,210 questions). The run used this repository's files and engine with the kit's runner and scorer at commit 25c6ec7b, in bf16, one request at a time: Winoground on 1 x H200, the other ten benchmarks on 1 x RTX PRO 6000 (96 GB). This is our own run of the public kit, not a leaderboard entry; the board's Full score also needs the maintainers' private tests. The run files are in runs/dormice-vision-0.3.

	public vision index	answered
all eleven benchmarks, this repository's files	78.38	12,210 / 12,210
the ten benchmarks other than Winoground	77.82	11,410 / 11,410

benchmark	questions	accuracy	skill
CV-Bench	2,038	88.42	80.90
BLINK	961	85.54	77.61
RealWorldQA	589	82.51	71.42
CharXiv	532	85.71	80.95
InfographicVQA	244	99.18	98.91
Mind2Web	973	95.17	93.56
Winoground	800	91.62	83.25
KIE (CORD + FUNSD)	2,465	98.05	96.79
Hateful Memes	1,213	77.41	54.82
R-Bench-M (weight 0.5)	665	41.65	29.97
MMMU-Pro vision (weight 0.25)	1,730	51.04	43.85

How to read these numbers:

• Two GPU types. Winoground was run on 1 x H200 and the other ten benchmarks on 1 x RTX PRO 6000, with the same files, engine and settings; runs/dormice-vision-0.3/parts keeps each part's environment and status. bf16 answers are not identical across GPU types, so a rerun on one GPU type will differ on some questions.
• Difference from our earlier read. An earlier read of the same weights through our training code on H200 GPUs gave 78.54 on all eleven benchmarks and 78.00 on the ten other than Winoground; this run gives 78.38 and 77.82. The answers are the same on 12,061 of the 12,210 questions: 147 of the ten benchmarks' 11,410 differ (90 of them MMMU-Pro vision questions) and 2 of Winoground's 800. On those 11,410 the number of input tokens is the same in both, and where the answers differ the two top options were close in the earlier read (median gap 0.03). To see how much of this is the GPU type, we also ran this repository's files on the 1,730 MMMU-Pro vision questions on an H200: the answers differ from this run's (RTX PRO 6000) on 87 questions and from the earlier H200 read on 44 (accuracy 51.27, against 51.04 and 51.79). So the GPU type changes more answers than the difference between this engine and our training code does, and neither is zero; we have not traced the second one further. The figures to go by for this repository's files are the ones in the tables above.
• This is not a state-of-the-art claim. This run's 78.38 is 0.03 below the 78.41 reported for RSI-JEV-27b, the model we compared against during development, and the earlier read's 78.54 is 0.13 above it. Differences that small are inside the noise of a single read: they do not show that either model is better. When this was written (2026-11-10), open submissions to the board reported public vision indexes of up to 82.91.
• Known limits. Hateful Memes (skill 55), R-Bench-M (30) and MMMU-Pro vision (44) are weak. On questions of those kinds the model is wrong often, and its probabilities should not be relied on.

Latency (our measurement; the board measures on its own private sample): the kit runner's per-request wall time for dormice:Engine on 1 x RTX PRO 6000 (96 GB), one request at a time, over the 11,410 questions of the ten benchmarks run there, started with empty kernel caches:

median	mean	80th percentile	95th percentile	slowest
199 ms	290 ms	423 ms	820 ms	1,372 ms

194 of the 11,410 requests took longer than 1,000 ms. Model loading, including the warm-up described above, took 109 s and is not part of these figures. The 800 Winoground requests on 1 x H200: median 342 ms, mean 350 ms, 80th percentile 355 ms, slowest 958 ms.

Overlap with evaluation data

No test or validation split of any benchmark in the suite was used for training, and no Winoground image or caption was used in training or in preparing the training data. Where a benchmark has an official training split, that split may be among the training sources (InfographicVQA, CORD, FUNSD, Multimodal-Mind2Web and Hateful Memes: train splits only). MMMU's dev and validation sets were left out because of their overlap with MMMU-Pro.

Training images were compared with the 10,742 images of the public suite that we hold as files (every benchmark except Winoground), by exact sha256 and by perceptual hash (pHash, distance of at most 6). There were no exact matches. Near matches were dropped from training: 694 in the first round of data (Hateful Memes 386, CORD 210, DocVQA 43, ChartNet 29, Mind2Web 11, others 15), 76 in the second and 27 in the third; later additions went through the same check. Winoground's images were not part of this comparison; the only training data of that kind are COCO captions with word-order-swapped negatives, built by us from COCO. The maintainers' private tests are not available to us, so no such check is possible against them.

Licence

The update, the head and the code in this repository are released under Apache-2.0. The base model, Qwen/Qwen3.8-27B, is Apache-2.0 and is not redistributed here. The training data come from the sources below, under the licences and terms listed; the images belong to their owners. Some of these sources carry use conditions that apply to the data itself; they are repeated here: InfographicVQA (research use only, ICDAR / RRC terms), FUNSD (non-commercial research and educational use), AI2D (non-commercial), ScienceQA (CC BY-NC-SA 4.0, non-commercial and share-alike), Hateful Memes (the dataset's own licence, research use) and TallyQA (treated here as non-commercial). None of that data is redistributed here.

Data attribution

dormice was trained on multiple-choice questions built from the public data sources below. Thanks to everyone who builds and maintains them. Records were reformatted into multiple-choice questions, and many were modified (answer options added, boxes or marks drawn on images, question text rendered into images), so none of the material should be read as an official copy of its source.

source	used for	reference	licence or terms
Hypersim	multi-view and correspondence questions	Roberts et al., 2021	CC BY-SA 3.0
MindCube (train)	spatial reasoning from several views	Yin et al., 2025	MIT
2.5VRD	relative depth of two objects	Su et al., 2021	annotations CC BY 4.0; images from Open Images V4
VSR (train)	spatial relations	Liu et al., 2023	CC BY 4.0; images from COCO
Nemotron-Image-Training-v3.5 (coco_count_spatial_qa, sat_spatial_qa)	counting, 2D relations, spatial questions	NVIDIA	OpenMDW-1.1; images from COCO
TallyQA (train)	counting	Acharya et al., 2019	Apache-2.0 (repository); treated here as non-commercial; images from Visual Genome and COCO
ChartNet (grounded_qa, reasoning_qa)	chart questions	ChartNet dataset card	CDLA-Permissive-2.0
DocVQA (train), via pixparse's copy on the Hub	document questions	Mathew et al., 2021	MIT (the copy's licence)
InfographicVQA (train)	infographic questions	Mathew et al., 2022	research use only (ICDAR / RRC terms)
CORD v2 (train)	receipt fields	Park et al., 2019	CC BY 4.0
FUNSD (train)	form fields	Jaume et al., 2019	non-commercial research and educational use
AI2D	science diagrams	Kembhavi et al., 2016	non-commercial (the release's licence file)
ScienceQA (train)	science questions	Lu et al., 2022	CC BY-NC-SA 4.0
Multimodal-Mind2Web (train)	web-page actions	Deng et al., 2023	annotations CC BY 4.0; OpenRAIL (dataset card)
Hateful Memes (train)	meme moderation	Kiela et al., 2020	Hateful Memes dataset licence (research use)
HateCheck, DynaHate, Measuring Hate Speech, Gab Hate Corpus, Contextual Abuse Dataset	text for synthetic memes, drawn by us on DOCCI photos	Röttger et al., 2021; Vidgen et al., 2021; Kennedy et al., 2020; Kennedy et al., 2022; Vidgen et al., 2021	CC BY 4.0
DOCCI	photos for the synthetic memes	Onoe et al., 2024	CC BY 4.0
Visual Genome	question answering on photos	Krishna et al., 2017	CC BY 4.0
VQA v2 (train)	question answering on photos	Goyal et al., 2017	CC BY 4.0; images from COCO
COCO (train2014, train2017), images and captions	images for the sources above; caption-matching questions with word-order-swapped negatives	Lin et al., 2014	annotations CC BY 4.0; images under their Flickr licences

Of the 192,000 questions in the largest of the three training sets, 56,674 come from the non-commercial sources named under Licence.