guanxuyu/visual-jev-4b-answer-sft

Visual Jev 4B — answer-supervised adapter

A LoRA adapter for Qwen/Qwen3-VL-4B-Instruct, post-trained for forced-choice visual decisions: one image, one shared text context, and a set of questions whose candidate answers are supplied at request time. The model scores the candidates; it does not write prose.

📄 Paper · 💻 Code and quickstart · 🌐 Project page

Try it

The simplest way to try the adapter is with the repository's inference quickstart. It downloads the Qwen3-VL-4B base and this adapter from Hugging Face on first run. The included example asks six different questions about one image and includes an incorrect_question choice for false assumptions (selected with 0.966 probability for the dog/person mismatch shown above).

git clone https://github.com/guanxuyu-sv/Visual-Jev.git
cd Visual-Jev
python3 -m venv .venv && source .venv/bin/activate
pip install torch torchvision
pip install -r code/requirements.txt
python code/examples/quickstart.py \
    --image assets/figures/demo_bottle.jpg \
    --request-file code/examples/bottle_questions.json --device auto

On Apple Silicon, --device auto selects MPS. For CUDA, install a PyTorch build that matches your CUDA runtime before installing the remaining requirements. See the reproduction guide for the training and evaluation workflow.

This is the system our paper recommends, and it is deliberately the simplest one we trained. We also built typed decision heads on top of the same interface and measured no accuracy advantage from them (+0.000 macro against this adapter at matched data and budget, three seeds each). Those heads are therefore not part of this release. The finding is the reason this adapter is what we publish: adapt the backbone, keep the readout the backbone already has.

What it improves, and what it does not

Equal-weight macro accuracy over four benchmarks, mean ± spread over three seeds. † marks a benchmark no post-training ever saw.

Benchmark	Original backbone	This adapter	n
GQA-Choice	0.880	0.916 ± 0.002	5,447
SNLI-VE	0.629	0.808 ± 0.006	1,368
TextVQA-Choice †	0.974	0.975 ± 0.002	1,742
TallyQA-Choice †	0.340	0.345 ± 0.014	338
Macro	0.706	0.761 ± 0.002

Read the table honestly: the entire gain sits on the two task families represented in training. On the two held-out families the adapter is indistinguishable from the untrained backbone — TextVQA moves by 0.001 and TallyQA by 0.005 against a seed spread of 0.014. If your task resembles relational VQA or visual entailment, this adapter helps; if it resembles counting, it does not, and TallyQA at 0.345 against a six-option chance rate of 0.167 is weak in absolute terms for both models.

Loading

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

BASE = "Qwen/Qwen3-VL-4B-Instruct"
ADAPTER = "guanxuyu/visual-jev-4b-answer-sft"   # seed 0 lives at the root

model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16,
                                                    device_map="cuda")
model = PeftModel.from_pretrained(model, ADAPTER).eval()
processor = AutoProcessor.from_pretrained(BASE)

# the other two seeds:
#   PeftModel.from_pretrained(model, ADAPTER, subfolder="seed1")

Reproducing the numbers above requires more than these weights

The scores are candidate-token logits read from the backbone's own LM head at one fixed readout position, not free-form generation. Prompt layout, readout position, and candidate tokenisation determine the result, so a custom prompt measures a different setup. The public code repository includes the prompt construction, inference path, and paper reproduction guide; use those to reproduce the reported procedure.

Two further cautions from our own measurements:

• Decisions were measured in bfloat16. Path-to-path logit deviations we traced are arithmetic, not modelling: they collapse by four orders of magnitude in float32, and the worst disagreeing cases have a decision margin of 0.000.
• This adapter was trained on two candidate counts only, K in {2, 4}, while the evaluation above presents K from 2 to 8. That gap matters less here than it would for a typed head: the readout is candidate tokens through the backbone's own LM head, with no fixed slot per option, so there are no slots left without gradient. Measured on the varied-K GQA evaluation it reaches 0.916, against 0.919 for a slot-indexed head trained on the varied-K data itself. Treat wider candidate sets as untested coverage rather than as a known failure.

Training

Base	Qwen/Qwen3-VL-4B-Instruct, vision tower frozen (exclude_modules: ".*visual.*")
Method	LoRA on the language tower, r=16, α=32, dropout 0.05
Targets	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable	33,030,144 parameters
Objective	full-vocabulary next-token cross-entropy at the readout position
Steps	3,000, batch size 8, grad checkpointing, lr 1e-4, 100 warmup
Images	max_pixels 200,704 (196 visual tokens)
Data	question records derived from GQA and SNLI-VE training splits; GQA Choice items presented K in {2, 4} options
Seeds	0 (root), 1 (seed1/), 2 (seed2/)

No new human annotation was collected. The training records are derived from publicly released benchmarks; no images are redistributed here, and the adapter is a set of low-rank parameter deltas carrying no image data. Using it requires obtaining the backbone under its own licence.

Limitations

• Helps only on task families seen in training (see the table).
• Counting remains weak in absolute terms.
• Single-machine study on consumer GPUs; all figures are reported with seed spread and sample sizes.
• English only.
• Choice mode normalizes over the options supplied for each question; it has no built-in abstention outside that set. To let it reject a false premise or insufficient evidence, include an option such as incorrect_question or not enough evidence. The quickstart's incorrect_question option catches the dog/person mismatch with 0.966 probability. That score is relative to the supplied choices, not a calibrated measure of image evidence.
• Separately, our trained evidence-sufficiency output, which detects when relevant image evidence has been removed, reduced macro accuracy by 0.013. That result concerns a separate output objective; it does not rule out request-time reject options.