leinenterprises/Herbert_v0.8

Herbert v0.8

A LoRA on Qwen/Qwen3.8-27B for typed decisions over text and images. It continues from Herbert v0.5 with three more training stages: a text decision mix, self-distillation from the base model's own reasoning, and synthetic plus annotation-derived visual questions.

Give it a state (text, images, or both), a question and a list of options; it returns a probability for every option from one forward pass. Nothing is generated.

Try it: live demo.

Results

Decision Index 0.3, public vision suite (12,210 rows, all answered), official kit:

Benchmark	jev-vision-27b	Herbert v0.5	Herbert v0.8
CV-Bench	76.8	79.2	81.5
BLINK	54.6	72.8	76.2
RealWorldQA	72.8	73.4	72.5
CharXiv	78.9	77.4	77.9
InfographicVQA	96.2	96.7	96.2
Mind2Web	84.4	83.1	86.3
Winoground	83.5	84.0	83.0
KIE (CORD + FUNSD)	91.3	92.5	92.5
Hateful Memes	52.0	52.7	50.4
R-Bench-M	24.7	31.2	35.4
MMMU-Pro vision	38.0	42.3	42.5
Public vision index	73.06	75.69	76.38

Run files: leinenterprises/herbert-results.

jev-bench (22,773 independent public decisions, scored with Jevify, base in 4-bit): macro accuracy 0.747, ECE 0.074 (jev-vision-27b 0.734 / 0.084; Jev 1.13.0 0.733 / 0.113 per the Jevify README). Run files.

Text: on a fixed 5,018-question sample of the public text suite (42 benchmarks), v0.8 scores within 0.5 points on average of its text-trained intermediate stage; the full text suite has not been run on this version yet.

How it answers

Options are labelled A, B, C, ... in the prompt; the answer is read from the label logits at the answer position and softmaxed over the listed options only. Up to 624 options per question (A-Z, a-z, 0-9, then single-token two-letter labels); larger questions are refused as unsupported, never truncated. Images up to 8 per request.

Run it

pip install vllm fastapi uvicorn peft transformers "decision-index[vision] @ git+https://github.com/apolinario/decision-index"
cd code
python -m train.merge --ckpt .. --out ../merged      # adapter -> plain bf16 model (one-time)
python -m serve.server --model ../merged --perms 1 --port 8000 --max-model-len 65536 --max-pixels 1638400

POST /v1/systemone (Decision Index wire format) or /v1/decisions (the kit's vision decide engine, images as a separate list).

Training

LoRA r=32, alpha=64 on the language model; vision tower frozen. Starting from Herbert v0.5:

1. Text decisions. A mix of public decision corpora (train splits only, licence-filtered, decontaminated against the Decision Index text suite by state and question hashes).
2. Self-distillation. Qwen3.8-27B with thinking on answered ~12k train-split questions; its answer distribution after reasoning became the target for the one-pass readout.
3. Visual questions with known answers. Our own renderer (3D scenes, charts, exam pages) plus questions built from COCO 2017 train annotations (body keypoints across different people, object counts, box-level spatial relations).

No benchmark items or evaluation splits were used, and every training image was near-duplicate filtered (perceptual hash) against the evaluation images.

Limits

Public-suite numbers are self-run with the official kit; the board's Full score also includes private tests run by the maintainers. Text is measured on a sample only; on the public text board this family trails text-first models.