leinenterprises/Herbert_v0.5

Herbert v0.5

A LoRA on Qwen/Qwen3.8-27B for typed decisions over text and images. It continues from leinenterprises/jev-vision-27b and adds training on synthetic decision questions whose answers are known by construction.

Give it a state (text, images, or both), a question and a list of options; it returns a probability for every option from one forward pass. Nothing is generated.

Results

Decision Index 0.3, public vision suite (12,210 rows, all answered), official kit:

Benchmark	jev-vision-27b (base)	Herbert v0.5
CV-Bench	76.8	79.2
BLINK	54.6	72.8
RealWorldQA	72.8	73.4
CharXiv	78.9	77.4
InfographicVQA	96.2	96.7
Mind2Web	84.4	83.1
Winoground	83.5	84.0
KIE (CORD + FUNSD)	91.3	92.5
Hateful Memes	52.0	52.7
R-Bench-M	24.7	31.2
MMMU-Pro vision	38.0	42.3
Public vision index	73.06	75.69

Run files: leinenterprises/herbert-results.

How it answers

Options are labelled A, B, C, ... in the prompt; the answer is read from the label logits at the answer position and softmaxed over the listed options only. Up to 624 options per question (A-Z, a-z, 0-9, then single-token two-letter labels); larger questions are refused as unsupported, never truncated. Images up to 8 per request.

Run it

pip install vllm fastapi uvicorn peft transformers "decision-index[vision] @ git+https://github.com/apolinario/decision-index"
cd code
python -m train.merge --ckpt .. --out ../merged      # adapter -> plain bf16 model (one-time)
python -m serve.server --model ../merged --perms 1 --port 8000 --max-model-len 65536 --max-pixels 1638400

POST /v1/systemone (Decision Index wire format) or /v1/decisions (the kit's vision decide engine, images as a separate list).

Training

• Starts from the jev-vision-27b adapter (LoRA r=32, alpha=64 on the language model; vision tower frozen).
• Added data: synthetic decision questions generated by our own renderer (3D scenes with depth, point correspondence across views, camera motion, spatial relations and counts; charts drawn from known data; exam-style pages with computed answers), labels exact by construction. No benchmark items and no evaluation splits were used.
• The previous mix (text decision corpus plus train-split vision data, see the base model's card) is replayed alongside it.

Limits

Public-suite numbers are self-run with the official kit; the board's Full score also includes private tests run by the maintainers.