Add LEXam (mcq_4_choices) eval result

#5
vLLM Semantic Router org

Why: d3 has no result on LEXam, a registered Hub benchmark whose 4-choice task is closed-set and fits a decision model directly.

Benefit: 61.75 on mcq_4_choices (all 1,655 items, 0 errors), second on the current board.

How: one /v1/systemone request per item with a single choice question; the answer is the argmax of the returned probabilities, with no sampling and no chain of thought. vLLM nightly on one A100 80GB, BF16. Model, dataset and image revisions are pinned in the YAML and the evidence repo: https://huggingface.co/datasets/HappyWindman555/d3-hub-evals

Trade-offs: the other entries are generative models, mostly with thinking and sampling, so the protocols differ. The YAML notes field says so. German items score 50.8 and English items 80.1.

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment