MANTA-27B

MANTA (Medical Answers, Natively from Text And images) is a medical decision model developed by the Gevaert Lab at the Department of Computational Medicine, Stanford University. Give it a medical image or text and a set of questions with their answer options: MANTA-27B returns a probability for every option of every question in one forward pass.

MANTA comes in two releases: MANTA-27B (this repository), the more accurate, and MANTA-4B, smaller and faster. Every results table of this card shows both releases; the text and the ranks speak of MANTA-27B.

Release	Backbone	GPU memory	Median latency	MANTA-Bench medical
MANTA-27B (this release)	Qwen3.8-27B	about 60 GB	148 ms	86.0 %
MANTA-4B	Qwen3.5-4B	about 16 GB	68 ms	85.3 %

MANTA-27B builds on the Qwen3.8-27B vision-language backbone and adds a dedicated decision head, trained from scratch, that scores every answer option directly, instead of generating text. Both the language and the vision towers are adapted end-to-end in a multi-stage training process designed for calibrated option probabilities.

MANTA-27B was trained on about 8.5 million decision requests, 68 % of them medical (text and images), with about 1,800 H100 GPU-hours of training steps.

• Medical decisions (MANTA-Bench): MANTA-27B 86.0 % accuracy, rank 1 of 20 decision models (best non-MANTA model: liquid-d1-ultra, 71.0 %); ECE-15 0.021 on the in-domain test sets.
• Frontier performance on clinical decisions on the same medical decisions (6,700 questions): MANTA-27B 86.1 % vs Gemini 3.8 Flash 76.0 %, Claude Sonnet 5.5 74.6 %, GPT-6 Luna (chat) 62.4 %, MedGemma 27B (local) 52.3 %, in one forward pass.
• Open-weight models (mean over the 6 rows of the summary table; A.7 marks the open models; Clef counts once, through its API and on our GPUs): MANTA-27B has the highest mean accuracy of the 11 open-weight models we compared (76.1 %; next: mammoth-1, 75.5 %); first among them on medical decisions (with MANTA-4B), out-of-domain medical data and general decisions; mammoth-1 first on medical QA, medical knowledge and reasoning and the exam sets with images.
• Medical QA (MedQA, MedMCQA, PubMedQA, HEAD-QA): MANTA-27B mean 83.6 %, rank 4 of 27 decision models.

_{Accuracy: four rows of the summary table in Evaluation (MANTA-Bench medical and general, medical QA, the exam sets; the same questions for every model). Latency: the median per request; API models: the server-side time that OpenRouter reports (it excludes the client network); MANTA-27B, MANTA-4B and MedGemma 27B: one H100 at batch size 1. For the QA groups and the exam sets, the median of the requests of that group.}

MANTA-27B is a research model, not a medical device. Do not use its outputs for diagnosis, treatment or any other decision about a patient without independent validation on your own data and review by qualified clinicians.

Model details

Developed by	Gevaert Lab, Department of Computational Medicine, Stanford University
Model type	Decision model (no text generation)
Backbone	Qwen3.8-27B (vision-language), adapted
Decision head	new, trained from scratch
Parameters	27.5B (BF16, 12 safetensors shards, 55.0 GB; sha256 in SHA256SUMS)
GPU memory	about 60 GB (one GPU)
Input	a case (text or JSON), optional images, 1 to 64 questions with their options; up to 4,096 prompt tokens
Output	a calibrated probability per answer option, in one forward pass
License	Apache 2.0 (LICENSE)
Base model license	the license of Qwen3.8-27B: LICENSE_base_model

Modalities

The training data consists of images and text. Physiological signals are rendered as images. 3D CT and MRI volumes are used as 2D slices, and medical videos as single frames.

Medical

Radiology

• X-ray: chest X-ray, digital radiography, mammography, panoramic dental radiography
• Computed tomography (CT): conventional and contrast-enhanced CT, micro-CT
• Magnetic resonance imaging (MRI): T1-weighted, contrast-enhanced T1, T2-weighted, FLAIR, proton density, diffusion-weighted, perfusion maps (CBF, CBV, MTT, Tmax, TTP), cardiac MRI
• Positron emission tomography (PET), FDG PET/CT
• Ultrasound: B-mode, transthoracic echocardiography, transrectal ultrasound (3D and micro-ultrasound)
• Angiography

Pathology and microscopy

• Histopathology: whole-slide imaging, histopathology and cytopathology images
• Cytology
• Light microscopy: brightfield, phase contrast
• Fluorescence microscopy: epifluorescence, confocal, synthetic fluorescence images
• Imaging flow cytometry
• Electron microscopy: transmission, scanning, serial-section, focused ion beam SEM, cryo-electron tomography

Endoscopy and surgery

• Colonoscopy, white-light endoscopy, wireless capsule endoscopy, bronchoscopy, cystoscopy (white-light and narrow-band imaging), laparoscopy, surgical microscopy

Ophthalmology

• Optical coherence tomography (OCT), fundus photography, ultra-widefield fundus photography, slit-lamp photography, fluorescein angiography

Dermatology

• Dermoscopy

Other optical imaging

• Hyperspectral imaging (RGB reconstructions), optoacoustic tomography

Physiological signals (rendered as images)

• Electrocardiography: 12-lead ECG plots rendered from signals, ECG printout images
• Electroencephalography: polysomnography (sleep recordings), scalp EEG
• Heart sounds (phonocardiography), lung sounds (auscultation), cough recordings: log-mel spectrograms

Text

• Biomedical and clinical text: exam and textbook questions, research abstracts and claims, clinical-trial records, drug labels and reviews, adverse-event and device reports, biomedical knowledge-base facts (genes, variants, drugs, diseases)

Evaluation only (in held-out evaluation data, not in training)

• Colour Doppler ultrasound, OCT angiography, colposcopy, confocal laser endomicroscopy, near-infrared imaging, super-resolution, TIRF, multiphoton, darkfield and specular microscopy

General domain

• Text: general-domain text in many languages, including code, tables and chess puzzles
• Images: natural photographs (incl. street-level views), satellite images, synthetic 3D scenes, diagrams, charts, maps, memes, AI-generated images
• Video: short clips as sequences of sampled frames
• Document scans and GUI screenshots are excluded

Quickstart

hf download gevaertlab/MANTA-27B --local-dir MANTA-27B && cd MANTA-27B
pip install -r requirements.txt    # the versions of the published evaluation (Python 3.12, CUDA 12.4)
python example.py                  # the two example requests in examples/

from manta import Manta            # request encoding and probabilities (manta/ in this repository)

model = Manta.from_pretrained(".")  # one GPU with 80 GB
result = model.predict({
    "state": "67-year-old man, 3 days of fever, productive cough, right pleuritic chest pain. SpO2 89 % on room "
             "air. Chest radiograph: right lower lobe consolidation.",
    "questions": {
        "diagnosis": {"type": "choice", "instructions": "What is the most likely diagnosis?",
                      "criteria": {"pneumonia": "Community-acquired pneumonia",
                                   "pulmonary_embolism": "Pulmonary embolism",
                                   "heart_failure": "Acute decompensated heart failure"}},
        "hypoxemia": {"type": "noul", "instructions": "Does the patient have hypoxemia?"},
        "severity": {"type": "score", "instructions": "How severe is the illness?",
                     "criteria": ["mild", "moderate", "severe", "critical"]},
    },
})
print(result["questions"]["diagnosis"]["probabilities"])

The model itself loads with transformers (AutoModel.from_pretrained(path, trust_remote_code=True, dtype=torch.bfloat16)); Manta adds the prompt format of the published evaluation and the softmax per question. A case longer than the prompt limit is cut at the end (truncated in the output).

Request field	Content
state	the case: a string or any JSON value
images	optional: file paths, URLs or data URIs, in prompt order
questions	question id → {"type", "instructions", "criteria"}, decided jointly; up to 64
choice criteria	option id → description; the answer is one option id
score criteria	ordered level descriptions, lowest first; option ids "0", "1", …
noul (yes/no)	criteria optional; option ids "true", "false"

Use short task names as question ids (diagnosis, modality) and put the questions about one case into one request. Choice options are shown in the fixed random order of the evaluation (option_order="eval"; "given" and "sorted" are available).

Evaluation

MANTA-Bench, our benchmark of decision questions from public data sets, has in-domain test sets (held-out test groups of the training sources) and out-of-domain test sets (data sets that no MANTA training used).

MANTA-27B (this release) and MANTA-4B against the best other decision model and four chat models: GPT-6 Luna (chat), Claude Sonnet 5.5 and Gemini 3.8 Flash through their APIs, at the lowest reasoning setting each API accepts, and MedGemma 27B, run locally. Every model of a row answers the same questions. MANTA answers in one forward pass, without reasoning; the other decision models answered zero-shot, through their APIs (Clef also with its open weights on our GPUs). Bold: the best value of the row. The appendix has the full tables: every model, all questions, calibration.

Benchmark	Questions	MANTA-27B	MANTA-4B	Best other decision model	Gemini 3.8 Flash	Claude Sonnet 5.5	GPT-6 Luna (chat)	MedGemma 27B (local)
MANTA-Bench medical	6,700	86.1	86.1	71.2 (liquid-d1-ultra)	76.0	74.6	62.4	52.3
MANTA-Bench medical, out-of-domain test sets	554	74.4	71.5	73.3 (liquid-d1-ultra)	73.6	79.1	69.9	52.3
Medical QA (MedQA, MedMCQA, PubMedQA, HEAD-QA)	1,978	82.7	75.0	85.9 (mammoth-1)	90.0	87.5	83.0	71.9
Medical knowledge and reasoning (MMLU-medical, MedXpertQA-Text)	976	59.5	51.4	73.7 (mammoth-1)	78.8	71.9	64.8	47.9
Medical exam sets with images	1,005	63.1	54.2	70.2 (liquid-d1-ultra)	87.1	78.6	63.3	47.5
MANTA-Bench general	5,792	90.7	90.1	86.8 (pplx-decider-v1.2)	88.5	86.0	79.6	74.0

_{Accuracy in %. Questions: the comparison subsets (MANTA-Bench: at most 20 requests per medical source and 50 per general source; QA suite and exam sets: at most 500 requests per benchmark), the questions that all models of the row answered. Medical QA: the mean of the benchmarks of the row, equal weight; exam sets: the mean of MedXpertQA-MM and WorldMedQA-V. Out-of-domain: data sets that no MANTA training used. MedGemma 27B ran locally (vLLM, temperature 0).}

_{MANTA-Bench general row: Not ranked (answered fewer than 90 % of the questions because of provider refusals): drex-max (54 %).}

Examples: from dataset labels to decision questions

Every training and evaluation request is converted from the original annotations of a public data set. The labels of a release (CSV rows, label files, folder names, boxes and masks) are first mapped to canonical label strings; fixed rules then turn each label into a question with its allowed options and target.

Question	Made from	Example
choice	any label space (also two-class spaces such as benign / malignant); above 24 classes, a seeded menu of 24 options that always holds the correct one	the finding of an endoscopy image
noul (yes / no)	labels that are yes / no answers (yes / no, true / false, present / absent); presence questions "Is there a class in the image?" for the classes of a box or mask label space that lists every instance	a biomarker present in an OCT scan
score (ordered levels)	a label space with ordered levels (lowest first); counts, in the bins 0, 1, 2, 3–4, 5–9, 10–19, 20–49, 50 or more	a stenosis grade, a cell count

• Regions. A box or mask becomes text coordinates in the question, on a 0–1000 scale relative to the image: {"bbox_2d": [x1, y1, x2, y2]}, or {"point_2d": [x, y]} for a thin or overlapping structure. Nothing is drawn on the image. Regions that are too small or too large are left out.
• Targets. Each target records where it comes from: human (the label of the release), derived (computed from the annotations, for example a presence or a count), votes (annotator votes). Captions and free-text answers are converted by a language model instead (target source model).
• Evaluation. The question ids are task names without the data-set name (disease, finding_in_box.box), and the options of a choice question are shown in a fixed random order (option_order="eval").

The four examples below come from the out-of-domain data sets of MANTA-Bench (A.1.1), which no MANTA training used. Each shows the image as the model receives it, the original annotation of the release, the labels we derive from it, and the request as it is evaluated, in the request format of the Quickstart. Each request asks more questions than shown here (for example the imaging technique); the examples keep one or two of them.

OLIVES (macular OCT B-scans): a table row → a choice and a noul

_{OLIVES (Prabhushankar et al., NeurIPS 2022 Datasets and Benchmarks; Zenodo 10.5281/zenodo.7105232), CC BY 4.0; resized.}

Original annotation (one row of the release, selected columns):

Disease Label: 1.0    DRT/ME: 1.0    IR HRF: 1.0    Fluid (IRF): 1.0    Fluid (SRF): 0.0    Vitreous debris: 0.0

Disease Label 1 is diabetic macular edema (release README); a biomarker column is 1 = present, 0 = absent. Our labels: disease: diabetic macular edema, bio_intraretinal_hyperreflective_foci: present, bio_subretinal_fluid: absent, ... (11 biomarkers).

{
  "state": {"context": "An optical coherence tomography (OCT) image of the retina of a human is provided (cross-sectional view)."},
  "images": ["olives.jpg"],
  "questions": {
    "disease": {
      "type": "choice",
      "instructions": "Based on the retinal microstructural appearance in this optical coherence tomography image, is the underlying disease diabetic retinopathy or diabetic macular edema?",
      "criteria": {"diabetic macular edema": null, "diabetic retinopathy": null}
    },
    "bio_intraretinal_hyperreflective_foci": {
      "type": "noul",
      "instructions": "In this optical coherence tomography B-scan of the macula, is intraretinal hyperreflective foci present or absent?",
      "criteria": {"true": "present", "false": "absent"}
    }
  }
}

Targets: disease = diabetic macular edema (human); bio_intraretinal_hyperreflective_foci = true (human). The biomarker labels are present / absent, so they become yes / no questions, with "present" and "absent" as the descriptions of true and false. No score: the labels have no order.

GRAZPEDWRI-DX (paediatric wrist radiographs): YOLO boxes → a region choice and a presence noul

_{GRAZPEDWRI-DX (Nagy et al., Sci Data 9, 222 (2022); figshare 14825193), CC BY 4.0; resized. The blue box is the region of the finding_in_box.box question, drawn here for the reader only: the model gets the coordinates as text, not a drawn box.}

Original annotation (the YOLO label file of the image: class, x-centre, y-centre, width, height, relative to the image; class 3 = fracture, 5 = periosteal reaction, 8 = text):

3 0.529747 0.457207 0.251899 0.121622
8 0.036076 0.420045 0.056962 0.076577
5 0.468354 0.571509 0.083544 0.161036
5 0.632911 0.540541 0.032911 0.209459

Our labels: fracture: fracture visible; boxes: fracture ×1, periosteal reaction ×2 (the text box is skipped). The fracture box becomes [404, 396, 656, 518] on the 0–1000 scale (x1 = 0.529747 − 0.251899 / 2 = 0.404, ...).

{
  "state": {"context": "A digital radiograph (x-ray imaging) of the wrist of a pediatric human patient is provided.",
            "age": "15 years", "sex": "male"},
  "images": ["grazpedwri_dx.jpg"],
  "questions": {
    "finding_in_box.box": {
      "type": "choice",
      "instructions": "Within the region of interest ({\"bbox_2d\": [404, 396, 656, 518]}), what is the best-matching radiographic finding?",
      "criteria": {"bone anomaly": null, "bone lesion": null, "foreign body": null, "fracture": null, "metal": null,
                   "periosteal reaction": null, "pronator quadratus sign": null, "soft tissue swelling": null}
    },
    "finding_box.fracture.present": {
      "type": "noul",
      "instructions": "Is there a fracture in the image?"
    }
  }
}

Targets: finding_in_box.box = fracture (human); finding_box.fracture.present = true (derived: the image has a fracture box, and the box classes list every finding of the image). The same request asks the presence of each of the 8 findings (here periosteal reaction yes, the six others no) and the finding in the second periosteal-reaction box. No score: the findings have no order and no count is asked.

CADICA (invasive coronary angiography): a stenosis box → a choice and a region score

_{CADICA (Mendeley Data p9bpx9ctcv, version 5), CC BY 4.0; resized. The orange box is the region of the stenosis_degree.box question, drawn here for the reader only: the model gets the coordinates as text, not a drawn box.}

Original annotation (the ground-truth file of the keyframe: x, y, width, height in pixels of the 512 × 512 frame, and the stenosis degree; the video is listed among the lesion videos of the patient):

176 286 49 53 p0_20

p0_20 is a stenosis below 20 %. Our labels: lesion: coronary lesion visible; boxes: coronary artery stenosis ×1. The box becomes [344, 559, 439, 662] (x1 = 176 / 512 = 0.344, ...).

{
  "state": {"context": "A biomedical image is provided."},
  "images": ["cadica.jpg"],
  "questions": {
    "lesion": {
      "type": "choice",
      "instructions": "Based on the contrast-filled coronary artery lumen in this angiographic frame, is a stenotic coronary lesion visible?",
      "criteria": {"coronary lesion visible": null, "no coronary lesion": null}
    },
    "stenosis_degree.box": {
      "type": "score",
      "instructions": "Within the region of interest ({\"bbox_2d\": [344, 559, 439, 662]}), what is the best-matching degree of luminal stenosis of the boxed coronary lesion?",
      "criteria": ["<20% stenosis", "20-50% stenosis", "50-70% stenosis", "70-90% stenosis", "90-98% stenosis",
                   "99% stenosis", "100% (total occlusion)"]
    }
  }
}

Targets: lesion = coronary lesion visible (human); stenosis_degree.box = level 0, <20% stenosis (human). The two lesion labels are not yes / no words, so they stay a two-option choice; the stenosis degrees are ordered, so they become a score.

GastroVision (GI endoscopy): a folder name → a choice with a 24-option menu

_{GastroVision (OSF 84e7f), CC BY 4.0; resized.}

Original annotation: none besides the folder of the image in the release, Normal stomach/. Our label: finding: normal stomach.

{
  "state": {"context": "A biomedical image is provided."},
  "images": ["gastrovision.jpg"],
  "questions": {
    "finding": {
      "type": "choice",
      "instructions": "Based on the endoscopic mucosal appearance, what is the best-matching finding, landmark or intervention in this GI endoscopy image?",
      "criteria": {"accessory tools": null, "angiectasia": null, "barrett's esophagus": null, "cecum": null,
                   "colon diverticula": null, "colon polyps": null, "colorectal cancer": null, "duodenal bulb": null,
                   "dyed-lifted-polyps": null, "dyed-resection-margins": null, "erythema": null,
                   "esophageal varices": null, "esophagitis": null, "gastroesophageal junction normal z-line": null,
                   "ileocecal valve": null, "mucosal inflammation large bowel": null, "normal esophagus": null,
                   "normal stomach": null, "pylorus": null, "resected polyps": null, "resection margins": null,
                   "retroflex rectum": null, "small bowel terminal ileum": null, "ulcer": null}
    }
  }
}

Target: finding = normal stomach (human). The release has 27 classes, so the question shows a seeded menu of 24 that holds the correct class. No noul or score: the classes are neither yes / no answers nor ordered.

Citation

Authors: Gevaert Lab, Department of Computational Medicine, Stanford University. This author line and the entry below are temporary: the technical report will name the authors, and this section will then point to it. Please cite the technical report once it is available.

@misc{manta2026,
  title        = {MANTA: Medical Answers, Natively from Text And images},
  author       = {{Gevaert Lab, Stanford University}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/gevaertlab/MANTA-27B}},
  note         = {Technical report forthcoming}
}

Acknowledgements

MANTA-27B was trained on Marlowe, Stanford's GPU-based computational instrument, supported by Stanford Data Science and Stanford Research Computing. Evaluations were also run on the Sherlock cluster of Stanford Research Computing.

Contact

Gevaert Lab, Department of Computational Medicine, Stanford University: the Community tab of this repository.

Appendix: full results

All models in a table answer the same questions. MANTA answers in one forward pass, without reasoning; the other decision models answered zero-shot, through their APIs (Clef also with its open weights on our GPUs). Accuracy in %, ECE-15: expected calibration error (15 bins); bold: the best value of the column in that table.

A.1 MANTA-Bench: medical decisions

At most 200 requests per source: the in-domain test sets (held-out test groups of the training sources; MANTA trained on their training splits, the other models answer zero-shot) and the out-of-domain test sets (data sets that no MANTA training used).

Model	Accuracy	ECE-15	In-domain acc.	In-domain ECE-15	Out-of-domain acc.	Out-of-domain ECE-15
MANTA-27B	86.0 [85.7, 86.4]	0.030	87.8	0.021	70.6	0.109
MANTA-4B	85.3 [85.0, 85.7]	0.039	87.3	0.028	68.5	0.139
liquid-d1-ultra	71.0 [70.6, 71.5]	0.013	71.2	0.013	69.3	0.035
mammoth-1	69.9 [69.5, 70.4]	0.051	70.2	0.049	67.2	0.075
pplx-decider-v1.2	69.9 [69.4, 70.3]	0.018	70.2	0.018	67.2	0.038
liquid-d1.1	69.6 [69.2, 70.1]	0.039	70.0	0.037	66.5	0.075
liquid-d1	69.5 [69.0, 69.9]	0.055	70.1	0.052	64.5	0.095
jev-1.14	69.0 [68.6, 69.5]	0.041	69.6	0.040	64.0	0.053
drex-max	68.6 [68.2, 69.1]	0.067	69.0	0.064	65.1	0.096
Clef (open weights, our GPUs)	68.6 [68.2, 69.1]	0.033	68.8	0.033	66.8	0.030
Clef (API)	68.4 [67.9, 68.8]	0.031	68.6	0.032	66.5	0.037
mercury-decide	67.6 [67.1, 68.1]	0.150	67.8	0.148	65.9	0.168
morph-reflex	67.4 [67.0, 67.9]	0.117	67.8	0.117	64.4	0.120
jev-1.13	67.0 [66.5, 67.4]	0.058	67.6	0.058	62.0	0.062
elephant-1	66.7 [66.3, 67.2]	0.070	67.4	0.064	60.8	0.121
clef-flash	66.3 [65.8, 66.7]	0.027	66.4	0.024	65.1	0.061
pplx-decider-flash-v1.2	65.3 [64.8, 65.7]	0.053	65.7	0.051	61.4	0.072
clef-omni	63.4 [62.9, 63.8]	0.026	63.3	0.030	64.1	0.043
gpt-6-luna	63.4 [62.9, 63.9]	0.153	63.2	0.157	64.8	0.127
joey-1	55.4 [54.8, 55.8]	0.061	55.4	0.061	54.6	0.063

_{57,163 questions that all 20 models answered (51,334 in-domain, 5,829 out-of-domain); accuracy in %, 95 % bootstrap interval over requests (1000 draws).}

Medical text questions only, all 27 models (also the text-only ones):

Model	Accuracy	ECE-15
MANTA-27B	87.7 [87.0, 88.4]	0.025
MANTA-4B	85.2 [84.4, 86.0]	0.038
liquid-d1-ultra	82.0 [81.1, 82.8]	0.030
pplx-decider-v1.2	79.6 [78.8, 80.5]	0.059
mammoth-1	79.5 [78.7, 80.4]	0.055
drex-max	79.4 [78.5, 80.2]	0.068
liquid-d1.1	79.3 [78.4, 80.1]	0.040
morph-reflex	79.0 [78.2, 79.9]	0.092
jev-1.14	78.5 [77.6, 79.4]	0.070
liquid-d1	78.4 [77.5, 79.2]	0.068
jev-1.13	77.8 [76.8, 78.7]	0.086
Clef (open weights, our GPUs)	77.7 [76.8, 78.6]	0.057
gpt-6-luna	77.7 [76.8, 78.6]	0.118
mercury-decide	77.6 [76.7, 78.6]	0.126
Clef (API)	77.3 [76.4, 78.2]	0.055
microsoft-decision-1	77.2 [76.2, 78.0]	0.086
drex-v2.0	76.4 [75.5, 77.4]	0.024
elephant-1	76.3 [75.4, 77.2]	0.055
clef-omni	75.8 [74.9, 76.7]	0.039
solar-decide	74.6 [73.7, 75.5]	0.183
clef-flash	73.9 [72.9, 74.7]	0.058
pplx-decider-flash-v1.2	73.5 [72.6, 74.4]	0.100
drex-v1.5	72.9 [71.9, 73.8]	0.090
tev1-4b	71.4 [70.4, 72.4]	0.096
kev-4b	70.8 [69.8, 71.7]	0.014
solar-decide-flash	69.2 [68.2, 70.2]	0.139
joey-1	63.7 [62.7, 64.7]	0.044

_{8,250 questions.}

A.1.1 MANTA-Bench: out-of-domain medical decisions per data set

The out-of-domain questions of A.1 (5,829 questions that all 20 models answered), split by data set: 9 data sets that no MANTA training used; every model answers them zero-shot.

Summary over data sets. Mean acc.: the unweighted mean of the per-data-set accuracies (each data set counts once, whatever its size; A.1 pools the questions instead). Firsts: the data sets where the model has the best accuracy (ties count for every tied model). Mean rank: the mean rank among the 20 models (1 = best; tied models share the better rank). Bold: the best value of the column.

Model	Out-of-domain (9): mean acc.	firsts	mean rank
MANTA-27B	64.4	3 / 9	4.7
liquid-d1-ultra	63.3	2 / 9	3.8
MANTA-4B	63.1	3 / 9	7.0
pplx-decider-v1.2	60.6	1 / 9	6.1
liquid-d1.1	60.0	0 / 9	8.4
mammoth-1	59.0	0 / 9	8.3
liquid-d1	58.7	0 / 9	10.0
Clef (open weights, our GPUs)	58.5	0 / 9	9.1
mercury-decide	58.4	0 / 9	9.7
drex-max	58.1	0 / 9	9.0
jev-1.14	58.0	1 / 9	10.2
Clef (API)	58.0	0 / 9	10.2
morph-reflex	57.9	0 / 9	10.3
gpt-6-luna	57.4	0 / 9	9.8
clef-flash	56.9	0 / 9	12.1
jev-1.13	55.7	0 / 9	13.8
pplx-decider-flash-v1.2	54.4	0 / 9	16.4
clef-omni	54.2	0 / 9	13.0
elephant-1	52.9	0 / 9	16.8
joey-1	46.4	0 / 9	16.6

_{Accuracy in %. Rows sorted by mean accuracy. Firsts and ranks use the accuracies as shown (one decimal).}

Accuracy per data set. Accuracy in %; bold: the best model of the row; MANTA rank: the rank of MANTA-27B among the 20 models. Question types: the questions of the data set per type (choice: one of several named options; noul: yes / no; score: ordered levels, lowest first). Columns in the order of A.1 (pooled accuracy). Data sets with few questions (the Questions column) give noisy accuracies.

Out-of-domain medical images: 9 data sets, 5,829 questions; MANTA-27B first on 3, MANTA-4B on 3; a MANTA release first on 6.

Dataset	Questions	Question types	MANTA-27B	MANTA-4B	liquid-d1-ultra	mammoth-1	pplx-decider-v1.2	liquid-d1.1	liquid-d1	jev-1.14	drex-max	Clef (open weights, our GPUs)	Clef (API)	mercury-decide	morph-reflex	jev-1.13	elephant-1	clef-flash	pplx-decider-flash-v1.2	clef-omni	gpt-6-luna	joey-1	MANTA rank
bbbc054_microglia	296	choice 296	53.7	63.5	43.2	34.1	42.6	42.6	39.9	33.1	34.8	34.5	31.1	35.8	29.1	39.9	33.1	37.2	35.5	43.2	43.2	35.5	2
cadica	766	choice 662, score 104	80.9	83.0	72.7	72.8	76.4	72.5	71.9	73.8	72.7	75.3	74.3	72.5	72.6	69.5	71.8	72.6	71.7	72.7	75.6	75.6	2
dsad	612	choice 612	56.7	49.5	59.3	56.4	48.2	56.4	53.4	59.8	50.0	49.0	52.8	51.3	47.5	50.5	40.7	50.5	43.1	46.4	40.5	34.0	3
fetal_planes_db_burgos_artizzu	273	choice 273	59.0	65.9	71.4	52.4	64.8	65.6	57.9	60.4	59.0	58.6	58.6	61.2	65.9	52.4	43.2	54.9	57.9	44.7	67.0	49.5	9
fpus23	411	choice 411	50.1	44.3	49.1	42.1	49.1	46.0	49.4	46.0	49.6	43.1	42.1	45.7	46.5	43.8	39.9	45.0	42.8	37.5	47.2	33.8	1
gastrovision	319	choice 319	63.6	66.8	65.8	65.5	66.8	62.4	64.9	60.8	63.3	66.1	63.3	64.9	65.2	61.1	62.4	60.2	57.7	59.9	50.8	46.7	9
grazpedwri_dx	1,379	choice 443, noul 936	82.6	81.1	74.8	77.3	73.4	71.1	67.7	70.5	68.3	77.2	76.5	72.9	69.4	68.3	67.8	74.1	66.6	76.5	74.8	68.7	1
mll23	254	choice 254	57.9	43.7	53.1	51.6	46.1	44.9	48.0	48.0	46.1	44.1	45.3	43.3	48.0	44.5	42.9	40.9	40.2	29.9	42.5	15.4	1
olives	1,519	choice 165, noul 1,354	74.7	69.8	80.0	78.6	78.3	78.3	75.0	69.8	79.1	78.3	78.2	78.3	76.4	71.4	74.0	77.0	73.8	77.0	75.2	58.1	14

A.2 Medical QA: MedQA, MedMCQA, PubMedQA, HEAD-QA

Held-out test splits; options in a fixed random order (published numbers use the original order).

Both MANTA releases trained on the training splits of MedQA, MedMCQA, PubMedQA, HEAD-QA; the tables report their test splits.

Model	MedQA	MedMCQA	PubMedQA	HEAD-QA	Mean	ECE-15 (mean)
liquid-d1-ultra	91.8	78.4	82.2	91.6	86.0	0.026
mammoth-1	89.9	78.2	81.8	91.2	85.3	0.040
liquid-d1.1	89.9	74.9	80.8	90.1	83.9	0.039
MANTA-27B	88.5	75.8	80.6	89.5	83.6	0.059
drex-max	89.1	76.2	79.6	88.7	83.4	0.050
jev-1.14	89.1	75.4	78.6	90.3	83.4	0.058
pplx-decider-v1.2	88.7	74.5	80.4	89.4	83.3	0.044
jev-1.13	87.6	76.3	77.4	90.0	82.8	0.060
gpt-6-luna	90.0	75.9	75.6	89.7	82.8	0.072
liquid-d1	86.5	73.7	81.4	88.4	82.5	0.061
morph-reflex	87.0	73.5	79.0	87.5	81.8	0.066
drex-v2.0	87.2	71.1	80.4	86.5	81.3	0.041
elephant-1	81.0	71.8	79.4	86.6	79.7	0.042
microsoft-decision-1	82.1	72.8	76.4	87.4	79.7	0.061
Clef (open weights, our GPUs)	84.9	70.2	75.0	86.9	79.3	0.058
mercury-decide	85.9	69.3	74.8	85.3	78.8	0.096
solar-decide	83.9	71.2	74.2	85.4	78.7	0.144
Clef (API)	83.6	68.6	75.0	85.9	78.3	0.056
clef-omni	78.0	68.7	74.8	85.2	76.7	0.052
MANTA-4B	75.2	67.8	78.6	82.6	76.1	0.091
pplx-decider-flash-v1.2	76.6	64.6	76.2	82.4	74.9	0.112
clef-flash	74.3	64.5	75.8	82.5	74.3	0.061
drex-v1.5	73.8	62.8	74.4	80.5	72.9	0.084
solar-decide-flash	71.1	59.4	70.8	75.6	69.2	0.103
tev1-4b	67.4	57.9	75.0	75.5	69.0	0.072
kev-4b	65.1	50.9	74.0	71.9	65.5	0.049
joey-1	48.9	44.6	70.4	57.2	55.3	0.068

_{Accuracy in % on the questions that all 27 models answered (MedQA 1,271, MedMCQA 4,114, PubMedQA 500, HEAD-QA 2,658). Mean: equal weight per benchmark.}

A.3 Medical knowledge and reasoning: MMLU-medical, MedXpertQA-Text

Official test splits, options in a fixed random order; no MANTA training used these benchmarks.

Model	MMLU-med.	MedXpertQA-Text	Mean	ECE-15 (mean)
mammoth-1	91.0	59.2	75.1	0.057
liquid-d1-ultra	91.4	42.6	67.0	0.100
elephant-1	86.6	45.9	66.2	0.062
drex-max	88.2	39.3	63.7	0.195
gpt-6-luna	88.6	37.6	63.1	0.138
pplx-decider-v1.2	88.9	36.7	62.8	0.149
liquid-d1.1	89.3	32.4	60.8	0.157
jev-1.14	91.6	29.8	60.7	0.171
jev-1.13	91.5	29.3	60.4	0.181
MANTA-27B	88.8	29.9	59.4	0.216
morph-reflex	86.3	32.3	59.3	0.220
Clef (open weights, our GPUs)	90.5	27.8	59.1	0.142
drex-v2.0	87.1	30.8	59.0	0.124
liquid-d1	87.9	29.7	58.8	0.177
microsoft-decision-1	87.3	29.2	58.2	0.187
Clef (API)	89.9	25.8	57.9	0.148
mercury-decide	86.5	26.8	56.6	0.258
solar-decide	86.5	26.7	56.6	0.292
clef-omni	93.3	18.6	56.0	0.166
clef-flash	91.9	19.0	55.5	0.193
pplx-decider-flash-v1.2	82.8	24.5	53.7	0.281
drex-v1.5	82.2	22.8	52.5	0.217
MANTA-4B	82.6	20.3	51.4	0.246
solar-decide-flash	79.0	17.1	48.1	0.236
tev1-4b	78.1	16.1	47.1	0.218
joey-1	59.3	30.9	45.1	0.065
kev-4b	73.2	14.6	43.9	0.122

_{Accuracy in % on the questions that all 27 models answered (MMLU-med. 1,788, MedXpertQA-Text 2,438). Mean: equal weight per benchmark.}

A.4 Medical exam sets with images: MedXpertQA-MM, WorldMedQA-V

No MANTA training used these benchmarks.

Model	MedXpertQA-MM	WorldMedQA-V (local)	WorldMedQA-V (English)	Mean	ECE-15 (pooled)
liquid-d1-ultra	59.2	84.3	78.7	70.3	0.102
mammoth-1	53.6	84.6	75.5	66.8	0.147
jev-1.14	47.6	88.1	82.2	66.3	0.148
liquid-d1.1	50.6	81.9	76.9	65.0	0.123
pplx-decider-v1.2	48.7	84.6	73.1	63.8	0.121
MANTA-27B	46.7	81.2	76.2	62.7	0.198
gpt-6-luna	46.0	83.6	74.5	62.5	0.200
jev-1.13	45.3	81.9	77.3	62.5	0.153
drex-max	48.4	79.2	72.7	62.2	0.221
liquid-d1	45.8	81.2	75.2	62.0	0.160
Clef (open weights, our GPUs)	42.1	80.2	75.5	60.0	0.177
morph-reflex	43.9	80.9	70.6	59.9	0.221
mercury-decide	40.5	80.5	72.7	58.6	0.322
elephant-1	40.4	75.8	73.4	57.5	0.176
Clef (API)	39.1	79.5	72.0	57.4	0.200
pplx-decider-flash-v1.2	35.2	71.0	72.7	53.5	0.293
MANTA-4B	32.2	75.4	72.4	53.1	0.285
clef-omni	31.8	73.7	72.0	52.4	0.186
clef-flash	33.0	71.7	71.0	52.1	0.197
joey-1	25.6	41.0	44.8	34.2	0.165

_{Accuracy in % on the questions that all 20 models answered (MedXpertQA-MM 1,577, WorldMedQA-V (local) 293, WorldMedQA-V (English) 286). Mean = (MedXpertQA-MM + WorldMedQA-V) / 2, with WorldMedQA-V the mean of its local and English versions (the same questions in two languages).}

A.5 General decisions

MANTA-Bench general text and image sources, with the same rules (in-domain: MANTA trained on their training splits; the other models answer zero-shot).

Model	Accuracy	ECE-15	In-domain acc.	In-domain ECE-15	Out-of-domain acc.	Out-of-domain ECE-15
MANTA-27B	91.3 [90.8, 91.7]	0.011	91.3	0.010	92.2	0.041
MANTA-4B	90.4 [89.9, 90.9]	0.009	90.4	0.009	90.7	0.040
pplx-decider-v1.2	87.2 [86.6, 87.7]	0.013	87.1	0.012	91.1	0.087
mammoth-1	86.1 [85.5, 86.7]	0.009	86.0	0.010	91.1	0.035
liquid-d1-ultra	85.6 [84.9, 86.2]	0.010	85.5	0.009	91.1	0.067
Clef (open weights, our GPUs)	85.0 [84.3, 85.6]	0.008	84.9	0.009	90.3	0.055
Clef (API)	84.6 [83.9, 85.3]	0.008	84.5	0.008	89.6	0.057
elephant-1	84.2 [83.5, 84.9]	0.014	84.2	0.013	88.1	0.039
pplx-decider-flash-v1.2	84.0 [83.3, 84.6]	0.012	83.9	0.012	90.9	0.102
morph-reflex	84.0 [83.4, 84.6]	0.065	83.9	0.067	89.4	0.025
liquid-d1	83.8 [83.1, 84.5]	0.027	83.7	0.028	88.6	0.043
clef-omni	83.6 [82.8, 84.3]	0.028	83.5	0.029	89.4	0.067
clef-flash	83.4 [82.6, 84.0]	0.009	83.3	0.009	89.4	0.065
liquid-d1.1	83.3 [82.7, 83.9]	0.008	83.2	0.008	91.5	0.048
mercury-decide	82.8 [82.1, 83.4]	0.082	82.7	0.083	87.9	0.034
jev-1.14	81.6 [80.9, 82.3]	0.028	81.5	0.028	90.7	0.030
jev-1.13	81.1 [80.4, 81.7]	0.039	80.9	0.039	90.7	0.039
joey-1	77.7 [76.7, 78.6]	0.024	77.5	0.024	85.4	0.038
gpt-6-luna	77.4 [76.6, 78.2]	0.102	77.7	0.103	65.9	0.051

_{23,231 questions that all 19 models answered (22,759 in-domain, 472 out-of-domain); accuracy in %, 95 % bootstrap interval over requests (1000 draws).}

_{Not ranked (answered fewer than 90 % of the questions because of provider refusals): drex-max (56 %).}

A.6 Comparison with chat models

GPT-6 Luna (chat), Claude Sonnet 5.5, Gemini 3.8 Flash and MedGemma 27B with the decision models, on subsets of A.1 to A.5; accuracy only (the chat models give no option probabilities).

MANTA-Bench medical, at most 20 requests per source:

Model	Accuracy	In-domain acc.	Out-of-domain acc.
MANTA-27B	86.1 [85.0, 87.2]	87.2	74.4
MANTA-4B	86.1 [85.0, 87.1]	87.4	71.5
Gemini 3.8 Flash	76.0 [74.6, 77.4]	76.2	73.6
Claude Sonnet 5.5	74.6 [73.2, 75.9]	74.2	79.1
liquid-d1-ultra	71.2 [69.8, 72.5]	71.0	73.3
pplx-decider-v1.2	70.5 [69.1, 71.9]	70.4	71.3
liquid-d1.1	70.4 [68.9, 71.8]	70.5	69.0
mammoth-1	70.3 [68.9, 71.7]	70.2	71.5
liquid-d1	69.9 [68.4, 71.3]	70.1	67.1
Clef (open weights, our GPUs)	69.5 [68.1, 70.9]	69.3	72.2
jev-1.14	69.5 [68.1, 70.9]	69.9	64.8
drex-max	69.4 [68.0, 71.0]	69.3	70.9
Clef (API)	69.1 [67.8, 70.5]	69.0	70.9
mercury-decide	68.5 [67.2, 69.9]	68.4	69.3
elephant-1	68.1 [66.7, 69.4]	68.2	66.2
morph-reflex	68.1 [66.7, 69.6]	68.0	68.6
jev-1.13	67.7 [66.4, 69.1]	67.9	65.7
clef-flash	66.3 [64.9, 67.8]	66.1	69.0
pplx-decider-flash-v1.2	65.8 [64.4, 67.2]	65.9	65.0
clef-omni	64.3 [62.9, 65.7]	63.7	70.4
gpt-6-luna	64.2 [62.7, 65.5]	63.7	69.3
GPT-6 Luna (chat)	62.4 [60.9, 63.9]	61.8	69.9
joey-1	56.6 [55.1, 58.0]	56.6	56.0
MedGemma 27B (local)	52.3 [51.0, 53.7]	52.3	52.3

_{6,700 questions that all 24 models answered; accuracy in %. Chat models (GPT-6 Luna chat, Claude Sonnet 5.5, Gemini 3.8 Flash) at the lowest reasoning setting their API accepts (Luna: none; Sonnet, Gemini: low), accuracy only; MedGemma 27B run locally (vLLM, temperature 0).}

Medical QA suite and exam sets, at most 500 requests per benchmark:

Model	Accuracy	MedQA	MedMCQA	PubMedQA	HEAD-QA	MMLU-med.	MedXpertQA-Text	MedXpertQA-MM	WorldMedQA-V (local)	WorldMedQA-V (English)
Gemini 3.8 Flash	86.5 [85.3, 87.5]	97.5	86.0	81.8	94.6	91.5	66.0	86.2	90.4	85.4
Claude Sonnet 5.5	81.6 [80.3, 82.7]	95.9	80.8	80.0	93.2	90.9	53.0	72.2	87.9	82.1
mammoth-1	78.2 [76.9, 79.5]	90.7	78.5	81.8	92.6	89.9	57.6	51.0	84.4	76.6
liquid-d1-ultra	77.5 [76.2, 78.8]	91.8	76.7	82.2	92.2	91.1	42.9	58.4	84.4	79.6
GPT-6 Luna (chat)	73.9 [72.5, 75.2]	89.9	75.7	75.6	90.8	89.2	40.4	49.2	81.2	73.4
liquid-d1.1	73.9 [72.5, 75.3]	90.9	74.9	80.8	89.8	88.4	32.7	49.7	81.6	78.1
jev-1.14	73.8 [72.5, 75.2]	88.7	75.5	78.6	90.0	90.9	31.3	45.4	87.6	83.2
drex-max	73.5 [72.2, 74.8]	88.5	73.9	79.6	88.8	87.8	42.2	47.9	78.7	73.7
pplx-decider-v1.2	73.5 [72.2, 74.8]	89.3	71.9	80.4	90.4	87.8	38.9	45.9	84.8	74.1
gpt-6-luna	73.5 [72.1, 74.8]	90.7	74.5	75.6	90.8	87.8	40.6	44.8	83.3	74.8
jev-1.13	72.7 [71.4, 74.2]	87.7	76.5	77.4	90.6	91.1	30.0	44.5	81.6	78.1
MANTA-27B	72.5 [71.2, 74.0]	87.7	73.5	80.6	89.2	88.6	30.4	47.0	81.2	77.4
liquid-d1	72.1 [70.7, 73.4]	88.7	73.1	81.4	88.0	87.6	31.5	43.9	80.9	75.9
elephant-1	71.5 [70.0, 72.8]	81.7	70.9	79.4	88.0	85.6	47.6	40.3	75.9	73.7
morph-reflex	71.1 [69.8, 72.5]	89.1	71.9	79.0	87.6	85.0	33.3	43.0	80.9	71.5
Clef (open weights, our GPUs)	70.0 [68.7, 71.5]	85.2	69.2	75.0	86.9	91.9	29.2	40.3	80.1	75.9
Clef (API)	68.5 [67.1, 70.0]	82.9	66.4	75.0	86.5	90.9	27.3	38.8	79.4	72.3
mercury-decide	68.3 [66.9, 69.8]	86.6	66.4	74.8	86.5	84.0	27.7	39.4	80.1	73.0
clef-omni	65.7 [64.2, 67.2]	76.5	69.6	74.8	87.3	92.3	17.2	29.8	73.4	72.3
pplx-decider-flash-v1.2	65.2 [63.7, 66.6]	78.2	67.2	76.2	83.1	79.1	26.5	34.5	70.9	73.0
MANTA-4B	64.5 [63.1, 66.0]	71.4	66.2	78.6	83.7	82.8	20.1	34.1	75.5	73.0
clef-flash	64.4 [63.0, 65.9]	73.0	63.4	75.8	83.3	89.7	20.5	32.7	71.3	72.3
MedGemma 27B (local)	60.4 [59.0, 61.9]	69.8	62.6	77.2	78.1	78.5	17.4	27.2	70.2	65.3
joey-1	48.3 [46.9, 49.9]	51.4	43.7	70.4	58.8	58.8	32.3	26.7	40.4	44.9

_{3,959 questions that all 24 models answered; accuracy in %. Chat models (GPT-6 Luna chat, Claude Sonnet 5.5, Gemini 3.8 Flash) at the lowest reasoning setting their API accepts (Luna: none; Sonnet, Gemini: low), accuracy only; MedGemma 27B run locally (vLLM, temperature 0).}

General decisions, at most 50 requests per source:

Model	Accuracy	In-domain acc.	Out-of-domain acc.
MANTA-27B	90.7 [89.6, 91.7]	90.6	96.0
MANTA-4B	90.1 [89.0, 91.2]	90.0	96.0
Gemini 3.8 Flash	88.5 [87.5, 89.6]	88.4	96.0
pplx-decider-v1.2	86.8 [85.7, 87.9]	86.7	94.0
mammoth-1	86.0 [84.9, 87.1]	85.9	93.0
Claude Sonnet 5.5	86.0 [84.8, 87.2]	86.0	88.0
liquid-d1-ultra	85.2 [83.9, 86.4]	85.0	93.0
Clef (open weights, our GPUs)	84.6 [83.4, 85.9]	84.4	94.0
Clef (API)	84.4 [83.2, 85.7]	84.3	93.0
elephant-1	84.4 [83.2, 85.6]	84.3	93.0
morph-reflex	83.9 [82.7, 85.1]	83.7	92.0
liquid-d1.1	83.4 [82.3, 84.6]	83.2	96.0
pplx-decider-flash-v1.2	83.4 [82.1, 84.7]	83.2	93.0
liquid-d1	83.2 [82.0, 84.5]	83.1	92.0
clef-omni	83.0 [81.6, 84.4]	82.8	92.0
clef-flash	82.8 [81.4, 84.2]	82.6	93.0
mercury-decide	82.7 [81.5, 83.9]	82.5	94.0
jev-1.14	81.7 [80.3, 83.1]	81.5	95.0
jev-1.13	80.8 [79.5, 82.2]	80.6	93.0
GPT-6 Luna (chat)	79.6 [78.0, 81.1]	80.1	47.0
gpt-6-luna	77.0 [75.5, 78.5]	77.3	59.0
joey-1	76.8 [74.8, 78.5]	76.5	92.0
MedGemma 27B (local)	74.0 [72.3, 75.8]	74.0	73.0

_{5,792 questions that all 23 models answered; accuracy in %. Chat models (GPT-6 Luna chat, Claude Sonnet 5.5, Gemini 3.8 Flash) at the lowest reasoning setting their API accepts (Luna: none; Sonnet, Gemini: low), accuracy only; MedGemma 27B run locally (vLLM, temperature 0).}

_{Not ranked (answered fewer than 90 % of the questions because of provider refusals): drex-max (54 %).}

A.7 Compared models: provider model versions

• MANTA-4B: the other MANTA release (gevaertlab/MANTA-4B)
• clef: cloudflare/clef (Cloudflare); open weights: Cloudflare/clef
• clef-flash: cloudflare/clef-flash (Cloudflare); open weights: Cloudflare/clef-flash
• clef-gpu: the open Clef weights, run on our GPUs with the MANTA evaluation code; open weights: Cloudflare/clef
• clef-omni: cloudflare/clef-omni-20261009 (Cloudflare); open weights: Cloudflare/clef-omni
• drex-max: nace-ai/drex-max-20261016 (DeepInfra)
• drex-v1.5: nace-ai/drex-v1.5-20261009 (DeepInfra); open weights: nace-ai/drex-v1.5
• drex-v2.0: nace-ai/drex-v2.0-20261009 (DeepInfra); open weights: nace-ai/drex-v2.0
• elephant-1: baseten/elephant-1-20261021 (BaseTen); open weights: baselabs/elephant-1
• gpt-6-luna: openai/gpt-6-luna-decisions-20261006 (OpenAI)
• jev-1.13: typesafe/jev-1.13-20260917 (TypeSafe)
• jev-1.14: typesafe/jev-1.14-20261014 (TypeSafe)
• joey-1: baseten/joey-1-20261029 (BaseTen); open weights: baselabs/joey-1
• kev-4b: jaredpalmer/kev-4b-20260924 (Phala); open weights: jaredpalmer/kev-4b
• liquid-d1: liquid/d1-20260930 (Liquid)
• liquid-d1-ultra: liquid/d1-ultra-20261015 (Liquid)
• liquid-d1.1: liquid/d1.1-20261027 (Liquid)
• mammoth-1: baseten/mammoth-1-20261021 (BaseTen); open weights: baselabs/mammoth-1
• mercury-decide: inception/mercury-decide-20260930 (Inception)
• microsoft-decision-1: microsoft/microsoft-decision-1-20261009 (Azure)
• morph-reflex: morph/reflex-20261103 (Morph)
• pplx-decider-flash-v1.2: perplexity/pplx-decider-flash-v1.2-20261016 (Perplexity); open weights: perplexity-ai/pplx-decider-flash-v1.2
• pplx-decider-v1.2: perplexity/pplx-decider-v1.2-20261016 (Perplexity); open weights: perplexity-ai/pplx-decider-v1.2
• solar-decide: upstage/solar-decide-20260928 (Upstage)
• solar-decide-flash: upstage/solar-decide-flash-20261008 (Upstage)
• tev1-4b: togethercomputer/tev1-4b-experimental-20260923 (Together); open weights: togethercomputer/Tev1-4B-experimental

Chat models (comparison with chat models only):

• claude-sonnet-5.5: anthropic/claude-sonnet-5.5 (Google)
• gemini-3.8-flash: google/gemini-3.8-flash (Google)
• gpt-6-luna-chat: openai/gpt-6-luna (Azure)
• medgemma-27b: google/medgemma-27b-it@2d3e00ea38b50018bf5dd3aa1009457cd2d5a48f (run on our GPUs with vLLM); open weights: google/medgemma-27b-it

_{Open weights: the public weights of the evaluated version in the builder's Hugging Face repository, from the builders' model cards, documentation and changelogs (checked on 9 Nov 2026).}

A.8 Reproduction

We compared the probabilities of this checkpoint with the predictions of the evaluation runs, on the same evaluation requests.

On 200 evaluation requests (493 questions) this checkpoint gives the saved option order for every question (0 mismatches), the same top option for 99.6 % of the questions and a largest probability difference of 0.063 against the predictions of the evaluation runs; against the unmerged training weights: 99.8 % same top option, largest difference 0.080. The differences come from bfloat16 rounding (the merged weights, one request per batch instead of the evaluation's packed batches); the questions whose top option differs are near-ties.