surogate/rune-26b-a4b-GGUF

Hugging Face | Training and Serving Engine | Launch Blog |
License: Apache 2.0 | Authors: Invergent

Rune 26B-A4B v3

Rune is a decision model for text and images with a context size of 262k: give it a state, which can be text, structured data or a picture with text, plus a question and a set of options, and it returns one option with a probability over all of them, in a single forward pass. It answers three kinds of questions:

• choice (pick one of N)
• noul (a probability that a statement is true)
• score (a position on an ordinal scale)

This is Rune v3 as bfloat16 safetensors, a standard Gemma4ForConditionalGeneration checkpoint (26.5B parameters, 8 of 128 experts active per token). It is served fastest by the surogate engine, whose decisions endpoint implements the protocol Rune was trained with. Every number on this page was measured, and the harnesses are named, so you can reproduce them.

The Rune v1 GGUF builds that this repository used to hold are still in its history, at revision 2a15504.

Accuracy

Decision Index is a public leaderboard of decision models. Its current edition, 0.2.1, scores the 0.2 suite (151,034 requests from 44 benchmarks) on a 38-benchmark panel: each benchmark's accuracy is rescaled so that chance scores 0 ("skill"), averaged within five areas, and the areas are weighted into the headline index. Rune v3's row is the leaderboard's: our submitted run of these bf16 weights, adopted after the maintainers' own reproduction matched it bit for bit. Index and area skill scores in %, as published on 26 September 2026:

System	Index	Raw	Knowledge & Reasoning	Language	Retrieval & Classification	Tools & Automation	Arts & Human Taste
Surogate Rune v3, thinking: true (bf16)	59.2*		48.0	64.0	64.7	72.4	41.3
Jev 1.13	57.89	68.08	51.3	62.0	55.4	75.1	37.7
Surogate Rune v3 (bf16)	57.44	67.30	43.4	63.1	63.5	71.2	41.9
AutoJev-27B	56.40	66.89	40.9	63.5	54.9	79.3	39.4
simple-jev (Qwen3.8-27B)	55.74	66.26	36.6	62.1	63.2	76.2	36.5
Decider chat (Qwen3.6-27B)	51.35	63.06	37.0	57.1	52.2	71.4	35.1
Winnow-12B (Q8_0)	50.02	61.91	33.8	56.0	54.0	71.0	30.0
JoshuaSP diffusiongemma (26B-A4B)	49.47	61.28	32.7	53.5	58.4	70.2	26.7
Jevfire	49.37	61.63	30.5	53.3	56.2	72.3	32.2
Decider 35B-A3B (NVFP4)	47.11	59.72	31.8	55.5	54.7	56.5	32.6

* Our estimate, not yet on the leaderboard: the leaderboard's per-benchmark 0.2.1 scores for Rune v3, each moved by the change thinking made to that benchmark in our full run of the suite with the official 0.2 scorer (the 0.2.1 scorer is not public yet). Rebuilt the same way, the published rows for Rune v3 (57.44) and Jev 1.13 (57.89) come out exactly.

The index counts unanswered requests as wrong. Rune v3 is second on the board, 0.45 points behind Jev 1.13, and with thinking on it would lead it by about 1.3. On the previous edition, 0.2 (40 benchmarks, a plain mean of the areas), Rune v3 scored 53.39, and 54.89 with thinking, against 51.67 for Jev 1.13. 0.2.1 takes RouterBench and SGD out of the panel, weighs Knowledge & Reasoning and Language at 26% each and Arts & Human Taste at 10%, and rescores five benchmarks.

Calibration

Measured on the Decision Index's 33 benchmarks that have a right or wrong answer per field (each benchmark weighted equally, every request). At the default temperature Rune is overconfident; reading the answers at temperature 2 makes the probabilities match accuracy:

decision temperature	accuracy	mean confidence	ECE	Brier
1 (default)	73.9%	86.4%	12.5%	0.385
2	73.9%	73.6%	2.2%	0.360

The temperature changes only the probabilities, never the chosen option. With surogate, pass --decision-temperature 2 to surogate serve.

Thinking

Thinking is opt-in, per request. With "thinking": true, a question Rune is unsure of thinks before it answers: if its one-pass confidence (the top option's probability; for a true/false question the larger of p and 1 - p) is below 0.7, Rune reasons for up to 512 tokens and then answers; every other question keeps its one-pass answer.

On the Decision Index 0.2 suite (our full run, scored with the official 0.2 scorer), about one question in ten thinks and the 0.2 index rises from 53.39 to 54.89, with Knowledge & Reasoning from 43.5 to 48.4; under 0.2.1's weights that is about 59.2 against 57.44 (see Accuracy). The largest gains in skill points: GSM8K +15.2, CRUXEval +11.4, CLadder +10.4, BBH +8.0, PhishNChips +5.1. ForecastBench, whose answers are probabilities, loses 9.8: a thought makes those probabilities more extreme.

A question that thinks takes about 5 seconds instead of 0.2 (see Speed); the others are unchanged. Thinking is not available together with images yet.

Images

The vision tower is kept, so Rune accepts images. With surogate, start the server with --vision and put the picture in the request's images array as a data URL; it is placed ahead of the text.

We measured image decisions with the item format and scoring of Image JevBench v0.1 on 1,196 items: the benchmark's 136 public items (its 8 example cards, and its 128 preview items rebuilt from their original sources) and 1,060 items we built the same way from the same openly licensed sources. This is our own run, not a leaderboard result: the benchmark's sealed items are private and were not used, so our items are a different sample. Accuracy in %, one request at a time on one RTX PRO 6000 with surogate 1.5.3, every item answered:

Family	Items	Chance	Rune v3, 1,120 image tokens	Rune v3, 280 image tokens (default)	Gemma 4 26B-A4B-it, 280 image tokens
All items	1,196	29.6	83.4	77.1	71.8
The benchmark's public items	136	27.5	79.4	75.7	73.5
ScreenSpot (click target on a screenshot)	139	20.0	89.2	85.6	83.5
ScreenSpot-Pro (professional apps, high resolution)	101	20.0	78.2	57.4	56.4
Android-in-the-Wild (next tap)	90	20.0	85.6	81.1	81.1
Multimodal-Mind2Web (web action)	102	20.0	83.3	68.6	68.6
FinQA (financial table)	121	25.8	85.1	74.4	39.7
Geometry3K (diagram)	121	25.0	74.4	66.9	58.7
ArxivQA (scientific figure)	110	25.0	62.7	58.2	63.6
CLEVR-HOPE (synthetic 3D scene)	110	50.0	70.9	66.4	62.7
Everyday photo (VizWiz photos)	102	50.0	94.1	97.1	93.1
Charts (our renders)	100	35.0	100.0	100.0	100.0
Documents (our renders)	100	38.0	96.0	95.0	90.0

The screen families ask which of five labelled markers drawn on the screenshot to click. The benchmark's own everyday photos are synthetic and mostly sealed, so that row is 100 real VizWiz photos plus the benchmark's 2 example photos; ArxivQA's answer labels were machine-generated.

Image probabilities are well calibrated at the default decision temperature (ECE, ten bins, top option):

	accuracy	mean confidence	ECE at temperature 1	ECE at temperature 2
Rune v3, 1,120 image tokens	83.4%	88.2%	5.1%	6.5%
Rune v3, 280 image tokens	77.1%	84.0%	6.9%	5.6%
Gemma 4 26B-A4B-it, 280 image tokens	71.8%	94.7%	22.9%	16.8%

The --decision-temperature 2 recommended above for text keeps image ECE within 1.5 points of temperature 1.

Image token budget. Gemma 4 turns each image into at most 280 tokens by default. Raising the image processor's budget to 1,120 (max_soft_tokens and image_seq_length in processor_config.json) gives Rune more detail: +6.3 points over all items (paired 95% interval +4.3 to +8.4), most of it on small screen targets (ScreenSpot-Pro +20.8) and dense pages and tables (Mind2Web +14.7, FinQA +10.7). It costs time: one image decision took 0.18 s at 280 tokens and 0.46 s at 1,120 (median, one request at a time, RTX PRO 6000).

Speed

Measured during the Decision Index run above: one NVIDIA RTX PRO 6000 Blackwell per server, bf16, surogate 1.5.3 with the MoE routing fix described below, four concurrent clients per GPU. A Decision Index request carries a shared state and often many questions, so these figures are per request, not per question:

	median	p95	mean
latency per request	388 ms	2.33 s	751 ms

One question per request, one request at a time, on an idle RTX PRO 6000 (surogate main):

	median	p90
a decision (no thinking)	0.09-0.18 s	0.2 s
a question that thinks (thinking: true, confidence below 0.7)	4.9 s	6.3 s

With thinking on, questions at or above the confidence gate answer as fast as without it. On the Decision Index's mix, where one question in ten thinks, that averages about 0.65 s per question.

Running it

Thinking is off, and that is the default: the chat template in these files disables it.

With surogate (recommended)

The weights take 51.6 GB, so they need a GPU with room for them plus the KV cache; we serve them on 96 GB RTX PRO 6000 cards:

hf download surogate/rune-26b-a4b-GGUF --local-dir rune-v3
surogate serve ./rune-v3 --host 0.0.0.0 --port 8000 \
  --max-model-len 32768 --max-num-seqs 8 --kv-capacity auto --decision-temperature 2

Under sustained concurrent load, surogate 1.5.3 can occasionally hit a non-finite router value in the MoE layers and stop with an illegal memory access. surogate's main branch includes a fix (#217): with it, an affected request fails cleanly and the server keeps running. Use a build that includes it.

Then ask it a decision (use the model id that GET /v1/models lists; add "thinking": true next to "questions" to let unsure questions think, which needs a surogate build that includes #240):

curl http://localhost:8000/api/alpha/decisions -H 'Content-Type: application/json' -d '{
  "model": "<id from /v1/models>",
  "state": "Customer wrote: the package arrived late and damaged, I want my money back.",
  "questions": {
    "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?",
               "criteria": {"true": "A refund is requested", "false": "No refund is requested"}}
  }
}'

With transformers

The checkpoint loads like any Gemma 4 checkpoint. A decision is read at the first generated position as a softmax over the option letters' logits, with the prompt rendered as surogate's decisions endpoint renders it (docs/inference/decisions.md).

There are no GGUF builds of v3 yet.

Files

file	size	contents
model-00001-of-00011.safetensors … model-00011-of-00011.safetensors	51.6 GB in total	bf16 weights, at most 5 GB per shard
model.safetensors.index.json		tensor-to-shard map
config.json, generation_config.json		Gemma4ForConditionalGeneration config
chat_template.jinja		chat template, thinking off by default
tokenizer.json, tokenizer_config.json, processor_config.json		tokenizer and image processor