span-02

span-02 is a decision model. Give it a state (text or JSON, plus up to 4 images) and a set of typed questions, and it returns a calibrated probability distribution for every answer. No free text, no parsing.

• Typed answers: pick one option (choice), yes/no (noul), a rating on a scale (score), or a number (number).
• Fast: a typical request answers in tens of milliseconds. When the model is unsure, it thinks briefly first, and the request still finishes well under a second.
• Small and efficient: a fine-tune of Gemma 4 26B-A4B (mixture of experts, ~4B active parameters), quantized to NVFP4/FP8 (16.4 GB) and served by its own Rust/CUDA engine, which is in this repo.
• Runs on one Blackwell GPU: RTX PRO 6000 or RTX 5090.

Decision Index 0.3 (public)	score
Text index	63.62 (all 140,178 requests answered)
Vision	70.69 (all 12,210 requests answered)

The weights only run with the included engine (engine/). They cannot be loaded with transformers or vLLM.

Quickstart

With Docker (needs an NVIDIA driver with CUDA 12.9 support and the NVIDIA container toolkit):

hf download respanai/span-02 --local-dir span-02 && cd span-02
docker build -t span-02 .
docker run --gpus all -p 8000:8000 -v "$PWD":/model:ro -e DLLM_MODEL_DIR=/model -v span02-data:/data span-02

Without Docker (CUDA toolkit 12.8+ and Rust; no GPU needed to build):

hf download respanai/span-02 --local-dir span-02 && cd span-02
(cd engine && NVCC=/usr/local/cuda/bin/nvcc cargo build --release -p dllm-server)
bash launch.sh        # serves on 0.0.0.0:8000

launch.sh also downloads the speculative-decoding draft model (google/gemma-4-26B-A4B-it-assistant). The server is ready in about 20 s; GET /health then returns {"ok": true, "model": "span-02"}.

Example

curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "state": {"customer": "My package arrived damaged and nobody answers my emails. I want a refund now.",
            "order": {"id": 123, "days_late": 4}},
  "questions": {
    "dept":   {"type": "choice", "instructions": "Which team should handle this?",
               "criteria": {"returns":  "Exchanges, wrong or damaged items",
                            "shipping": "Delivery status, delays, lost packages",
                            "billing":  "Charges, invoices, payment problems"}},
    "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
    "anger":  {"type": "score", "instructions": "How frustrated is the customer?",
               "criteria": ["Calm or positive", "Mildly annoyed", "Angry or threatening to leave"]},
    "late":   {"type": "number", "instructions": "How many days late is the order?"}
  }
}'

{"model": "span-02",
 "answers": {
   "dept":   {"type": "choice", "choice": "returns", "confidence": 0.945,
              "probabilities": {"returns": 0.963, "shipping": 0.024, "billing": 0.013}},
   "refund": {"type": "noul", "noul": 0.99999},
   "anger":  {"type": "score", "score": 1.9999, "confidence": 0.9998,
              "legend": {"0": "Calm or positive", "1": "Mildly annoyed", "2": "Angry or threatening to leave"},
              "probabilities": {"0": 0.000008, "1": 0.00012, "2": 0.99987}},
   "late":   {"type": "number", "number": 4.0, "text": "4", "confidence": 0.99}},
 "usage": {"input_tokens": 540, "reasoning_tokens": 0, "output_tokens": 2, "latency_ms": 45.2}}

(Values are illustrative.)

The API follows TypeSafe's System One API, so its clients work unchanged; number, reasoning and images are span-02 extensions that those clients can ignore.

How it works

1. Read the answer directly. Each question is one prompt; the answer is read from the model's probabilities for the option labels in a single forward pass. All questions of a request share the state, and the engine computes it once.
2. Think if unsure. If the top answer has probability below 0.8, the model first writes a short thought, then the answer is read again. The final distribution is the average of the two: 0.5 × before thinking + 0.5 × after thinking.
3. Stay on time. Thoughts are limited by a time budget (700 ms by default) and decoded with speculative decoding (Gemma's public MTP draft, verified greedily), at about 480 thought tokens/s on an RTX PRO 6000.

Number answers are generated token by token after the (optional) thought, with the same speculative decoding.

API reference

POST /v1/systemone takes a request and returns the answers. GET /health reports readiness.

Request

field
state	required	Any JSON value. The model treats it as data, not instructions.
questions	required	{id: question}. Answers come back in the same order.
assets	optional	Images keyed by id: {"type": "image", "format": "png" | "jpg" | "jpeg", "data_base64": "..."}. Mentioning an id in the state or a question is optional; see Images.
images	optional	A list of images as data URIs or base64 (used by the Decision Index kit). At most 4 images in total.
reasoning	optional	{"mode": "auto" | "on" | "off", "budget_ms": 0-600000, "trace": true | false}
model	optional	Accepted and ignored; responses always say span-02.

• mode: auto (default) thinks only when unsure; on always thinks; off never thinks.
• budget_ms: the thinking time limit for this request; 0 means no limit. If omitted, on thinks until the thought ends (at most 4,096 tokens) and auto uses the server default (--budget-ms, 700).
• trace: return each thought's text.
• Bodies can be JSON, multipart/form-data (a request part plus one file part per image, referenced as {"part": name}), or compressed with Content-Encoding: zstd or gzip. Unknown top-level fields are ignored.

Question types

type	criteria	answer
choice	{key: description}, 1 to 679 options	choice, confidence, probabilities
noul	optional {"true": ..., "false": ...}	noul = probability of true
score	ordered list of 2 to 10 levels, low to high	score = Σ i·pᵢ, legend, probabilities, confidence
number	none	number (null if the text is not a number), text, confidence

• choice confidence = (p_max − 1/n) / (1 − 1/n): 0 for a uniform guess, 1 for certainty.
• score confidence = 1 − (expected distance from the most likely level) / (the same for a uniform guess).
• number confidence = exp(mean log-probability of the answer tokens). Number questions think only with mode: "on", because the think gate is trained on option questions.

Response

• answers[id]: the typed answer. If the question thought, it also has reasoning: {"tokens": N} (plus trace when requested).
• usage: input_tokens (the prompt tokens of every question; the shared state counts once per question and each image its image tokens + 2), reasoning_tokens (thought tokens), output_tokens (number-answer tokens; other answers are read in one pass and generate none), latency_ms (server time).
• image_tokens: total image tokens, present only when the request has images.

Images

The model was trained with all images (up to 4) before the text, referred to as "Image 1", "Image 2", and so on. The server shows images first, then assets in the order sent, and replaces every mention of an asset id in the state, instructions and criteria with that image's label. For example {"state": {"photo": "a3f1c2e8-found"}, "assets": {"a3f1c2e8-found": ..., "77b0d4a1-poster": ...}} reaches the model as {"photo": "Image 1"}, and "the dog in this picture: 77b0d4a1-poster" becomes "the dog in this picture: Image 2". Assets that are never mentioned are still shown. Images are PNG or JPEG, at most 16 MiB each, and are resized to the image budget (280 tokens per image by default, 560 in the Decision Index settings).

Errors

Errors look like {"error": {"type": "...", "message": "..."}}.

status	type	when
400	invalid_request	invalid JSON, or a broken compressed or multipart body
400	invalid_asset	unknown asset type or format, bad base64, not a PNG/JPEG, more than 4 images
400	max_tokens_exceeded	the request does not fit the context or the KV cache ("maximum context length")
401	authentication_error	missing or wrong key (only when the server sets AR_API_KEY)
404	not_found	any other path
413	request_too_large	body over 64 MB, or an image over 16 MiB
415	unsupported_encoding	a Content-Encoding other than zstd or gzip
422	validation_error	missing fields, bad question types or criteria, too many options, bad reasoning
529	overloaded	the server is busy; retry shortly
500	server_error	an internal failure

Performance

Latency (Decision Index settings, one RTX PRO 6000, one request at a time)

Client wall time per request, including HTTP, in ms. The Decision Index requires median, mean and p80 of at most 1,000 ms each.

request mix	n	median	mean	p80	accuracy (choice questions)
DI public text 0.3, stratified sample (42 benchmarks)	400	71	308	712	0.923
DI public vision: CV-Bench, BLINK, RealWorldQA	150	186	291	642	0.747
image rows of 6 DI vision benchmarks	210	292	389	699	0.652
think-heavy set (78% of questions think)	150	616	514	770	0.562

On the full public text run: median 36 ms, mean 241 ms, p95 850 ms.

Throughput (64 requests in flight)

Short single-question requests, 700 ms thinking budget. RTX PRO 6000 with --max-rows 16384 --think-max-rows 8192 --kv-gb 48 --pages 24000; RTX 5090 with the defaults.

GPU	requests/s, no thinking	requests/s, auto thinking	p99, auto
RTX PRO 6000	167	150	0.85 s
RTX 5090	127	102	0.70 s

On the same RTX 5090, the engine is 1.3 to 7× faster than vLLM without thinking and keeps a lower tail latency with it.

Hardware

• An NVIDIA Blackwell GPU with compute capability 12.0 (sm_120a): RTX PRO 6000 Blackwell (96 GB) or RTX 5090 (32 GB). Other architectures are not supported: the kernels use sm_120a FP4/FP8 tensor-core instructions.
• NVIDIA driver with CUDA 12.9 support (R575 or newer).
• GPU memory in use: about 28 GB with the default settings; the Decision Index settings below use about 41 GB.

Quantization

part	format
MoE experts	NVFP4 W4A4: e2m1 weights with an FP8 scale per 16 elements and an FP32 global scale; activations NVFP4 with static per-layer global scales
attention and dense MLP	FP8 e4m3 W8A8: per-row weight scales, dynamic per-token activation scales
tied embedding / LM head	FP8 e4m3, per row
vision tower, router, norms, KV cache	bf16

Decision Index settings

The Decision Index submission runs launch.sh as shipped:

--max-len 131072 --budget-ms 0 --tau 0.8 --cap 1024 --think-ms 750 --think-cost think_cost.json
--image-soft-tokens 560 --serial --no-prefix-cache --spec-max-branches 16 --pages 12000 --kv-gb 24

• Deterministic thinking budget: --budget-ms 0 --think-ms 750 --think-cost think_cost.json. The wall clock is never read. Instead, a fixed cost model turns a 750 ms target into a thought-token cap that depends only on the request (prompt length, images, number of thinking questions), so the result does not depend on the machine or its load. A short single question gets about 330 thought tokens; requests with many thinking questions or long prompts get fewer. --cap 1024 is the hard limit.
• One request at a time, no prefix cache: --serial --no-prefix-cache, so an answer does not depend on other requests. Requests where several questions think at once can still vary very slightly in their post-thought probabilities between runs.
• Image budget: --image-soft-tokens 560, about 1.3 MP with the aspect ratio kept.
• KV cache: --pages 12000 --kv-gb 24, sized so every Decision Index request fits on its own.

Limits

• Choice questions: 1 to 679 options. Score questions: 2 to 10 levels.
• Images: at most 4 per request.
• Context: 131,072 tokens per question (prompt + images + thought). Long prompts stay cheap: the sliding-window layers keep only their 1,024-token window.
• KV cache: a request too large for the whole KV cache is rejected with 400 max_tokens_exceeded. With the default --kv-gb 9 (sized for a 32 GB RTX 5090) this can happen for requests with several long questions, for example 4 questions of 8K tokens each. On a larger GPU, raise --kv-gb and --pages (e.g. --kv-gb 48 --pages 24000 on an RTX PRO 6000).
• Number answers: at most 32 tokens. Thoughts: at most 4,096 tokens (1,024 in the Decision Index settings).
• Request body: at most 64 MB. Nothing is truncated; requests over a limit are rejected.

Files

file	contents
model.safetensors, config.json	quantized weights (16.4 GB) and the Gemma 4 config
tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json	tokenizer and chat template
processor_config.json	Gemma 4 image processor at 560 image tokens
draft_vocab_16384.json	pruned vocabulary for the speculative-decoding draft
think_cost.json	cost model for the deterministic thinking budget (calibrated on an RTX PRO 6000)
engine/	the dllm-server source (Rust + CUDA); see engine/README.md for build and flags
launch.sh	downloads what is missing and starts the server with the Decision Index settings
Dockerfile, .dockerignore	two-stage image: CUDA build, then a small runtime
NOTICE	licenses and attributions

License

The weights are a derivative of Google's Gemma 4 (google/gemma-4-26B-A4B-it), distributed under the Gemma 4 license (Apache 2.0). The engine in engine/ is Apache-2.0; it ports algorithms from vLLM, SGLang, FlashInfer and other projects, credited in NOTICE and in the source file headers.