span-02

span-02 is a decision model: it answers typed questions about a state (JSON or text, plus up to 4 images) with a full probability distribution over the options, in the Decision Index /v1/systemone wire format. It is a fine-tune of Gemma 4 26B-A4B (mixture of experts, about 4B active parameters per token), quantized (NVFP4 experts, FP8 dense layers) and served by a custom Rust/CUDA engine for Blackwell GPUs (sm_120: RTX 5090, RTX PRO 6000) that is included in this repo (engine/).

Before answering it may think. If the top probability of the no-think answer is below 0.8, the model writes a greedy thought. Its length limit comes from a deterministic thought budget: a fixed cost model (think_cost.json) turns a 750 ms target into a per-question token cap that depends only on the request (prompt length, images, number of questions and of thinking questions), so the same request always gets the same thought. After a thought the answer is read out again, and the final distribution is the average of the two: 0.5 x the no-think distribution + 0.5 x the post-thought distribution, for every question type. A choice answer is the option with the highest final probability. Thoughts and number answers are decoded with speculative decoding (Gemma's public MTP draft). The verification is greedy, so both are token-identical to plain greedy decoding.

Outputs are typed: choice, noul (yes/no probability), score (ordinal levels) and number (a free-form number). The model never returns free text as an answer.

Quantization

This checkpoint is engine-native. It cannot be loaded by transformers or vLLM.

part	format
MoE experts	NVFP4 W4A4 (e2m1 weights, FP8-e4m3 scale per 16 elements, FP32 global scale; activations NVFP4 with static per-layer global scales)
attention and dense MLP linears	FP8 e4m3 W8A8 (per-output-row weight scale, dynamic per-token activation scale)
tied embedding / LM head	FP8 e4m3 per row
vision tower, router, norms, KV cache	bf16

model.safetensors is 16.4 GB. GPU memory in use at serving time is about 26 GB.

Hardware

• An NVIDIA Blackwell GPU with compute capability 12.0 (sm_120a). Tested on an RTX PRO 6000 Blackwell (96 GB), where the latency below was measured, and on an RTX 5090 (32 GB).
• NVIDIA driver with CUDA 12.9 support (R575 or newer).
• To build: CUDA toolkit 12.8+ (nvcc) and Rust. No GPU is needed at build time.

Other GPU architectures are not supported, because the kernels use the sm_120a block-scaled FP4/FP8 tensor-core instructions.

Files

file	what
model.safetensors, config.json	quantized weights and the Gemma-4 config
tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json	tokenizer and chat template
processor_config.json	Gemma-4 image processor at the served budget: max_soft_tokens 560 (about 1.3 MP)
draft_vocab_16384.json	pruned 16384-token vocabulary for the MTP draft head
think_cost.json	fixed cost model of the deterministic thought budget (--think-cost; calibrated on an RTX PRO 6000)
engine/	source of dllm-server (Rust + CUDA; see engine/README.md)
launch.sh	finds or downloads the weights and the draft, then starts the server with the settings below
Dockerfile, .dockerignore	two-stage image: CUDA devel build, then CUDA base runtime
NOTICE	licenses and attributions

The MTP draft model is not in this repo. launch.sh downloads the public google/gemma-4-26B-A4B-it-assistant.

Quickstart

Docker:

hf download lihe07/span-02 --local-dir span-02 && cd span-02
docker build -t span-02 .
docker run --gpus all -p 8000:8000 -v "$PWD":/model:ro -e DLLM_MODEL_DIR=/model -v span02-data:/data span-02

Without -v "$PWD":/model ... DLLM_MODEL_DIR, the container downloads this repo itself. Pass -e HF_TOKEN for that while the repo is private.

Without Docker (CUDA 12.8+ toolkit and Rust installed):

hf download lihe07/span-02 --local-dir span-02 && cd span-02
(cd engine && NVCC=/usr/local/cuda/bin/nvcc cargo build --release -p dllm-server)
bash launch.sh          # serves on 0.0.0.0:8000; PORT / HOST / AR_API_KEY are optional

Startup takes about 10 s plus about 7 s of warmup. GET /health returns {"ok":true,"model":"span-02"} when the server is ready.

API

POST /v1/systemone; the format is TypeSafe's System One API (docs.typesafe.ai/api) plus the extensions marked below. GET /health returns {"ok": true, "model": "span-02"}.

Request fields:

• state (required): any JSON value (string, object, array, ...).
• questions (required): {question id: question}; answers come back in this order.
• model: accepted and ignored (e.g. "jev-latest"); the response model is always the served id span-02.
• assets: TypeSafe image assets, {placeholder: {"type": "image", "format": "png" | "jpg" | "jpeg", "data_base64": ...}}, or {"part": name} with a multipart/form-data body (part request = the JSON request, one file part per image). Referencing an asset is optional: a placeholder that occurs in state or in a question's instructions / criteria (any string value, also mid-string; object keys are never replaced) is replaced by the image's label. See "Images" below.
• images (extension, sent by the Decision Index kit's vision runner): a list of images as data URIs or bare base64. At most 4 images per request in total (images + assets).
• reasoning (extension): {"mode": "auto" | "on" | "off", "budget_ms": <ms>, "trace": <bool>}, all optional. mode: auto (default) uses the tau 0.8 gate, on thinks for every question, off never thinks. budget_ms (0 to 600000) is the wall-clock thinking budget of the request; 0 means none (a thought ends at the end-of-thought token, EOS or the cap). Without it, mode: "on" thinks to completion (up to the cap) and other requests use the server's --budget-ms. The deterministic --think-ms cap of the submission settings applies regardless; budget_ms only adds or removes a wall-clock deadline (the submission server has none), so requests without it stay deterministic. trace: true returns each thought's text.
• Other top-level fields are ignored.
• Bodies may be compressed with Content-Encoding: zstd (the TypeSafe Python SDK default) or gzip.

Question types:

• choice: criteria maps each option key to its description (1 to 679 options). The answer is the argmax, its confidence = (p_max - 1/n) / (1 - 1/n), and the probabilities in criteria order.
• noul: the probability that the statement is true. criteria is optional: {"true": ..., "false": ...}.
• score: criteria is an ordered array of 2 to 10 level descriptions, from low to high. The answer is score = sum_i i x p_i, with the per-level probabilities, a legend (the levels as sent) and a confidence.
• number (extension): a free-form number, no criteria. The model writes the number greedily after its (optional) thought, as it was trained (the plain decimal, e.g. 391 or 2.5, at most 32 tokens). The answer is {"type": "number", "number": 391.0, "text": "391", "confidence": 0.98}: number is the parsed value (null if the text is not a number), confidence = exp(mean log-probability of the generated answer tokens). The think gate is trained for option questions only, so number questions think only with "reasoning": {"mode": "on"}.

Request:

curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "model": "span-02",
  "state": {"customer": "My package arrived damaged and nobody answers my emails. I want a refund now.",
            "order": {"id": 123, "days_late": 4}},
  "questions": {
    "dept":   {"type": "choice", "instructions": "Which team should handle this?",
               "criteria": {"returns": "Exchanges, wrong or damaged items",
                            "shipping": "Delivery status, delays, lost packages",
                            "billing": "Charges, invoices, payment problems"}},
    "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
    "anger":  {"type": "score", "instructions": "How frustrated is the customer?",
               "criteria": ["Not frustrated: calm or positive", "Somewhat frustrated: mild annoyance",
                            "Very frustrated: angry, hostile or threatening to leave"]}},
  "reasoning": {"mode": "auto", "trace": true}}'

Response:

{"model": "span-02",
 "answers": {
   "dept":   {"type": "choice", "choice": "returns", "confidence": 0.945,
              "probabilities": {"returns": 0.963, "shipping": 0.024, "billing": 0.013}},
   "refund": {"type": "noul", "noul": 0.99999},
   "anger":  {"type": "score", "score": 1.9999,
              "legend": {"0": "Not frustrated: calm or positive", "1": "Somewhat frustrated: mild annoyance",
                         "2": "Very frustrated: angry, hostile or threatening to leave"},
              "probabilities": {"0": 0.000008, "1": 0.00012, "2": 0.99987}, "confidence": 0.9998}},
 "usage": {"input_tokens": 412, "reasoning_tokens": 0, "output_tokens": 0, "latency_ms": 41.7}}

(The token counts and latency in this response are illustrative.)

• An answer whose question thought has "reasoning": {"tokens": <thought tokens>, "trace": "<thought text>"} (trace only when the request asked for it); answers that did not think have no reasoning field.
• usage holds request totals: input_tokens = the prompt tokens of every question (the shared state is counted once per question; each image counts its soft tokens + 2), reasoning_tokens = thought tokens generated, output_tokens = answer tokens generated (the tokens of number answers; choice, noul and score answers are read from one forward pass and generate none), latency_ms = server time from arrival to response.
• With images, the response also has a top-level image_tokens field (the total image soft tokens).

Images

The model was trained with all images (up to 4) in front of the text, in order, and with the text calling them "Image 1", "Image 2", ... So the server puts the images first and then the assets in the order they are sent, and replaces every occurrence of an asset placeholder in the state, instructions and criteria by that image's label. For {"state": {"photo": "a3f1c2e8-found"}, "assets": {"a3f1c2e8-found": ..., "77b0d4a1-poster": ...}} the model reads {"photo": "Image 1"}, and an instruction "... as in this picture: 77b0d4a1-poster?" becomes "... as in this picture: Image 2?". Each asset is shown once, however often it is mentioned; an asset that is never mentioned is shown all the same (one unreferenced asset renders exactly like one images entry). A score legend echoes the placeholder, not the label. Requests without assets are rendered exactly as before. Assets are PNG or JPEG (the format must match the bytes), at most 16 MiB each; they are resized to the image budget, not rejected for their dimensions.

Errors

{"error": {"type": ..., "message": ...}}, one status per kind:

status	type	when
400	invalid_request	invalid JSON, a malformed compressed or multipart body
400	invalid_asset	a rejected asset: unknown type or format, bad base64, not a PNG/JPEG, missing or unused multipart part, more than 4 images
400	max_tokens_exceeded	the prompt (+ images + thought reservation) exceeds the context, or the request cannot fit in the whole KV cache; the message contains "maximum context length"
401	authentication_error	missing or wrong bearer token (only with AR_API_KEY)
404	not_found	any path other than POST /v1/systemone and GET /health
413	request_too_large	body over 64 MB (also after decompression), an image over 16 MiB
415	unsupported_encoding	a Content-Encoding other than zstd / gzip, or more than one layer
422	validation_error	schema errors: missing fields, bad question types or criteria, too many options, bad reasoning, more than 4 or undecodable images
529	overloaded	concurrent requests hold the KV cache; retry after a short delay
500	server_error	an internal failure

The capacity errors carry the Decision Index capacity markers, so the DI client records unsupported.

Serving settings (Decision Index submission)

--max-len 131072 --budget-ms 0 --tau 0.8 --cap 1024 --think-ms 750 --think-cost think_cost.json
--image-soft-tokens 560 --serial --no-prefix-cache --spec-max-branches 16 --pages 12000 --kv-gb 24

• --budget-ms 0: there is no wall-clock limit; the clock is never read for thinking decisions.
• --think-ms 750 --think-cost think_cost.json: deterministic thought budget. Each request's per-question thought cap = floor((750 - fixed) / per_token(n)), where fixed is the modelled prompt, image and readout time of that request and per_token(n) the modelled time per thought token with n of its questions thinking together. It is a pure function of the request, so answers do not depend on the machine or its load. A thought ends at the end-of-thought token, at EOS, or at that cap (at most --cap 1024). A short single question gets about 330 thought tokens, a request with many thinking questions or a long prompt fewer.
• --tau 0.8: think when the no-think top probability is below 0.8. The final distribution is then 0.5 x no-think + 0.5 x post-thought (choice, score and noul alike).
• --spec-draft (passed by launch.sh): MTP speculative decoding of thoughts and number answers with an adaptive draft depth and a pruned 16384-token draft vocabulary. It is lossless (token-identical to greedy decoding).
• --image-soft-tokens 560: the Gemma-4 image budget (about 1.3 MP; aspect ratio kept).
• --serial --no-prefix-cache --spec-max-branches 16: each answer is a deterministic function of its request. It does not depend on load or on earlier requests (for up to 16 questions of one request thinking at once). --serial runs one request at a time; concurrent clients queue.
• --max-len 131072: the context limit per question (prompt + images + thought).
• --pages 12000 --kv-gb 24: KV cache pools (full-attention layers 12000 x 16 = 192K tokens, 3.9 GB; sliding-window layers get the rest of --kv-gb 24, ~20 GB), sized so that every Decision Index request fits on its own under --serial (the largest needs about 3.7K full pages: many questions with long tails). The pool size does not change any answer. The server uses about 41 GB of GPU memory.

Latency

Measured with the submission settings on one RTX PRO 6000 Blackwell, one request at a time, as client wall time per request including HTTP (ms). The Decision Index rule requires the median, mean and p80 each to be at most 1000 ms.

request mix	n	median	mean	p80	accuracy (choice questions)
DI public text 0.3, stratified sample (42 benchmarks), DI kit	400	71	308	712	0.923 (2725 questions)
DI public vision: CV-Bench, BLINK, RealWorldQA, DI kit	150	186	291	642	0.747
image rows of 6 DI vision benchmarks (incl. MMMU-Pro vision), DI kit	210	292	389	699	0.652
think-heavy question set (78 % of questions think)	150	616	514	770	0.562

The rule is met on every mix measured, with margin. Against a plain 256-token cap on the same machine and requests, accuracy is the same within noise (pooled paired difference -0.03 points, 95 % CI [-0.39, +0.34]) and the tail is much shorter (DI text requests over 1 s: 34 -> 4 of 400; slowest 2.8 s -> 1.07 s). Against a blend that puts the post-thought half on the post-thought top option only (one-hot), the averaged distributions have the same accuracy within noise (+0.13 points, 95 % CI [-0.06, +0.35], 3,707 choice questions, same thoughts) and are better calibrated (log-loss 0.409 -> 0.397, Brier score 0.200 -> 0.193; on the questions that thought 1.368 -> 1.290 and 0.734 -> 0.689). Speculative decoding runs at about 480 thought tokens/s at batch 1 on real thoughts on this GPU (2.1 ms per token).

Capacity limits

Requests over these limits are rejected with the DI capacity markers. Nothing is truncated.

• Choice questions: 1 to 679 options.
• Images: at most 4 per request (images: PNG, JPEG, GIF, BMP, TIFF or WebP; assets: PNG or JPEG). Each image uses at most 560 soft tokens.
• Context: the prompt tokens + 562 per image + the reserved thought tokens + 3 must fit in 131072 tokens (--max-len 131072; the sliding-window layers keep only their 1024-token window, so long prompts stay cheap). The reservation is the deterministic cap of a lone thinking question of that size; at long prompts it is small or zero (the cost model leaves no thought time).
• KV cache: a request whose prompt pages (shared prefix + every question's own tail) exceed the whole KV pool is rejected with 400 max_tokens_exceeded ("maximum context length"); none of the DI 0.3 public requests does with the pools above.
• Request body: at most 64 MB.

License

The weights are a derivative of Google's Gemma 4 (google/gemma-4-26B-A4B-it) and are distributed under the Gemma 4 license terms, which are Apache 2.0: ai.google.dev/gemma/docs/gemma_4_license. The serving engine in engine/ is Apache-2.0. It ports algorithms from vLLM, SGLang, FlashInfer and other projects, which are credited in NOTICE and in the source file headers.