electroglyph/d1-2B-onnx

d1-2B — ONNX

ONNX text builds of LiquidAI's d1-2B (a 2B decision model on LFM3-VL), runnable with onnxruntime 1.31.1. No SGLang, no generation server needed at inference.

This is a decision model, not a chatbot: you send a state plus typed questions and get back calibrated answers in one forward pass — zero tokens generated. The main script is score_d1.py, which speaks the checkpoint's own System One API (system_one(state, questions)).

The builds preserve the checkpoint's serving attention: the 6 full-attention layers run the packed set mask (state bidirectional, each question sees the state plus its own question), exactly as trained (d1_attention: set, d1_set_size: 16). Temperature 1.337 and the readout tables from calibration.json are applied, so outputs are comparable with the PyTorch serving.

Decision scoring

pip install -r requirements.txt          # GPU line, or swap to the CPU line
python score_d1.py --request examples/support.json --model ./model-cuda-int8 --cuda

A request is {"state": ..., "questions": {...}} with the same schema as model.system_one. Three question types (logits-only build — reads, boxes and estimates need hidden states the graph does not output):

• noul — yes/no (Should the agent offer a refund? → {"noul": 0.71})
• choice — pick one (priority → urgent at 0.73)
• score — rubric levels, answered at the median level by default (--score-readout mean|median|argmax)

Example 1: support triage (examples/support.json)

python score_d1.py --request examples/support.json --model ./model-cuda-int8 --cuda

{
  "answers": {
    "refund": {"type": "noul", "noul": 0.7133},
    "priority": {"type": "choice", "choice": "urgent", "confidence": 0.7339,
      "probabilities": {"low": 0.1145, "normal": 0.1516, "urgent": 0.7339}},
    "tone": {"type": "choice", "choice": "angry", "confidence": 0.6472,
      "probabilities": {"angry": 0.6472, "neutral": 0.3306, "happy": 0.0222}},
    "satisfaction": {"type": "score", "score": 0.0, "confidence": 0.5005,
      "probabilities": {"0": 0.5005, "1": 0.4200, "2": 0.0534, "3": 0.0149, "4": 0.0112},
      "legend": {"0": "very dissatisfied", "1": "dissatisfied", "2": "neutral",
                 "3": "satisfied", "4": "very satisfied"}}
  },
  "usage": {"input_tokens": 211, "output_tokens": 0}
}

Read: refund leans yes but it's the close call (0.71), ticket is urgent, customer is angry, satisfaction median level 0 with real mass on 1 (0.42).

Example 2: content moderation (examples/moderation.json)

python score_d1.py --request examples/moderation.json --model ./model-cuda-int8 --cuda

{
  "answers": {
    "violates": {"type": "noul", "noul": 0.9859},
    "category": {"type": "choice", "choice": "scam", "confidence": 0.9798,
      "probabilities": {"legit": 0.0021, "spam": 0.0160, "scam": 0.9798, "abuse": 0.0021}},
    "severity": {"type": "score", "score": 4.0, "confidence": 0.5112,
      "probabilities": {"0": 0.0565, "1": 0.0310, "2": 0.0973, "3": 0.3040, "4": 0.5112}}
  },
  "usage": {"input_tokens": 196, "output_tokens": 0}
}

Read: violates policy (0.986), scam not mere spam (0.98), severity critical (median level 4, though 3 holds 0.30 — have a human confirm severe-vs-critical).

Notes for decision use

• Score answers default to the median level; --score-readout mean gives the expectation instead.
• Up to 16 questions per request read together (d1_set_size); larger requests run as multiple passes automatically. Text only — pictures, audio and video need the PyTorch serving in the source repo.
• Near-0.5 noul scores are coin flips: escalate, don't threshold blindly.

Contents

path	what
model-cuda-int8/	CUDA int8 build (recommended on GPU): model.onnx + model.onnx.data (~2.1 GB). Worst miss vs original 0.015
model-cpu-int8/	CPU int8 build: model.onnx + model.onnx.data (~2.3 GB). Worst miss vs original 0.024
model-cuda-int4/	CUDA int4 build (smallest): model.onnx + model.onnx.data (~1.2 GB). True int4, k-quant, block 32, only the LM head at int8. Worst miss vs original 0.082
examples/	ready-made decision requests (support.json, moderation.json)
onnx_d1.py	shared runner: OnnxD1 (single-forward prefill, block_bias feed)
score_d1.py	decision scoring (this README's main script)
compare_d1.py	accuracy + speed check vs the original (needs torch + graphics card)
d1ref/	the checkpoint's own engine modules (byte-identical copies: rendering, set-attention mask, calibration, readouts)
calibration.json	the checkpoint's temperature + readout tables (used by the scorer)
orig_tokenizer/	pristine tokenizer
requirements.txt	runtime deps
config.json (in each build)	source architecture config (Hub convention)
LICENSE	LFM Open License v1.0 (same as the source repo — see below)

How it was built

torch.onnx.export cannot handle this model (hybrid conv/attention layers with custom kernels), and the checkpoint's D1Model architecture is unknown to stock tooling. The path is Microsoft's onnxruntime-genai ModelBuilder (0.19.0, onnxruntime 1.31.1), which supports LFM3, against a copy of the checkpoint with two local compat tweaks (weights symlinked, source untouched): the config architecture renamed to its backbone (Lfm3VlForConditionalGeneration), plus shims for the copy of transformers/​kernels this environment ships (reference-torch fallback for one decorator spelling, VL auto-class registration).

# text-only exports (the vision tower is not exported)
python -m onnxruntime_genai.models.builder \
  -i <patched-checkpoint> -o model-cuda-int8 -p int8 -e cuda \
  --extra_options text_only=true block_size=32
python -m onnxruntime_genai.models.builder \
  -i <patched-checkpoint> -o model-cpu-int8 -p int8 -e cpu \
  --extra_options text_only=true accuracy_level=4 block_size=32
python -m onnxruntime_genai.models.builder \
  -i <patched-checkpoint> -o model-cuda-int4 -p int4 -e cuda \
  --extra_options text_only=true block_size=32 algo_config=k_quant \
  matmul_mixed_precision=last_matmul:int8

Then per build, graph surgery for the attention mask: ORT's GQA op has no mask input, so each of the 6 full-attention GQA nodes was replaced with an exact query-chunked eager subgraph (8 chunks, shared block_bias input) replicating the node's QK-RMSNorm and NeoX RoPE; sliding-window GQA and the conv layers stay stock (causal in every mode). Weights are untouched.

All builds are text-only decoders; pictures, audio and video need the PyTorch serving.

Running it: raw onnxruntime (not onnxruntime-genai)

The runner drives the graph with plain onnxruntime (single forward, empty past states — decisions never decode). genai_config.json is kept as exported.

Verification

Measured on this machine (RTX 3090 Ti graphics card + processor) with compare_d1.py: the original (full precision, graphics card) against each build on the same 3 requests / 10 questions. Times are per request, after warmup. Same answers means the top-probability answer matches throughout (see the median-readout note under the table).

model	file size	support	moderation	biggest miss vs original	same answers
original (full precision)	~3.6 GB	0.02 s	0.02 s	—	—
model-cuda-int8	2.1 GB	0.07 s	0.06 s	0.015	10 of 10
model-cuda-int4	1.2 GB	0.07 s	0.06 s	0.082	10 of 10
model-cpu-int8	2.3 GB	0.44 s	0.40 s	0.024	10 of 10

• The 8-bit builds are the faithful ones (worst miss 0.015 graphics, 0.024 processor).
• The small 4-bit build matches on every top answer but smears close calls: moderation severity reads median level 3 vs original 4 (top answer is still 4 on both). Treat smeared levels as "have a human confirm".
• Self-test (refund/tone/satisfaction): original 0.9823/0.6582/0.7151 vs cuda-int8 0.9821/0.6476/0.7001, cpu-int8 0.9824/0.6457/0.6909 (all within 0.025 of the original), cuda-int4 within 0.09.

Notes / limits

• Provider matrix is strict: model-cuda-int8 and model-cuda-int4 need CUDAExecutionProvider (--cuda), model-cpu-int8 needs CPU. The CUDA builds keep the QK-norms inside GQA, which the CPU kernel does not implement (verified: cross-EP runs fail); the CPU export hoists them out. Tested: CUDA builds live-run on a 3090 Ti, CPU build on CPU.
• INT8 barely moves probabilities vs bf16; the small INT4 smears close calls (see severity above). The int8 builds are the threshold-safe choice.
• score_d1.py serves noul/choice/score only; reads, boxes, estimates and media need the PyTorch serving (the graph outputs logits only).
• No chat generation: the runner is prefill-only (decisions never decode).

License

• Weights (*.onnx.data), config.json, d1ref/ modules, calibration.json, tokenizer: LFM Open License v1.0, same as the source repo (LiquidAI/d1-2B) — see LICENSE. Note its commercial-use term: commercial use is licensed only below $10M annual revenue (qualified non-profits doing non-commercial/research work are exempt from the threshold).
• Runner scripts (onnx_d1.py, score_d1.py): MIT.