hyperquader/ministral-3-3b-instruct-genai-int4

Ministral 3 3B Instruct — ONNX Runtime GenAI, int4

This is a format conversion, not a fine-tune. Nothing was trained. The weights are mistralai/Ministral-3-3B-Instruct-2512, converted to an ONNX Runtime GenAI model directory and quantised to int4 so a desktop can load it without PyTorch. If you want the model, use the original; this exists because Amber Linux's broker loads genai directories and there was no such build to fetch.

Mistral publishes an ONNX export of this model, but in the transformers.js layout — decoder_model_merged, embed_tokens, no genai_config.json — which onnxruntime-genai does not load. That is the gap this fills.

What is in it

property	value
base	mistralai/Ministral-3-3B-Instruct-2512 (Apache 2.0)
format	onnxruntime-genai model directories, one per execution provider
precision	int4
model type	mistral3_text — the language tower; the vision encoder is not converted
context	262144

Built with the onnxruntime_genai.models.builder from the version this directory's genai_config.json records. Convert and run with the same genai version: a builder ahead of the runtime writes a config the runtime cannot load, and it says so in a way that reads like a broken model.

That version has no native upstream release, only Python wheels — so pip install onnxruntime-genai-cuda runs it, and for anything that is not Python, amberlin-runtime publishes a linux amd64 build of the pinned commit as Debian packages.

Two builds, one tokenizer

cpu/ and cuda/ each hold a complete graph, its weights and its own genai_config.json. The tokenizer, the tokenizer config and the chat template are byte-identical between them and sit at the root, so they are stored once.

Take the root files and one directory, and flatten it — genai loads a single directory and will not look into a subfolder:

hf download hyperquader/ministral-3-3b-instruct-genai-int4 --local-dir m \
  --include 'cuda/*' --include 'tokenizer*' --include 'chat_template.jinja'
mv m/cuda/* m/ && rmdir m/cuda

Repeat --include per pattern. Several patterns after one flag are accepted and the first is then silently ignored, which downloads a directory that looks complete and will not load.

A CUDA build does load on a CPU if provider_options is emptied, which makes the two look interchangeable. They are not: on the same machine that path runs at 21 tok/s against the CPU build's 33. The graphs are quantised differently, which is also why the CPU build is the larger of the two.

Measured here

Greedy decode, 128 tokens, best of three runs, onnxruntime-genai 0.16.0:

device	on disk	load	decode
CPU — i7-13700K	2.28 GB	3.9 s	33 tok/s
CUDA — RTX 5070 Ti	2.01 GB	1.0 s	323 tok/s

Each row is its own build, not one artefact measured twice.

Tool calling, without any training

A call is [TOOL_CALLS]name[ARGS]{…} — both markers are single special tokens, so the arguments are a plain JSON object and a call ends where that object balances.

Scored on 119 held-out prompts against a live tool server, with the 13-tool catalogue in the prompt and generation constrained by a grammar built from the tool schemas:

measure	result
chose the right tool	98 / 103
declined correctly when no tool applied	16 / 16
total correct	114 / 119
catalogue cost	2,837 prompt tokens

Two things that number depends on, stated because they are easy to miss. The grammar constrains form, never judgement: it makes a malformed call and an out-of-enum argument impossible to emit, and it does not make the model choose correctly. And the catalogue is doing the work — offered no catalogue, this model takes the free-text branch on every prompt that wanted a call. A grammar is not a substitute for telling a model the tools exist.

The remaining failures are mostly arguments rather than routing, and a measurable part of them are disagreements with the reference answer rather than mistakes.

Licence and attribution

Apache 2.0, inherited from the base model. The weights are Mistral AI's work; this repository contributes the conversion and the measurements. Cite mistralai/Ministral-3-3B-Instruct-2512 for the model itself.