seimanu/decider-4b-NPU2

decider-4b for the Ryzen AI NPU

Mapika/decider-4b — a decision model — converted to run on the Ryzen AI NPU with FastFlowLM. Built on Qwen/Qwen3.5-4B-Base.

A decision model does not generate text. It reads a state and one or more typed questions, each with an explicit option list, and returns a distribution over the options. It is called from software, not chatted with.

What is here

model.q4nx             4,02 GB   the weights, converted for the NPU
config.json                      model configuration for flm
tokenizer.json                   tokenizer
tokenizer_config.json            tokenizer configuration

Tested with

manueldomke/FastFlowLM — a build of FastFlowLM with two commits on top: a fix for a sample the engine draws and throws away (PR #776), and the patch described below, which passes the sampler's logits through as logprobs (PR #777). Upstream ROCm/FastFlowLM runs these models too; it just cannot report the probabilities.

Everything here was built from commit 5717973 of that fork and run on a Ryzen AI 7 PRO 350, which is the machine the timings in those two pull requests come from.

How to run it

Copy this directory to ~/.config/flm/models/Decider-4B-NPU2/ and add an entry to your flm model list:

"qwen3.5:decider4b": {
  "name": "Decider-4B-NPU2", "url": "", "file_url": "", "size": 0,
  "flm_min_version": "1.0.3",
  "files": ["config.json", "model.q4nx", "tokenizer.json", "tokenizer_config.json"],
  "vlm": false, "default_context_length": 32768, "max_prefill_len": 4096,
  "details": {"format": "NPU2", "family": "qwen3.5", "think": false,
              "parameter_size": "4B", "quantization_level": "Q4_K"}
}

Then flm run qwen3.5:decider4b.

The prompt format matters. The model expects the questions and their options first, then the state, then an answer slot:

Question: Which team should handle this
Options:
(A) billing
(B) technical support
(C) sales

Context:
My card was charged twice for the same purchase.

Answer with the letter of one option only.

Answer: (

Using it from software

A decision model is called from code, not chatted with, and flm speaks the OpenAI completion API. systemone.py in this repository is the piece in between: it takes a request in System One's shape, builds the prompt the model expects, asks flm for one token, and turns the logits into an answer.

python3 systemone.py --port 6291 --upstream 52630 --model qwen3.5:decider2b

It needs the patched flm from the two pull requests above -- without the logprobs patch it has no distribution to read.

Request:

POST http://127.0.0.1:6291/v1/systemone
{"state": "My card was charged twice.",
 "questions": {"q1": {"type": "choice",
                      "instructions": "Which team should handle this?",
                      "criteria": {"A": "billing", "B": "technical support"}}}}

Answer:

{"object": "structured_decision",
 "answers": {"q1": {"type": "choice", "choice": "billing", "label": "A",
                    "confidence": 0.66, "probabilities": {"billing": 0.66,
                                                          "technical support": 0.34}}}}

Three question types: choice (pick one option), noul (yes/no) and score (a level on a scale; reported as the expected level, not the most likely one).

The model's own temperatures are in the file and belong on the caller's side -- the distribution flm reports is before the temperature, and the adapter applies the one the model was trained with. They differ per model and are listed at the top of systemone.py.

A fault in the request comes back as HTTP 422 and not as a value under answers: an empty question would otherwise read like a model that cannot decide.

How it was converted

With FLM_Q4NX_Converter from the decider-4b-v2.1-Q4_K_M GGUF, using the qwen3.5-4B architecture.

Five fixes to that converter were necessary — without them the model either refuses to load in flm or answers with a constant token for every input. A pull request to FLM_Q4NX_Converter fixes them: the bf16 prefill copies of the linear attention were computed and thrown away; tied embeddings left model.embed_tokens.weight out of the file; the target tensor type from the config was ignored whenever the source type was a known format; there was no Q4_K encoder; and linear_out_proj was set to Q4_1 in the 2B and 0.8B configs where the 4B has Q4_K.

Reading the probabilities

flm writes a hard-coded {"logprobs", nullptr} into its responses, although the sampler keeps the top-k of every sampled token. The patch that passes them through is PR #777, which builds on PR #776. With it, logprobs carries one entry per token of the answer, in order, each with the token's log-probability over the whole vocabulary, before the temperature. The model's own temperatures are choice 1.110, noul 1.560, score 1.287 and belong on the caller's side.

Without that patch the model still answers correctly — it returns the letter, just not its probability.

Limits

At most 26 options per question. The answer is read from the first generated token, and A-Z are one token each. From the 27th option the labels become two characters and no longer fit into a single token, so the distribution collapses.

The first entry is the decision. The model answers with one token, so the first entry carries the whole answer; the ones after it describe the rest of the sentence, which for a decision model is usually not generated at all.

Quantised to Q4_K, so slightly less accurate than the bf16 original. That is the usual trade for running on the NPU; no measurement of the difference is claimed here.

License

Apache-2.0, inherited from Mapika/decider-4b.