peonist-ai/halogen-qwen3.8-flash-next

halogen-flash, Qwen3.8-Flash-Next for AMD Strix Halo

Weights for halogen-flash-server, a dedicated inference engine for Qwen3.8-Flash-Next on AMD Strix Halo (gfx1151). One model family, one GPU, every kernel written for this silicon.

These weights are in halogen's own .hgn format and will not load in transformers, vLLM, or llama.cpp. They exist to be mounted into the halogen-flash-server container.

What is here

qwen38-flash-next-w4b.hgn              115.55 GiB   checkpoint (1198 tensors)
qwen38-flash-next-w4b.overlay.hgn        2.31 GiB   quality sidecar, KEEP THIS
qwen38-flash-next-w4b.overlay-speed.hgn  2.22 GiB   optional: the speed arm
tokenizer/                                          flat tokenizer directory

The checkpoint is TWO files. The sidecar is a patch overlay: a subset of tensors re-quantized more carefully, read in place of the base file's copies. The engine loads <checkpoint>.overlay.hgn automatically when it sits beside the checkpoint, you do not configure anything, you just keep both files. Drop it and the model still runs, roughly 5–9% worse on perplexity, saying so in one line of startup output.

Download both. Together they are ~118 GiB, and the model needs ~68 GiB resident plus the memory-mapped n-gram table.

Use

hf download peonist-ai/halogen-qwen3.8-flash-next --local-dir ~/halogen-models

podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --security-opt seccomp=unconfined --ipc=host --ulimit memlock=-1:-1 \
  -v ~/halogen-models:/models:ro \
  ghcr.io/peonist-ai/halogen-flash-server:0.1.0

This repo carries the tokenizer, so one -v is all you need. On Docker rather than Podman, replace --group-add keep-groups with --group-add video --group-add render: keep-groups is a Podman keyword that Docker resolves against the container's /etc/group and fails to find.

An OpenAI-compatible endpoint comes up on :8731. Full documentation is in the server repository.

Precision

4-bit weights are a correctness precondition, not an optimization: Qwen3.8-Flash-Next is 125B total parameters plus a 51B n-gram embedding table, and neither BF16 (335 GiB) nor FP8 (173 GiB) fits in Strix Halo's 124 GB of unified memory. The question was never whether to quantize but where to spend the bits.

base file	115.55 GiB, 1198 tensors
trunk + experts	Q4C-P (4-bit, per-column groups)
n-gram embedding table	FP8, 47.7 GiB, a lookup, paged rather than resident
rank-1 / conv1d / PLE projections	BF16 pass-through
quality sidecar	723 non-expert tensors re-quantized activation-aware, plus the twelve o_proj at 8-bit

The sidecar is where most of the quality lives, and it is small because the loss was concentrated. Measuring each tensor family against its own BF16 ceiling put nearly all of the non-expert quantization cost in twelve o_proj tensors, 106 MB, 0.09% of the file. Those twelve now ship at 8 bits, which measures as a statistical tie with full BF16 rows. The mechanism is calibration: a 4-bit trunk is over-confident, and the whole gain sits in the hardest quartile of predictions.

Speculative decoding

The checkpoint carries the model's MTP head (a full Flash-Next layer , 512 experts and its own indexer). Speculative decoding with it is byte-identical to serial greedy decode: the head only proposes, and a token is emitted only if the trunk would have produced it. It is a speed optimization with no quality cost, on by default, and disableable per request.

Provenance and license

Derived from Qwen3.8-Flash-Next. These weights are a derivative work and are governed by the license of the original model, see the base model for those terms. Verify them yourself before commercial use; nothing here grants rights the upstream license does not. Unlike halogen's 27B checkpoint, no third-party quantization was imported: every quantized value here was produced by us from the BF16 originals.

The halogen-flash engine is separate, closed source, and distributed under its own terms, see the server repository.