Orca-Qwen3.8-Flash-Next-W4A16-Merlin

orcarouter's uncensored Qwen3.8-Flash-Next, rebuilt tensor-for-tensor into halt95's Merlin W4A16 layout so it runs on the stock Merlin 3.0.0 engine on four RTX 3090s at the full 262,144-token context. The routed experts are Mochgolf's AutoRound INT4 quant of the same orcarouter weights, repacked from auto_gptq into compressed-tensors. The FP8 KV scales were recalibrated on this checkpoint.

This model does not refuse. It is an abliterated derivative. You are responsible for how you deploy it and for any filtering your use case needs.

Where every tensor comes from

Component	Source	Format here
Routed experts	Mochgolf AutoRound (0.15.0, 200 iters, 128×2048 pile-10k, Intel's recipe) of orcarouter@d9dbbb7f, at Mochgolf@97e37898	INT4 sym g128, compressed-tensors pack-quantized, BF16 scales
MTP draft experts	orcarouter BF16	INT4 sym g128 RTN (mtp-routed-experts-int4.safetensors)
GDN in_proj_qkv / in_proj_z / out_proj	orcarouter BF16	INT8 per-channel RTN (scale floored at 1e-8 like halt95)
PLE n-gram table	orcarouter BF16	FP8 E4M3, one global BF16 scale (amax/448)
Everything else (attention, shared experts, routers, vision, embeddings…)	orcarouter BF16	BF16, unchanged
FP8 KV scales	calibrated on this checkpoint (2K–255K-token traffic + images, margin 1.10)	qsa_kv_scales.json

Config, tensor names (227,569) and file layout match halt95/Qwen3.8-Flash-Next-W4A16-Merlin@d9d99c3. Every tensor orcarouter did not modify is byte-identical to that build. PROVENANCE.txt records the exact source revisions, and convert-stats-*.jsonl holds the per-tensor conversion statistics.

Running it

Built for and served on stock Merlin 3.0.0 (halt95's vLLM 0.32.0 fork, tag merlin-v3.0.0, profile fn-quad-3090) with no local patches. Merlin's launcher picks up the checkpoint's own qsa_kv_scales.json.

docker run -d --name orca-flash --gpus all --shm-size 8g \
  --ulimit memlock=-1 --ulimit stack=67108864 -p 8000:8000 \
  -e MERLIN_MODEL=flash-next -e CHECKPOINT_DIR=/models/checkpoint \
  -e KV_BYTES=1950000000 -e MODEL_NAME=flash-next-orca \
  -v /path/to/Orca-Qwen3.8-Flash-Next-W4A16-Merlin:/models/checkpoint:ro \
  <merlin-engine 3.0.0 image> --gpu-memory-utilization 0.80

• KV_BYTES=1.95e9 per rank gives a 438K-token pool. Merlin ships 4.1e9. The smaller pin and --gpu-memory-utilization 0.80 leave about 2 GB free per card for other GPU work.
• Merlin defaults kept: MTP K=3, prefix caching, RecoverSSM, FP8 KV, reasoning effort medium, and Froggeric's Qwen3.8 chat template v22.5. chat_template.jinja in this repo is the checkpoint's own template, if you serve without Merlin's.
• Host RAM: the PLE worker keeps about 46 GB resident, and the launcher wants about 69 GiB available at boot.
• Boot time: about 4 minutes from NVMe.

Evaluation

Measured on 4× RTX 3090 against the previous build on the same box (a re-quantized leoncca AWQ checkpoint on a patched Merlin v2.5.2), with the same scripts:

	This build	Previous build
WikiText-2 perplexity	3.67	6.07
Code perplexity	1.87	2.66
GSM8K-250, 5-shot, no thinking	98.0% flexible / 98.0% strict	95% / 77% (first 100)
Needle retrieval at 25K / 99K / 156K / 249K	all found	all found (to 156K)
Refusals	0/10	0/10
Decode, 1 stream, no thinking	136.5 tok/s (MTP K=3)	70 tok/s (MTP off)
Peak VRAM per card, under load	21.97–22.61 GB	23.36–24.09 GB

It also passed 52 concurrent and multi-turn tool-calling agent sessions with zero HTTP errors and no output corruption.

Expert quantization error (dequantized vs BF16, mean relative error, gate/up/down):

• AutoRound: 0.108 / 0.107 / 0.106
• Plain round-to-nearest on the same matrices: 0.116 / 0.115 / 0.112

Files

• model-dense-L*.safetensors, model-experts-L*.safetensors: per-layer dense and routed-expert shards
• model-plefp8-*.safetensors, model-ple-static.safetensors: PLE table
• model-top.safetensors, model-visual.safetensors, model_extra_tensors.safetensors, mtp-routed-experts-int4.safetensors
• qsa_kv_scales.json: FP8 KV scales for this checkpoint
• recipe/: the converter (convert_orca_merlin.py), the byte-identity audit against halt95 (audit_vs_halt95.py) and the KV calibration driver (calibrate_kv.py), as used. They expect the original build machine's paths, so treat them as documentation rather than a turnkey pipeline.

License and acknowledgements

Qwen/Qwen3.8-Flash-Next is under the Qwen Community License 1.0 (included as LICENSE). halt95's and Mochgolf's releases use the same license, and orcarouter's repo is tagged Apache-2.0. This derivative is distributed under the Qwen Community License. Its main extra condition applies to very large commercial products or services: above 100M monthly active users or US$20M monthly revenue, the model name must be shown prominently in the product's interface.

Built from the work of the Qwen team (base model), orcarouter (refusal removal), Mochgolf (AutoRound expert quant) and halt95 (the Merlin layout and engine).