dongnhdev/Qwen3.8-Flash-Next-OrcaUncensored-IQ2-Light

Qwen3.8-Flash-Next OrcaUncensored — DS4 IQ2 (Light)

Qwen3.8-Flash-Next in the light DS4-IQ2 packaging of ivanfioravanti/Qwen3.8-Flash-Next-DS4-IQ2: an IQ2 main (IQ2_XXS gate/up, Q2_K down padded to 768, embedded MTP block) plus an external demand-paged PLE Q4_1 sidecar instead of a ~95 GiB resident BF16 n-gram. Runs resident and zero-swap on a 64 GiB Apple Silicon box (measured on M5 Pro), 8K→220K context.

Performance (M5 Pro 64 GiB, measured)

model	MTP-off	single-stream --mtp	notes
Q8 dense (exact)	~30–32 t/s	~38 t/s	most accurate
Q4_K imat dense	~36 t/s	~42–43 t/s	+~14% vs Q8, near-Q8 quality

• MTP (--mtp) adds ~+17% single-stream over MTP-off; draft acceptance ~67%.
• Adaptive draft depth (engine env DS4_QWEN4_MTP_DEPTH, default auto): drafts a 2nd token on deterministic/structured/code continuations for a further +5–6%, and falls back on free-form prose so it never regresses. Output stays autoregressive-exact (verify only commits argmax-matching drafts → zero quality change).
• Multi-session throughput (engine --batched-session N, concurrent requests, full quality): ~67 t/s @ 8 streams, ~80 t/s @ 16 streams aggregate (~2× a single stream). Decode is memory-bandwidth-bound on this hardware, so aggregate throughput plateaus around ~80 t/s rather than scaling linearly; a single stream cannot exceed ~43 t/s here (that ceiling needs a smaller model or more MTP heads, not tuning). Concurrency is bounded by ctx × sessions KV vs 64 GiB (ctx 4096 fits 16 streams; large ctx needs fewer).

Two dense-quant mains (pick one; both share the same PLE sidecar)

The IQ2 experts are identical across both; they differ only in how the per-layer dense projections are quantized:

main	dense	size	decode vs Q8	logit cosine vs Q8	pick when
...-Q2KDownPad768-MTP.gguf	Q8_0	41.73 GiB	baseline	1.000 (exact)	most accurate output
...-DenseQ4Kselimat-MTP.gguf	selective Q4_K + imatrix	40.68 GiB	+~14%	0.949 (argmax preserved)	faster decode, near-Q8 quality

The DenseQ4Kselimat variant requantizes the less-sensitive dense projections Q8_0→Q4_K (routed to DS4's fast dense Q4_K GEMV) while keeping the most quality-sensitive full-attention q/k/v/output projections at Q8_0, and weights each dense tensor's Q4_K quantization by an importance matrix (activation energy per input column). Net: ~+14% decode at logit cosine ~0.949 vs Q8 (argmax preserved, coherent). Encode-only change (same runtime kernel/bytes as plain Q4_K; the imatrix costs nothing at inference). imatrix source: the published llama.cpp calibration unsloth/Qwen3.8-Flash-Next-GGUF (same qwen4exp tensor decomposition).

Files

• Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-MTP.gguf — Q8 dense main (exact). 44,806,612,448 bytes, sha256 e078c60abfdc9c5dd849eddb41660e5fc8f1b0da2f1200c643a8d3ec50324e8a.
• Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-DenseQ4Kselimat-MTP.gguf — imatrix-weighted selective-Q4K dense main (faster, near-Q8). 43,677,591,168 bytes, sha256 b1dd08509231126b5f1596603fd2589ff8bdf8eae221420f3e4409110fee8bd4.
• Qwen3.8-Flash-Next-PLE-Q4_1.gguf — PLE Q4_1 n-gram sidecar, shared by both mains (required). Byte-identical between base Qwen and this fine-tune (abliteration does not touch the n-gram table), so it is Ivan's sidecar verbatim.
• Qwen3.8-Flash-Next-PLE-Q4_0.gguf — experimental PLE sidecar requantized Q4_1→Q4_0 with gguf-tools/ple_requant.py (−3 GiB). Not recommended: round-trip relL2 0.138 vs Q4_1, and measured no speed or context gain (PLE is CPU-mmap, not Metal-wired). Drop-in for --ple. 28,800,141,632 bytes, sha256 68920ba4887e7e463263ce0c771aacd1902b315c89145ab090e08e3c2d69d611.
• Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-SelQ4Down41-48-DenseQ4Kselimat-MTP.gguf — "unc8" selective tier: 7 of 48 trunk down layers (41–47) Q2_K→Q4_K, MTP blk.48 down Q8_0. 45,774,743,232 bytes, sha256 cef12cb8871635fe94d95731806b8a73f0cece5e11566ddd75050f2a90864bbd.
• Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-SelQ4Down33-48-DenseQ4Kselimat-MTP.gguf — "unc16" selective tier: 15 of 48 down layers (33–47) Q4_K. 47,662,180,064 bytes, sha256 af96a237633305fad2abd4c3a8fa537a0514d1c0400251605fce6d4f9931b448.
• Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-SelQ4Down18-48-DenseQ4Kselimat-MTP.gguf — "unc31" selective tier: 30 of 48 down layers (18–47) Q4_K. 51,201,124,096 bytes, sha256 7f45767af63a45b17971870d2288d8dbdcd92236f4961c69f531df165aaadef8.
• Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q4KDownPad768-DenseQ4Kselimat-MTP.gguf — "unc48L" (Complete), strongest uncensor: all 48 trunk down layers Q4_K, MTP blk.48 down Q8_0. Needs SSD expert streaming on 64 GiB (see "Running unc48L at 256K"). 55,447,856,768 bytes, sha256 20b4c617a5e7a8d3976d328daf04c48c0f250f47c9a1e7ad91265e698616c41d.
• Qwen3.8-Flash-Next-draft-vocab-vi-en-code-64k.txt — optional MTP draft vocabulary (65,568 token ids, most frequent first; 390,414 bytes, sha256 f8b0dc7c28f173c4b071c986cf44d59ff4f169ad5a6442a88ea240e3b1fbe652). Shared by every main; see "MTP draft vocabulary" below.

Uncensor tiers: how strong each one is (2026-09-21; unc31/unc48L fit updated 2026-09-23)

Uncensor strength is set by how many trunk down-expert layers survive the abliteration delta — 2-bit (Q2_K) down experts erase most of it, Q4_K preserves it. The tiers:

tier	Q4_K down layers	uncensor strength (10-prompt probe, greedy)	fits on M5 Pro 64 GiB
Q2K main	0	weakest: only low-band relaxes (profanity)	220K–256K
unc8	7 (41–47)	≈ Q2K baseline: mid-band (phishing/hotwire) still refused	256K, ~42 t/s @4K
unc16	15 (33–47)	≈ Q2K baseline: mid-band still refused	256K, 40.1 t/s @4K
unc31	30 (18–47)	mid-band full: phishing/hotwire/lockpick comply; break-in + extreme (meth/IED) still refused	256K with SSD streaming (K=36): 46.5 t/s short prompt
unc48L (Complete)	48 (0–47)	strongest (extreme-band, disclaimer-led)	256K with SSD streaming (K=32): 42–43 t/s short prompt, **34 t/s prose at 256K full**, 49.3 GiB peak

Pick unc48L for the strongest uncensor, unc31 for mid-band uncensor with a little more speed and faster prefill, unc8 for the fastest build, or the Q2K mains for the lightest footprint. unc31 and unc48L at long context need the SSD-streaming engine below; the older engine OOMs them at 256K.

Running unc48L at 256K (SSD expert streaming)

The ds4-metal develop engine keeps the first K routed layers resident and streams the experts of the rest through a locked RAM cache; KV is 4-bit by default. On M5 Pro 64 GiB:

DS4_QWEN4_STREAM_FULL_LAYERS=32 DS4_QWEN4_PLE_PREFETCH_FULL=0 \
DS4_QWEN4_MTP_DRAFT_VOCAB=Qwen3.8-Flash-Next-draft-vocab-vi-en-code-64k.txt \
ds4-server -m Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q4KDownPad768-DenseQ4Kselimat-MTP.gguf \
    --ple Qwen3.8-Flash-Next-PLE-Q4_1.gguf -c 262144 --prefill-chunk 2048 --mtp \
    --ssd-streaming --ssd-streaming-cache-experts 6GB

Measured (2026-09-23, greedy, MTP + 64K draft vocab, server warm; peak = system-wide wired):

unc48L config	short prompt VI/EN/code	full context, prose answer	full context, counting (MTP-friendly)	peak wired	expert cache
K=36, 6 GB	43.5 / 41.9 / 43.6 t/s	—	37.8 @226K	52.6 GiB	capped at 0.22 GiB (over the wire limit)
K=32, 6 GB	42.9 / 41.6 / 43.0 t/s	34.1 @256K	43.8 @226K, 43.1 @256K	48.9–49.3 GiB	fully locked
K=32, 8 GB	+2% vs 6 GB	34.3 @256K	43.7 @226K	50.9–51.4 GiB	fully locked
K=30, 6 GB	≈ K=32	—	38.4 @256K	49.0 GiB	fully locked

The counting column (answer + "count from 1 to 400") is what MTP predicts best, so it overstates real use. Decode speed is nearly flat with context length: the same reasoning-heavy prose task (summarize a C file, thinking on) runs 34.4 / 35.1 / 36.0 / 34.1 t/s at 37K / 71K / 135K / 256K (unc31: 35.9 at 135K). Simple short chats run ~42 t/s. What changes the speed is the kind of answer (how well MTP predicts it), not the context length.

Use K=32. Prefill at 256K runs ~310 t/s (about 14 min for a full 256K prompt); unc31 prefills ~30% faster. Needle retrieval passed in every full-context run, swap stayed flat.

MTP draft vocabulary (optional, ~+4% decode)

The MTP draft only needs its argmax, so it can be scored over the most frequent output rows instead of all 248K. DS4_QWEN4_MTP_DRAFT_VOCAB=<file> gathers those rows into a small draft head (~178 MiB); the verify pass still uses the full head, so every committed token is still checked by the full model. The Qwen3.8 tokenizer puts Vietnamese at high token ids, so a plain id prefix (DS4_QWEN4_MTP_DRAFT_ROWS) collapses Vietnamese draft acceptance (71% -> 36%); this list ranks ids by measured frequency over Vietnamese and English Wikipedia, code and Python instead, and covers 100% / 99.4% / 97.0% of the model's own Vietnamese / English / code output.

Measured on unc31 (M5 Pro 64 GiB, 8K, greedy, 500 tokens, two runs each):

prompt	full head	this list
Vietnamese	35.5 t/s (71.0% accept)	36.7 t/s (71.0%)
English prose	35.3 t/s (68.3%)	36.9 t/s (68.3%)
code	36.1 t/s (72.7%)	37.5 t/s (70.3%)

DS4_QWEN4_MTP_DRAFT_VOCAB=Qwen3.8-Flash-Next-draft-vocab-vi-en-code-64k.txt ds4 -m ... --mtp ...

Built with gguf-tools/frspec_vocab.py (ds4-metal fork, branch develop).

Runtime (DS4 Metal engine)

Runs on the ds4-metal engine (llama.cpp-lineage Metal fork for qwen4exp). The build used here adds, over the stock light runtime: the external --ple sidecar loader combined with native multi-session batched decode, the fast Q4_K dense GEMV, adaptive MTP draft depth, and correctness fixes (accurate GDN-gate softplus, an isfinite guard in the expert reduction, ignore_eos honored under --mtp, and a clearer tool-call parse error). The --ple sidecar is CPU-mmap demand-paged (each token faults ~one page per hash head); it fits 64 GiB beside the resident model, whereas the embedded-BF16 n-gram lineage does not.

Usage

# exact (default)
ds4 -m Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-MTP.gguf \
    --ple Qwen3.8-Flash-Next-PLE-Q4_1.gguf --mtp -c 4096 -p "..."

# faster (imatrix-weighted selective dense Q4_K)
ds4 -m Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-DenseQ4Kselimat-MTP.gguf \
    --ple Qwen3.8-Flash-Next-PLE-Q4_1.gguf --mtp -c 4096 -p "..."

# serving many concurrent requests (ds4-server, ~2x aggregate)
ds4-server -m ...DenseQ4Kselimat-MTP.gguf --ple ...PLE-Q4_1.gguf --mtp \
    --batched-session 16 -c 4096 --host 127.0.0.1 --port 8000

Long contexts: add --prefill-chunk 2048 to stay zero-swap at 128K–220K on 64 GiB.

Lineage / method

Uncensored weights from orcarouter/Qwen3.8-Flash-Next-Uncensored (rev 8336e613ea508b13c2159bd0f68965d97a606b95), quantized to Ivan's DS4-IQ2 layout, then repackaged into this light form by stripping the embedded n-gram tensor (retained tensors byte-identical; external Q4_1 sidecar substituted). The DenseQ4Kselimat main is derived from the Q8 main by imatrix-weighted Q4_K requant of the dense projections (attn q/k/v/output kept Q8_0).

Behavior — uncensoring is PARTIAL and tier-dependent

This is an abliterated fine-tune; the 2-bit light-quant dilutes most of the abliteration, so the base Q2K mains behave much closer to the original base model than to a full abliteration (they relax mild refusals but keep most harmful refusals). The SelQ4 tiers progressively restore the abliteration delta on the down experts (table above): unc31 is the first tier where the mid-band requests comply; the extreme categories (drug/weapon synthesis, IED) and break-in stay refused on every tier except unc48L (Complete), which only reaches them with a disclaimer. Treat every tier as a research artifact, not an unrestricted model. Provided for research; you are responsible for lawful use.