Never realized this was even possible. Sometimes when we are so distracted in our lives, a beacon of light like this opens up a whole another world. Thank you for sharing your knowledge and your work. I glad I caught it on Reddit.

Glad I saw your reddit post before the stupid mods removed them on r/LocalLLaMA/. Easier discovery my ass. Anyways, just gave these quants a spin and they are way better than the custom quants I've been trying to spin up. Thank you for sharing them! They are godsend for 16gb vram cards. Would love more details on what you did to each layers in detail. but happy to get what I got.

oh nice, glad they're working well for you, especially on a 16GB card which is basically the whole reason I was tuning it in the first place.

quick rundown on the layer stuff: so i ran a per-tensor-category KL-divergence sweep first (quantizing one weight group at a time, checking drift against the Q8_0 source across general/code/math/toolcalling prompts) to figure out where each category's safe floor actually sits. then I validated the combined model as a whole and honestly got some surprises. like a couple of components (attn_v, attn_gate individually) did basically nothing when protected even though they looked rough on their own. meanwhile attn_qkv and ffn_down turned out to be the two strongest levers in the whole thing.

"Qwen3.8-27B-Bedrock-v4 — Tensor Recipe

FULL-ATTN BLOCKS (17: 3,7,11,15,19,23,27,31,35,39,43,47,51,55,59,63,64)
attn_q [5120,12288] IQ3_S
attn_k [5120,1024] IQ3_S
attn_v [5120,1024] Q5_K
attn_output [6144,5120] IQ3_S

DELTANET BLOCKS (48: all others, 0-64)
attn_qkv [5120,10240] Q4_K
attn_gate [5120,6144] Q4_K
ssm_alpha [5120,48] IQ4_XS
ssm_beta [5120,48] IQ3_XXS
ssm_out [6144,5120] Q4_K

ALL BLOCKS (0-64)
ffn_gate [5120,17408] IQ4_XS
ffn_up [5120,17408] Q4_K
ffn_down [17408,5120] Q4_K

GLOBAL
output.weight [5120,248320] IQ4_XS
token_embd.weight [5120,248320] IQ4_XS

EXCLUDED (F32, untouched)
ssm_conv1d, ssm_a, ssm_dt.bias, all norm tensors, block 64 nextn.* head

Size: 14,235.04 MiB / 4.37 BPW (13.91 GiB on disk)"

I'm also enjoying your tightrope quant with a 131072 q4_0 context on 16GB VRAM in Linux. Without MTP so it fits. Thank you.

Why would you leave the mtp layers untouched? I thought they would be the safest to compress as the quality change at most affects prediction speed and not token quality.

Thank you for sharing bedrock's recipie. Tho I generally use tightrope for the extra context. I use llama.cpp with kv cache q5_1 compression and gets about 69k context at 70-100 t/s with my 5070 ti. Love the performance and quality.

btw you should really link this gguf to the main qwen3.8 27b page so more people can find this gem.
https://huggingface.co/docs/hub/model-cards#specifying-a-base-model

Why would you leave the mtp layers untouched? I thought they would be the safest to compress as the quality change at most affects prediction speed and not token quality.

Thank you for sharing bedrock's recipie. Tho I generally use tightrope for the extra context. I use llama.cpp with kv cache q5_1 compression and gets about 69k context at 70-100 t/s with my 5070 ti. Love the performance and quality.

btw you should really link this gguf to the main qwen3.8 27b page so more people can find this gem.
https://huggingface.co/docs/hub/model-cards#specifying-a-base-model

To reply like I did to a similar comment here's what I said before "MTP is purely a speculative decoding feature designed for speed, not quality. It predicts several future tokens ahead of time so they can be verified in parallel, which helps throughput if your hardware supports it well. For coding specifically that precision-heavy stuff like brackets and syntax often breaks acceptance rates enough that you either get zero benefit or worse output than just running the standard heads straight. Right now Ive only calibrated the regular variant since most people on consumer GPUs dont actually get meaningful speedups from MTP anyway, and when they do they usually prefer keeping quality stable over chasing extra tokens per second. If demand for a calibrated MTP build shows up Ill revisit it but for now I figured standard was the right focus for this project."

If people want it I will go back and add it but I wanted to get this model out for people to try I know what it's like to have to wait for a model to fit on your hardware.

And i'll link the main Qwen card soon just forgot to thanks for telling me though!

Why would you leave the mtp layers untouched? I thought they would be the safest to compress as the quality change at most affects prediction speed and not token quality.

Thank you for sharing bedrock's recipie. Tho I generally use tightrope for the extra context. I use llama.cpp with kv cache q5_1 compression and gets about 69k context at 70-100 t/s with my 5070 ti. Love the performance and quality.

btw you should really link this gguf to the main qwen3.8 27b page so more people can find this gem.
https://huggingface.co/docs/hub/model-cards#specifying-a-base-model

Small correction to what I said earlier MTP is lossless. The main head verifies every draft token, so even a badly quantized MTP head can only make things slower (low acceptance rate), it never changes the actual output. That's exactly why I left the MTP heads uncalibrated for now: they're the most sensitive part of the quant, and on consumer GPUs the speedup is marginal anyway, so I'd rather ship standard-first with stable quality. Happy to add a calibrated MTP build if people want one.

Got it. not much space savings, potentially messing up prediction effectiveness. Not worth squeezing the MTP.

I also recent came across your model buried in a comment on Reddit. I've also been enjoying some use of your model thus far - was able to fit Bedrock-v4 on my 16 GB VRAM card while having a modest 64k context with slight asymmetric kv cache quantization (Q8_0/Q5_1) without tanking the speed. Thanks for that.

That said, Unsloth just released 3.0 of their GGUFs: https://unsloth.ai/docs/basics/dynamic-3.0-ggufs
Would a recalibration off of the new versions see any potential gains, and would you be considering it if so?

I also recent came across your model buried in a comment on Reddit. I've also been enjoying some use of your model thus far - was able to fit Bedrock-v4 on my 16 GB VRAM card while having a modest 64k context with slight asymmetric kv cache quantization (Q8_0/Q5_1) without tanking the speed. Thanks for that.

That said, Unsloth just released 3.0 of their GGUFs: https://unsloth.ai/docs/basics/dynamic-3.0-ggufs
Would a recalibration off of the new versions see any potential gains, and would you be considering it if so?

Not gonna lie I was thinking about it, I read they haven't updated the Q8 Quant I used for this model yet but once they do update it I will most likely redo the steps and update the files or make a new card for the model's thanks for the support!

I also recent came across your model buried in a comment on Reddit. I've also been enjoying some use of your model thus far - was able to fit Bedrock-v4 on my 16 GB VRAM card while having a modest 64k context with slight asymmetric kv cache quantization (Q8_0/Q5_1) without tanking the speed. Thanks for that.

That said, Unsloth just released 3.0 of their GGUFs: https://unsloth.ai/docs/basics/dynamic-3.0-ggufs
Would a recalibration off of the new versions see any potential gains, and would you be considering it if so?

Not gonna lie I was thinking about it, I read they haven't updated the Q8 Quant I used for this model yet but once they do update it I will most likely redo the steps and update the files or make a new card for the model's thanks for the support!

Thanks for creating Bedrock-v4! It’s been performing really well for me, reaching ~14 t/s on my RTX 3060 12GB with GPU + RAM offloading.

Please keep the current quants available when you update them. The new Unsloth Dynamic 3.0 quants seem slower for this kind of hybrid setup.