unsloth

/

Qwen3.5-35B-A3B-GGUF

like 858

Follow

Unsloth AI 27.7k

Reddit thread in question: https://www.reddit.com/r/LocalLLaMA/comments/1resggh/

Update: Now fixed, see: https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks

Hey @ubergarm ! Great work as usual! I'm investigating currently on UD-Q4_K_XL - I recently switched to using MXFP4, but as you noted my script most likely had some issues somewhere - I will update the community asap.

Thank you again for the investigation and also the rest of the community's patience - we highly appreciate it.

For now , MXFP4_MOE which is also partially dynamic is the correct option as of now, or using Q4_K_M which also uses our imatrix calibration dataset.

I will update everyone hopefully soon - and thank you to the community for the constant support .

It happens! Oh man I have so many little copy paste issues between the scripts, and with like five Qwen models coming out in a week all with similar but slightly different names its been a lot haha...

Thanks Daniel and you and your bro and team get some rest! Cheers!

Thank you immensely!

Q2 and Q3 also have tensors set to MXFP4.

Furthermore, at least Qwen3 Coder Next's UD GGUFs have the same issue, if not more.

Qwen 3 Coder Next has other weights forced to MXFP4 in even the non-UD quants as well.

Can someone help me understand this issue? I thought MXFP4 was not a "bad" quant type but it seems to cause trouble here?

@CHNtentes

There are no bad quant types, just incorrect choices for specific tensors. Generally MXFP4 is used specifically for routed experts ffn_(gate|down|up)_exps and not for attn.* tensors. Also the label mixes like UD-Q4_K_XL implies there would not be any MXFP4 tensors but be a mix of q4_K and similar.

Accidentally using MXFP4 in the wrong place could lead to lower quality than could otherwise be achieved.

You can see some perplexity and KLD data in the reddit thread linked above which shows a measurable discrepancy. This is what alerted folks to the potential issue.

@Fizzarolli

Thanks for checking further, hopefully the bug didn't propagate too far or long, but yes that looks unfortunate.

Thank you @Fizzarolli for the help! I'm checking why MXFP4 got injected in the XL quant specifically - it definitely seems like my switch to MXFP4 had some issues - so sorry on the issues - will update everyone soon

I honestly don't understand much about this, but I also intuitively feel that the presence of MXFP4 quanta in the layers can degrade the generation results. If I remember correctly, this coarser quantization isn't suitable for all.

@danielhanchen

I remember in your post you say XL quants are for the combination of best quality and speed. How are you measuring the mix of speed and quality? I see q6_K and q5_K being slow. My hunch is that mxfp4 that uses mxfp4 and q8 might be fastest and best quality even though mxfp4 itself might not be most accurate. Is there possibility of iq4_nl and q8 mix?

Here is analysis on speed on my machine for vulkan. Based on your back-end and GPU it might be different for you.

My Conclusion for speed based on quant-bench:

My quant-bench Data:

bash  ./quant-bench -d vulkan0
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
Using device: Vulkan0

=== Prompt Processing (Prefill) Phase (Batch Size = 512) ===

=== Token Generation (Decoding) Phase (Batch Size = 1) ===

@engrtipusultan
I just want to note that not all quants maintain good speed when running in hybrid or even pure CPU mode.

@engrtipusultan
I just want to note that not all quants maintain good speed when running in hybrid or even pure CPU mode.

Sorry I did not understand that, can you explain more.

I have AMD APU so I am personally using vulkan only no offloading to CPU. I think same might be the case for apple devices where there is unified memory.
Also I do understand speed of quant differ on different hardware and backend. For example on hardware where BF16 or MXFP4 is supported they are much faster. Similary I quants are slower on certain backends etc.

Similarly hybrid inference CPU plus GPU might have different results.

So MXFP4 layers shouldnt be in UD quants? vide : https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF?show_file_info=Qwen3-Coder-Next-UD-Q4_K_XL.gguf

@engrtipusultan
There used to be a table in the llama.cpp repository, but it seems to be outdated and removed. I couldn't find it. I only remember that not all quants had good performance on all backends. For example, IQ quants was only for GPU, it was slower on CPU and hybrid. I don't know how it is now.

@engrtipusultan
There used to be a table in the llama.cpp repository, but it seems to be outdated and removed. I couldn't find it. I only remember that not all quants had good performance on all backends. For example, IQ quants was only for GPU, it was slower on CPU and hybrid. I don't know how it is now.

It is still there. Hard to find :P . Based on testing, it seems outdated, atleast for Vulkan backend on my hardware it is incorrect as on 4bit IQNL is showing fastest whereas it says iq quants are suppose to be slow on vulkan backend.

Qwen3.5-122B-A10B also has the same issue, it seems MXFP4 is widely used in UD quantization, and it has caused the performance of 122B Q3-level quantization to be much lower than expected, with generation producing garbled text or repetition. I haven't seen this kind of loss in 100B-level models for a long time.

Will post some results soon - again apologies on the delay and sorry on the issues - I'll update everyone soon on MXPF4 vs Q4 specifically.

But overall:

1. All our quants work fine, just some with MXFP4 layers are slightly less performant than Q4 variants
2. UD-Q4_K_XL is the main issue - I would use MXFP4 instead for example, which still uses some of our dynamic methodology and uses our calibration dataset
3. There is a tool calling bug which I will also fix - this is not part of our quants, but related to a generic issue about the model.

MXP4 layers make it crash in Vulkan llama CPP with Intel Arc iGPU and GPU.

@danielhanchen Do you plan on updating ggufs for 397B, 112B and 35B in the next few hours? I've been waiting for the dust to settle before downloading several GB's worth of data :-) Thanks for your work

@CHNtentes

There are no bad quant types, just incorrect choices for specific tensors. Generally MXFP4 is used specifically for routed experts ffn_(gate|down|up)_exps and not for attn.* tensors. Also the label mixes like UD-Q4_K_XL implies there would not be any MXFP4 tensors but be a mix of q4_K and similar.

Accidentally using MXFP4 in the wrong place could lead to lower quality than could otherwise be achieved.

You can see some perplexity and KLD data in the reddit thread linked above which shows a measurable discrepancy. This is what alerted folks to the potential issue.

@ubergarm Why do you say that MXFP4 is not a bad quant type? I think MXFP4 being used for experts is simply because the GPT-OSS models are natively trained that way, not because the distribution of the weight values in experts specifically "conform" to MXFP4. There is a comment by ikawrakow about using MXFP4 for PTQ, and while I lack any experience with AI/ML, I agree with him that only having a single exponent bit is really suboptimal for accuracy.

Honestly, I believe the purpose of FP4 and other low-precision floats assigning most of the bits to the exponent field is just to cover the widest dynamic range possible while minimizing the need for extra scaling factors (which will complicate direct hardware implementations).

Hey all we just updated all our quants! https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/discussions/10

I wrote a large discussion in https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwen3535ba3b_unsloth_dynamic_ggufs_benchmarks/

Sorry for the pings but thought y'all should know!
CC: @Firepal3D @itsthenewmeta @SuperbEmphasis @AI-Joe-git @engrtipusultan @lingyezhixing @eleius @anfieldiro @dagbs @elitehawkturd @henry2man @cmh @void-mckenzie @pixelmelt @Dampfinchen @LaikaFramework @usbphone @Tho465 @ryan4yin @klimekop6 @bgonzaga @Chief-Inspector @DevilaN @Pentium95 @MaCgI @alexaione

TiwitMuffbiscuit here. Thanks a lot! I know it’s a lot of work, but it’s also a lot of good data to work with. You even went as far as testing against all other major quants and left everything available for us to see—that’s a first. Beautiful graphs.

I’ve downloaded everything I needed to put into the Reddit post; I’m just waiting for the god-damn computer to finish a task. Seven hours to go, then I’ll be able to do some tests. In the meantime, I’ll update the post adequately.

I have Vega8 and Vulkan backend, poor gpu club, in past, testing Q8 has always better PP than Q4 and Q4 always have better TG than Q8.

Shared tests show that all have almost same TG and PP also PP is marginally higher at q4 than q8. Which reverse of what I tested in past.

Can other community members confirm what are the case for them ?

Qwen3.5-122B-A10B also has the same issue, it seems MXFP4 is widely used in UD quantization, and it has caused the performance of 122B Q3-level quantization to be much lower than expected, with generation producing garbled text or repetition. I haven't seen this kind of loss in 100B-level models for a long time.

I have been seeing those same issues with Q8 as well.

Qwen3.5-122B-A10B also has the same issue, it seems MXFP4 is widely used in UD quantization, and it has caused the performance of 122B Q3-level quantization to be much lower than expected, with generation producing garbled text or repetition. I haven't seen this kind of loss in 100B-level models for a long time.

Qwen3-Coder-Next as well. Hope to see new UD GGUFs...