How good is the MTP on this model?
I was doing some tests with a MTP I extracted myself and using it with https://huggingface.co/el4/GRM-3.2-Sky-ONYX-GGUF and was getting only a minor speedup compared to Ornith (~39->51 tps compared to 60-70 for Ornith). Is this in line with how this model performs as well with MTP?
The MTP in this one is giving me massive gains. On my 3090 system with spec-draft-n-max set to 3, I am getting average throughput of 160 t/s and very high acceptance rates (80%~) using the IQ4_NL version.
EDIT: I should add that I am using Beellama instead of upstream llama.cpp and have spec-dm-controller set to "profit".
Huh, that's about as good as Ornith. I must be doing something wrong lmao.
Is it possible to upload a mtp only gguf?
I'd use your model, but the current GPU setup I'm using (2 1080tis) need a smaller more compressed version to fit. And for some reason all the MTP GGUF scripts don't seem to want to inject MTP for me into the ONYX model.
@gerryri
Not sure how well that would work with two cards, but I think IQ4_XS might work for you with reduced KV size -- something like this plus your usual split options :
${LLAMA_SERVER} \
--model OrionLLM.GRM-3.2-Sky.IQ4_XS.mtp.gguf \
--ctx-size 128000 \
--no-mmproj \
--flash-attn on \
--n-gpu-layers all \
--kv-offload \
--kv-unified \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--temperature 1.0 \
--top-k 20 \
--top-p 0.95 \
--min-p 0 \
--reasoning auto \
--reasoning-preserve \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
--cache-type-k-draft q8_0 \
--cache-type-v-draft q8_0
This ( in fact, probably even --ctx-size 150000 --cache-type-k q8_0 --cache-type-v q8_0 ) shall have a good chance to fit in 22 GiB (but again, not sure how the split would would work -- never tested that). And as for choosing the optimal value for the draft queue -- --spec-draft-n-max 4 is not necessarily the best fit for a 1080ti, and likely the best way to set it is to experiment:
Hmm, I'll try that. Haven't specifically tried that version at all. With the other Qwen 3.5 35b * models, it works pretty good with the dynamic quant (I use 256k and q4 context), it split very well in my experience because it's a MoE. The Qwen 27b * do not work well though.
Maybe I can get a script successfully extracting MTP from this model, since it's a more standard quant. It did not work with the APEX type ones lol.