Jun-LoRA-E4B-MTP-GGUF

MTP / speculative-decoding draft model for efficiencyx/Jun-LoRA-E4B-GGUF. Not a chat model, it only proposes tokens that Jun then verifies.

This is Google's stock drafter, not one trained on Jun. The weights are google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant unchanged, converted to GGUF and quantized to Q4_K_M. Fine-tuning a drafter on Jun (tried on E4B) did not beat stock in llama.cpp, so this repo just puts the stock one where JunOS looks for it: the chat repo's name with -MTP added, same quant tag.

• gemma-4-E4B-it-qat-assistant-Q4_K_M.gguf, 77 MB, Q4_K_M
• QAT branch, matching Jun's base (unsloth/gemma-4-E4B-it-qat-q4_0-unquantized). A drafter from the non-QAT branch loads fine and guesses much worse.
• Tensors, shapes and architecture metadata are identical to the Q8_0 conversion that llama.cpp already serves, only the quant types differ.

Speed

The same weights at Q8_0 (amaranus/Gemma-4-E4B-it-qat-assistant-MTP-Q8_0-GGUF), RTX 3060, Jun E4B Q4_K_M as the target, q8_0 KV cache, draft_num_predict=1, medians of 12 requests:

setup	greedy tok/s	temp 0.7 tok/s	acceptance (greedy / temp 0.7)
Jun alone	64.4	64.3
with the drafter	79.1	75.5	0.545 / 0.461

Drafting 2 tokens per pass was no faster on this card.

Use

JunOS's ./mtp-autotune.sh falls back to it when OLLAMA_MTP is empty. With llama.cpp directly:

llama-server -m gemma-4-E4B-it-qat-q4_0-unquantized.Q4_K_M.gguf \
  --spec-type draft-mtp --spec-draft-model gemma-4-E4B-it-qat-assistant-Q4_K_M.gguf --spec-draft-n-max 1

Made with

llama.cpp 8e7f22b:

python convert_hf_to_gguf.py gemma-4-E4B-it-qat-q4_0-unquantized-assistant \
  --outtype bf16 --outfile gemma-4-E4B-it-qat-assistant-BF16.gguf
llama-quantize gemma-4-E4B-it-qat-assistant-BF16.gguf gemma-4-E4B-it-qat-assistant-Q4_K_M.gguf Q4_K_M