flashback2k/FlashModel-Qwen3.5-9B-GGUF

⚡ FlashModel-Qwen3.5-9B — GGUF

imatrix GGUF quants of FlashModel-Qwen3.5-9B, a reasoning / coding / tool-calling fine-tune of Qwen/Qwen3.5-9B distilled from open-weight frontier teachers.

• All quants use an importance matrix computed on this fine-tune (not borrowed from the base model).
• *_K_XL quants use a dynamic per-tensor layout mirrored tensor-for-tensor from unsloth/Qwen3.5-9B-GGUF (see How these were made).
• Every file was loaded and ran a generation test in llama.cpp before upload.
• Text-only: the fine-tune does not include the vision tower, so there is no mmproj.

Provided Quants

(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)

Link	Type	Size/GB	Notes
GGUF	IQ3_M	4.42	smallest here
GGUF	Q3_K_M	4.62	lower quality
GGUF	Q3_K_XL	5.05	dynamic layout
GGUF	IQ4_XS	5.20	good size/quality trade-off
GGUF	Q4_K_S	5.35	fast
GGUF	IQ4_NL	5.42
GGUF	Q4_K_M	5.63	fast, recommended
GGUF	Q4_K_XL	5.97	dynamic layout, recommended
GGUF	Q5_K_S	6.31
GGUF	Q5_K_M	6.47
GGUF	Q5_K_XL	6.74	dynamic layout, highest precision here
imatrix	imatrix	0.01	importance matrix used for every quant above

Which one should I pick?

Your free VRAM / RAM	Pick
≥ 10 GB	Q5_K_XL
~8 GB	Q4_K_XL — default choice
~7 GB	IQ4_XS or Q4_K_S
≤ 6 GB	Q3_K_XL, then IQ3_M

Leave room for the KV cache: long reasoning traces need context. Size numbers are file sizes, not total memory use.

Usage

llama.cpp

# download one quant
huggingface-cli download flashback2k/FlashModel-Qwen3.5-9B-GGUF \
  FlashModel-Qwen3.5-9B-Q4_K_XL.gguf --local-dir .

# chat (thinking mode, recommended sampling from Qwen3.5)
llama-cli -m FlashModel-Qwen3.5-9B-Q4_K_XL.gguf --jinja -ngl 99 -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5

# OpenAI-compatible server with tool calling
llama-server -m FlashModel-Qwen3.5-9B-Q4_K_XL.gguf --jinja -ngl 99 -c 32768 --port 8080

Or pull straight from the Hub: llama-server -hf flashback2k/FlashModel-Qwen3.5-9B-GGUF:Q4_K_XL --jinja

Ollama

ollama run hf.co/flashback2k/FlashModel-Qwen3.5-9B-GGUF:Q4_K_XL

LM Studio

Search for flashback2k/FlashModel-Qwen3.5-9B-GGUF and pick a quant.

Recommended sampling

These are the base model's official recommendations (Qwen3.5 model card); they were not re-tuned for the fine-tune.

Mode	temperature	top_p	top_k	min_p	presence_penalty
Thinking, general	1.0	0.95	20	0.0	1.5
Thinking, precise coding	0.6	0.95	20	0.0	0.0
Non-thinking, general	0.7	0.8	20	0.0	1.5

Always pass --jinja so the embedded Qwen3.5 chat template (thinking blocks, tool calls) is used.

Reasoning-budget tag

Training system prompts started with a reasoning-budget tag chosen from the length of the teacher's reasoning:

<|reasoning_budget|>medium<|/reasoning_budget|>

Values: off, low, medium, high, xhigh, max. The tag's effect on output length has not been measured yet; treat it as experimental.

About the fine-tune

Base	Qwen/Qwen3.5-9B
Method	LoRA r=128 (RSLoRA, α=32) on attention + MLP projections, merged
Data	9,638 examples / 45M tokens, loss on assistant turns only
Context in training	up to 16,384 tokens (longer examples dropped, never truncated)
Schedule	1 epoch, 600 steps, lr 5e-5 cosine
Held-out eval loss	0.5745 (step 100) → 0.5586 (step 600)

Training data and teachers

Only open-weight teachers whose licenses allow training on their outputs:

Share	Domain	Dataset	Teacher
40%	Math (answers verified against references)	nvidia/Nemotron-SFT-Math-v4	DeepSeek-V4-Pro
22%	Competitive programming (Python)	nvidia/Nemotron-SFT-Competitive-Programming-v2	DeepSeek-R1-0528
16%	Multi-turn tool calling (judge-filtered)	nvidia/Nemotron-SFT-Agentic-v2	GLM-4.6 / DeepSeek-V3.2
11%	Instruction following	nvidia/Nemotron-SFT-Instruction-Following-Chat-v3	GPT-OSS-120B
11%	Science reasoning	nvidia/Nemotron-SFT-Science-v2	GPT-OSS / Kimi-K2 / DeepSeek-V3.2

Datasets © NVIDIA, CC BY 4.0 (some Math StackExchange-derived samples CC BY-SA 4.0).

⚠️ Status and known limitations

• No benchmarks yet. Lower held-out loss means the model imitates the teachers more closely; it does not by itself prove it beats the stock Qwen3.5-9B. Comparative evals are planned and will be added here.
• English-centric. All training data was English. On non-English prompts (e.g. Russian) the model often reasons in English.
• Math and code examples longer than 16k tokens were excluded, which skews those domains toward shorter problems.
• Inherits the base model's limitations and biases.

How these were made

1. Merged weights → BF16 GGUF with llama.cpp convert_hf_to_gguf.py (build b11005-82324fc50).
2. imatrix: llama-imatrix on the BF16 model over ~131k tokens (256 × 512) of held-out training-distribution text (math, code, tool calls, instructions, science). A Q8_0 source produced non-finite activations — the stock Qwen3.5-9B Q8_0 does too — so BF16 was used.
3. Standard quants: llama-quantize --imatrix.
4. *_K_XL quants: Unsloth has not published its Dynamic quantization tool, but the resulting per-tensor types are visible in its GGUF headers. The types of all 427 tensors were read from unsloth/Qwen3.5-9B-GGUF (UD-Q3_K_XL, UD-Q4_K_XL, UD-Q5_K_XL), applied with llama-quantize --tensor-type-file, and verified tensor-for-tensor after quantization. The layout keeps the most sensitive tensors at higher precision — e.g. in Q4_K_XL: Gated DeltaNet ssm_out at Q8_0, ssm_alpha/ssm_beta at F16, attn_v at Q6_K, about half of ffn_down at Q6_K. The imatrix is this model's own. These are not official Unsloth quants and are not affiliated with or endorsed by Unsloth.
5. Each quant loaded in llama.cpp (CUDA) and generated a reply before upload.

Credits

Qwen team for Qwen3.5 · NVIDIA for the Nemotron SFT datasets · DeepSeek, Zhipu AI (GLM), OpenAI (GPT-OSS) and Moonshot AI (Kimi) for open-weight teachers · ggml-org/llama.cpp · Unsloth for the published dynamic-quant layouts · quant-table format inspired by mradermacher.