open-zzrl/Qwen3.5-9B-Trace-Inversion-GGUF

🚀 Live Demo

Try the interactive Kaggle Notebook (no setup required): Qwen3.5 Trace-Inversion Demo

🧠 Qwen3.5-9B-Trace-Inversion (GGUF)

This repository contains GGUF quantized versions of the Qwen3.5-9B-Trace-Inversion model.

These files are optimized for local inference using llama.cpp, Ollama, LM Studio, or any other GGUF-compatible runner.

🎯 What does this model do?

This model is trained on the Trace Inversion technique. Instead of solving a problem from scratch, it acts as a Reasoning Generator.

You provide it with a Question and its known Answer, and it reconstructs the step-by-step logical reasoning (<think>...</think>) that connects the two.

📦 Available Quantizations

We have generated a wide range of quants to fit any hardware, from low-end laptops to powerful workstations.

File	Quant Type	Size	Recommended For
model-f16.gguf	F16	~17.1 GB	Maximum fidelity (Original weights).
model-q8_0.gguf	Q8_0	~9.5 GB	High RAM setups (32GB+). Nearly identical to F16.
model-q6_k.gguf	Q6_K	~7.4 GB	Excellent balance of size and quality.
model-q5_k_m.gguf	Q5_K_M	~6.5 GB	Great quality, recommended for 16GB RAM.
model-q5_k_s.gguf	Q5_K_S	~6.3 GB	Slightly smaller than Q5_K_M.
model-q4_k_m.gguf	Q4_K_M	~5.4 GB	Best Overall. Recommended for most users (8-12GB VRAM).
model-q4_k_s.gguf	Q4_K_S	~5.1 GB	Slightly smaller than Q4_K_M.
model-q4_0.gguf	Q4_0	~5.1 GB	Older 4-bit format. Q4_K_M is usually better.
model-iq4_xs.gguf	IQ4_XS	~5.0 GB	Advanced 4-bit quantization. Great quality-to-size ratio.
model-q3_k_l.gguf	Q3_K_L	~4.7 GB	Large 3-bit. Good quality for low VRAM.
model-q3_k_m.gguf	Q3_K_M	~4.4 GB	Medium 3-bit. Noticeable quality drop, but works on low-end devices.
model-iq3_s.gguf	IQ3_S	~4.2 GB	Advanced 3-bit quantization.
model-q3_k_s.gguf	Q3_K_S	~4.1 GB	Small 3-bit.
model-q2_k.gguf	Q2_K	~3.6 GB	Very low quality. Only for extreme memory limits.

(Note: Lower-bit I-quants like IQ1, IQ2, and IQ3_XXS were omitted because they require an importance matrix (imatrix) to generate properly, which was not available during the conversion process.)

🚀 How to use

🦙 Ollama

Create a Modelfile with the following content (make sure to update the path to your downloaded GGUF file):

FROM ./model-q4_k_m.gguf

# Set the correct prompt template
TEMPLATE """<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
<think>

Stop generating when the model outputs these tokens

PARAMETER stop "<|im_end|>" PARAMETER stop ""