sizzlebop/CodeThink-V4-Qwen3-4B-Mix-GGUF

CodeThink-V4-Qwen3-4B-Mix GGUF

This repository provides GGUF quantizations for modrill/CodeThink-V4-Qwen3-4B-Mix.

CodeThink-V4-Qwen3-4B-Mix is a code reasoning model built on Qwen/Qwen3-4B-Base fine-tuned using a Mix Distillation recipe (0.2 traces from Qwen3-30B-A3B-Thinking and 0.8 traces from Qwen3-4B-Thinking across 4,155 competitive programming and code reasoning problems). The final LoRA adapter was merged into full weights at the 2-epoch endpoint.

All GGUF files were converted from the original safetensors weights using llama.cpp at native BF16 precision, followed by quantization into standard k-quant variants.

---

Available Files and Quantizations

File	Quant Type	Size	Description / Recommendation
CodeThink-V4-Qwen3-4B-Mix-BF16.gguf	BF16	8.22 GB	Full precision base conversion. Highest fidelity reference weights.
CodeThink-V4-Qwen3-4B-Mix-Q8_0.gguf	Q8_0	4.37 GB	Near-lossless 8-bit quantization. Recommended for evaluation and high accuracy.
CodeThink-V4-Qwen3-4B-Mix-Q6_K.gguf	Q6_K	3.38 GB	High quality retention with minimal degradation. Excellent balance for general development.
CodeThink-V4-Qwen3-4B-Mix-Q5_K_M.gguf	Q5_K_M	2.94 GB	Balanced medium 5-bit quant. Strong code generation quality with moderate memory footprint.
CodeThink-V4-Qwen3-4B-Mix-Q4_K_M.gguf	Q4_K_M	2.53 GB	Fast, lightweight 4-bit quant. Recommended default for everyday coding and local tool use.
CodeThink-V4-Qwen3-4B-Mix-Q3_K_M.gguf	Q3_K_M	2.09 GB	Compact footprint when memory headroom is strictly limited.
CodeThink-V4-Qwen3-4B-Mix-Q2_K.gguf	Q2_K	1.67 GB	Maximum compression for testing in ultra-constrained environments.

---

Technical Specifications

• Architecture: Qwen3 (Qwen3ForCausalLM)
• Parameter Count: 4.41B
• Context Window: 32,768 tokens
• Attention Configuration: 32 attention heads, 8 key-value heads, head dimension 128
• Base Precision: bfloat16
• License: Apache-2.0

---

Prompt Format

This model uses the standard ChatML / Qwen conversation format, supporting explicit <think> reasoning blocks:

<|im_start|>system
You are a helpful coding assistant.<|im_end|>
<|im_start|>user
Write a Python function to check if a number is prime.<|im_end|>
<|im_start|>assistant
<think>
Need to verify if n <= 1, then check divisors up to sqrt(n).
</think>

def is_prime(n: int) -> bool:
    if n <= 1:
        return False
    if n <= 3:
        return True
    if n % 2 == 0 or n % 3 == 0:
        return False
    i = 5
    while i * i <= n:
        if n % i == 0 or n % (i + 2) == 0:
            return False
        i += 6
    return True
<|im_end|>

---

Quickstart Guide

Option 1: llama.cpp CLI

Run local interactive inference directly with llama-cli:

# Run 4-bit quantized model
llama-cli -m CodeThink-V4-Qwen3-4B-Mix-Q4_K_M.gguf \
  -p "<|im_start|>user\nWrite a quicksort implementation in Python.<|im_end|>\n<|im_start|>assistant\n" \
  -n 512 \
  -ngl 99

Option 2: llama.cpp Server

Launch an OpenAI-compatible HTTP server:

llama-server -m CodeThink-V4-Qwen3-4B-Mix-Q4_K_M.gguf \
  --port 8080 \
  -c 32768 \
  -ngl 99

Query the endpoint:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "CodeThink-V4-Qwen3-4B-Mix",
    "messages": [
      {"role": "user", "content": "Explain how Dijkstra algorithm works."}
    ],
    "temperature": 0.6
  }'

Option 3: Ollama

Create an Ollama model using a custom Modelfile:

FROM ./CodeThink-V4-Qwen3-4B-Mix-Q4_K_M.gguf

TEMPLATE """{{- if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{- end }}
{{- range .Messages }}
<|im_start|>{{ .Role }}
{{ .Content }}<|im_end|>
{{- end }}
<|im_start|>assistant
"""

PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
PARAMETER temperature 0.6
PARAMETER top_p 0.95

Build and run with Ollama:

ollama create codethink-4b -f Modelfile
ollama run codethink-4b "Write a binary search function in Rust."

Option 4: LM Studio

1. Copy the desired .gguf file into your local LM Studio models folder.
2. Select CodeThink-V4-Qwen3-4B-Mix-Q4_K_M.gguf from the model picker.
3. Use the ChatML prompt preset with system and user markers.

---

Original Model and Citation

• Original Hugging Face Checkpoint: modrill/CodeThink-V4-Qwen3-4B-Mix
• Base Architecture Reference: Qwen/Qwen3-4B-Base
• Mix Distillation Methodology: Li et al., 2025, "Small Models Struggle to Learn from Strong Reasoners", arXiv:2502.12143