pgmharikrishnan/grug-35b-v2-MLX-4bit

Grug 35B v2 — MLX 4-bit

This repository provides Grug 35B v2 quantized to 4-bit format for native execution on Apple Silicon via MLX and mlx-lm.

• Original Model: ProCreations/grug-35b-v2 (Qwen3.5 MoE, 35B total / ~3B active, 256 experts × 8/tok, 40 layers)
• Model Size: ~18 GB (split across 4 safetensors shards)
• Quantization Scheme: 4-bit affine quantization (q_group_size=64, bfloat16 non-quantized weights, ~4.503 bits/weight effective). MoE router layers (mlp.gate, shared_expert_gate) are kept at 8-bit by stock mlx-lm MoE policy — routing stays precise while bulk expert weights go 4-bit.
• Vision: text-only conversion (vision tower excluded, same as the 27B MLX). For a vision-enabled MLX build see leonsarmiento/grug-35b-v2-6bit-XL-mlx.
• Conversion note: the official checkpoint ships unfused per-expert weights, which stock mlx_lm.convert 0.31.3 rejects. Experts were stacked/concatenated into the fused layout (experts.gate_up_proj + experts.down_proj) before the identical stock load → bf16 cast → quantize → save flow. Fusion verified exact (layer-0 expert-0 gate byte-identical pre-quantize).

---

Why Grug v2?

Grug v2 trains the model to compress internal reasoning into ultra-concise mental checks, eliminating <think> token bloat from the response stream while maintaining reasoning depth and tool-calling precision. v2.1 added long-hunt data, deep-think and stuck-loop escape on top of the 35B MoE base.

---

Apple Silicon Metal Smoke Test (Mac Studio M4 Max, 128 GB)

Single-prompt verification run (same prompt shape as the 27B card):

Metric	Grug 35B v2 (Native MLX 4-bit)
Task	numbers_closer_than_threshold — correct itertools.combinations solution
<think> block	concise, single-sentence style
Generation	128 tokens @ 124.8 tok/s
Prompt prefill	27 tokens @ 16.2 tok/s
Peak memory	19.7 GB

<think>
Use itertools.combinations to compare every pair. abs difference < threshold indicates close pair.
</think>

```python
from itertools import combinations

def numbers_closer_than_threshold(numbers, threshold):
    """
    Return True if any two numbers in the list are closer than `threshold`.
    """
    for a, b in combinations(numbers, 2):
        if abs(a - b) < threshold:
            return True
    return False

---

## Usage

### 1. Installation

```bash
pip install -U mlx-lm

2. Python API

from mlx_lm import load, generate

model_id = "pgmharikrishnan/grug-35b-v2-MLX-4bit"
model, tokenizer = load(model_id)

prompt = "Implement a python function to find the longest palindromic substring."
response = generate(
    model,
    tokenizer,
    prompt=prompt,
    max_tokens=512,
    verbose=True
)
print(response)

3. Command Line Generation

mlx_lm.generate \
    --model pgmharikrishnan/grug-35b-v2-MLX-4bit \
    --prompt "Write a Python function to check if numbers in a list are closer than a threshold." \
    --max-tokens 512

4. Local OpenAI-Compatible Serving

mlx_lm.server \
    --model pgmharikrishnan/grug-35b-v2-MLX-4bit \
    --port 8080

Serving note: do NOT point mlx-serve's pinned 4-model resident set at this repo without recomputing the memory budget — it is a fifth resident model on a 4-model cap (config/mlx-serve/models.conf).

---

Quantization Details

• Tool: mlx_lm.convert 0.31.3 flow (with unfused→fused expert pre-pass, see note above)
• Bits: 4-bit (routers 8-bit, stock MoE policy)
• Group Size: 64
• Quantization Mode: affine
• Weight Dtype: bfloat16
• Effective Bits per Weight: 4.503

---

Related

• 27B sibling (same recipe): pgmharikrishnan/grug-v1.1-qwen-3.8-27b-MLX-4bit
• QAT GGUF (llama.cpp, not the source of this conversion): ProCreations/grug-35b-qat-q4-gguf