Laghu-MLX-4bit (unofficial community 4-bit conversion)

This is a 4-bit MLX quantization of udgamlabs/Laghu-MLX, the MLX build of udgamlabs/Laghu, a 152M-parameter small language model trained by Udgam Labs. All credit for the model goes to Udgam Labs. This repo only changes the weight precision.

Quality caveat. This is an unofficial conversion, not released by Udgam Labs. Udgam Labs deliberately chose not to publish a 4-bit MLX build because it loses quality clearly. They report mean KL 0.337 and top-1 agreement 69.9% against their F16 reference. Their bf16 MLX build scores KL 0.0009 and 98.0%. I did not run quality benchmarks on this 4-bit copy. For quality, use the bf16 udgamlabs/Laghu-MLX. This copy is here for speed and memory experiments.

Quantization

• Tool: mlx_lm.convert -q --q-bits 4 --q-group-size 64 (affine, group size 64)
• Effective size: 4.506 bits per weight (model.safetensors is about 86 MB, vs 303.6 MB for upstream bf16)
• Versions: mlx-lm 0.32.2, mlx 0.32.4, Python 3.12
• Architecture: gemma4_text, max context 16,384 tokens, 512-token sliding window
• The tokenizer, chat template and generation config are unchanged from upstream

Usage

pip install -U mlx-lm
mlx_lm.generate --model pratikgx/Laghu-MLX-4bit --prompt "Who are you?" --max-tokens 256

from mlx_lm import load, generate

model, tokenizer = load("pratikgx/Laghu-MLX-4bit")
messages = [{"role": "user", "content": "Write a haiku about autumn rain."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True))

The chat template supports an enable_thinking switch. Upstream recommends temperature 0.7, top-p 0.95 and top-k 64, and keeping context at 16,384 tokens or less.

Speed (bf16 vs this 4-bit copy)

Measured on an Apple M4 Pro (16 GPU cores, 24 GB) running macOS 27.0.1, with mlx 0.32.4 and mlx-lm 0.32.2. Settings: greedy decoding, prompts taken from Pride and Prejudice, 256 generated tokens (ignore EOS), prefill step 2048, 1 warmup run, then the median of 3 runs.

Prompt tokens	bf16 decode	4-bit decode	bf16 prefill	4-bit prefill	bf16 peak mem	4-bit peak mem
512	364 tok/s	402 tok/s	11,516 tok/s	11,844 tok/s	0.66 GB	0.38 GB
2,048	356 tok/s	409 tok/s	11,264 tok/s	10,825 tok/s	1.17 GB	1.11 GB
8,192	328 tok/s	397 tok/s	9,198 tok/s	8,876 tok/s	1.23 GB	1.15 GB
16,000	303 tok/s	379 tok/s	7,554 tok/s	7,069 tok/s	1.26 GB	1.15 GB

4-bit decodes about 10–25% faster. Prefill speed is about the same. Peak memory is lower only at short prompts, because the KV cache dominates at longer contexts.

The quality benchmarks (GSM8K, IFEval, ARC-Easy, HellaSwag) were run on the bf16 model only, not on this 4-bit copy. Raw results, settings and the harness are in the dataset pratikgx/launch-bench-results.

License and attribution

Apache-2.0, the same as upstream. See NOTICE, copied unchanged from udgamlabs/Laghu-MLX: the weights and tokenizer are © 2026 Udgam Labs LLP, and the chat template is Google's Gemma 4 template (Apache-2.0).