mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs

Altworld's creative-writing model, quantized for a Mac. A mixed-precision MLX quant of HemmingwayAI/Astrea-R8-Chat-9B, a Qwen3.5-9B fine-tune built for chat and fiction. 8.0 GB on disk, down from 18.8 GB bf16. See the original card for what the model is trained to do and how Altworld evaluates it.

OptiQ quantizes the language tower to mixed 4/8-bit and keeps the vision tower at bf16 in a sidecar, so one checkpoint does both text and images.

What it is

Property	Value
Base	HemmingwayAI/Astrea-R8-Chat-9B (Qwen3.5-9B fine-tune)
Language tower	per-layer 4/8-bit: 116 layers at 4-bit, 134 at 8-bit (250 total)
Vision tower	bf16, kept in optiq_vision.safetensors (333 tensors)
On disk	7.10 GB language + 0.91 GB vision
Group size	64
Context	262,144 tokens

Following llama.cpp's naming for mixed quants, the "4bit" label denotes the predominant precision, not the weighted average.

How the bit-widths were chosen

The per-layer allocation was not measured on this model. It was transferred from mlx-community/Qwen3.5-9B-OptiQ-4bit, whose allocation came from a KL-divergence sensitivity sweep over a six-domain calibration mix. The two language towers share an architecture exactly, and all 250 tensors matched with none unmatched.

Checked after conversion

Run through this quant and its parent, Qwen3.5-9B-OptiQ-4bit, on Apple Silicon with greedy decoding and thinking off:

Prompt	Astrea	Parent
Capital of France, one word	Paris	Paris
120 km in 1.5 h, speed in km/h	80	80
is_palindrome(s) ignoring case and non-letters	correct function	correct function
What makes a good short-story opening, two sentences	coherent answer in its own wording	coherent answer

Image input: shown a red rectangle, a blue circle and the number 42, it answered "The image contains a red square, a blue circle, and the number 42 written in black."

Serve it with n-gram drafting

Measured on a 24 GB M4, batch 1, greedy, median of three runs each:

Workload	Plain	--ngram-draft 16
Edit (rename a parameter across a file and return it)	16.6 tok/s	44.9 tok/s (2.7x)
Code generation	16.7 tok/s	18.0 tok/s (1.08x)
Prose from a 1,024-token prompt	16.6 tok/s	16.2 tok/s (0.98x)

pip install mlx-optiq
optiq serve --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit --ngram-draft 16

This checkpoint has no MTP head, so --mtp does not apply.

Use it

# chat, images, and every app that talks to optiq serve
optiq serve --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit --ngram-draft 16
optiq lab --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit

Text only, with stock mlx-lm:

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit")
messages = [{"role": "user", "content": "Open a short story set on a night ferry."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=400))