dhara-250m-OptiQ-8bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs

An OptiQ mixed-precision 8-bit quant of codelion/dhara-250m, the second member of OptiQ's Diffusion LLM family, for Apple Silicon.

dhara is a tri-mode 250M model: one set of weights that decodes three ways, standard autoregressive (left-to-right), block-diffusion (fill a block of tokens and iteratively un-mask it), and self-speculation (draft a block with the diffusion forward, verify with the AR forward). It is a custom architecture stock mlx-lm can't load (it adds Canon depthwise-conv layers, QK-norm after RoPE, and a logit soft-cap); OptiQ ships a vendored, mlx-native port that registers with mlx-lm and is bit-exact to the reference.

At 250M, dhara is a base to fine-tune, the way Google's Gemma-270M is, small enough to LoRA on-device for one task, not a general assistant.

Demo: dhara-chat Space — chat with it and compare all three decoding modes.

Install

pip install mlx-optiq

Usage

import optiq  # registers the dhara architecture with mlx-lm
from mlx_lm import load, generate

model, tok = load("mlx-community/dhara-250m-OptiQ-8bit")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Explain the Mediterranean climate."}],
    tokenize=False, add_generation_prompt=True)
print(generate(model, tok, prompt))

Block-diffusion and self-speculation are handled by the OptiQ runtime. optiq serve --model mlx-community/dhara-250m-OptiQ-8bit serves an OpenAI/Anthropic-compatible API; --mtp routes through the self-speculative path. LoRA fine-tuning uses the standard optiq lora train autoregressive trainer.

The quant, 8-bit preserves the reference model

At 250M there is no redundancy to spend, so this quant is tuned for fidelity to the reference, not for the smallest possible file. OptiQ's mixed precision holds the layers that need it at bf16 and drops the rest to 8-bit.

Measured against the bf16 reference on the same calibration prompts:

Variant	Size	bpw	KL vs bf16 ↓	Reproduces bf16 output
bf16 (reference)	460 MB	16	—	—
uniform 4-bit	130 MB	4.53	0.0608	no
uniform 8-bit	266 MB	8.52	0.0007	partly
dhara-250m-OptiQ-8bit	357 MB	10.25	0.0005	yes

Decoded with the reference implementation's settings (greedy, repetition penalty 1.3), autoregressive and self-speculative decode are byte-identical to bf16; block-diffusion matches at 0.87 similarity (its confidence-threshold un-masking is chaotic, and more precision does not tighten it).

A 4-bit dhara stays fluent but answers differently from the reference — coherent text, different model. On a 250M base that is meant to be fine-tuned, matching the reference matters more than the last 200 MB.

The full 6-benchmark Capability Score (MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop) is unchanged:

Variant	Capability	MMLU	GSM8K	IFEval
bf16 (reference)	8.34	24.7	1.6	23.3
dhara-250m-OptiQ-8bit	8.33	24.5	1.7	23.8

BFCL, HumanEval and HashHop sit at 0 for every variant including bf16; this is a genuine 250M ceiling, not a quantization or harness artifact. Note the Capability Score is flat across every quant of this model, 4-bit included, because a 250M base has no benchmark headroom for quantization damage to show up in, which is why the KL and output-fidelity numbers above are the ones that decide the bit budget.

Scores are reported honestly. dhara-250m is meant to be fine-tuned on a specific task, where these base scores are the starting point, not the product.

Decode modes, self-speculation is the default

dhara decodes three ways from one set of weights. The recommended default is self-speculation (--mtp): it drafts a block in one parallel forward and verifies it autoregressively (two forwards per round, no commit pass), so the emitted output is identical to plain AR decode while committing ~3–4 tokens per round, AR accuracy at ~1.4× the speed of token-by-token AR. The model is overhead-bound (a 32-token forward costs about the same as a 1-token forward), so quantization buys size, not throughput.

Mode	Speed (M3 Max)	Character
self-speculation (--mtp)	~1.4× AR	recommended, output identical to AR, several tokens/round
autoregressive	~130 tok/s	the exact reference; pair with a repetition penalty (greedy can loop)
block-diffusion	parallel	prefix-cached; bidirectional (infilling), trades denoising steps for speed

Self-spec and block-diffusion are prefix-cached (KV + Canon-conv state), so each step processes only the new block, O(block) per step, not O(sequence).

Quantization details

OptiQ measures each layer's quantization sensitivity (KL divergence vs the bf16 reference on calibration data) and assigns per-layer bit-widths under a target budget. This quant: 99 weight tensors at 8-bit + 125 held at bf16, 10.25 bits-per-weight.

Because dhara is tri-mode, the calibration probes both forwards it serves, causal AR and block-diffusion, across the mask trajectory a block actually travels (100% masked down to 25%). Self-speculation needs no separate calibration: it composes the other two forwards.

The Canon depthwise convs, QK-norm, and logit soft-cap are not Linear modules, so they stay at bf16 automatically, only the attention and MLP projections are quantized.

Quantize your own

This quant was produced by mlx-optiq. Point it at any Hugging Face model to get the same sensitivity-aware mixed precision:

pip install mlx-optiq

# small model, no redundancy to spend: 16 = keep the layer at bf16
optiq convert <hf-model-id> --target-bpw 10 --candidate-bits 8,16

# larger model with headroom
optiq convert <hf-model-id> --target-bpw 5.0 --candidate-bits 4,8

optiq lab   # full local workbench: chat, compare, quantize, fine-tune

License + provenance

Derived from codelion/dhara-250m. See the Diffusion LLM family guide for details.