Inkling-Small-mlx-3bit

An MLX 3bit build of thinkingmachines/Inkling-Small — 264 B total parameters, ~12 B active; 42 layers, 256 routed experts (top-6) + 2 shared; natively multimodal (image + audio in, text out).

Sized to run fully resident on a 128 GB Mac — no expert-offload, no expert pruning, no layer streaming. The whole model sits in unified memory and decodes at conversational speed.

Why this model is the interesting one for Apple Silicon: speed follows the active parameter count, not how hard you compress. Inkling-Small activates ~12 B params per token, so this tier reads roughly 9.8 GB per token. The 975 B Inkling, by contrast, needs SSD expert-offload even at 2-bit and manages ~0.2–0.4 tok/s.

Tiers

Pick the tier that fits your machine. Sizes are on-disk and measured from the built artifacts; peak/speed columns are filled in only where a benchmark has actually run.

tier	target Mac	recipe	on-disk	peak	load	prefill tok/s
4bit	192 GB	experts 4-bit, non-expert 8-bit	153.5 GB	—	—	—
3bit ← this repo	128 GB	experts 3-bit, non-expert 8-bit	120.9 GB	—	—	—
2bit	96 GB	experts 2-bit, non-expert 8-bit	88.4 GB	—	—	—

Speeds are not yet measured. On-disk sizes above are real (taken from the built artifacts), but load/peak/tok-s columns stay empty until serving/bench_mlx.py has run on the target hardware — we would rather ship a blank column than a guess. Run it yourself with the command below and please open a discussion with your numbers.

Why MLX on Apple Silicon

MLX is Apple's array framework for Apple Silicon. For a big sparse MoE like this one, the practical differences from a CPU/GPU-split runtime are:

	what it means here
Unified memory	The GPU reads the same DRAM as the CPU, so a 120 GB build needs 120 GB of system memory — not VRAM plus a host copy. There is no PCIe transfer per layer, which is what makes a >100 GB model practical on a desktop at all.
Lazy, mmap'd loading	mx.load memory-maps safetensors, so weights page in on demand instead of being read and copied up front.
Native quantized matmul	Quantized weights are multiplied in their packed form (quantized_matmul, and gather_qmm for MoE expert gathers) rather than dequantized to fp16 first — so low-bit tiers save bandwidth at runtime, not just on disk.
Per-module precision	nn.quantize(..., class_predicate=...) lets one checkpoint mix bit-widths per tensor class, which is exactly how these tiers keep attention and the router high-precision while the experts go low.
Small dependency surface	pip install mlx mlx-lm and a single Python model file. No compile step, no separate server binary.

Honest limits: MLX's ecosystem is younger than llama.cpp's, it has no importance-matrix ("i-quant") style calibrated quantization yet, and it runs on Apple Silicon only. If you want a GGUF build instead, one exists at unsloth/Inkling-Small-GGUF.

Memory notes

peak (once measured) is peak unified-memory use, not file size — leave headroom for the KV cache and the OS. Two things matter on a machine near its limit:

• Raise the Metal wired limit. It defaults to ~75% of RAM, which is below this tier's footprint on a 128 GB machine:

  sudo sysctl iogpu.wired_limit_mb=180000   # ~180 GB on a 192 GB Mac; scale to your RAM
  

• macOS will kill a process that stays too large for too long (jetsam), even while memory pressure looks fine. If long generations die silently, move down a tier.

Quantization recipe

experts_only: the routed experts (and the vision/audio matmuls) carry the tier's bit-width, while attention, token/output embeddings, RMSNorms, the router gate, the per-layer short-convolutions and the relative-position bias stay high-precision (8-bit).

That asymmetry is the point, and it is nearly free: non-expert weights are only 5.9 B of the 264 B params (~6 GB at 8-bit), yet they are read in full on every token, while just 6 of 256 experts are. Bits spent there cost ~2% of the file and protect the paths every token depends on. Expert precision degrades gracefully; attention and router precision degrade globally.

Usage

# pip install mlx mlx-lm transformers        (+ scipy for audio, pillow for images)
# this repo bundles the inkling_mlx/ loader, so there is nothing else to install
from inkling_mlx.load import load
from inkling_mlx.generate import greedy_generate
from transformers import AutoTokenizer

path = "./Inkling-Small-mlx-3bit"
model, config = load(path)
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
ids = tok("The capital of France is")["input_ids"]
print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))

Images are cut into 40 px patches (one soft-token each) and audio is d-mel encoded at 20 tokens/s; both go through InklingProcessor. See the source repo for a multimodal runner.

Benchmark it yourself

python serving/bench_mlx.py --model ./Inkling-Small-mlx-3bit --tier 3bit

Reports load time, prefill tok/s, decode tok/s at several context lengths, and peak memory.

Attribution

• Base model: thinkingmachines/Inkling-Small, Apache-2.0.
• The bundled inkling_mlx/ loader is vendored from PipeNetwork/inkling-mlx, Apache-2.0.
• Quantized with mlx; see the recipe above for exactly which tensors were touched.