Mellum2-12B-A2.5B-Instruct-mlx-4bit

This is a 4-bit MLX quantization of JetBrains/Mellum2-12B-A2.5B-Instruct, the instruction-tuned Mixture-of-Experts coding assistant from JetBrains. It is derived from the full-precision jedisct1/Mellum2-12B-A2.5B-Instruct-mlx conversion.

Unlike its sibling Thinking model, the Instruct model answers directly without a <think> reasoning block. Mellum 2 uses 64 experts with 8 active per token (about 2.5B active parameters out of 12B), a mix of sliding-window and full-attention layers, and a 131,072-token context window.

A 4-bit quant tuned for reliable tool calling

A flat 4-bit quantization puts the attention and output layers at 4 bits, and those are exactly where the model copies literal source text verbatim into a tool argument. Quantizing them too aggressively is what makes a 4-bit coding model start emitting malformed tool calls.

This quantization avoids that. The 64 experts are the overwhelming majority of the weights, so they are quantized to 4 bits to keep the file small, while the parts that govern formatting and routing are kept at 8 bits:

• attention projections (q_proj, k_proj, v_proj, o_proj) — 8-bit
• token embeddings and lm_head — 8-bit
• the per-layer expert router (mlp.gate) — 8-bit
• the MoE experts (switch_mlp) — 4-bit

All quantization uses a group size of 64, which works out to about 4.85 bits per weight. The disk footprint is the same as a flat 4-bit model, but tool calling holds up.

Tool calling was verified end to end against a live mlx_lm.server driven by the swival agent harness, run side by side with the full-precision model. Across repeated multi-step coding tasks the 4-bit weights matched the full model's behavior: well-formed read_file, edit_file, write_file, list_files, and shell-command calls, and no malformed tool calls. Generation stops cleanly on <|im_end|> (the eos_token_id is set to [0, 28], which is what lets agent harnesses see a proper tool_calls finish reason).

If you want output that matches the original model as closely as possible and can spare the memory, use the 8-bit quantization instead.

Requirements

The mellum architecture is not supported by the stock mlx-lm code yet.

Until it is supported upstream, install this fork of mlx-lm from source:

pip install git+https://github.com/jedisct1/mlx-lm

Or run it directly with uv:

uvx --from git+https://github.com/jedisct1/mlx-lm mlx_lm.server

Use with mlx-lm

Quick test:

uvx --from git+https://github.com/jedisct1/mlx-lm \
  mlx_lm.generate --model jedisct1/Mellum2-12B-A2.5B-Instruct-mlx-4bit \
  --prompt "Write a Python function that reverses a linked list." \
  --max-tokens 16384 \
  --temp 0.6 --top-p 0.95 --top-k 20

Starting the server:

uvx --from git+https://github.com/jedisct1/mlx-lm \
  mlx_lm.server --model jedisct1/Mellum2-12B-A2.5B-Instruct-mlx-4bit \
  --max-tokens 16384 \
  --temp 0.6 --top-p 0.95 --top-k 20

The recommended sampling settings from JetBrains are temperature=0.6, top_p=0.95, top_k=20.

Using this setup with the Swival.dev harness

Install swival.dev:

uv tool install swival

Then point it at the running server:

swival --provider llamacpp --model jedisct1/Mellum2-12B-A2.5B-Instruct-mlx-4bit

License

Apache 2.0, inherited from the original model.