MiniCPM5-2B-4bit-NQ

Numen Quant (NQ) 4-bit build of openbmb/MiniCPM5-2B, quantized for Apple silicon with MLX.

Modifications

This model is a modified version of openbmb/MiniCPM5-2B: its weights have been modified and quantized to 4-bit (group size 64) by Numen. The architecture, tokenizer and chat template are unchanged. Distributed under the Apache License 2.0 (see LICENSE), as is the original.

Use with mlx

pip install -U mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("numen-tech/MiniCPM5-2B-4bit-NQ")

messages = [{"role": "user", "content": "Hello"}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, verbose=True))

Evaluation

model	wino	boolq	arc-c	arc-e	hella	obqa	piqa	siqa	mean
BF16 (original)	61.3	82.4	43.9	72.6	59.5	36.2	72.9	44.2	59.12
MLX affine 4-bit	60.2	81.7	42.2	69.1	57.6	34.8	71.4	43.0	57.50
NQ 4-bit	61.4	82.5	42.7	70.2	57.6	37.0	72.4	43.9	58.47

Zero-shot, lm-evaluation-harness 0.4.13 via mlx_lm.evaluate; acc_norm where the task defines it, acc otherwise. Single run per model; per-task standard errors are 0.5–2.2 points. MLX affine 4-bit is mlx_lm.convert -q --q-bits 4 --q-group-size 64 from the same checkpoint.