sergolsub/mlx-bespoke-v3-9B

Bespoke-Nimble-9B-v3 (MLX 8-bit)

8-bit MLX conversion of bespokelabs/Bespoke-Nimble-9B-v3 for Apple Silicon (mlx_lm).

Origin and license

• Original weights: https://huggingface.co/bespokelabs/Bespoke-Nimble-9B-v3
• License: cc-by-nc-4.0 — carried over from the upstream weights; if anything conflicts, the upstream repository's LICENSE prevails.
• Conversion: 8-bit quantization for MLX only; no retraining, no architecture changes.

Non-commercial. CC BY-NC 4.0: commercial use is not permitted; keep the upstream attribution when redistributing.

Stream-converted (LoRA merge over the torch base + affine-int8); the upstream declares cc-by-nc-4.0 in its card metadata and ships no LICENSE file, so the license lives in this card's frontmatter

Measured (Mac mini M4 Pro 48 GB, MLX 8-bit)

• Short decision — 3 warmup + median of 5: 525 ms
• On disk 11.38 GB · load 0.66 s · peak resident memory 12.11 GB
• Decision latency: median 826 ms · sustained 1.85 short decisions/s
• Quality after 8-bit quantization (our eval): accuracy 83.0% (8-bit) vs 83.0% (bf16) · noul 94.9% vs 94.9% · choice 76.0% vs 76.0% · score 70.8% vs 70.8%
• Calibration PPL 2.042 (8-bit) vs 2.036 (bf16)
• Decision Index 0.4.1: 51.89

Usage

from huggingface_hub import snapshot_download
path = snapshot_download("sergolsub/mlx-bespoke-v3-9B")  # weights + the loader .py
import sys; sys.path.insert(0, path)
from bespoke_mlx import Bespoke
model = Bespoke(path)
# one separate call per task type (each question is its own object):
print(model.decide(
    "I was charged twice for order #4411 and the app still shows it as unpaid.",
    {"refund": {"type": "noul", "instructions": "Does the customer want a refund?"}}
))
print(model.decide(
    "Order #7821 shipped on Monday but tracking has not updated for 5 days.",
    {
        "team": {"type": "choice", "instructions": "Which team should handle this?",
                  "criteria": {"billing": "Payments and refunds",
                               "shipping": "Delivery and tracking"}}
    }
))
print(model.decide(
    "Export failed: disk quota exceeded on the worker.",
    {
        "severity": {"type": "score", "instructions": "How severe is the impact?",
                      "criteria": ["cosmetic", "annoying", "blocks"]}
    }
))

Files

• *.safetensors — 8-bit weights (MLX layout)
• config.json, tokenizer*, chat_template.jinja — standard transformers/MLX set
• *_mlx.py / decision_config.json / joint_config.json — decision/joint-head inference glue (present when the model has a decision mode)

Quantization quality (KL vs bf16)

• Weights compare against the bf16 originals tensor by tensor (histogram KL over 512 bins, dequantized 8-bit vs source).
• Tensors compared: 775 · mean KL 2.04e-04 · max KL 4.91e-03 · nonzero 341
• Worst layer: model.visual.blocks.19.attn.proj.weight (KL 4.91e-03)
• Full per-tensor table: kl_audit.json in this repository (tensor, KL, max abs err, RMSE).

Checksums (SHA-256)

SHA-256 of every file in this repository except this README itself: LFS digests pulled from the HF metadata, small files (config, tokenizer, …) hashed on demand — the safetensors shards included. Re-check any download with sha256sum <file> and compare.

File	SHA-256	Size
.gitattributes	34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930	1,570 B
bespoke_mlx.py	380165abf19c4bb670c06df88167d3998fb0dedcc37ef2683edaa87afc8a952a	15,832 B
chat_template.jinja	a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715	7,756 B
config.json	b407e81ab63127284077e154eed6f2f09917ed132c9636033fa9b6bbfc1a2d79	2,744 B
kl_audit.json	b8d503360222b70984b4072a0b9a002908340b1cc1b99fc4cfb98d7adce4cc07	70,912 B
merges.txt	a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d	3 MB
model-00001-of-00011.safetensors	678af6a255e485113a37149d6ac77c2b129e364261cfc4e7401fed67ebb352eb	2.03 GB
model-00002-of-00011.safetensors	5f5f1a4c8d2a90a089a0bf24e252645910e911612637b9f48764bea4c3cb435f	1.08 GB
model-00003-of-00011.safetensors	3441285c498897cc05c1512eca284737e75067ebd564387bf69590e35cc94726	1.12 GB
model-00004-of-00011.safetensors	0e497c604fae9118e2cde662526dbc617745d0dd4101624dfc72f378d9e97c82	1.12 GB
model-00005-of-00011.safetensors	11962bafff2c1214b78581c20630f59ce19c9d599fc148c727b692aa4c1692d9	1.12 GB
model-00006-of-00011.safetensors	2d5ba55d8632fd7713e5c32e69d81c8c688b31a81fe5eca6cdcb64935984fc3a	1.09 GB
model-00007-of-00011.safetensors	e199987de17011ed9cec3b80285ca7d720b7cbd73832fda7f99329f250d75198	1.11 GB
model-00008-of-00011.safetensors	c385773bf9d166d1aca91e358dcf368bbcdba2a13d949e2104747ca5dc540c1e	1.08 GB
model-00009-of-00011.safetensors	e85302f7e4a54bfcae1c09631a5c750b851ac03549be7f7d12700a4dfed679ad	1.08 GB
model-00010-of-00011.safetensors	faa89134dac02240fc88ffe0d23662f5d6aa54338223da4e036639cdb1562621	503 MB
model.safetensors.index.json	7cc746f8ed856cc0d7fc4735796ade7e14d00a9b06342006654268c6b7eb7911	131,885 B
preprocessor_config.json	27225450ac9c6529872ee1924fcb0962ff5634834f817040f444118116f4e516	390 B
tokenizer.json	5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42	13 MB
tokenizer_config.json	316230d6a809701f4db5ea8f8fc862bc3a6f3229c937c174e674ff3ca0a64ac8	16,710 B
video_preprocessor_config.json	7768af27c1fafa9cc9011c1dc20067e03f8915e03b63504550e11d5066986d13	385 B
vocab.json	ce99b4cb2983d118806ce0a8b777a35b093e2000a503ebde25853284c9dfa003	7 MB