sergolsub/mlx-elephant-v1-9B

baselabs elephant-1 (MLX 8-bit)

8-bit MLX conversion of baselabs/elephant-1 for Apple Silicon (mlx_lm).

Origin and license

• Original weights: https://huggingface.co/baselabs/elephant-1
• License: mit — carried over from the upstream weights; if anything conflicts, the upstream repository's LICENSE prevails.
• Conversion: 8-bit quantization for MLX only; no retraining, no architecture changes.

Measured (Mac mini M4 Pro 48 GB, MLX 8-bit)

• Standard stand — load 5.08 s · peak RSS 12.21 GB
• Text 512 tokens (492 tok) — avg of 5 after 3 warm-up: binary 1425 ms · multichoice 1430 ms · score 1435 ms · peak RSS 12.21 GB
• Image 512x512 + text — avg of 5 after 3 warm-up: binary 2407 ms · multichoice 2404 ms · score 2400 ms · peak RSS 12.21 GB
• Box: not supported
• On disk 11.4 GB · load 5.32 s · peak resident memory 12.21 GB
• Decision latency: median 657 ms · sustained 2.56 short decisions/s
• Quality after 8-bit quantization (our eval): accuracy 92.2% (8-bit) vs 92.2% (bf16) · noul 95.0% vs 95.0% · choice 92.0% vs 92.0% · score 88.0% vs 88.0%
• Calibration PPL 1.769 (8-bit) vs 1.771 (bf16)
• Decision Index 0.4.1: 62.11

Usage

# our qev port — same runtime the /v1/systemone server uses
from huggingface_hub import snapshot_download
path = snapshot_download("sergolsub/mlx-elephant-v1-9B")  # weights + the loader .py
import sys; sys.path.insert(0, path)
from onejev_mlx import OneJev
model = OneJev(path)
# one separate call per task type (each question is its own object):
print(model.decide(
    "I was charged twice for order #4411 and the app still shows it as unpaid.",
    {"refund": {"type": "noul", "instructions": "Does the customer want a refund?"}}
))
print(model.decide(
    "Order #7821 shipped on Monday but tracking has not updated for 5 days.",
    {
        "team": {"type": "choice", "instructions": "Which team should handle this?",
                  "criteria": {"billing": "Payments and refunds",
                               "shipping": "Delivery and tracking"}}
    }
))
print(model.decide(
    "Export failed: disk quota exceeded on the worker.",
    {
        "severity": {"type": "score", "instructions": "How severe is the impact?",
                      "criteria": ["cosmetic", "annoying", "blocks"]}
    }
))

Files

• *.safetensors — 8-bit weights (MLX layout)
• config.json, tokenizer*, chat_template.jinja — standard transformers/MLX set
• *_mlx.py / decision_config.json / joint_config.json — decision/joint-head inference glue (present when the model has a decision mode)

Quantization quality (KL vs bf16)

• Weights compare against the bf16 originals tensor by tensor (histogram KL over 512 bins, dequantized 8-bit vs source).
• Tensors compared: 760 · mean KL 2.16e-04 · max KL 5.57e-03 · nonzero 249
• Worst layer: model.language_model.layers.0.linear_attn.in_proj_a.weight (KL 5.57e-03)
• Full per-tensor table: kl_audit.json in this repository (tensor, KL, max abs err, RMSE).

Checksums (SHA-256)

SHA-256 of every file in this repository except this README itself: LFS digests pulled from the HF metadata, small files (config, tokenizer, …) hashed on demand — the safetensors shards included. Re-check any download with sha256sum <file> and compare.

File	SHA-256	Size
.gitattributes	34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930	1,570 B
chat_template.jinja	a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715	7,756 B
config.json	156ce11b92b3890a0a95d21789c38f38e9760d6669d520dc8a68ba8472635295	3,318 B
constraints.txt	7afb5334b5d57ef09e3d8ac2fc670f07182c50ffb51035d5b6eb7898f863edfb	1,416 B
engine.py	72373353379c53b7cabfb5d05b6fef3c898d48c87cc386e03fc3de9c07e67ca7	2,799 B
generation_config.json	290866a6d7d4a4442fe83ba2434a31517446f40a01de500ff97b4354e7e5d082	116 B
kl_audit.json	5043aa63379faff0a21239f5332ddfcd54b7639a8bdb38cd34e6575b185d5090	64,436 B
model-00001-of-00003.safetensors	3ca094e32f467f13f5ffb0da6bf620e6729c9b4706a6958ea5aeea4dfadd838e	4.30 GB
model-00002-of-00003.safetensors	e9b5447ac842ab0e0171a0fd38b1d38dfdb1f542630e6a5d70adff284610999d	4.32 GB
model-00003-of-00003.safetensors	f0877ec041cec2855d1712e00dfb46dd9e64daffb71caeffa04db74a32d25c96	2.75 GB
model.safetensors.index.json	64571659bd78a67f35288acd258b54308bcd613bf0e04743e6adf16d739072b0	115,858 B
onejev_mlx.py	5ab1bfe822d381b3ac56ccc378df312ebecf5b96ba8469b22b357d20ba348e8f	19,173 B
processor_config.json	bfbc24af59a3e73a9cd0653b8d4ae758dfaec4e3e6c15dfb1c6c8ee8d5683c85	1,220 B
readout.json	96ea8a5c6ecfc30c20cdd369762db5997c5f4fb9821ce0a0318941343eb5e8ba	63 B
serve.patch	f234c125f1d38d82104b9e1f24e75bde51e4401928acfba2fbf5c59ef2fb46b8	41,045 B
setup.sh	6bf9514c6b80f585a94f9a40c03f87c27cce9e2de8d2ba4cf4b41434babca62a	1,285 B
tokenizer.json	a5cd9732badce41de57e6efce8302930ded1c1188c5f81feb2bd6c24c4a1941f	20 MB
tokenizer_config.json	2984c03425a28d242744e234a5003786fd289b3713db79fe0958c65eb59d85b2	1,505 B