randmaru/Mellum-4b-base-mlx-mxfp4

This is an MXFP4 MLX quantization of JetBrains/Mellum-4b-base for Apple Silicon inference.

---

MXFP4 vs 4Bit Quantization Comparison

Parameter	MXFP4	4Bit
Quantization format	4‑bit floating point with microscaling, group 32, shared exponent E8M0	4‑bit integer (INT4/NF4)
Tensor types	U8, U32, BF16	BF16, U32
Parameter size (safetensors)	~2.14 GB (2,135,555,369 bytes)	~2.26 GB (2,261,175,474 bytes)
Total storage (all files)	~2.14 GB (2,142,623,163 bytes)	~2.27 GB (2,270,778,309 bytes)
Hardware support	Most efficient on GPUs with microscaling / FP8 tensor core support	Broad support, but often requires specialized INT4 kernels
Apple Silicon compatibility	Designed with hardware microscaling support in Apple Neural Engine / GPU	Works, but without specialized Neural Engine optimization
Inference speed	Higher on compatible hardware: FP path, lower dequantization overhead, higher throughput	Kernel‑dependent; usually lower or comparable at similar quality
Quality	Better preserves dynamic range, less degradation on outliers	Higher risk of accuracy loss on outliers at the same bitrate

Key takeaways:

• Parameter size: The MXFP4 version has a slightly smaller safetensors file (~2.14 GB vs ~2.26 GB). This reflects the actual file sizes from the respective repositories; the MXFP4 format with microscaling and floating‑point storage (U8/U32/BF16) results in a more compact representation in this case compared to the integer version (BF16/U32).
• Total storage: MXFP4 also occupies less total disk space (~2.14 GB vs ~2.27 GB) when summing all files in the repository. This indicates more efficient packaging and/or fewer auxiliary files.
• Performance: MXFP4 typically delivers higher inference throughput (tokens/s) on hardware with microscaling/FP8 support, especially on Apple Silicon. The shared exponent per group of 32 elements reduces dequantization overhead and enables the use of floating‑point tensor cores, which can be faster than conventional INT4 kernels.
• Apple Silicon optimization: The MXFP4 format was developed with the hardware capabilities of Apple chips (M‑series) in mind, particularly the Neural Engine, which natively supports microscaling. Therefore, MXFP4 is the recommended choice for MacBook and offers the best balance of speed and quality on these devices.
• Quality: MXFP4 preserves the dynamic range of weights better, so generation quality can be higher at the same compression level.

Actual speed depends on the backend, GPU, batch size, and quantization implementation.