Qwen2.5-7B-Instruct MLX ladder + group-size sweep
Collection
Complete MLX grid for Qwen2.5-7B-Instruct — full bit ladder (bf16/8/6/4/3/2-bit) + 4-bit group-size sweep at gs=32/64/128. • 8 items • Updated
Qwen2.5-7B-Instruct-MLX-bf16
Full non-quantized MLX bfloat16 conversion of Qwen/Qwen2.5-7B-Instruct. The clean reference build: HF bf16 in, MLX bf16 out — nothing chained, nothing converted from GGUF.
Apple Silicon only. GGUF Q4_K_M is a llama.cpp quant — MLX has no literal Q4_K_M mode. Don't conflate them.
Use
pip install mlx-lm
mlx_lm.generate --model zaydiscold/Qwen2.5-7B-Instruct-MLX-bf16 \\
--prompt "Explain quantum entanglement in one paragraph" --max-tokens 200
Conversion
python -m mlx_lm convert \
--hf-path Qwen/Qwen2.5-7B-Instruct \
--mlx-path ./Qwen2.5-7B-Instruct-MLX-bf16 \
--dtype bfloat16
Credits
- Source: Qwen/Qwen2.5-7B-Instruct
- MLX conversion: zaydiscold
Part of a Qwen2.5-7B-Instruct MLX quant ladder + group-size perplexity sweep. See the sibling repos under zaydiscold for other bit levels and group sizes — perplexity numbers are coming as a separate dataset repo.
M1 16GB smoke note
This bf16 reference artifact uploaded cleanly, but generation smoke failed on the Flow.swiss M1 16GB host with process exit -6. Use a larger Apple Silicon machine for bf16 runtime.
- Downloads last month
- 41
Model size
8B params
Tensor type
BF16
·
Hardware compatibility
Log In to add your hardware
Quantized
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support