Qwythos-9B-v2-oQ4-mtp

This model was quantized using oQ (oMLX v0.5.1) mixed-precision quantization.

Quantization details

• Model type: qwen3_5
• Bits: 4
• Group size: 64
• Format: MLX safetensors

Environment

• Hardware: M5 MacBook Air 32GB
• Inference Framework: oMLX v0.5.1
• Settings:
  • Thinking: Disabled
  • Chat template parameter: enable_thinking=false (forced)
  • TurboQuant KV Cache: Disabled (if enable: reduces speed)
  • Native MTP: Enabled (key speed improvement)

Performance Benchmarks

Note: Results are for reference only and may vary depending on hardware, software configuration, and workload.

Single Request Results

Test	TTFT(ms)	TPOT(ms)	pp TPS	tg TPS	E2E(s)	Throughput	Peak Mem
pp1024/tg128	1496.9	21.44	684.1 tok/s	47.0 tok/s	4.242	271.5 tok/s	6.11 GB
pp4096/tg128	5907.3	22.32	693.4 tok/s	45.2 tok/s	8.768	481.7 tok/s	6.73 GB

Continuous Batching (pp1024 / tg128)

Batch	tg TPS	Speedup	pp TPS	pp TPS/req	TTFT(ms)	E2E(s)
1x	47.0 tok/s	1.00x	684.1 tok/s	684.1 tok/s	1496.9	4.242
2x	48.4 tok/s	1.03x	635.0 tok/s	317.5 tok/s	3225.2	8.519
4x	76.4 tok/s	1.63x	604.0 tok/s	151.0 tok/s	6616.7	13.485

Intelligence Benchmark

Note: Each benchmark round tests only 30 questions. Results are for reference only.

Benchmark	Accuracy	Correct	Total	Time(s)	Think
MMLU	66.7%	20	30	36.4	No
TRUTHFULQA	83.3%	25	30	13.8	No
GSM8K	83.3%	25	30	147.8	No
MATHQA	40.0%	12	30	30.3	No
HUMANEVAL	80.0%	24	30	163.5	No

Summary

Benchmark	Accuracy
MMLU	66.7%
TRUTHFULQA	83.3%
GSM8K	83.3%
MATHQA	40.0%
HUMANEVAL	80.0%
Average	70.7%