Ornith-1.0-9B-MTP-int4-AutoRound

This repository contains an INT4 (W4A16, group_size 128/64) quantized version of deepreinforce-ai/Ornith-1.0-9B with an integrated MTP (Multi-Token Prediction) head, optimized using Intel AutoRound with 1000 iterations.

---

⚡ Key Highlights

• Intel AutoRound Optimization (1000 Iterations): Unlike legacy heuristic quantization methods (e.g., outdated GPTQ configurations) that introduce perplexity spikes and degrades agentic reasoning, Intel AutoRound optimizes weight rounding via signed gradient descent. Running 1000 iterations ensures maximal quality preservation for complex code reasoning, terminal scaffolding, and multi-step tool use.
• Injected MTP Head (Speculative Decoding): Includes an integrated Multi-Token Prediction head for native speculative decoding. This enables ~1.5x speedups in single-stream decoding without quality loss.
• Reduced VRAM Footprint: Shrinks the ~19 GB BF16 model down to ~5.5–6.0 GB, enabling high-performance deployment on single consumer GPUs (e.g., RTX 3090/4090, Intel Arc, or cloud instances).

---

📊 Model Details

Attribute	Details
Base Model	deepreinforce-ai/Ornith-1.0-9B
Quantization Scheme	INT4 Weight-Only (W4A16)
Quantization Engine	intel/auto-round (1000 iterations)
Feature Additions	Multi-Token Prediction (MTP) sidecar head injected
License	MIT

---

🚀 Quickstart & Serving

vLLM with MTP Speculative Decoding

To leverage both INT4 memory savings and MTP acceleration in vLLM:

vllm serve urakozz/Ornith-1.0-9B-MTP-int4-AutoRound \
  --port 8000 \
  --tensor-parallel-size 1 \
  --max-model-len 128k \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --served-model-name ornith-9b