rajib-sarwar/Llama-3.2-1B-Instruct-4bit-MLX

⚡️ Llama-3.2-1B-Instruct (4-Bit MLX)

This repository hosts a 4-bit quantized artifact of Meta's Llama-3.2-1B-Instruct foundation model, engineered specifically for high-throughput, low-latency on-device inference on Apple Silicon using Apple's open-source MLX framework.

---

📊 Hardware Benchmarks & Performance Telemetry

Telemetry was recorded running native INT4 execution passes across unified memory on Apple Silicon:

Metric	Measured Telemetry	Engineering Impact
Model Parameters	1.23 Billion	Base foundation transformer weights
Quantization Precision	INT4 (~4.501 bits/weight)	Compressed from 16-bit float
Storage Footprint	~695 MB (.safetensors)	Reduced from ~2.48 GB FP16 baseline
Peak Memory Allocation	0.771 GB – 0.828 GB	Safely below iOS Jetsam (OOM) memory limits
Cold Start Prefill	44.64 tokens/sec	High I/O penalty loading tensors from disk
Warm Start Prefill	778.79 tokens/sec	~17.4x throughput increase (Cached in RAM)
Sustained Throughput	~319.62 – 321.88 tokens/sec	Real-time autoregressive token generation

🧠 KV Cache Memory Footprint (Grouped-Query Attention)

Thanks to Grouped-Query Attention (GQA with 8 KV heads vs. 32 query heads), memory allocation scales minimally over extended context lengths:

Generated Output Length	Peak Unified Memory	Measured KV Cache Overhead
300 Tokens	0.737 GB	Base runtime
600 Tokens	0.718 GB	Active memory recycling
1,200 Tokens	0.759 GB	~41 MB cache delta

• Theoretical Cache Growth: $$2\times 16\text{ layers}\times 8\text{ KV heads}\times 64\text{ dim}\times 1200\text{ tokens}\times 2\text{ bytes}\approx \mathbf{37.5}\text{ MB}2\; \backslash times\; 16\backslash text\{\; layers\}\; \backslash times\; 8\backslash text\{\; KV\; heads\}\; \backslash times\; 64\backslash text\{\; dim\}\; \backslash times\; 1200\backslash text\{\; tokens\}\; \backslash times\; 2\backslash text\{\; bytes\}\; \backslash approx\; \backslash mathbf\{37.5\backslash text\{\; MB\}\}$$
• Edge Safety: The combined INT4 weights + KV cache footprint remains under 0.8 GB, ensuring safe execution on constrained iOS runtimes without Jetsam memory termination risks.

---

📈 Latency Dynamics: Cold Start vs. Warm Start Prefill

A primary bottleneck in on-device AI infrastructure is the latency disparity between uninitialized storage reads and active Unified Memory caching:

1. Cold Start Prefill (44.64 tok/s) — High I/O Storage Penalty
┌─────────────┐   Disk Read & Bus Transfer   ┌────────────────┐   Compute Pass   ┌──────────────┐
│ NVMe / SSD  │ ───────────────────────────► │ Unified Memory │ ───────────────► │ GPU Compute  │
└─────────────┘           (~22.4 ms/tok)     └────────────────┘                  └──────────────┘

2. Warm Start Prefill (778.79 tok/s) — Zero Storage Latency (~17.4x Acceleration)
                                             ┌────────────────┐   Direct Stream  ┌──────────────┐
                                             │ Resident Cache │ ───────────────► │ GPU Compute  │
                                             └────────────────┘   (~1.28 ms/tok) └──────────────┘

Prefill Latency Comparison (Tokens per Second)
──────────────────────────────────────────────────────────────────────────────
Cold Start (Disk I/O)    | ██ (44.64 tok/s)
Warm Start (RAM Cached)  | ████████████████████████████████████ (778.79 tok/s) [17.4x]
──────────────────────────────────────────────────────────────────────────────

---

🚀 Quickstart & Inference

1. Installation

Ensure an Apple Silicon Mac with macOS 14+ and an isolated virtual environment:

pip install mlx-lm huggingface_hub

2. Direct CLI Generation

Run text generation directly from your terminal using the MLX engine:

mlx_lm.generate \
    --model rajib-sarwar/Llama-3.2-1B-Instruct-4bit-MLX \
    --prompt "Explain how 4-bit quantization reduces memory pressure on Apple Silicon." \
    --max-tokens 300

3. Native Python Pipeline

from mlx_lm import load, generate

# Load the 4-bit model directly from Hugging Face Hub
model, tokenizer = load("rajib-sarwar/Llama-3.2-1B-Instruct-4bit-MLX")

prompt = "Explain how 4-bit quantization reduces memory pressure on Apple Silicon."

response = generate(
    model, 
    tokenizer, 
    prompt=prompt, 
    max-tokens=300, 
    verbose=True
)

print(response)

---

🛠️ Quantization Pipeline & Methodology

Converted from official meta-llama/Llama-3.2-1B-Instruct bfloat16 tensors using mlx_lm.convert:

# 1. Force snapshot metadata caching
python3 -c "from huggingface_hub import snapshot_download; snapshot_download('meta-llama/Llama-3.2-1B-Instruct')"

# 2. Execute 4-bit quantization
mlx_lm.convert \
    --hf-path meta-llama/Llama-3.2-1B-Instruct \
    -q \
    --q-bits 4 \
    --upload-repo rajib-sarwar/Llama-3.2-1B-Instruct-4bit-MLX

---

🛡️ Edge Safety & On-Device Memory Design

• Jetsam Compliance: Operating at < 830 MB peak memory overhead ensures mobile applications running on iOS/iPadOS can load and stream responses without risking operating system Jetsam Out-of-Memory (OOM) termination crashes.
• Unified Memory Bus Optimization: Quantizing weights to 4-bit cuts required memory bandwidth by ~70%, minimizing battery drain and preventing thermal throttling on constrained devices.

---

📄 License & Attribution

This model is a quantized derivative of Meta's Llama 3.2 family. Usage is subject to the Meta Llama 3.2 Community License Agreement.