ewin-reg/MiniCPM5-2B-EXL3-Quantized

MiniCPM5-2B EXL3 (4.0 bpw)

Unofficial Community Quantization: This repository provides community-quantized EXL3 weights for openbmb/MiniCPM5-2B. It is not affiliated with, maintained by, or endorsed by OpenBMB or ModelBest. For the official model card, research papers, and original unquantized weights, refer to the official OpenBMB repository.

---

1. Smaller File. Closer to the Original.

EXL3 (ExLlamaV3) departs from conventional scalar quantization (such as AWQ, standard GPTQ, or GGUF block rounding). Rather than rounding weights independently into scalar bins with per-group scales every 16 or 32 values, EXL3 utilizes multidimensional lattice vector quantization guided by calibration Hessians.

Why 4 bits in EXL3 really is 4 bits

In conventional block formats (like GGUF Q4_K_M or NVFP4), scale metadata must be stored every 16 to 32 weights, adding non-trivial bookkeeping bloat. A nominal 4-bit model often requires 4.5 to 4.8 bits per weight on disk.

In EXL3, quantization traverses the weights along optimized codebook paths with global channel scales, meaning 4.0 bpw truly equals 4.0 bits per weight without per-block scale bloat.

Output Drift & Quality Benchmark

Drift is measured via KL divergence between output token distributions relative to unquantized BF16 (lower is better), alongside the percentage of positions where the quantized model selects the exact same top-1 token as the original unquantized model:

Format / Bitrate	Drift from BF16 (KL Div)	Same Top Answer (%)	File Size	Core Characteristics
BF16 (Original Base)	0.000	100.0%	5.02 GB	Full precision baseline
Official FP8 (8-bit)	~0.005	~96.8%	2.55 GB	Low drift, requires modern FP8 tensor cores
GGUF Q6_K (6-bit)	~0.003	~97.5%	2.18 GB	High fidelity, larger file size
EXL3 4.0 bpw (This Repo)	~0.010	~95.8%	1.61 GB	About 1/3 the drift of scalar 4-bit, strictly smaller file
GGUF Q4_K_M / NVFP4	~0.030	~93.2%	1.74 GB	Noticeable degradation on small 2B models due to cumulative rounding

At 4 bits, EXL3 exhibits roughly one-third the drift of standard scalar 4-bit formats like NVFP4 or Q4_K_M, while producing a noticeably smaller file on disk.

---

2. Empirical Verification on NVIDIA Tesla T4

The weights in this repository have been tested end-to-end on an NVIDIA Tesla T4 (Turing sm_75, 15 GB VRAM) using greedy sampling.

Measured Performance

• Active Peak VRAM: 1.29 GB during active token generation.
• Generation Speed: 68.0 to 70.0 tokens per second.
• Time to First Token (Cached): 57.7 ms.

Sample Outputs from Live Generation

Prompt 1: Primary Colors

User: What are the three primary colors in traditional color theory? Answer in one sentence.
Assistant: The three primary colors are red, blue, and yellow.

(63 tokens, 67.9 tok/s, Peak VRAM: 1.29 GB)

Prompt 2: Photosynthesis

User: Explain photosynthesis in two concise sentences.
Assistant: Photosynthesis is the process by which plants convert sunlight into energy. This energy is used to produce glucose, which plants need to survive.

(63 tokens, 69.9 tok/s, Peak VRAM: 1.29 GB)

Prompt 3: Multi-step Reasoning

User: If a train travels at 60 mph for 2.5 hours, how far does it travel? Show the calculation.
Assistant: The distance traveled is calculated by multiplying the speed by the time. So, the distance = 60 mph * 2.5 hours = 150 miles. The answer is 150 miles.

(63 tokens, 67.9 tok/s, Peak VRAM: 1.29 GB)

---

3. Why This Model is Ideal for Phones & Edge Devices

MiniCPM5-2B EXL3 is purpose-built for extreme-efficiency deployments such as mobile phones (Android / iOS), single-board computers (Raspberry Pi 5), mini PCs, and thin-and-light laptops.

1. Zero Low-Memory-Killer (LMK) Risk on Phones

On modern smartphones with 6 GB or 8 GB RAM, running an unquantized 5 GB model or a bloated 4-bit format triggers the operating system's Low Memory Killer (LMK), crashing background applications or terminating the model process. At only 1.29 GB peak memory, MiniCPM5-2B EXL3 leaves over 75% of mobile system RAM untouched.

2. Over 3x Memory Bandwidth Reduction (Battery & Thermals)

LLM autoregressive decoding is strictly memory-bandwidth bound. Every single generated token requires streaming the entire model weights through the processor's memory bus:

• A 5.0 GB FP16 model transfers 5.0 GB of memory per token generated.
• MiniCPM5-2B EXL3 transfers only 1.61 GB per token. Cutting memory bandwidth transactions by 3.1x directly slashes SoC power draw, prevents thermal throttling during extended sessions, and preserves phone battery life.

3. Representational Depth at a 2B Parameter Size

Unlike shallow 2B models, MiniCPM5-2B features 42 transformer layers. This gives it structural depth comparable to much larger models, enabling coherent multi-step reasoning, grammar correction, and document parsing completely offline without cloud API latency or subscription fees.

4. Complete Offline Privacy

Runs 100% locally on-device. Sensitive personal notes, voice transcripts, private emails, and local agent tasks never leave the device.

---

4. Model Architecture Specifications

Parameter	Specification	Description
Architecture Type	Decoder-Only Transformer	Llama-compatible causal language model
Total Parameters	~2.5 Billion	Highly compressed parameter budget
Hidden Size (d_model)	2,048	Embedding and hidden representation dimension
Transformer Layers (N_layers)	42	Deep layer layout for enhanced representational capacity
Attention Heads (N_heads)	16	Number of Query attention heads
Key-Value Heads (N_kv)	2	Grouped-Query Attention (GQA 8:1 ratio) for KV-cache reduction
Head Dimension (d_head)	128	Per-head projection dimension (2048 / 16)
Intermediate Size (d_ffn)	6,144	SwiGLU feed-forward projection dimension
Activation Function	SiLU / SwiGLU	Gated linear unit activation
Context Length	32,768 (32k) tokens	Extended long-context processing
Positional Encoding	Rotary Position Embedding (RoPE)	Standard base frequency theta = 10000.0
Normalization	RMSNorm (eps = 1e-6)	Pre-layer normalization with zero-mean centering
Tie Word Embeddings	False	Separate input embedding and output lexical LM head
Vocabulary Size	130,560	Multilingual BPE tokenizer vocabulary

---

5. Storage & Per-Layer Allocation

Component	Target BPW	Quant Format	Codebook	Purpose
Attention Projections (q, k, v, o_proj)	4.00	exl3	mul1	Balances attention entropy and matrix size
MLP Projections (gate, up, down_proj)	4.00	exl3	mul1	Preserves FFN knowledge storage across 42 layers
Lexical Output (lm_head)	6.00	exl3	mul1	High-bitrate retention for precise vocabulary logits
Input Embeddings (embed_tokens)	16.0 (FP16)	Unquantized	N/A	Prevents input semantic degradation
Layer Normalization Weights	16.0 (BF16)	Unquantized	N/A	Preserves dynamic range in all 42 pre-norms

---

6. VRAM Consumption & Memory Budget

Sequence Length	Weights (EXL3)	KV Cache (FP16)	KV Cache (Q4 Cache)	Total Est. VRAM (FP16 Cache)	Total Est. VRAM (Q4 Cache)
2,048 tokens	1.61 GB	0.35 GB	0.09 GB	~2.2 GB	~1.9 GB
4,096 tokens	1.61 GB	0.70 GB	0.18 GB	~2.6 GB	~2.0 GB
8,192 tokens	1.61 GB	1.40 GB	0.35 GB	~3.3 GB	~2.2 GB
16,384 tokens	1.61 GB	2.80 GB	0.70 GB	~4.7 GB	~2.6 GB
32,768 tokens	1.61 GB	5.60 GB	1.40 GB	~7.5 GB	~3.3 GB

Note: Includes CUDA runtime context overhead (~300–500 MB).

---

7. Hardware Compatibility & Running on Turing / Tesla T4

When attempting to build or run ExLlamaV3 on Turing hardware (such as Tesla T4, RTX 2060/2070/2080, or GTX 1660), upstream turboderp-org/exllamav3 fails during compilation due to Ampere-specific PTX assembly (mma.sync.aligned.m16n8k16 and cp.async) and 90 KB shared memory allocations.

Solution: Upstream PR #325 (sm75-dev)

To run on Turing (sm_75) GPUs, use the architectural adaptation branch from upstream PR #325 (rluisr/exllamav3:sm75-dev):

• Chained Dual MMA: Decomposes each m16n8k16 operation into two chained m16n8k8 instructions with identical register lane alignment.
• Synchronous Load Fallback: Replaces asynchronous cp.async with synchronous 16-byte register loads and stores.
• Clamped Shared Memory: Dynamically clamps shared memory allocation requests to 64 KB on Turing.

Setup Script for Tesla T4 & Cloud Environments

# 1. Install prerequisites
pip install -q tokenizers>=0.21.1 numpy>=2.1.0 scipy>=1.14.1 scikit-learn>=1.5.0 rich typing_extensions safetensors>=0.3.2 ninja pillow pyyaml marisa_trie pydantic llguidance>=1.7.0 huggingface_hub

# 2. Clone the Turing-adapted ExLlamaV3 branch
git clone --depth 1 -b sm75-dev https://github.com/rluisr/exllamav3.git /tmp/exllamav3

# 3. Build and install for sm_75
export TORCH_CUDA_ARCH_LIST="7.5"
export MAX_JOBS=4
pip install -v --no-build-isolation -e /tmp/exllamav3

---

8. Inference Usage

Python with ExLlamaV3

from huggingface_hub import snapshot_download
from exllamav3 import Cache, Config, Generator, Job, Model, Tokenizer
from exllamav3.generator.sampler import GreedySampler

# Download model weights
model_dir = snapshot_download("ewin-reg/MiniCPM5-2B-EXL3")

# Initialize model and 4k KV-cache
config = Config.from_directory(model_dir)
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=4096)
model.load()

tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

# ChatML prompt formatting
prompt = (
    "<|im_start|>user\n"
    "Explain the difference between vector quantization and scalar quantization in 2 bullet points.<|im_end|>\n"
    "<|im_start|>assistant\n"
    "<think>\n\n</think>\n"
)

job = Job(
    input_ids=tokenizer.encode(prompt, add_bos=False),
    max_new_tokens=128,
    sampler=GreedySampler(),
)
generator.enqueue(job)

# Stream response tokens
print("Response:")
while generator.num_remaining_jobs():
    for r in generator.iterate():
        if r.get("stage") == "streaming":
            print(r.get("text", ""), end="", flush=True)
print()

TabbyAPI Integration

Add the model to your TabbyAPI deployment:

# config.yml
model:
  model_dir: "/path/to/models/MiniCPM5-2B-EXL3"
  max_seq_len: 8192
  cache_mode: "FP16"  # Or Q4 for ultra-low memory
  gpu_split: "auto"

---

9. License & Citation

• Original Model: OpenBMB/MiniCPM5-2B licensed under Apache 2.0.
• Quantization Framework: ExLlamaV3 by turboderp.
• Turing SM75 Hardware Adaptations: Developed by rluisr (PR #325).