malachkubihd45/MiniCPM5-1B-mixed4-8

MiniCPM5-1B → LiteRT-LM (mixed 4/8-bit)

MiniCPM5-1B-mixed4to8.litertlm — mixed-precision quantized, ready for the LiteRT-LM runtime (CPU/GPU, Android/iOS/desktop/web).

Result

Artifact	Size
Source BF16 safetensors	2.16 GB
fp32 .litertlm (intermediate)	4.33 GB
Final mixed-precision .litertlm	0.94 GB

Precision map (weights; activations float, dynamically quantized at runtime)

Tensor	Precision	Scheme
lm_head / unembedding (decode_logits_output, 130560×1536)	8-bit	per-channel symmetric
gate_proj (all 24 layers) — most quantization-sensitive FFN weight	8-bit	per-channel symmetric
External embedder table (130560×1536)	8-bit	per-channel symmetric
q/k/v/o_proj, down_proj (all layers)	4-bit	blockwise-64 (group size 64)
Norms / rotary embeddings / activations	fp32	untouched

Constraints honored: nothing below 4-bit, nothing above 8-bit. Expected average ≈ 4.3 bits/weight for the transformer body.

Files produced

• MiniCPM5-1B-mixed4to8.litertlm — final container (LlmMetadata + HF tokenizer (zlib) + quantized prefill/decode TFLite + quantized external embedder). Verified with litert-lm-peek and smoke-tested with the LiteRT-LM runtime.
• mixed_4to8_recipe.json — per-model_type mapping recipe (container-level form).
• recipe_prefill_decode.json / recipe_embedder.json — bare-tflite recipes used for the mmap-based low-RAM quantization path.
• quantize_and_pack.py — quantizes out_fp32/*.tflite and rebuilds the .litertlm by swapping only the TFLite sections (metadata preserved byte-exact).
• out_fp32/ — fp32 export (contains embedder.tflite, llm_metadata.pb, tokenizer, chat template; the 3.5GB fp32 model.tflite can be deleted).
• out_quant/ — quantized tflite sections (model_quant.tflite, embedder_quant.tflite).

Toolchain (pinned, known-good for LlamaForCausalLM)

Python 3.12 · torch 2.12.1 (CPU) · litert-torch 0.9.1 · ai-edge-quantizer 0.7.0 · ai-edge-litert 2.1.5 · transformers 5.9.0 · litert-lm / litert-lm-builder 0.12.0

uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python torch --index-url https://download.pytorch.org/whl/cpu
uv pip install --python .venv/bin/python "litert-torch==0.9.1" "ai-edge-quantizer==0.7.0" \
    "ai-edge-litert==2.1.5" "transformers==5.9.0" "litert-lm==0.12.0" litert-lm-builder sentencepiece

Reproduce

# 1) Float32 export (no default quantization), embedder externalized, 4k cache.
#    lightweight conversion keeps peak RAM low on CPU-only machines.
.venv/bin/litert-torch export_hf . ./out_fp32 \
  --quantization_recipe="" \
  --externalize_embedder=True \
  --prefill_lengths=128,1024 \
  --cache_length=4096 \
  --experimental_lightweight_conversion=True

# 2) Mixed-precision quantize + container rebuild (mmap-backed, ~3GB peak RAM)
.venv/bin/python quantize_and_pack.py

Run

uv tool install litert-lm        # or use .venv/bin/litert-lm
litert-lm run MiniCPM5-1B-mixed4to8.litertlm \
  --backend=cpu \
  --prompt="What is 2+2? Answer with just the number."

Smoke tests on host CPU passed (model outputs its <think>…</think> reasoning block then the answer; correct on both probes).

Notes:

• This box is headless: the litert-lm CLI needs libEGL.so.1/libGLESv2.so.2. If missing, extract the libegl1, libegl-mesa0, libgles2, libgbm1 debs into a local dir and export LD_LIBRARY_PATH=<dir>.
• To change which tensors are 8-bit, edit the regexes in recipe_prefill_decode.json (decode_logits_output, .*gate_proj.*) and re-run step 2. Rules later in the list win (AEQ applies last-match priority).