MiniCPM-V-4 · LiteRT INT8 (on-device, Snapdragon 8850 CPU)

MiniCPM-V-4-int8.litertlm — an on-device LiteRT-LM build of MiniCPM-V-4, quantized to INT8 (weight-only, GPTQ + Hadamard rotation) and packaged as a single .litertlm bundle for CPU inference on Snapdragon 8850.

Model

For the base model, architecture, capabilities and license, refer to the original model card:

https://huggingface.co/openbmb/MiniCPM-V-4

This repository only provides an edge-optimized LiteRT deployment of that model:

• Format: .litertlm (LiteRT-LM bundle: tokenizer + LLM prefill/decode + embedder + navit SigLIP vision encoder + resampler).
• Quantization: INT8 weight-only, per-channel, GPTQ + Hadamard rotation (activation-outlier smoothing).
• Vision: official multi-slice preprocessing (thumbnail + sub-tiles, 64 tokens per slice, up to 9 slices), navit SigLIP.

Performance (Snapdragon 8850, CPU backend)

Measured on-device (arm64 CPU, --backend=cpu), single image, 4-slice input:

Metric	Value
Prefill speed	40.9 tokens/sec
Decode speed	16.6 tokens/sec
Vision encode (4 slices)	~2.04 s
Time to first token	~7.0 s (284-token prefill incl. vision)
Init (model load + compile)	~2.2 s

Accuracy — MME benchmark (on-device, 8850 CPU)

Full MME evaluation running the INT8 bundle on-device:

Group	Score
Perception	1568
Cognition	491
Total	2059

Usage

Run with a LiteRT-LM CPU runner:

LD_LIBRARY_PATH=. ./litert_lm_main \
  --model_path=./MiniCPM-V-4-int8.litertlm \
  --backend=cpu \
  --image_path=./image.jpg \
  --input_prompt="What is in this image?"

License

Follows the license of the base model — see https://huggingface.co/openbmb/MiniCPM-V-4.