Instructions to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
# Run inference directly in the terminal:
llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
# Run inference directly in the terminal:
llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
# Run inference directly in the terminal:
./build/bin/llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828

Use Docker

docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828

Llama new
LM Studio
Jan

vLLM

How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828

Ollama
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Ollama:
```
ollama run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
```
Unsloth Desktop

How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Docker Model Runner:
```
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
```

Lemonade

How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828

Run and chat with the model

lemonade run user.MiniCPM-V-4.7-1B-RKNN3-RK1828-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

MiniCPM-V-4.7-1B-RKNN3-RK1828

MiniCPM-V 4.7 1B converted to RKNN3 and running end-to-end on the Rockchip RK1828 NPU — both the text tower and the vision tower, with real image questions answered on device.

⚠️ Read this first. The vision tower in this repository contains a runtime-contract adapter: MiniCPM-V 4.7 natively emits 8×8 = 64 visual tokens per 448×448 image, but the RKNN3 multimodal callback requires image_ph * image_pw = 14×14 = **196** tokens. A nearest-neighbour 8×8 → 14×14 resample was inserted into the exported graph to satisfy that contract. This build is therefore NOT equivalent to the Hugging Face original MiniCPM-V 4.7 1B. Do not use it to benchmark or reproduce the original model's accuracy. See How it was made.

This repository is not just six binary files. It includes the launch script, request examples, conversion scripts and a deployment guide, so that the model can actually be run and reproduced.


Base model	`openbmb/MiniCPM-V-4.7-1B` (Apache-2.0)
Hardware	RK1828 / RK1820 NPU (tested `RM1828MC0-F`, firmware 1.1.0)
Host	Windows PC or RK3588 / RK3576 board; Ubuntu x86 cannot drive the module directly
Runtime	`rknn3_110` (`rkllm3-server`), versions must match toolkit / proxy / firmware
Quantization	`w4a16`, `normal`, `group32`, 8 cores; head `w6a16`
NPU footprint	1269 MB (exclusive differential measurement)

What's in the repository

Path	Contents
`rknn/`	6 conversion artifacts + `start.cmd` (1.36 GiB)
`host/`	`send_vlm_req.cmd`, two ready-made request bodies, pure-stdlib test-image generator
`docs/DEPLOY.md`	Deployment, API contract, 10 pitfalls actually hit, verified results
`convert/`	Portable copies of the two vision-export scripts + the conversion recipe
`SHA256SUMS`	Checksums of every file above

Quick start

<root>\
  rknn\                     <- this repo
  runtime\rknn3_110\bin\    <- rkllm3-server.exe
  runtime\rknn3_110\lib\    <- DLLs

cd /d <root>\rknn
start.cmd                          :: defaults 0.0.0.0:8899 (RK_HOST / RK_PORT override)
cd /d <root>\host
send_vlm_req.cmd orange            :: HTTP=200, response in resp_orange.json

The server is ready only when a real inference returns — the weight-sync log can sit still for ~2 minutes. Do not poll rknn-smi during loading.

Full details (including the Session-1 requirement, proxy reuse and DLL/PATH rules) are in docs/DEPLOY.md.

API

OpenAI-compatible POST /v1/chat/completions. content must be an array; the server constructs the image placeholder tokens itself:

curl -s http://127.0.0.1:8899/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":[
        {"type":"text","text":"图里是什么？"},
        {"type":"image_url","image_url":{"url":"data:image/png;base64,<BASE64>"}}
      ]}],"max_tokens":512}'

Do not put <image> / <|image_pad|> into the text. Use curl.exe --max-time; PowerShell Invoke-WebRequest hangs on this server.

A correct response has timings.prompt_n ≈ 218 = 196 visual tokens + text tokens — that is the proof the vision embeddings really entered the LLM.

Verified on device

Check	Result
Orange image → colour	`这张图是橙色。`
Black square → shape/colour	`黑色正方形`
Cross-checks (orange asked for shape; square asked for colour)	correct — not language-prior guessing
Pure text (capital of China)	`北京`
Chinese ASR + this VLM coexisting	3458 MB (0.6B ASR) / 3353 MB (1.7B ASR), all requests HTTP 200
Throughput	~90 tok/s generation, ~1700 tok/s prefill

finish_reason=stop on simple cases. Complex images may return length — always check finish_reason, not just HTTP 200.

How it was made

Text tower — MiniCPM-V 4.7 reuses the Qwen3.5-0.8B architecture with vocab_size=248094 and an independent lm_head. Extraction + zoo Qwen3_5 export_llm.py / export_rknn.py. The top-level tie_word_embeddings must be set to false too — otherwise HF tie_weights() silently binds the 254M-parameter head to the embedding and the model emits word salad.
Vision tower — vision_tower + merger exported to ONNX with transformers>=5.18 (MiniCPMV4_7ForConditionalGeneration), then RKNN with uint8 NHWC input and mean/std = 0.5*255. Output is 3D [1, N, 1024]; a 2D [N,1024] export is mis-parsed as "N images of 1024 tokens".
196-token adapter — nearest-neighbour resample 8×8 → 14×14 inside the graph (Gather/Reshape only, no Resize op). This changes graph structure only: the .weight SHA256 is byte-identical to the unadapted 64-token build, because no weights are touched.

See convert/README.md for the recipe and docs/DEPLOY.md §5 for the pitfalls.

Limitations

Not equivalent to the original 4.7 (196-token adapter, see the warning above).
canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense multi-object counting were not verified.
OCR is "characters readable" at best (RK1828 → RK₁₈₂₈).
Only 448×448 single-image input was tested.
1B model with a <think> chain: raising max_tokens is not monotonically safer (256 worked on one OCR case where 512 stalled inside <think>).
Known upstream issue: on ASR device error, NPU memory can leak (mem_sync ret=-10 retry loop); workaround is to kill the ASR process.

Checksums

sha256sum -c SHA256SUMS

File	SHA256
`MiniCPM-V-4.7-1B-llm.rknn`	`948e487bff9b2324b3016c665aa071feff05ac735e078c20ecb4a0ac7b825a1b`
`MiniCPM-V-4.7-1B-llm.weight`	`72fd0b16b8016d63a7d8b92f8d159ad7ea68a783eaca89b915bd0e240ae22dab`
`MiniCPM-V-4.7-1B-llm.embed.bin`	`511c5794dbf3d927fb17c5efc0052fababafdd35185be2961f6a855c1852b335`
`MiniCPM-V-4.7-1B-llm.tokenizer.gguf`	`e028b0a11710ee5e0402c3829e871c3bd5c568540fe377febdd1a515d45b1544`
`MiniCPM-V-4.7-1B-vision.rknn`	`9e081464857fad825281ae8778defb882424cbf5d46a37acb6de59dd5cbc99d6`
`MiniCPM-V-4.7-1B-vision.weight`	`9c1939dd21e511c72ef6fa6333d43d962acc067d5302ed625815abeaf1cc1c03`

License & attribution

Apache-2.0, inherited from openbmb/MiniCPM-V-4.7-1B. Please credit OpenBMB for the original model. The RKNN3 conversion, the 196-token adapter and the host scripts in this repository are provided under the same license, without warranty.

Built with airockchip/rknn3-model-zoo V1.1.0.

中文摘要

把 openbmb/MiniCPM-V-4.7-1B 转成 RKNN3，在 RK1828 上跑通完整图文链路（文本塔 + 视觉塔 + rkllm3-server --model2），真机 5/5 带图与纯文本请求全部 HTTP 200，NPU 独占占用 1269 MB。

必须知道的取舍：4.7 视觉塔原生只出 8×8=64 个视觉 token，而 RKNN3 runtime 的 multimodal 回调要求 image_ph * image_pw = 14×14 = 196。为满足契约，导出图里加了 8×8→14×14 的最近邻重采样适配层（只改图结构、不改权重， .weight 的 SHA256 与 64-token 版完全相同）。因此 本产物不等价于 HF 原版 4.7，不能拿它复现原版精度； canvas mRoPE / 多分辨率切砖 / 高分辨 OCR / 密集计数均未验证。

项	结果
橙图问颜色	`这张图是橙色。`
黑方块	`黑色正方形`
交叉对照（排除语言先验猜答案）	正确
纯文本	`北京`
与 ASR 共存	0.6B 3458 MB / 1.7B 3353 MB（A/B/A 可复现）
视觉 token 注入证据	`timings.prompt_n ≈ 218`

仓库含 6 个转换产物 + start.cmd + 请求脚本 + 转换脚本 + 部署文档，拿到即可跑；部署与十个实测踩过的坑见 docs/DEPLOY.md。复杂图务必看 finish_reason，别只看 HTTP 200。

Downloads last month: 73

GGUF

Model size

0 params

Architecture

qwen35

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828

Base model

openbmb/MiniCPM-V-4.7-1B

Quantized

(55)

this model