oliver0804/WeVisOne-VL-2B-RK3588-rkllm

WeVisOne-VL-2B — RKLLM + vision RKNN for the RK3588 NPU

tencent/WeVisOne-VL-2B (Qwen3-VL architecture) converted for the RK3588 / RK3588S NPU:

file	runs on	format
WeVisOne-VL-2B_RK3588_w8a8.rkllm	NPU via librkllmrt 1.3.1	LLM, w8a8, rkllm-toolkit 1.3.1
WeVisOne-VL-2B_vision_RK3588.rknn	NPU via librknnrt 2.5.0	vision encoder, fp16

Weights are modified (converted); the base model is Apache-2.0.

Two-model pipeline

This is not a single file: the text model runs through the RKLLM runtime, the vision encoder through the RKNN runtime. The glue (not included here as a ready-made script) must:

1. run the vision .rknn on the image (fixed 448×448 preprocessing),
2. the encoder returns, per vision token, an embedding plus 3 deepstack feature maps — concatenate them along the feature dimension (Qwen3-VL style, [n_tokens, 4·hidden] fp32),
3. build the prompt with <|vision_start|> + one <|image_pad|> per vision token + <|vision_end|>,
4. inject the float32 embedding block through librkllmrt's image-embed API (RKLLMInput/image_embed, n_image_tokens), which the 1.3.1 runtime supports.

The conversion scripts are included under conversion/ for reproducibility (vision ONNX export, vision RKNN build, LLM w8a8 conversion).

Benchmark (LLM part on ROCK 5C, clocks locked)

generate	memory (LLM + vision)
14.1 tok/s	~4.3 GB

Strength: object detection / grounding boxes on natural images are the most accurate of the models tested on this board (better than Qwen3.5-2B). Weakness: GUI-element grounding on screenshots is weak (13 % accuracy with two zoom iterations vs 53 % for Qwen3.5-2B on a 32-sample ScreenSpot-v2 subset) — it was not trained on that task format.

License

Apache-2.0, inherited from the base model (see LICENSE).