MiniCPM5-2B-onnx

ONNX export of openbmb/MiniCPM5-2B for onnxruntime-web / WebGPU.

Two graphs per precision under onnx/:

• fp16/prefill/model.onnx — prompt tokens -> logits + present KV
• fp16/decode/model.onnx — token + past KV -> logits + updated KV
• int4/prefill/model.onnx / int4/decode/model.onnx — weight-only 4-bit (block 64, symmetric) quantization of the same graphs (~quarter memory)

Weights are ONNX external-data shards sitting next to each model.onnx (the graphs exceed the 2 GB protobuf limit). Keep each graph's directory together when serving.

Architecture

Vanilla Llama: 42 layers, hidden 2048, GQA 16 query heads / 2 KV heads, head_dim 128, vocab 130560, untied embeddings, RoPE theta 5e6 (full rotary), max context 131072.

Inputs

• input_ids [B, T] int64
• position_ids [B, T] int64 (absolute positions; prefill: 0..T-1; decode: past_len)
• attn_mask [B, T, total] additive float (0 keep / -inf mask), pure causal
• past_k_{i} / past_v_{i} [B, 2, past_len, 128] — zero-length for prefill (note: the prefill graphs declare only input_ids, position_ids, attn_mask — unused zero-length past inputs are pruned at export)

Outputs

• logits [B, T, 130560]
• present_k_{i} / present_v_{i} [B, 2, total_len, 128]

Loop: prefill once, then feed present_* back as past_* each decode step, slicing position_ids and attn_mask accordingly. Greedy sampling host-side from logits.

Chat template

The chat template ships only as a separate chat_template.jinja file at the repo root — it is NOT embedded in tokenizer_config.json (newer Hub convention). If your runtime reads the template from tokenizer_config.json and finds none, load chat_template.jinja explicitly. The model is a ChatML-style assistant (<|im_start|> / <|im_end|>, with an enable_thinking flag controlling the <think> / </think> reasoning block; /think and /no_think soft switches are special tokens 130080/130081). Plain-completion prompts also work, but expect better quality with the chat format. The uploaded file is byte-identical to the upstream template in openbmb/MiniCPM5-2B.

Output order note

present_* outputs are interleaved per layer: the trace order is logits, present_k_0, present_v_0, present_k_1, present_v_1, ... — index them as outputs[1 + 2*i] (k) and outputs[2 + 2*i] (v) when consuming by position.

WebGPU: onnxruntime-web webgpu EP. fp16 graphs ~5 GB each; int4 graphs ~1.5 GB each — use int4 for consumer GPUs.