ozhyhinas/MiniCPM5-2B-q4f16_1-MLC

MiniCPM5-2B, q4f16_1, for MLC LLM and WebLLM

openbmb/MiniCPM5-2B quantized to q4f16_1 with MLC LLM, plus a WebGPU model library compiled against the TVM runtime that ships in WebLLM 0.2.84. Drop-in for WebLLM: point an appConfig entry at this repo and the model runs in the browser. Text only; the base model has no vision tower.

Built by Oleh Zhyhinas for LatexGen, which converts spoken math and PDF pastes to LaTeX on-device. On its 105-item text benchmark MiniCPM5 2B scored 52% judge-correct against 26% for Qwen3.5 2B and 13% for Qwen3 1.7B at the same size, with the biggest gains on prose passages that mix words and formulas.

Use with WebLLM

import * as webllm from "@mlc-ai/web-llm"; // 0.2.84

const appConfig = {
  model_list: [{
    model: "https://huggingface.co/ozhyhinas/MiniCPM5-2B-q4f16_1-MLC",
    model_id: "MiniCPM5-2B-q4f16_1-MLC",
    model_lib: "https://huggingface.co/ozhyhinas/MiniCPM5-2B-q4f16_1-MLC/resolve/main/libs/MiniCPM5-2B-q4f16_1-MLC-webgpu.wasm",
    vram_required_MB: 1900,
    low_resource_required: true,
    required_features: ["shader-f16"],
    overrides: { context_window_size: 4096 },
  }],
};
const engine = await webllm.CreateMLCEngine("MiniCPM5-2B-q4f16_1-MLC", { appConfig });
const reply = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Write x squared plus one in LaTeX." }],
  temperature: 0,
  extra_body: { enable_thinking: false }, // MiniCPM5 is a hybrid thinking model
});

enable_thinking: false makes WebLLM prefix the answer with an empty <think>\n\n</think> block, which is how the official chat template turns reasoning off. Leave it out to let the model think first.

What is in the repo

File	Purpose
params_shard_*.bin, tensor-cache.json	q4f16_1 weights, 42 shards, 1.42 GB, 4.5 bits per parameter
mlc-chat-config.json	MLC config: model_type: llama, context window 4096, prefill chunk 1024, ChatML conversation template with MiniCPM5's stop tokens (</s> = 1, `<
tokenizer.json, tokenizer_config.json	copied from the base model
libs/MiniCPM5-2B-q4f16_1-MLC-webgpu.wasm	WebGPU model library for WebLLM 0.2.84

How it was built

MiniCPM5-2B is a plain LlamaForCausalLM (42 layers, hidden 2048, 16 attention heads, 2 KV heads, vocab 130560, untied embeddings, rope theta 5e6), so MLC's stock llama definition handles it without changes. The conversation template is MLC's qwen3 ChatML template with the stop tokens swapped, which matches the base model's chat_template.jinja (including the empty think block for non-thinking mode and stripping of <think> blocks from history).

mlc_llm gen_config openbmb/MiniCPM5-2B --quantization q4f16_1 --model-type llama \
  --conv-template qwen3 --context-window-size 4096 --prefill-chunk-size 1024 -o out/
# then edit out/mlc-chat-config.json: stop_token_ids [1, 130073], stop_str ["</s>", "<|im_end|>"]
mlc_llm convert_weight openbmb/MiniCPM5-2B --quantization q4f16_1 -o out/
mlc_llm compile out/mlc-chat-config.json --device webgpu -o out/libs/MiniCPM5-2B-q4f16_1-MLC-webgpu.wasm

Compiler: mlc-llm at commit ed1c7f65 with its bundled TVM (c04f730a, v0.20 line), the same pair WebLLM 0.2.84 was built from, with Emscripten 3.1.56. Compiling against a newer MLC produces a library the 0.2.84 runtime rejects. One TVM patch was needed: the dlight GPU fallback rule keeps BlockRV/LoopRV handles across several decompose_reduction calls in one PrimFunc, and the first rewrite invalidates the rest (ValueError: The block no longer exists in the IRModule). The fix records block and loop names and reacquires the handles with sch.get_sblock(name) before each decomposition; the compiled kernels are unchanged.

SHA-256 of the model library: see release-manifest.json.

License

Apache 2.0, inherited from the base model. The quantized weights are a derivative of openbmb/MiniCPM5-2B.

Quantized, compiled and published by Oleh Zhyhinas (GitHub, LinkedIn). Questions and issues: open a discussion on this repo.