fatih-can/MiniCPM5-2B-MLC

MiniCPM5-2B for WebLLM

openbmb/MiniCPM5-2B compiled for WebGPU with MLC-LLM, in two quantization tiers plus runtime-selectable context windows. All artifacts were compiled from source with a pinned 0.20-line toolchain, so each wasm loads in WebLLM 0.2.84 / 0.2.85.

This repo is a mono-repo: every variant lives in its own subfolder, in the standard WebLLM layout (mlc-chat-config.json, tokenizer*.json, params_shard_*.bin, tensor-cache.json, libs/<name>.wasm). Just point a WebLLM model entry at the subfolder URL and model_lib at the wasm.

Variants

Subfolder	Quantization	Params	Bits/param	Role
q4f16_1/	q4f16_1	1.4 GB	~4.5	Default
q4f16_autoawq/	q4f16_autoawq	2.1 GB	~4.5, group 128, AWQ	Quality

Every wasm is compiled with a conv_template matching MiniCPM5's ChatML template (stop tokens </s> = 1, <|im_end|> = 130073, strip_reasoning_in_history). The model is a hybrid thinking model: pass extra_body: { enable_thinking } to toggle its thinking block.

Removed: q3f16_1

The q3f16_1/ subfolder was dropped. On WebGPU it produces token soup on every device we tested, in the same browser/GPU where q4f16_1 answers coherently and deterministically.

• every params_shard_*.bin matches the md5 recorded in its own tensor-cache.json (33/33), and the shard sizes sum exactly to the declared ParamBytes;
• tokenizer.json / tokenizer_config.json are byte-identical to the base model and to the working q4f16_1 folder, and the two mlc-chat-config.json files differ only in the "quantization" string;
• dequantizing layer 0's down_proj with the exact q3f16_1 packing (10 × 3-bit per uint32, (q - 3) * scale, group_size = 40) correlates 0.97 with the real tensor from openbmb/MiniCPM5-2B (q4: 0.995), with uniform error and no structural spikes.

Diffing the two compiled wasms kernel-by-kernel, 84 of 89 shader functions are byte-identical and only the quantized-matmul kernels differ — so the breakage is confined to MLC's int3 path. Upstream has acknowledged it: in mlc-llm#2700 a q3f16_1 user hits a fused_dequantize_take1 assertion and the answer is "We will look into the potential issue of q3f16_1 ... would you mind trying the 4-bit quantization q4f16_1?", and web-llm#630 ("is q3f16 supported in web-llm?") is still open — which is why WebLLM ships no q3f16 models.

Use q4f16_1 instead.

Context windows

Each wasm is compiled for a 32768-token window. At runtime you can shrink it with overrides.context_window_size (4k / 8k / 16k / 32k) — so one wasm per quant covers every context preset, with no extra downloads.

Use with WebLLM 0.2.85

import * as webllm from "@mlc-ai/web-llm"; // 0.2.84 / 0.2.85

const appConfig = {
  model_list: [
    {
      model: "https://huggingface.co/fatih-can/MiniCPM5-2B-MLC/resolve/main/q4f16_1",
      model_id: "MiniCPM5-2B-q4f16_1-MLC",
      model_lib: "https://huggingface.co/fatih-can/MiniCPM5-2B-MLC/resolve/main/q4f16_1/libs/MiniCPM5-2B-q4f16_1-MLC-webgpu.wasm",
      vram_required_MB: 2000,
      overrides: { context_window_size: 32768 },
    },
    // Same pattern for q4f16_autoawq with its own model_id / wasm.
  ],
};
const engine = await webllm.CreateMLCEngine("MiniCPM5-2B-q4f16_1-MLC", { appConfig });
const reply = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Write x squared plus one in LaTeX." }],
  temperature: 1.0,
  top_p: 0.95,
  // extra_body: { enable_thinking: false }, // to turn reasoning off
});

How it was built

Built on a Linux box with a from-source MLC-LLM 0.20.0 toolchain:

• mlc-llm v0.20.0, TVM at the pinned 3rdparty/tvm (b628d91f), emsdk 3.1.56, LLVM 18.

• One TVM patch to tvm/s_tir/dlight/gpu/fallback.py: record block/loop names and re-acquire handles via sch.get_sblock(name) before each decompose_reduction (fixes "The block no longer exists in the IRModule" when a PrimFunc has multiple reduction blocks).

• Per quant, the layout input is built with:

  mlc_llm gen_config openbmb/MiniCPM5-2B --quantization {QUANT} --model-type llama \
    --conv-template qwen3 --context-window-size 32768 --prefill-chunk-size 1024 -o out/
  # then: stop_token_ids [1, 130073], stop_str ["</s>", "<|im_end|>"]
  mlc_llm convert_weight openbmb/MiniCPM5-2B --quantization {QUANT} -o out/params
  mlc_llm compile out/mlc-chat-config.json --device webgpu \
    -o "out/libs/MiniCPM5-2B-{QUANT}-webgpu.wasm"
  

The underlying weights are the base model's; only the quantized byte layout and the compiled WebGPU kernel differ per subfolder. SHA-256 for every file is tracked by Hugging Face on each commit.