dstefl/EuroLLM-9B-Instruct-2512-q4f16_1-MLC

EuroLLM-9B-Instruct-2512 · q4f16_1 · MLC/WebGPU

utter-project/EuroLLM-9B-Instruct-2512 converted to the MLC format so it runs fully client-side in a browser via web-llm. No such build existed publicly; this repo is the conversion, not a fine-tune — the weights are the original model's, quantized to q4f16_1 (~4.9 GB). Note this is the open 2512 revision; the original EuroLLM-9B-Instruct repo is gated and is not what you get here.

What is in this repo

• Quantized weight shards + tokenizer + mlc-chat-config.json (context_window_size: 32768; conversation template chatml, matching the base model's <|im_start|> chat template).
• EuroLLM-9B-Instruct-2512-q4f16_1-MLC-webgpu.wasm — the model library compiled for WebGPU against the exact TVM generation web-llm 0.2.84 ships its runtime for (mlc-llm 2008fe8, 2026-05-11). A library built against a newer TVM fails at load with LinkError: TVMFFIGetCustomAllocator — that pin is the hard-won part.

Measured at compile time (mlc_llm's own report): 5 998 MB without KV cache, 0.16 MB per KV token — ≈ 6.7 GB GPU memory at a 4 096-token window. Plan for an 8 GB+ card; shader-f16 WebGPU support is required.

How to use (web-llm)

import { CreateMLCEngine } from "@mlc-ai/web-llm";

const engine = await CreateMLCEngine("EuroLLM-9B-Instruct-2512-q4f16_1-MLC", {
  appConfig: {
    model_list: [
      {
        model: "https://huggingface.co/dstefl/EuroLLM-9B-Instruct-2512-q4f16_1-MLC",
        model_id: "EuroLLM-9B-Instruct-2512-q4f16_1-MLC",
        model_lib:
          "https://huggingface.co/dstefl/EuroLLM-9B-Instruct-2512-q4f16_1-MLC/resolve/main/EuroLLM-9B-Instruct-2512-q4f16_1-MLC-webgpu.wasm",
        overrides: { context_window_size: 4096 },
      },
    ],
  },
});

Grammar-constrained JSON (XGrammar) works — it is the same runtime as every prebuilt web-llm model. The overrides above trades the 32k window for the 6.7 GB memory figure; raise it only with the VRAM to back it (0.16 MB × tokens).

How it was built

TVM + mlc_llm compiled from source — this time on owned hardware (a 6-vCPU VPS, overnight in tmux), not CI: same pinned commit 2008fe8, LLVM 18, uv-managed CPython 3.12, 3rdparty/tvm submodule, editable --no-deps installs, emsdk + web/prep_emcc_deps.sh, then convert_weight → gen_config → compile --device webgpu. The reproducible script lives in dstefl/householdsim (private) as scripts/eurollm-mlc-build-source.sh.

Honest quality note

Verified on real hardware (RTX 3090, web-llm 0.2.84) before publication: weights load, shaders compile, generation works, ~10–15 s per short structured-JSON completion. On our own 20-sentence Czech smart-home command corpus it scored 11/20 in pure-model mode where the 1.7B conversion scored 1/17 — a real capability jump, measured, not benchmarked. The 2512 revision's card publishes no benchmark table, so no external figure is quoted here; the EuroLLM paper (arXiv 2409.16235) covers the original 9B revision, not this one.

License & attribution

Base model © UTTER project, released under Apache 2.0; this quantized conversion is distributed under the same license. Not affiliated with the UTTER project. Converted 2026-09-01 for wattburg.com (a browser household-electricity simulator that runs its AI fully client-side).