Qwen3-30B-A3B LiteRT-LM Models

This repository is a conversion of Qwen/Qwen3-30B-A3B to LiteRT-LM, written with the LiteRT Tensor API (LiteRT dcb657e8 with a small patch in the conversion scripts) and packed with litert-lm-builder 0.19.0, for a LiteRT-LM built from source with two open LiteRT pull requests: on an Apple M4 Max (128 GB) with WebGPU, the 2,048-token file decodes 54.8 tokens/s and prefills 1,043 tokens/s, and the 4,096-token file 47.6 and 778 tokens/s (2026-11-03).

Qwen/Qwen3-30B-A3B is Qwen's mixture-of-experts model: 48 layers with 128 experts each, 8 of them used for each token, 30.5B parameters of which 3.3B are activated (source model card). By default its chat template turns thinking on, so each reply starts with a thought.

LiteRT-LM 0.19.0, the latest release on 2026-11-03, does not run these files. Each layer's experts are one moe custom op with SiLU, and the moe kernels of LiteRT-LM 0.19.0 accept only GELU: from PyPI, on the CPU it stops with moe node #85 unsupported activation='silu', and on the GPU with moe only supports activation='gelu_tanh' (or its legacy spelling 'gelu'). SiLU in the moe op is in two LiteRT pull requests, open on that day: #11192 (the GPU kernel) and #11188 (the XNNPACK kernel, CPU). The build that ran every number on this page is under Build the runtime.

Available Artifacts

File	Quantization	Context	Size
qwen3_30b_a3b_mixed_int4_ekv2048.litertlm	experts int4 blockwise (block 32, fp32 scales); attention q / k / v / o and lm_head int8 per channel; router, norms and token table fp32; fp32 KV cache	2048	20.65 GB
qwen3_30b_a3b_mixed_int4_ekv4096.litertlm	the same	4096	20.65 GB

The two files hold the same weights (the files differ in 2,892 bytes); the KV cache holds 2,048 or 4,096 tokens. Both declare fp32 activations.

Conversion Notes

• The decoder is written with the LiteRT Tensor API one tensor at a time, from the bf16 checkpoint at revision ad44e777. The converter path would export the whole model in fp32 first; the writer never holds an fp32 graph.
• Each layer's 128 experts are one TFLite custom op moe, an op that LiteRT runs with XNNPACK on the CPU and ML Drift on the GPU: only the 8 routed experts are computed for each token. The router, its softmax, the top 8 and their renormalization are ordinary ops in the graph. Int4 block scales below 2^-14 are raised to 2^-14.
• The residual add after each moe op is written as (x + moe) + moe * 0 (the writer's --moe_fusion_guard, numerically exact), so the moe output has a second consumer and ML Drift does not fuse the add into the moe kernel. Fused, the kernel's shader does not compile on WebGPU (Validation error lines, then one repeated token) or on Metal (an error at engine creation), with LiteRT dcb657e8 and, by the same code, main as of 2026-11-02: the same linking as LiteRT-LM issue #4055 on the SwiGLU kernel. Every GPU number and check on this page is with that form.
• Signatures prefill_128, prefill_1024 and decode.
• The metadata is LiteRT-LM's own Qwen3 configuration (models/qwen3 at LiteRT-LM dab2e708) verbatim: the chat template, the stop token <|im_end|> (151645), the qwen3 model type, a thought channel, and the sampler of the source's generation_config.json (type TOP_P: top_p 0.95, top_k 20, temperature 0.6). In the 2,048-token file, max_num_tokens is 2048. The tokenizer section is the checkpoint's tokenizer.json, byte for byte.
• fp32 activations are declared because fp16 activations loop on the GPU: with them, every one of the 8 check questions run (6 on the 4,096-token file, 2 on the 2,048-token file) repeated a short passage of its thought, from one clause to five sentences, until the token limit (the 2,048-token output limit on the 4,096-token file, the end of the context on the 2,048-token file), without an answer.
• Int4 attention and lm_head weights were not faster than int8 ones: with the same 128-wide expert blocks, decode was 41.9 against 41.1 tokens/s on WebGPU at 4,096 tokens, within the spread of the 3 processes. With them, one of the 12 check questions looped until the 2,048-token limit, so the files keep them int8. Experts in 128-wide blocks decoded slower than in 32-wide blocks (41.1 against 47.6 tokens/s).
• The conversion scripts and how each part was checked: john-rocky/hf-to-litertlm: qwen3_30b_a3b_work. A rebuild of the 2,048-token file from those scripts on 2026-11-03 gave every section of this file byte for byte; the whole file is the same bytes only when the header's uuid and creation time are copied from this file (make_bundle.py --ids-from), as REPRODUCE.md describes.

Performance

Apple M4 Max (128 GB, macOS 27.0), litert_lm_advanced_main built as below, 1,024 prefill tokens and 256 decode tokens, measured on 2026-11-03.

File	Backend	Prefill (tok/s)	Decode (tok/s)	Peak footprint
qwen3_30b_a3b_mixed_int4_ekv2048.litertlm	GPU (WebGPU)	1,043.42	54.81	32.75 GB
qwen3_30b_a3b_mixed_int4_ekv4096.litertlm	GPU (WebGPU)	778.11	47.64	34.14 GB
qwen3_30b_a3b_mixed_int4_ekv4096.litertlm	CPU (XNNPACK)	56.30	4.16	5.37 GB

• Command: --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --num_iterations=4 --cache_dir=:nocache, and on the GPU --sampler_handles_input=false.
• GPU rows: the median of 3 processes, each the mean of iterations 2 to 4. CPU row: 1 process with --num_iterations=2, iteration 2.
• Peak footprint: the largest physical footprint of the process read by vmmap about every 20 s (decimal GB). The physical footprint does not count clean pages of mapped files.
• No phone was tried: each file is 20.65 GB.

Accuracy

12 questions (8 short questions, a word problem, a Python function, an ordered list and a Japanese-to-English sentence) and a 3-turn conversation whose third turn asks for a number given in the first, greedy (--sampler_top_k=1), up to 2,048 tokens per answer, thinking on, judged against Transformers with the bf16 weights:

File	Backend	Same verdict as Transformers	Ends with the stop token	3 turns
qwen3_30b_a3b_mixed_int4_ekv2048.litertlm	GPU (WebGPU)	12 / 12	12 / 12	pass
qwen3_30b_a3b_mixed_int4_ekv4096.litertlm	GPU (WebGPU)	12 / 12	12 / 12	pass
qwen3_30b_a3b_mixed_int4_ekv4096.litertlm	CPU (XNNPACK)	12 / 12	12 / 12	pass

On the GPU, the 2,048-token file's 12 replies and its 3-turn conversation were byte for byte the 4,096-token file's. Every GPU check ran with --sampler_handles_input=false; the default GPU sampler path was compared on one 64-token reply only.

Each moe op carries activation='silu' (the source's hidden_act), checked against Transformers before the whole model: on the first two layers of the real weights with fp32 experts, over 185 positions of four prompts, the CPU picks Transformers' top-1 token at every position with a largest logit difference of 2.7e-5 (WebGPU: every position, 4.0e-5), and the router picks the same 8 experts at every position of both layers.

Build the runtime

LiteRT-LM dab2e708 with LiteRT dcb657e8 and ml-drift 76492bb9, with the two pull requests applied. --override_repository skips the patch commands that LiteRT-LM's WORKSPACE applies to the LiteRT and ml-drift archives, so apply_ws_patch_cmds.py (in the conversion scripts' directory) applies them by hand; it needs GNU sed installed as gsed:

git clone https://github.com/google-ai-edge/LiteRT-LM.git && cd LiteRT-LM
git checkout dab2e708dc5750549685692863039c5a954c3a35
git lfs pull --include "prebuilt/macos_arm64/*"
cd ..
git clone https://github.com/google-ai-edge/LiteRT.git && git -C LiteRT checkout dcb657e8a537b2dc944e53e14c7eb43bcd9ededa
git clone https://github.com/google-ai-edge/ml-drift.git && git -C ml-drift checkout 76492bb9412030ec93cc5b607df26ab5cf787490
python3 apply_ws_patch_cmds.py LiteRT-LM/WORKSPACE litert LiteRT
python3 apply_ws_patch_cmds.py LiteRT-LM/WORKSPACE ml_drift ml-drift
gh pr diff 11192 -R google-ai-edge/LiteRT | git -C LiteRT apply
gh pr diff 11188 -R google-ai-edge/LiteRT | git -C LiteRT apply
cd LiteRT-LM
bazel build --config=macos_arm64 -c opt --define=litert_runtime_link_mode=dynamic \
  --override_repository=litert=$PWD/../LiteRT --override_repository=ml_drift=$PWD/../ml-drift \
  //runtime/engine:litert_lm_advanced_main @litert//litert/c:litert_runtime_c_api_dylib \
  @litert//litert/runtime/accelerators/gpu:ml_drift_webgpu_accelerator_dylib

Put litert_lm_advanced_main, libLiteRt.dylib and libLiteRtWebGpuAccelerator.dylib from bazel-bin in one directory with libwebgpu_dawn.dylib and libGemmaModelConstraintProvider.dylib from prebuilt/macos_arm64/; the CLI does not start without the last two, also on the CPU. These are the steps of the build that ran every number on this page (2026-10-28), with only the targets the CLI needs here; they were not run again for this page. On 2026-11-03 gh pr diff gave the same two diffs that the 2026-10-28 build applied.

Run

The command of the GPU checks above:

./litert_lm_advanced_main --backend=gpu --model_path=qwen3_30b_a3b_mixed_int4_ekv2048.litertlm \
  --max_output_tokens=2048 --min_log_severity=2 --sampler_top_k=1 --sampler_handles_input=false \
  --cache_dir=:nocache --input_prompt="What is 17 + 25? Answer briefly."

On the CPU, use --backend=cpu and drop --sampler_handles_input=false; the CPU checks above gave --cache_dir a directory in place of :nocache. The CLI prints the model's thought between [thought] and [/thought], then the answer.

License

Apache License 2.0, the license of Qwen/Qwen3-30B-A3B.

Citation

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report},
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388},
}