mlboydaisuke/Qwen3-4B-LiteRT-gpu-composites

Qwen3-4B — LiteRT-LM bundles exported with litert-torch's GPU composite flags

Test exports made for the LiteRT team's GPU-composite work (2026-09-10; re-exported 2026-09-11 with the team's fixed-prefill recipe, see Export). Not a release: on LiteRT-LM 0.17.0's macOS GPU paths the full-flag file does not produce correct output yet (see Status below). Weights int4 blockwise-32, activations fp32 (dynamic_wi4b32_afp32).

Files

file	flags	bytes	sha256
Qwen3-4B_dynamic_wi4b32_afp32_11flags.litertlm	all eleven flags of the export command below	2,270,694,352	e9a4e2674b53d47c0024b5e71d859e4f5f4b5c99cc9a34ddbfabefeea50cc93f
Qwen3-4B_dynamic_wi4b32_afp32_9flags.litertlm	the same minus --use_sdpa_composite and --use_qkv_norm_rope_composite	2,270,290,496	9a4a758ce08957d1136b8706abeff8c71deff629cbe191e4e99741f926bd7dfe

2026-09-21: chat template updated to accept the 0.18 content-parts form. It is the v0.17.0 pbtext's template plus a format_content macro (dual/4155a0aa5616.jinja), swapped in after the export below with swap_template.py. The prompt rendered from string content, the weights, the tokenizer and the executor metadata are byte-identical; the sha256 column is the swapped files'. The pre-swap files are this repo's previous commit.

Signatures: prefill_1024 (fixed) and decode; cache 32771 (from --cache_length=32768 with --enable_gpu_dynamic_cache; the exporter rounds GPU-dynamic sizes up to the next prime). Bundle metadata from LiteRT-LM's models/qwen3/LlmMetadataProto.pbtext (max_num_tokens 4096, sampler TOP_P k=20 p=0.95 t=0.6, stop <|im_end|>). The 2026-09-10 versions of these files (prefill_1031 from --enable_gpu_dynamic_prefill, exporter-generated metadata with max_num_tokens 32771) are in this repo's history at commit 301fd8d1748ffbbd9f492cff86eb30defe0bd47c. Composites in the 11-flag graph: odml.qkv_norm_rope, odml.swiglu, odml.cache_update, odml.sdpa_transposed (decode), with fused QKV / gate-up projections and a bool mask.

Export

litert-torch upstream/main 6d4c622 (2026-09-08; use_swiglu_composite is not in the 0.9.4 wheel), litert-lm-builder 0.17.0 (main's builder needs it), ai-edge-litert 2.2.0, ai-edge-quantizer 0.9.0, torch 2.13.0, transformers 5.14.1.

python -m litert_torch.generative.export_hf \
  --model=Qwen/Qwen3-4B --output_dir=out/p1024 \
  --quantization_recipe=dynamic_wi4b32_afp32 \
  --litert_lm_llm_metadata_override=<LiteRT-LM checkout>/models/qwen3/LlmMetadataProto.pbtext \
  --enable_gpu_dynamic_cache=True \
  --apply_gpu_composites=True --fuse_gate_up=True --fuse_qkv=True \
  --use_qkv_norm_rope_composite=True --use_swiglu_composite=True \
  --use_sdpa_composite=True --use_bool_mask=True \
  --bundle_litert_lm=True --prefill_lengths=1024 --cache_length=32768

The 9-flag file drops the two --use_*_composite flags named in the table.

Status on LiteRT-LM 0.17.0 (macOS, Mac Studio M4 Max)

The speeds below were measured 2026-09-10/11 on the previous versions of these files (dynamic prefill prefill_1031, exporter-generated metadata; graphs otherwise identical). The current fixed-1024 files were re-checked 2026-09-11: 9-flag files 8/8 on the 8-question check, 11-flag failure modes unchanged, and the 0.6B A/B against its previous 9-flag file (three interleaved passes, same host) read the same speed within 3 %, and the 4B file's graph differs from its previous version in the same way.

file	WebGPU path — PyPI litert-lm 0.17.0 wheel	Metal path — litert_lm_main built from the v0.17.0 tag with --//runtime/executor:gpu_accelerator=metal, plus the release's prebuilt libLiteRtMetalAccelerator.dylib
11 flags	does not run: odml.sdpa_transposed and odml.qkv_norm_rope fail WGSL validation, output is !!	compiles, output wrong (blank); 72× GpuModelBuilder::UpdateOutputTensors(output of qkv_norm_rope) shape mismatch ({1,32,1024,1,128} vs {1,16,1024,1,128}) plus 36× the same for flash_decode_sdpa
9 flags	decode 117 tok/s, prefill 1.41k tok/s (litert-lm benchmark --backend gpu -p 1024 -d 128 --cache no, medians of 3 processes); 8-question sanity check 8/8	decode 103 tok/s, prefill 1.4k (996-token prompt, context 4096, decode to EOS, medians of 3); 8/8

Android and iOS: not measured.

Run (macOS)

pip install litert-lm==0.17.0
litert-lm run Qwen3-4B_dynamic_wi4b32_afp32_9flags.litertlm --backend gpu --thinking false --cache no --prompt "What is the capital of Japan? Answer briefly."
litert-lm benchmark Qwen3-4B_dynamic_wi4b32_afp32_9flags.litertlm --backend gpu -p 1024 -d 128 --cache no