Qwen3.5-0.8B-weight-space-programming — LiteRT-LM

RikiyaT/Qwen3.5-0.8B-weight-space-programming converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime.

The source checkpoint is Qwen/Qwen3.5-0.8B with weights edited by RikiyaT ("weight-space programming", no training); see the source model card for the method and its MATH-500 / GSM8K / IFEval results. All credit for the model goes to the Qwen team and to RikiyaT. This repository only changes the file format and the numeric precision.

Text decoder only. The upstream checkpoint is multimodal; the vision tower and MTP weights are not included.

Files

File	Quantization	Size	sha256
Qwen3.5-0.8B-weight-space-programming_int8.litertlm	dynamic int8 (channelwise) weights, float activations; the 36 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b) kept float	792,896,640 B	0f49e7b8b16f91cbf1401a385d6bf68d273560af04d75f739dd40bba2b8aa194

KV budget 4096 tokens (only the 6 full-attention layers keep a KV cache; the 18 GatedDeltaNet layers carry constant-size state). Prefill signatures: 128 and 1024.

Requirements

LiteRT-LM ≥ 0.19.0 (min_runtime_version: "0.19.0" in the bundle). Older runtimes cannot load it: LiteRT-LM 0.14 stops with Encountered unresolved custom op: gated_delta_update.

pip install litert-lm==0.19.0
litert-lm run ./Qwen3.5-0.8B-weight-space-programming_int8.litertlm --backend cpu \
  --thinking false --prompt "What is the capital of South Korea? Answer in one sentence."
litert-lm run ./Qwen3.5-0.8B-weight-space-programming_int8.litertlm --backend cpu \
  --thinking true --prompt "What is 17 × 23?"

The bundle declares thinking (<think>…</think> as the thought channel) and Qwen-style tool calls (<tool_call><function=…><parameter=…>), and pads prefill with <|endoftext|> so padding does not corrupt the recurrent state.

Verification (2026-11-04)

litert-lm run 0.19.0, CPU backend, x86-64 Linux container, --seed 0, the bundle's default sampler (top-p 0.95, top-k 20, temperature 0.6), one fresh process per prompt:

Prompt	Thinking	Output
대한민국의 수도는 어디인가요? 한 문장으로 답하세요.	off	서울특별시
다음 문장을 영어로 번역해줘: 고도 백 미터로 상승합니다.	off	It climbs to 100 meters.
What is 17 × 23? Answer with the number only.	off	51 (wrong)
What is 17 × 23?	on	thought closes, answers 391 (28.5 s)
What is $2^{10} + 24$? … put your final answer within \boxed{}.	on	does not finish: the thought computes 1024 + 24 = 1048, then repeats one sentence until the 4096-token budget runs out

Known limitations

• Long reasoning can loop. LiteRT-LM's sampler has no presence/repetition penalty; the source card's own evaluation used presence_penalty=1.5 and still reports 31.6 % of MATH-500 answers hitting the 32k-token limit. Use a --thinking-budget, or thinking off for short tasks.
• Arithmetic without thinking is unreliable (see the 17 × 23 row).
• LiteRT-LM 0.19.0 carries the recurrent state across conversations on one engine: a second conversation starts from the GatedDeltaNet state the first one left behind, so its output differs from a fresh engine's. Fixed upstream in google-ai-edge/LiteRT-LM@e1f86b05 after 0.19.0. Until a release includes it, create a new Engine per conversation (litert-lm run starts one process per prompt and is not affected).
• An int4 build (blockwise-64 weights) was produced and rejected: it looped on the translation prompt and in the 17 × 23 thought. It is not published.

Conversion

• litert-torch 0.10.0 export_hf (built-in qwen3_5 support), transformers 5.22.1, torch 2.14.1 (CPU), ai-edge-quantizer 0.10.1.
• LLM metadata: LiteRT-LM v0.19.0 models/qwen3_5/LlmMetadataProto.pbtext. litert-torch 0.10.0 packages with the 0.18 builder, whose schema lacks tool_call_config; that block was restored by unpacking and re-packing with litert-lm 0.19.0 (unpack / pack). The TFLite section is unchanged by the repack.
• Quantization recipe: dynamic_wi8_afp32 with a no-quantize override for Linear_in_proj_a / Linear_in_proj_b (+1.7 MB). Keeping these projections float follows the published litert-community/Qwen3.5-0.8B recipe.

litert-torch export_hf <checkpoint> <out> --cache_length=4096 --prefill_lengths=128,1024 \
  --quantization_recipe=recipe_wi8_gdn_ab_float.json \
  --litert_lm_llm_metadata_override=LlmMetadataProto.pbtext

License and changes

Apache-2.0, inherited from the source model. Changes from the original work: text decoder converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; vision tower and MTP weights omitted; tokenizer packaged unmodified; LiteRT-LM's Qwen 3.5 chat template and metadata used. Not affiliated with Alibaba / the Qwen team or with RikiyaT.