litert-community

/

Qwen2.5-1.5B-Instruct

like 51

Follow

LiteRT Community (FKA TFLite) 8.21k

This PR adds an int4 (wi4b64) dynamic-range quantized .litertlm variant: Qwen2.5-1.5B-Instruct_wi4b64_ekv4096.litertlm (790 MB, external KV cache length 4096).

Toolchain / pipeline (unpack → quantize → repack):

1. litert-lm-peek dump of the official f32 .litertlm release in this repository.
2. ai-edge-quantizer 0.8.0 post-training dynamic-range dynamic_wi4b64_afp32 (int4 weights, blockwise-64, fp16 scales, fp32 activations) applied full-scope to the TFLite section.
3. Rebuilt with litert-lm-builder 0.14.0; tokenizer and LlmMetadata mirrored from the official q8 release. Resulting .litertlm format 1.5.0.

Source checkpoint: litert-community/Qwen2.5-1.5B-Instruct f32 release.

Validation: inference-validated on Windows x86_64 CPU — 59.7 tok/s prefill / 11.8 tok/s decode — with correct Korean QA output.

Runtime tested: LiteRT-LM v0.14.0, Windows x86_64 (CPU/GPU); the project's Android arm64 (Snapdragon 865) pipeline runs the same format (device throughput not benchmarked for this file).

Known caveat: the metadata carries lowercase model_type (tf_lite_prefill_decode) vs uppercase in the official files — loads fine on LiteRT-LM v0.14.0, noted for completeness.

Produced by the LiteRT-LM-Unity project (https://github.com/Leuconoe/LiteRT-LM-Unity, release v0.14.0-unity).