Ternary-Bonsai-1.7B — LiteRT-LM (INT2 / ternary, mobile GPU)

Ternary (INT2) .litertlm bundles of prism-ml/Ternary-Bonsai-1.7B-unpacked, packaged for the LiteRT-LM runtime and Google AI Edge Gallery.

These run on the mobile GPU accelerator, not the NPU.

Recommended: bonsai-1.7b-int2pc-4k-gpu.litertlm. The plain, unmodified build with no optimizations applied. Everything else in this repo is experimental — longer context or lighter activations, but whether a given one loads depends on your app's LiteRT / LiteRT-LM version and dependencies. Start with the recommended build; reach for an experimental one only if you specifically need what it offers.

Builds

file	ctx	weights	activations	notes
bonsai-1.7b-int2pc-4k-gpu.litertlm	4096	INT2 per-channel	fp32	RECOMMENDED — plain, no optimizations
bonsai-1.7b-v2max-sdpa-fp16-int2all-32k-fixed2.litertlm	32768	INT2 (all)	fp16	experimental — fp16 activations, fused SDPA
bonsai-1.7b-v2sdpa-int2all-32k-fixed2.litertlm	32768	INT2 (all)	fp32	experimental — fused SDPA
bonsai-1.7b-int2pc-32k-crashfix.litertlm	32768	INT2 per-channel	fp32	experimental — long context, no SDPA fusion
bonsai-1.7b-int2pc-32k-mp-crashfix.litertlm	32768	INT2 per-channel	fp32	experimental — multi-prefill buckets

32768 is the model's full context — max_position_embeddings for Ternary-Bonsai-1.7B is 32768, so the long-context builds here sit at the ceiling rather than truncating it.

All bundles carry Bonsai's own chat template (Qwen3 ChatML with the reasoning block intact).

Why several variants

The GPU accelerator runs a float graph — INT2 is a storage format, and compute happens in fp16/fp32. Two consequences shape this list:

• Per-channel ternary dequantizes coherently. Block-quantized weights mix scales inside a single GEMM, which is why the per-channel builds are the conservative choice.
• fp16 activations are lighter but less widely supported. The fp16 bundles are the most likely to meet a runtime that won't take them.

Support ranges by app, so the full set is published rather than a single "best" build.

Sampling defaults

Every bundle ships these in its LlmMetadata, so a LiteRT-LM host picks them up without any configuration:

parameter	value
type	TOP_P
top-k	20
top-p	0.85
temperature	0.5

These are the values the bundles were built with. Override them in your host if you want different behaviour.

Usage

Any LiteRT-LM host — the AI Edge Gallery app, or litert_lm_main — with the GPU backend selected.

Provenance and licensing

• Base model: prism-ml/Ternary-Bonsai-1.7B-unpacked, Apache-2.0
• Built from Qwen3-1.7B, Copyright 2024 Alibaba Cloud, Apache-2.0
• These bundles are quantized, repackaged derivatives. No weights were retrained.

Created using Bonsai by Prism ML.

Training data: None was used here. These are post-training quantizations and repackagings of the released Bonsai checkpoint; no additional training, fine-tuning, or calibration data was involved. For the base model's training data, see the upstream Prism ML and Qwen3 model cards.

PII: No dataset was collected, processed, or shipped as part of this conversion, so no personally identifiable information is present in these artifacts beyond whatever the upstream released weights already encode.