infosave/Bonsai-8B_2bit_cmf

Bonsai-8B — CMF q1t (2.32 GB, runs anywhere)

Bonsai-8B — packed into a single CMF file with q1t encoding (ternary 1-bit/2-bit representation).

Because the model was trained for extreme sparsity/low-bit representation, q1t is its native representation: generation is token-for-token identical to the same model stored at higher precision. The whole model is one 2.32 GB file that memory-maps straight off disk and decodes at ~20 tok/s on an Apple M-series MacBook through a whole-token Metal graph.

What is CMF?

CMF (Cortiq Model Format) is a single-file LLM container with a small pure-Rust runtime — no Python, no torch, no C++ toolchain, no CUDA install:

• One file carries the weights, tokenizer and chat template, and checks its own integrity.
• mmap-first: pages load lazily on first touch; start-up is fast and RAM stays near the file size.
• Per-tensor quantization (q8 / q4 / vbit / q1t for 1-bit/2-bit trained models).
• O(1) attention option (--o1): convert attention to a constant-memory streaming operator — no retraining, weights byte-identical.
• Skills: one file can carry a swarm of specialists sharing a base model, with self-routing. See the skills guide.

Format spec and engine: https://github.com/infosave2007/cmf Mobile app (Flutter, runs CMF on-device): https://github.com/infosave2007/cmfmobile

Install

Prebuilt binaries (macOS arm64/x86_64, Linux, Windows): https://github.com/infosave2007/cmf/releases/latest

Or with a Rust toolchain:

cargo install cortiq-cli   # needs >= 0.5.5

Download & run

# download the model file (~2.32 GB)
huggingface-cli download infosave/Bonsai-8B_2bit_cmf bonsai-8b-q1t.cmf --local-dir .

# chat (the file carries its own chat template)
cortiq run bonsai-8b-q1t.cmf --prompt "Привет"

# Apple Silicon: run the whole-token Metal graph (~20 tok/s)
CMF_GPU=1 cortiq run bonsai-8b-q1t.cmf --prompt "Привет"

# raw completion mode (no chat template)
cortiq run bonsai-8b-q1t.cmf --prompt "The capital of France is" --raw --greedy

Run as a server

cortiq serve speaks the OpenAI API, so existing clients and SDKs work unchanged:

CMF_GPU=1 cortiq serve bonsai-8b-q1t.cmf --port 8080   # + web dashboard on /

curl localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "cmf",
  "messages": [{"role": "user", "content": "Explain mmap in one sentence."}]
}'

Performance (Apple M-series MacBook)

Mode	Speed
decode, CMF_GPU=1 (whole-token Metal graph)	~20.2 tok/s
decode, CPU only	~4.2 tok/s
resident memory	≈ file size (mmap)

Provenance

Converted from prism-ml/Bonsai-8B (Apache-2.0) with:

cortiq convert --model prism-ml/Bonsai-8B --quant q1t --output bonsai-8b-q1t.cmf