Bonsai 2 PQ2 for vLLM
Packed PQ2 safetensors conversion of Prism ML's Bonsai 2 27B. Requires the Bonsai 2 vLLM plugin and vLLM 0.29.0.
Validated: A100 40GB, BF16 text, 2,048-token context, one executing request. The fast profile uses a separate DFlash2 drafter and about 34 GiB of GPU memory. Vision, multi-GPU and longer contexts are not validated.
Quick start
Requires Linux, Python 3.12, Git, uv, a CUDA 13-compatible driver, CUDA toolkit (nvcc) and a C++ compiler.
git clone --branch v0.1.0 https://github.com/wonder-dot-ai/bonsai2-vllm-plugin.git
cd bonsai2-vllm-plugin
uv sync --locked
uv run --locked python tools/download_models.py
uv run --locked bonsai2-serve
OpenAI-compatible API: http://127.0.0.1:8000/v1, model name: bonsai2.
First startup may take several minutes to compile kernels.
See the plugin README for benchmarks, configuration and conversion details.
Apache-2.0. Created using Bonsai by Prism ML; upstream notices are included in NOTICE.txt.
- Downloads last month
- 149