Instructions to use webmp3/Sakura-TwIL-LM3-Pro-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use webmp3/Sakura-TwIL-LM3-Pro-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M

Use Docker

docker model run hf.co/webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use webmp3/Sakura-TwIL-LM3-Pro-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "webmp3/Sakura-TwIL-LM3-Pro-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "webmp3/Sakura-TwIL-LM3-Pro-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M

Ollama
How to use webmp3/Sakura-TwIL-LM3-Pro-GGUF with Ollama:
```
ollama run hf.co/webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M
```
Unsloth Desktop

How to use webmp3/Sakura-TwIL-LM3-Pro-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use webmp3/Sakura-TwIL-LM3-Pro-GGUF with Docker Model Runner:
```
docker model run hf.co/webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M
```

Lemonade

How to use webmp3/Sakura-TwIL-LM3-Pro-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Sakura-TwIL-LM3-Pro-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use webmp3/Sakura-TwIL-LM3-Pro-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use webmp3/Sakura-TwIL-LM3-Pro-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "webmp3/Sakura-TwIL-LM3-Pro-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Sakura — TwIL-LM3-Pro (GGUF, three sizes)

TwIL-LM3-Pro is a 3.66B reasoning model for formal-logic tasks from webAI, built from IBM's granite-4.2-3b with LoRA fine-tuning, weight interpolation and GRPO. Non-commercial use only: the weights and these quantizations are under the webAI Non-Commercial License ver. 1.0 (research and education), a copy is in LICENSE. This repository holds three GGUF files of different size made from the BF16 weights with our measured mixed-codec method. Pick one on the Files page; the table below says what each file is. Independent community quantization, not an official webAI release. Part of the Sakura Mini line.

Quality at a glance

Same method, same texts, same machine for every file; lower KL divergence means closer to the BF16 original:

Sakura-TwIL-LM3-Pro-Q4_K_M-2.06GiB.gguf vs llama.cpp default Q4_K_M (2.09 GiB): KL divergence to the BF16 original 57 % lower on average over all 3 texts (0.03 GiB smaller).
Sakura-TwIL-LM3-Pro-Q5_K_M-2.39GiB.gguf vs llama.cpp default Q5_K_M (2.43 GiB): KL divergence to the BF16 original 44 % lower on average over all 3 texts (0.04 GiB smaller).
Sakura-TwIL-LM3-Pro-Q6_K-2.75GiB.gguf vs llama.cpp default Q6_K (2.80 GiB): KL divergence to the BF16 original about equal (3 % lower on average over all 3 texts, but worse on English text and perplexity, better on developer text), 0.05 GiB smaller. This is a tie, not a win.

Why: the bit budget per tensor is allocated from measured sensitivity (see How it was made), not from fixed rules. Full table below.

Which file should I take?

File	For
`Sakura-TwIL-LM3-Pro-Q4_K_M-2.06GiB.gguf` (2.06 GiB, 4.83 bpw)	smallest: for tight memory budgets; expect visible quality loss
`Sakura-TwIL-LM3-Pro-Q5_K_M-2.39GiB.gguf` (2.39 GiB, 5.61 bpw)	middle: best balance of size and quality for most machines
`Sakura-TwIL-LM3-Pro-Q6_K-2.75GiB.gguf` (2.75 GiB, 6.45 bpw)	largest: closest to the BF16 original of the three

The files

File	Size	bits/weight	KLD en	KLD dev	KLD wiki	same top token (avg)	Tensor types by size
`Sakura-TwIL-LM3-Pro-Q4_K_M-2.06GiB.gguf`	2.06 GiB	4.83	0.0178	0.0125	0.0358	95.1 %	Q5_K 53%, IQ4_XS 40%, Q4_K 6%
`Sakura-TwIL-LM3-Pro-Q5_K_M-2.39GiB.gguf`	2.39 GiB	5.61	0.0067	0.0047	0.0123	97.0 %	Q5_K 73%, Q6_K 19%, IQ4_XS 5%, Q8_0 2%
`Sakura-TwIL-LM3-Pro-Q6_K-2.75GiB.gguf`	2.75 GiB	6.45	0.0045	0.0030	0.0091	97.7 %	Q6_K 80%, Q5_K 13%, Q8_0 6%

"Tensor types by size" lists the share of the file's bytes per ggml type (all are standard llama.cpp types, so the files run in stock llama.cpp). Quality columns: mean KL divergence of the quantized model's next-token distribution against the BF16 source on held-out text (lower is better) and how often the most likely token is unchanged (higher is better).

Comparison at similar size

Same texts, same context, same chunks, measured by us with llama-perplexity (BF16 source = reference; PPL of the BF16 model: en: 2.088, dev: 1.786, wiki: 11.904). The reference quants are: webAI (author's own quants), measured the same way. Sorted by size.

Quant	Source	Size	KLD en	KLD dev	KLD wiki	PPL en	same top token (avg)
`Sakura-TwIL-LM3-Pro-Q4_K_M-2.06GiB.gguf`	this repository	2.06 GiB	0.0178	0.0125	0.0358	2.157	95.1 %
Q4_K_M	webAI (author's own quants)	2.09 GiB	0.0464	0.0367	0.0625	2.139	92.8 %
`Sakura-TwIL-LM3-Pro-Q5_K_M-2.39GiB.gguf`	this repository	2.39 GiB	0.0067	0.0047	0.0123	2.137	97.0 %
Q5_K_M	webAI (author's own quants)	2.43 GiB	0.0116	0.0102	0.0193	2.132	96.4 %
`Sakura-TwIL-LM3-Pro-Q6_K-2.75GiB.gguf`	this repository	2.75 GiB	0.0045	0.0030	0.0091	2.134	97.7 %
Q6_K	webAI (author's own quants)	2.80 GiB	0.0043	0.0036	0.0089	2.125	97.6 %

Note on perplexity: on the English text the perplexity of our smallest file (2.157) is slightly higher than that of the author's Q4_K_M (2.139) although its KL divergence to BF16 is much lower; KL divergence measures agreement with the BF16 distribution, perplexity measures fit to the text, and the two can differ by a little.

File names carry the nearest standard size class (Q4_K_M, Q5_K_M, Q6_K) so the hub shows the right hardware fit; the files are mixed-codec builds with the tensor types listed above, not the llama.cpp default recipe.

Method: 12 chunks of 512 tokens per text; texts: English held-out text, developer-text held-out set, English encyclopedia held-out text (not used in any calibration). Single measurement on one machine; small KLD differences at similar size are not a quality ranking.

How it was made

The BF16 GGUF was converted by us from the original Safetensors (revision 4214aaa4) with the current convert_hf_to_gguf.py of llama.cpp; the importance matrix was computed by us (llama-imatrix, 96 chunks of 512 tokens on English chat, developer and encyclopedia text).
Every large weight matrix was quantized once per candidate type (Q2_K, IQ2_S, IQ3_XXS, IQ3_S, Q3_K, IQ4_XS, Q4_K, Q5_K, Q6_K, Q8_0) with llama-quantize and the importance matrix; the error of each choice was estimated per matrix (importance-weighted) and scaled by a sensitivity factor measured with real KL divergence: every group of tensors (for example the FFN down projections, the attention value projections, the first and last layers) was moved alone to a lower-bit type, and the KL divergence it caused was compared with its quantization error.
An exact budget allocation (multiple-choice knapsack over bytes) picked one type per matrix for each target size; the final model was assembled from the stored tensors without re-quantizing. We do not publish per-tensor choices. Norms and small tensors stay at high precision.

Limits

Measured only with KL divergence and perplexity on short held-out texts and not with downstream benchmarks; do not read it as a task-quality claim.
Quantization always costs some quality; the smallest file costs the most.
License: these files are modified (quantized) versions of TwIL-LM3-Pro and fall under the same webAI Non-Commercial License ver. 1.0: use for non-commercial research and educational purposes only, the license text must stay with any redistribution. The base model granite-4.2-3b is Apache-2.0 (IBM). This is a community build, not endorsed by webAI.
The quality numbers measure agreement with the BF16 model on general English text (KL divergence); we did not run the formal-logic benchmarks of the original model card on these files, so task accuracy after quantization is unmeasured.

Credits

Original model: TwIL-LM3-Pro (webAI), webAI Non-Commercial License ver. 1.0 license (included as LICENSE).
BF16 conversion and importance matrix: own work (llama.cpp). Reference quants: the author's published TwIL-LM3-Pro-Q4_K_M.gguf, Q5_K_M and Q6_K from webAI-Official/TwIL-LM3-Pro.
GGUF format and tools: ggml-org/llama.cpp (MIT).

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura — TwIL-LM3-Pro(GGUF,三种大小)

TwIL-LM3-Pro 是 webAI 推出的 3.66B 形式逻辑推理模型,基于 IBM 的 granite-4.2-3b,经 LoRA 微调、权重插值和 GRPO 训练得到。仅限非商业用途:权重及这些量化版本适用 webAI Non-Commercial License ver. 1.0(仅限研究与教育),许可证副本见 LICENSE。本仓库包含 三个不同大小的 GGUF 文件,由 BF16 权重通过我们实测得出的混合编码方法制作。请在 Files 页面选择其中一个;下表说明每个文件的用途。这是独立的社区量化版本,不是 webAI 的官方发布。属于 Sakura Mini 系列。

质量概览

每个文件使用相同的方法、相同的文本、同一台机器;KL 散度越低,越接近 BF16 原始模型:

Sakura-TwIL-LM3-Pro-Q4_K_M-2.06GiB.gguf 对比 llama.cpp 默认 Q4_K_M (2.09 GiB):相对于 BF16 原始模型,在全部 3 个文本上平均 KL 散度 低 57 %(小 0.03 GiB)。
Sakura-TwIL-LM3-Pro-Q5_K_M-2.39GiB.gguf 对比 llama.cpp 默认 Q5_K_M (2.43 GiB):相对于 BF16 原始模型,在全部 3 个文本上平均 KL 散度 低 44 %(小 0.04 GiB)。
Sakura-TwIL-LM3-Pro-Q6_K-2.75GiB.gguf 对比 llama.cpp 默认 Q6_K (2.80 GiB):相对于 BF16 原始模型,KL 散度 基本持平(在全部 3 个文本上平均低 3 %,但在英文文本和困惑度上更差,在开发者文本上更好),小 0.05 GiB。这是平手,不是胜出。

原因:每个张量的比特预算根据实测敏感度分配(见 How it was made),而不是固定规则。完整表格见下。

我该选哪个文件?

文件	适用场景
`Sakura-TwIL-LM3-Pro-Q4_K_M-2.06GiB.gguf` (2.06 GiB, 4.83 bpw)	最小:适合内存紧张的情况;质量损失较明显
`Sakura-TwIL-LM3-Pro-Q5_K_M-2.39GiB.gguf` (2.39 GiB, 5.61 bpw)	中等:对大多数机器来说,大小与质量的最佳平衡
`Sakura-TwIL-LM3-Pro-Q6_K-2.75GiB.gguf` (2.75 GiB, 6.45 bpw)	最大:三者中最接近 BF16 原始模型

文件详情

文件	大小	比特/权重	KLD en	KLD dev	KLD wiki	最高概率词不变(平均)	按大小排列的张量类型
`Sakura-TwIL-LM3-Pro-Q4_K_M-2.06GiB.gguf`	2.06 GiB	4.83	0.0178	0.0125	0.0358	95.1 %	Q5_K 53%, IQ4_XS 40%, Q4_K 6%
`Sakura-TwIL-LM3-Pro-Q5_K_M-2.39GiB.gguf`	2.39 GiB	5.61	0.0067	0.0047	0.0123	97.0 %	Q5_K 73%, Q6_K 19%, IQ4_XS 5%, Q8_0 2%
`Sakura-TwIL-LM3-Pro-Q6_K-2.75GiB.gguf`	2.75 GiB	6.45	0.0045	0.0030	0.0091	97.7 %	Q6_K 80%, Q5_K 13%, Q8_0 6%

“按大小排列的张量类型”列出各 ggml 类型占文件字节数的比例(均为标准 llama.cpp 类型,因此这些文件可在原版 llama.cpp 中运行)。质量列:量化模型的下一个词分布相对于 BF16 源模型在留出文本上的平均 KL 散度(越低越好),以及最可能的词保持不变的频率(越高越好)。

相近大小的对比

相同的文本、相同的上下文、相同的分块,由我们用 llama-perplexity 测量(以 BF16 源模型为参照;BF16 模型的 PPL:en: 2.088, dev: 1.786, wiki: 11.904)。参照量化版本来自:webAI(作者自己的量化版本),以同样方式测量。按大小排序。

量化版本	来源	大小	KLD en	KLD dev	KLD wiki	PPL en	最高概率词不变(平均)
`Sakura-TwIL-LM3-Pro-Q4_K_M-2.06GiB.gguf`	本仓库	2.06 GiB	0.0178	0.0125	0.0358	2.157	95.1 %
Q4_K_M	webAI(作者自己的量化版本)	2.09 GiB	0.0464	0.0367	0.0625	2.139	92.8 %
`Sakura-TwIL-LM3-Pro-Q5_K_M-2.39GiB.gguf`	本仓库	2.39 GiB	0.0067	0.0047	0.0123	2.137	97.0 %
Q5_K_M	webAI(作者自己的量化版本)	2.43 GiB	0.0116	0.0102	0.0193	2.132	96.4 %
`Sakura-TwIL-LM3-Pro-Q6_K-2.75GiB.gguf`	本仓库	2.75 GiB	0.0045	0.0030	0.0091	2.134	97.7 %
Q6_K	webAI(作者自己的量化版本)	2.80 GiB	0.0043	0.0036	0.0089	2.125	97.6 %

关于困惑度的说明:在英文文本上,我们最小文件的困惑度 (2.157) 略高于作者的 Q4_K_M (2.139),尽管它相对于 BF16 的 KL 散度要低得多;KL 散度衡量与 BF16 分布的一致程度,困惑度衡量对文本的拟合程度,两者可能有小幅差异。

文件名中带有最接近的标准大小等级(Q4_K_M、Q5_K_M、Q6_K),以便 Hugging Face 显示正确的硬件适配信息;这些文件是混合编码构建,使用上表所列的张量类型,而不是 llama.cpp 的默认方案。

方法:每个文本 12 个分块,每块 512 个 token;文本:英文留出文本、开发者文本留出集、英文百科留出文本(未用于任何校准)。在一台机器上的单次测量;相近大小下 KLD 的微小差异不构成质量排名。

制作方法

BF16 GGUF 由我们用当前 llama.cpp 的 convert_hf_to_gguf.py 从原始 Safetensors(版本 4214aaa4)转换而来;重要性矩阵由我们计算(llama-imatrix,在英文对话、开发者文本和百科文本上共 96 个分块,每块 512 个 token)。
每个大型权重矩阵针对每个候选类型(Q2_K、IQ2_S、IQ3_XXS、IQ3_S、Q3_K、IQ4_XS、Q4_K、Q5_K、Q6_K、Q8_0)用 llama-quantize 和重要性矩阵量化一次;按矩阵估计每种选择的误差(按重要性加权),并乘以一个 用真实 KL 散度测得的敏感度系数:每一组张量(例如 FFN 下投影、注意力 value 投影、首尾层)单独降到较低比特类型,将其造成的 KL 散度与其量化误差进行比较。
精确的预算分配(针对字节数的多选背包问题)为每个目标大小为每个矩阵选出一种类型;最终模型由已存储的张量组装而成,无需重新量化。我们不公布逐张量的选择。归一化层和小张量保持高精度。

局限

仅用 KL 散度和短留出文本上的困惑度测量,未做下游基准测试;请勿将其解读为任务质量的结论。
量化总会损失一些质量;最小的文件损失最大。
许可证: 这些文件是 TwIL-LM3-Pro 的修改(量化)版本,适用相同的 webAI Non-Commercial License ver. 1.0:仅限非商业的研究和教育用途,再分发时必须附带许可证文本。基础模型 granite-4.2-3b 为 Apache-2.0 (IBM)。这是社区构建的版本,未获 webAI 认可。
质量数字衡量的是在一般英文文本上与 BF16 模型的一致程度(KL 散度);我们没有在这些文件上运行原模型卡中的形式逻辑基准测试,因此量化后的任务准确率 未经测量。

致谢

原始模型:TwIL-LM3-Pro (webAI),webAI Non-Commercial License ver. 1.0 许可证(以 LICENSE 附上)。
BF16 转换和重要性矩阵:自有工作 (llama.cpp)。参照量化版本:作者发布的 TwIL-LM3-Pro-Q4_K_M.gguf、Q5_K_M 和 Q6_K,来自 webAI-Official/TwIL-LM3-Pro。
GGUF 格式和工具:ggml-org/llama.cpp (MIT)。