webmp3/Sakura-SenseNova-Loop-9B-Preview-GGUF

Sakura · GGUF quantization of sensenova/SenseNova-Loop-9B-Preview

Independent community quantization. Not an official release of the upstream author.

These files need a patched llama.cpp. The architecture sensenova_loop is not part of stock llama.cpp, Ollama or LM Studio; they stop with "unknown model architecture". Use the fork https://github.com/GuideboardLabs/llama.cpp-sensenova-loop at commit fad174ecd8bf22746a626c58e67e193f095aaa46 (the commit these files were made and tested with).

1. Model Overview

SenseNova-Loop-9B-Preview is a looped transformer (layers 4-27 run twice with shared weights, two-stream mHC residual routing, hybrid linear/full attention). This repository contains the text decoder only: the vision tower and the multi-token-prediction head are not exported, so there is no mmproj file and no image input.

2. Original Model / Attribution

• Original: sensenova/SenseNova-Loop-9B-Preview at revision eda5d06eec244f9bb078fd193255fc615720a1f9 (SHA-256 of every weight file checked against the Hub listing). License: Apache-2.0.
• Converted and quantized by webmp3 with the fork above, directly from the original safetensors (no third-party GGUF in between). Upstream authors retain their license and attribution.

3. Files

File	Quant	Size	Mean KLD vs Q8_0	Same top token	Short test output (greedy)
Sakura-SenseNova-Loop-9B-Preview-Q8_0.gguf	Q8_0	9.54 GB	-	-	'user\nThe capital of France is\nassistant\n\nOkay, so I need to figure out the capital '
Sakura-SenseNova-Loop-9B-Preview-Q5_K_M.gguf	Q5_K_M	6.47 GB	0.0160	94.71 %	'user\nThe capital of France is\nassistant\n\nOkay, so I need to figure out the capital '
Sakura-SenseNova-Loop-9B-Preview-Q4_K_M.gguf	Q4_K_M	5.63 GB	0.0392	90.83 %	'user\nThe capital of France is\nassistant\n\nOkay, so I need to figure out the capital '
Sakura-SenseNova-Loop-9B-Preview-Q3_K_M.gguf	Q3_K_M	4.64 GB	0.1490	83.07 %	'user\nThe capital of France is\nassistant\n\nThe user is asking a simple factual questi'
Sakura-SenseNova-Loop-9B-Preview-Q2_K.gguf	Q2_K	3.84 GB	0.3240	76.90 %	'user\nThe capital of France is\nassistant\n\nThe user is asking a simple factual questi'

Q8_0 is the reference (made without a matrix). The K-quants were made with an importance matrix. The router tensors (mhc_*) and the convolution kernels stay F32 in every file; the fork pins them. On this text Q3_K_M and especially Q2_K are far from Q8_0 (mean KLD 0.149 and 0.324 against 0.016 for Q5_K_M); use Q4_K_M or larger where you can.

4. How it was made

• Converted with the fork's convert_hf_to_gguf.py --outtype f32 from the original weights (no rounding step before quantization).
• Importance matrix: calibration text hfqa-original-calibration-v2 (SHA-256 bc7216ea05c8bd05d6b9e94d813ec9eb651a2375288c08c06135a3f874d886f2, 6144 bytes, 11 chunks of 128 tokens), computed once on the Q8_0 file with the fork's llama-imatrix; it covers 248 of 322 2-D weight tensors (the loop block is stored once, so both visits feed the same statistics). The other 74 are the router and convolution tensors (mhc_*, ssm_conv1d), which stay F32 and are never quantized, and token_embd.weight and output.weight, which have no matrix statistics and were quantized without them. Matrix SHA-256 5e061de9af7921d8e0d687b9fd4b929e9b358b93dfdfe6cf105b8b786183326f. The calibration text is short; no quality gain is claimed.
• Every file was checked: the tensor types equal the plan of the fork's own llama-quantize --dry-run (the standard check of this project uses stock llama.cpp and cannot read this architecture, so it was replaced by this one), the file loads and generates text with the fork, and the upload was read back from the Hub by SHA-256.

5. Measured and not measured

• Mean KLD and top-token agreement against the Q8_0 file were measured with the fork's llama-perplexity on a short held-out text (9 chunks of 128 tokens). They compare the quants with Q8_0, not with the original BF16/F32 model, and a short text says little about other languages or code.
• Not measured: benchmarks, long-context behaviour (the upstream model advertises 262144 tokens; only short contexts were run here), the vision and MTP parts, GPU backends, speed.
• Greedy decoding was not compared with the reference transformers implementation here.

6. Run it

git clone https://github.com/GuideboardLabs/llama.cpp-sensenova-loop && cd llama.cpp-sensenova-loop && git checkout fad174ecd8bf22746a626c58e67e193f095aaa46
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON && cmake --build build -j --target llama-cli llama-server llama-quantize
./build/bin/llama-server -m Sakura-SenseNova-Loop-9B-Preview-Q4_K_M.gguf -ngl 99 -c 4096 -fa on --port 8080

CPU or GPU: a plain build (no GPU backend) runs the model on the CPU from RAM only, and -ngl is what moves layers to a GPU. According to its own description the fork implements its fused mHC router operations for the CPU and Vulkan backends (not CUDA or ROCm), so the command above builds the Vulkan backend. We ran and measured everything on the CPU; no GPU backend was tested for these files.

7. License Information

Source license: apache-2.0. The fork is MIT-licensed llama.cpp code with the SenseNova-Loop additions of its author.

中文说明 · 樱花 Sakura（点击展开）

这些文件需要打了补丁的 llama.cpp。 架构 sensenova_loop 不在官方 llama.cpp、Ollama 或 LM Studio 中，它们会报错 "unknown model architecture"。请使用 https://github.com/GuideboardLabs/llama.cpp-sensenova-loop 的提交 fad174ecd8bf22746a626c58e67e193f095aaa46（本仓库文件即由此版本制作和测试）。

• 原始模型：sensenova/SenseNova-Loop-9B-Preview，修订 eda5d06eec244f9bb078fd193255fc615720a1f9，权重文件的 SHA-256 已与 Hub 清单核对。许可证 Apache-2.0。

• 仅包含文本解码器：视觉塔和多词元预测头未导出，没有 mmproj 文件，也不支持图像输入。

• 直接从原始 safetensors 转换（--outtype f32），再量化。Q8_0 为参考文件（无重要性矩阵）；K 量化使用重要性矩阵：校准文本 hfqa-original-calibration-v2（6144 字节，11 个块，每块 128 个词元），在 Q8_0 上用补丁版 llama-imatrix 计算一次，覆盖 248/322 个二维权重；其余 74 个是保持 F32、不会被量化的路由器和卷积张量（mhc_*、ssm_conv1d），以及没有矩阵统计、未使用矩阵量化的 token_embd.weight 和 output.weight。校准文本很短，不声称质量提升。路由器张量（mhc_*）和卷积核在所有文件中保持 F32。

• 校验：张量类型与补丁版 llama-quantize --dry-run 的计划一致（本项目的常规检查使用官方 llama.cpp，无法读取该架构，故以此代替）；文件能加载并生成文本；上传后已按 SHA-256 回读。

• 已测量：在一段很短的英文留出文本（9 个块，每块 128 个词元）上，各量化相对 Q8_0 的平均 KLD 和首选词元一致率（见上表）。Q3_K_M，尤其是 Q2_K，与 Q8_0 相差很大，可能时请使用 Q4_K_M 或更高。未测量：基准测试、长上下文（上游标称 262144，这里只运行了短上下文）、视觉与 MTP、GPU 后端、速度；也没有与参考 transformers 实现比较贪心解码。

• GPU：不带 GPU 后端的普通构建只在 CPU 上、从内存运行；-ngl 才会把层放到 GPU。该分支只为 CPU 和 Vulkan 实现了融合的 mHC 路由算子（没有 CUDA、ROCm），上面的命令因此启用 Vulkan 后端。我们的运行和测量全部在 CPU 上完成，没有测试任何 GPU 后端。