jonpark0/Qwen2.5-7B-Instruct-krafla

Qwen2.5-7B-Instruct for Krafla

INT4 and INT8 conversions of Qwen/Qwen2.5-7B-Instruct for Krafla, a dependency-free C11 runtime for CPU-only inference on x86-64 (SSE2 up to AVX-512 VNNI) and ARM (ARMv7 NEON up to i8mm and SVE).

Files

File	Weights	Size	Perplexity	Prefill	Decode	SHA-256
Qwen2.5-7B-Instruct-Q4_0.mlb	Q4_0, every matrix 4-bit	4086 MiB	9.92	26 tok/s	9.9 tok/s	633c617eb9eb641a…
Qwen2.5-7B-Instruct-Q4_0-mixed.mlb	Q4_0 with v,o,down in Q8_0	5189 MiB	9.57	21 tok/s	7.7 tok/s	58ff032171765a5c…
Qwen2.5-7B-Instruct-Q8_0.mlb	Q8_0, every matrix 8-bit	7718 MiB	8.99	16 tok/s	5.2 tok/s	5e960b5aa7cc27ee…
tokenizer.mlt	tokenizer	3468 KiB				df642efa797ea90e…

Measured on a GitHub runner (AMD EPYC 7763 64-Core Processor, 4 threads) with Krafla's avx2 build. Perplexity is measured on the wikitext-2 test set, 4 chunks of 512 tokens (tools/perplexity.py --ctx 512 --chunks 4). Lower is better, and it is only comparable between files of the same model. Speed comes from ml_bench -n 128 (128 prompt tokens, then 128 decoded tokens). The full hashes are in SHA256SUMS.

Run

Download a release of Krafla (version 0.8.0 or later; Linux x86-64, AArch64 and ARMv7, Windows x86-64) and two files from here:

curl -LO https://huggingface.co/jonpark0/Qwen2.5-7B-Instruct-krafla/resolve/main/Qwen2.5-7B-Instruct-Q4_0.mlb
curl -LO https://huggingface.co/jonpark0/Qwen2.5-7B-Instruct-krafla/resolve/main/tokenizer.mlt
./ml_run Qwen2.5-7B-Instruct-Q4_0.mlb -z tokenizer.mlt --chat qwen -t 4

# or serve it with the OpenAI API at http://127.0.0.1:8080/v1
./ml_server Qwen2.5-7B-Instruct-Q4_0.mlb -z tokenizer.mlt --chat qwen -t 4

The files record the model's 32768-position context. ml_run and ml_server use 4096 positions of it unless you pass --ctx N, which sets the size of the KV cache.

The release's ml_run runs the most capable build your CPU supports. A greedy one-turn reply from Qwen2.5-7B-Instruct-Q4_0.mlb to "What is the capital of France? Answer in one sentence.":

The capital of France is Paris.

How these files were made

With Krafla 0.8.0 (commit 0b527a0) on 2026-11-03, from the safetensors of Qwen/Qwen2.5-7B-Instruct:

python3 tools/convert_tokenizer.py Qwen2.5-7B-Instruct/tokenizer.json tokenizer.mlt
python3 tools/convert.py hf Qwen2.5-7B-Instruct Qwen2.5-7B-Instruct-Q4_0.mlb --wtype q4
python3 tools/convert.py hf Qwen2.5-7B-Instruct Qwen2.5-7B-Instruct-Q4_0-mixed.mlb --wtype q4 --q8 v,o,down
python3 tools/convert.py hf Qwen2.5-7B-Instruct Qwen2.5-7B-Instruct-Q8_0.mlb --wtype q8

Q4_0 and Q8_0 keep one scale per block of 32 weights (f16 scales: 4.5 and 8.5 bits per weight). See the user guide for the formats and the comparison with llama.cpp.

License

The license of the original model, apache-2.0, applies (copied in this repository).