jonpark0/SmolLM2-135M-Instruct-krafla

SmolLM2-135M-Instruct for Krafla

INT4 and INT8 conversions of HuggingFaceTB/SmolLM2-135M-Instruct for Krafla, a dependency-free C11 runtime that runs language models on CPUs without a GPU: x86-64 from SSE2 up to AVX-512 VNNI and ARM from ARMv7 NEON up to i8mm and SVE.

Files

File	Weights	Size	Perplexity	Prefill	Decode	SHA-256
SmolLM2-135M-Instruct-Q4_0.mlb	Q4_0, every matrix 4-bit	72 MiB	31.63	1043 tok/s	242.6 tok/s	2537e91869a148c3…
SmolLM2-135M-Instruct-Q8_0.mlb	Q8_0, every matrix 8-bit	136 MiB	23.54	732 tok/s	154.1 tok/s	808808d568c4bf88…
tokenizer.mlt	tokenizer	1138 KiB				02f356d659655d21…

Measured on a GitHub runner (Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz, 4 threads) with Krafla's avx2 build. Perplexity: wikitext-2 test set, 4 chunks of 512 tokens (tools/perplexity.py --ctx 512 --chunks 4); lower is better, and only files of the same model compare. Speed: ml_bench -n 128, 128 prompt tokens and 128 decoded tokens. Context: 4096 positions. The full hashes are in SHA256SUMS.

Run

Download a release of Krafla (version 0.6.0 or later; Linux x86-64, AArch64 and ARMv7, Windows x86-64) and two files from here:

curl -LO https://huggingface.co/jonpark0/SmolLM2-135M-Instruct-krafla/resolve/main/SmolLM2-135M-Instruct-Q4_0.mlb
curl -LO https://huggingface.co/jonpark0/SmolLM2-135M-Instruct-krafla/resolve/main/tokenizer.mlt
./ml_run SmolLM2-135M-Instruct-Q4_0.mlb -z tokenizer.mlt --chat smollm -t 4

ml_run picks the fastest build your CPU supports. A greedy one-turn reply from SmolLM2-135M-Instruct-Q4_0.mlb to "What is the capital of France? Answer in one sentence.":

The capital of France is Paris.

How these files were made

With Krafla 0.6.0 (commit 31e3769) on 2026-11-02, from the safetensors of HuggingFaceTB/SmolLM2-135M-Instruct:

python3 tools/convert_tokenizer.py SmolLM2-135M-Instruct/tokenizer.json tokenizer.mlt
python3 tools/convert.py hf SmolLM2-135M-Instruct SmolLM2-135M-Instruct-Q4_0.mlb --wtype q4 --max-seq 4096
python3 tools/convert.py hf SmolLM2-135M-Instruct SmolLM2-135M-Instruct-Q8_0.mlb --wtype q8 --max-seq 4096

Q4_0 and Q8_0 keep one scale per block of 32 weights (f16 scales: 4.5 and 8.5 bits per weight). See the user guide for the formats and the comparison with llama.cpp.

License

The license of the original model, apache-2.0, applies (copied in this repository).