Same size as unsloth. Up to 9.4x lower quality loss on English text.

MS (Mad Shift) quants of Qwen/Qwen3.5-2B, for the ms-llama.cpp fork.

  • Accuracy: 4.5-9.4x lower KL divergence than unsloth's smallest file of the same or larger size, for every one of the seven files (1).
  • On English text, Q4 size beats Q6_K: MS4.3 (1.30 GB, between unsloth's Q4_K_M and UD-Q4_K_XL in size) is closer to the original model than unsloth's Q6_K (1.57 GB): 0.00219 vs 0.00277 (1). MS3.5 beats Q4_K_M from 190 MB smaller.
  • Chinese (not in the calibration): MS still beats every paired unsloth file on held-out Chinese text, by 1.1-5.0x (4.5-9.4x on English). Chinese was deliberately left out of the calibration (English came first; future conversions will be multilingual). MS5 is also ahead of the larger Q6_K on Chinese; MS4.3 is behind it.
  • Top-1: the most likely next token matches the original BF16 model's 91.3-98.4% of the time, vs 80.8-96.4% for those same unsloth files: 54-73% fewer mismatches (1). MS4.3 98.13% vs Q6_K 97.51%.
  • vs EXL3: for the same KL divergence, MS files are 39-59% of the size of EXL3 files (UnstableLlama's EXL3 files and one 6.00 conversion of our own); every MS file is smaller than every EXL3 file measured.
  • Memory: every file runs in about 3 GB or less at a 32k context (estimate).

(1) KL divergence from the original BF16 model, mean over four English texts; top-1 = how often the quantized model's most likely next token is the BF16 model's. All 19 of unsloth's Qwen3.5-2B GGUFs (unsloth/Qwen3.5-2B-GGUF, revision f6d5376b), measured on stock llama.cpp.

KL divergence vs size

Top-1 agreement vs size

MS vs EXL3: KL divergence vs size

MS (Mad Shift) was created by Matthew Priester (GitHub, ORCID). If you use these files or build on the format, please credit MS (Mad Shift) by Matthew Priester.

Run it now

MS files need the MS fork of llama.cpp, ms-llama.cpp: stock llama.cpp, Ollama and LM Studio cannot load them.

1. Install the engine (Windows, NVIDIA GPU; once; PowerShell):

$d="$env:LOCALAPPDATA\ms-llama"; $z=if($env:PROCESSOR_ARCHITECTURE -eq "ARM64"){"arm64-cuda"}else{"x64-cuda12"}; ni -ItemType Directory -Force $d|Out-Null; curl.exe -fL -o "$d\ms.zip" "https://github.com/devvexus/ms-llama.cpp/releases/latest/download/ms-llama-win-$z.zip"; if($?){tar -xf "$d\ms.zip" -C $d}

Linux and macOS: build from source, see the ms-llama.cpp README.

2. Run any file (each needs about 3 GB of memory or less). It downloads once, then serves a chat page and an API on http://localhost:8080. Replace <file> with a name from the table:

& "$env:LOCALAPPDATA\ms-llama\llama-server.exe" -hf devvexus/Qwen3.5-2B-MS --hf-file <file> -c 32768 -np 1 --port 8080

-np 1 serves one chat at a time: every extra parallel slot needs its own copy of the model's recurrent state, which can push layers off the GPU and slow generation down a lot. After the first run, you can add --offline so the server starts without checking Hugging Face; that check can take a few minutes. On Windows the first download may print 'failed to create symlink ... switching to degraded mode'; it is harmless: the Hugging Face cache just can't use symlinks, so updated files may take extra disk space.

<file> Size
MS5/Qwen3.5-2B-MS5.gguf 1441 MB
MS4.3/Qwen3.5-2B-MS4.3.gguf 1301 MB
MS4/Qwen3.5-2B-MS4.gguf 1262 MB
MS3.7/Qwen3.5-2B-MS3.7.gguf 1165 MB
MS3.5/Qwen3.5-2B-MS3.5.gguf 1088 MB
MS3/Qwen3.5-2B-MS3.gguf 858 MB
MS2.7/Qwen3.5-2B-MS2.7.gguf 775 MB

A .gguf you downloaded yourself: -m <file>.gguf.

Images (optional). The vision encoder, vision/mmproj-Qwen3.5-2B-F16.gguf (0.67 GB, F16), lets the model read images. -hf does not load it by itself: add --mmproj-url with its address, or --mmproj <file> for a downloaded copy. It works with every MS file and adds about 0.9-1.1 GiB of GPU memory when on (nothing when off). The vision encoder is not MS-encoded and was not part of our evaluation; MS-encoding it is future work. For example:

& "$env:LOCALAPPDATA\ms-llama\llama-server.exe" -hf devvexus/Qwen3.5-2B-MS --hf-file <file> -c 32768 -np 1 --port 8080 --mmproj-url https://huggingface.co/devvexus/Qwen3.5-2B-MS/resolve/main/vision/mmproj-Qwen3.5-2B-F16.gguf

Use it in your apps

While the server runs:

  • Claude Code: $env:ANTHROPIC_BASE_URL="http://localhost:8080"; $env:ANTHROPIC_AUTH_TOKEN="local"; claude Claude Code's own prompt is large (about 24k tokens in our test with Claude Code 2.1.325), so give the server more context for it: -c 49152 instead of -c 32768.
  • OpenAI-compatible apps: base URL http://localhost:8080/v1, any API key.
  • Apps that run llama.cpp themselves: point them at %LOCALAPPDATA%\ms-llama.
  • Ollama and LM Studio can't load MS files.

Files

MS file Size KLD unsloth file (size) its KLD MS is Top-1, MS / unsloth
MS5 1441 MB 0.00142 UD-Q5_K_XL (1467 MB) 0.00640 4.5x closer 98.36% / 96.41%
MS4.3 1301 MB 0.00219 UD-Q4_K_XL (1340 MB) 0.01472 6.7x closer 98.13% / 94.87%
MS4 1262 MB 0.00270 Q4_K_M (1281 MB) 0.02007 7.4x closer 98.02% / 93.94%
MS3.7 1165 MB 0.00376 IQ4_XS (1173 MB) 0.02756 7.3x closer 97.52% / 92.92%
MS3.5 1088 MB 0.00730 Q3_K_M (1107 MB) 0.06848 9.4x closer 97.00% / 89.00%
MS3 858 MB 0.02793 UD-IQ2_M (860 MB) 0.23063 8.3x closer 93.32% / 80.79%
MS2.7 775 MB 0.04361 UD-IQ2_M (860 MB) 0.23063 5.3x closer 91.26% / 80.79%

Each quant is in its own folder; the optional vision encoder is vision/mmproj-Qwen3.5-2B-F16.gguf (668 MB, F16, see Images). Sizes in MB (10^6 bytes). Each MS file is compared with the smallest unsloth file of the same or larger size (unsloth/Qwen3.5-2B-GGUF, revision f6d5376b). Neither the MS files nor unsloth's Qwen3.5-2B files contain the model's MTP layer, so file sizes compare directly. Checksums: SHA256SUMS.

Per-text KL divergence and every unsloth file

KL divergence per text and top-1 agreement (how often the quantized model's most likely next token is the BF16 model's):

MS file wiki web code math Top-1
MS5 0.00154 0.00137 0.00138 0.00141 98.36%
MS4.3 0.00293 0.00201 0.00249 0.00131 98.13%
MS4 0.00323 0.00232 0.00364 0.00160 98.02%
MS3.7 0.00465 0.00358 0.00414 0.00268 97.52%
MS3.5 0.00985 0.00727 0.00857 0.00352 97.00%
MS3 0.03043 0.02851 0.02987 0.02293 93.32%
MS2.7 0.05064 0.04520 0.04303 0.03557 91.26%

All of unsloth's files on the same eval (revision f6d5376b):

unsloth file Size (MB) Mean KLD wiki web code math Top-1
UD-Q6_K_XL 1864 0.00169 0.00166 0.00151 0.00255 0.00103 98.33%
Q6_K 1575 0.00277 0.00320 0.00285 0.00299 0.00204 97.51%
UD-Q5_K_XL 1467 0.00640 0.00763 0.00669 0.00684 0.00444 96.41%
Q5_K_M 1435 0.00705 0.00790 0.00699 0.00855 0.00477 96.32%
Q5_K_S 1385 0.00870 0.00963 0.00866 0.01009 0.00643 95.92%
UD-Q4_K_XL 1340 0.01472 0.01663 0.01482 0.01721 0.01020 94.87%
Q4_1 1294 0.02533 0.02827 0.02442 0.03137 0.01725 93.37%
Q4_K_M 1281 0.02007 0.02235 0.01963 0.02377 0.01455 93.94%
Q4_K_S 1218 0.02402 0.02709 0.02416 0.02804 0.01678 93.48%
Q4_0 1215 0.03868 0.04405 0.03945 0.04255 0.02868 91.75%
IQ4_NL 1213 0.02714 0.03131 0.02857 0.02965 0.01902 93.07%
IQ4_XS 1173 0.02756 0.03171 0.02875 0.03052 0.01925 92.92%
UD-Q3_K_XL 1159 0.04540 0.05203 0.04761 0.04888 0.03308 91.16%
Q3_K_M 1107 0.06848 0.07982 0.06987 0.07449 0.04975 89.00%
Q3_K_S 1031 0.11840 0.14122 0.12376 0.12050 0.08812 85.76%
UD-Q2_K_XL 967 0.19697 0.22487 0.20646 0.19608 0.16048 82.04%
UD-IQ3_XXS 932 0.13256 0.15108 0.13215 0.14557 0.10144 85.02%
UD-IQ2_M 860 0.23063 0.25809 0.23265 0.24466 0.18711 80.79%
UD-IQ2_XXS 768 0.59367 0.64371 0.57999 0.65169 0.49930 69.07%

Medians and the worst chunk of every text, for every file: kld_results.json.

Memory per file

Memory needed to run each file fully on a GPU (or in RAM on CPU), approximate, with the default batch size. The context cache grows with the context length; a smaller -ub lowers the compute buffers.

Quant File size Memory at 8k context Memory at 32k context Free memory to have (32k)
MS5 1441 MB ~2.6 GB ~2.9 GB 4 GB
MS4.3 1301 MB ~2.4 GB ~2.7 GB 3 GB
MS4 1262 MB ~2.4 GB ~2.7 GB 3 GB
MS3.7 1165 MB ~2.3 GB ~2.6 GB 3 GB
MS3.5 1088 MB ~2.2 GB ~2.5 GB 3 GB
MS3 858 MB ~2.0 GB ~2.3 GB 3 GB
MS2.7 775 MB ~1.9 GB ~2.2 GB 3 GB

Speed

On an NVIDIA RTX Spark N1X laptop GPU (GB10, Blackwell, 6144 cores), Windows 11 on ARM; MS and unsloth files both on ms-llama.cpp, each MS file against the smallest unsloth file of the same or larger size (and MS4.3 against Q6_K):

MS file unsloth file Generation, tokens/s Prompt processing, tokens/s
MS5 UD-Q5_K_XL 119.1 vs 120.9 (0.99x) 10327 vs 11170 (0.92x)
MS4.3 UD-Q4_K_XL 126.5 vs 127.5 (0.99x) 11305 vs 11936 (0.95x)
MS4.3 Q6_K 126.5 vs 117.5 (1.08x) 10997 vs 10612 (1.04x)
MS4 Q4_K_M 127.5 vs 131.0 (0.97x) 11412 vs 11815 (0.97x)
MS3.7 IQ4_XS 131.6 vs 136.4 (0.96x) 11400 vs 12469 (0.91x)
MS3.5 Q3_K_M 135.4 vs 138.2 (0.98x) 11406 vs 10191 (1.12x)
MS3 UD-IQ2_M 159.1 vs 154.4 (1.03x) 11314 vs 9817 (1.15x)
MS2.7 UD-IQ2_M 168.2 vs 154.4 (1.09x) 11359 vs 9905 (1.15x)

Generation: chat benchmark without MTP (20 chat prompts x 256 tokens, greedy). Prompt processing: llama-bench -p 512 -n 0 -ngl 99, MS and unsloth files alternating in one session (median of 3).

EXL3

EXL3 (exllamav3) is a trellis format. We measured UnstableLlama's Qwen3.5-2B EXL3 files and one 6.00bpw conversion of our own (exllamav3 v0.0.28, default settings) against the same BF16 model:

File Size Mean KLD Top-1
2.10bpw 1.76 0.274546 79.08%
3.10bpw 1.93 0.057093 90.58%
4.00bpw 2.09 0.016215 94.91%
5.00bpw 2.26 0.004471 97.19%
6.00bpw, own conversion 2.43 0.001518 98.26%
6.00bpw 2.43 0.001559 98.19%
8.00bpw 2.78 0.000433 98.83%

Sizes are the file bytes needed for text generation (vision tower excluded), like the MS sizes. The EXL3 files were scored with exllamav3 on an RTX 4080 SUPER against a reference regenerated from the same BF16 file; on that machine the MS and unsloth files agree with this card's values within 1.4 standard errors on every text. exllamav3 with the BF16 weights scores 0.00042 against llama.cpp's BF16 logits (mean of the four texts); the EXL3 values include it. The chart shows the better of the two 6.00bpw files.

How the comparison was made

KL divergence (KLD) and top-1 agreement against the BF16 model, measured with llama-perplexity --kl-divergence (512-token context, 128 chunks per text, the second half of each chunk scored: 255 tokens per chunk, 32,640 per text) on four English texts that were not used to make the files: wiki (Wikipedia articles), web (FineWeb-Edu), code (real code patches from SWE-Gym) and math (MATH test problems and solutions). The headline number is the mean over the four texts; lower KLD is better. Top-1 agreement is how often the quantized model's most likely next token is the BF16 model's. The texts, with their sources and licenses, are published as devvexus/ms-eval-texts (the math text as a script that rebuilds it). Neither the MS files nor unsloth's Qwen3.5-2B files contain the model's MTP layer, so file sizes compare directly.

Why KL divergence: it compares the quantized model's whole next-token distribution with the original model's at every position of held-out text. It is not a task score: it measures how closely a file follows the original model on held-out text.

unsloth's files: unsloth/Qwen3.5-2B-GGUF at revision f6d5376b, measured on stock llama.cpp 36a0e2ffc. MS files: measured on ms-llama.cpp 6085028c3 with GGML_CUDA_MMMS_ACC=f32. Both against the same BF16 reference logits. All numbers are our own measurements.

A few text passages dominate their text's mean for every quant (for example drawing code inside a math solution).

Check it yourself: any English text works. Make reference logits with the BF16 model once, then compare any quant against them (MS files with the fork's build, other GGUFs with the same build or stock llama.cpp):

./build/bin/llama-perplexity -m Qwen3.5-2B-BF16.gguf -f text.txt -c 512 --kl-divergence-base base.kld
./build/bin/llama-perplexity -m Qwen3.5-2B-MS5.gguf -f text.txt -c 512 --kl-divergence-base base.kld --kl-divergence

What MS is

MS (Mad Shift) is a GGUF weight format for llama.cpp. MS uses the bitshift-trellis construction and the 1MAD decoding function described in QTIP (Tseng, Sun, Hou and De Sa, NeurIPS 2024).

Every large matrix is stored trellis-coded after a random Hadamard rotation (undone at run time); a tensor with k bits (k = 1..6) costs k + 0.0625 bits per weight. The fork has CPU and CUDA kernels for it. The files are made with the bitshift-trellis construction described in QTIP + our own calibration-aware, layer-by-layer conversion.

Names: an MSN file is built to beat unsloth's QN-class files at the same or a smaller size; N is a size class, not the average bits per weight.

How these files were made

  • Conversion: the bitshift-trellis construction described in QTIP + our own calibration-aware, layer-by-layer conversion, from Qwen's original weights. No tensors are copied from other GGUFs.
  • Make your own MS files: these files are made with the MS converter, open source (Apache 2.0): mad-shift-converter. A lightweight MS encoder for ms-llama.cpp's llama-quantize, built for fast conversion, is in development, so that anyone can make MS files of most models on their own hardware.
  • Embedding table (shared with the output head): MS-coded; from MS4.3 down, the most frequent tokens get higher precision.
  • Vision encoder: vision/mmproj-Qwen3.5-2B-F16.gguf, converted from Qwen's original weights with llama.cpp's convert_hf_to_gguf.py (--mmproj --outtype f16); not MS-encoded.

Limitations

  • The vision encoder is not MS-encoded and was not part of our evaluation (see Images).
  • Needs the MS fork; GPU (CUDA) and CPU (x86 and ARM) are supported there.
  • Accuracy was measured on four English texts and one held-out Chinese text. The split embedding table keeps rarely used tokens (much of the non-English vocabulary) at lower precision, so the lead is smaller on Chinese (see Chinese above).

Citation

Technical report: Priester, M. (2026). MS (Mad Shift): Trellis-Coded GGUF Quantization for llama.cpp (Version 1.0). Zenodo. https://doi.org/10.5281/zenodo.24145285

Please cite MS (Mad Shift). MS uses the bitshift-trellis construction and the 1MAD decoding function described in QTIP; please cite QTIP for them:

@misc{priester2026madshift,
  author       = {Matthew Priester},
  title        = {{MS (Mad Shift)}: trellis-coded GGUF quantization for llama.cpp},
  year         = {2026},
  howpublished = {\url{https://github.com/devvexus/ms-llama.cpp}}
}
@inproceedings{tseng2024qtip,
  title     = {{QTIP}: Quantization with Trellises and Incoherence Processing},
  author    = {Tseng, Albert and Sun, Qingyao and Hou, David and De Sa, Christopher},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2024}
}

License

The model files are licensed under Apache 2.0, like the base model Qwen/Qwen3.5-2B (Copyright 2026 Alibaba Cloud): see LICENSE and NOTICE. They are modified (quantized) versions of Qwen3.5-2B. The MS engine, ms-llama.cpp, is MIT-licensed. MS (Mad Shift) was created by Matthew Priester; please keep the NOTICE file when you redistribute these files. Checksums: SHA256SUMS.

This project is not affiliated with or endorsed by ggml-org/llama.cpp, Qwen/Alibaba, unsloth or NVIDIA. Other product names (Claude Code, Ollama, LM Studio and others) are trademarks of their owners, named only to describe compatibility.

Downloads last month
2,834
GGUF
Model size
2B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

We're not able to determine the quantization variants.

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for devvexus/Qwen3.5-2B-MS

Finetuned
Qwen/Qwen3.5-2B
Quantized
(339)
this model

Collection including devvexus/Qwen3.5-2B-MS

Paper for devvexus/Qwen3.5-2B-MS