Instructions to use devvexus/Qwen3.5-2B-MS with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use devvexus/Qwen3.5-2B-MS with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf devvexus/Qwen3.5-2B-MS:F16 # Run inference directly in the terminal: llama cli -hf devvexus/Qwen3.5-2B-MS:F16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf devvexus/Qwen3.5-2B-MS:F16 # Run inference directly in the terminal: llama cli -hf devvexus/Qwen3.5-2B-MS:F16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf devvexus/Qwen3.5-2B-MS:F16 # Run inference directly in the terminal: ./build/bin/llama cli -hf devvexus/Qwen3.5-2B-MS:F16
Use Docker
docker model run hf.co/devvexus/Qwen3.5-2B-MS:F16
- Llama new
- LM Studio
- Jan
- vLLM
How to use devvexus/Qwen3.5-2B-MS with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "devvexus/Qwen3.5-2B-MS" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "devvexus/Qwen3.5-2B-MS", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/devvexus/Qwen3.5-2B-MS:F16
- Ollama
How to use devvexus/Qwen3.5-2B-MS with Ollama:
ollama run hf.co/devvexus/Qwen3.5-2B-MS:F16
- Unsloth Desktop
- Pi
How to use devvexus/Qwen3.5-2B-MS with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf devvexus/Qwen3.5-2B-MS:F16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "devvexus/Qwen3.5-2B-MS:F16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use devvexus/Qwen3.5-2B-MS with Docker Model Runner:
docker model run hf.co/devvexus/Qwen3.5-2B-MS:F16
- Lemonade
How to use devvexus/Qwen3.5-2B-MS with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull devvexus/Qwen3.5-2B-MS:F16
Run and chat with the model
lemonade run user.Qwen3.5-2B-MS-F16
List all available models
lemonade list
- Hermes Agent
How to use devvexus/Qwen3.5-2B-MS with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf devvexus/Qwen3.5-2B-MS:F16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default devvexus/Qwen3.5-2B-MS:F16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use devvexus/Qwen3.5-2B-MS with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf devvexus/Qwen3.5-2B-MS:F16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "devvexus/Qwen3.5-2B-MS:F16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Same size as unsloth. Up to 9.4x lower quality loss on English text.
MS (Mad Shift) quants of Qwen/Qwen3.5-2B, for the ms-llama.cpp fork.
- Accuracy: 4.5-9.4x lower KL divergence than unsloth's smallest file of the same or larger size, for every one of the seven files (1).
- On English text, Q4 size beats Q6_K: MS4.3 (1.30 GB, between unsloth's Q4_K_M and UD-Q4_K_XL in size) is closer to the original model than unsloth's Q6_K (1.57 GB): 0.00219 vs 0.00277 (1). MS3.5 beats Q4_K_M from 190 MB smaller.
- Chinese (not in the calibration): MS still beats every paired unsloth file on held-out Chinese text, by 1.1-5.0x (4.5-9.4x on English). Chinese was deliberately left out of the calibration (English came first; future conversions will be multilingual). MS5 is also ahead of the larger Q6_K on Chinese; MS4.3 is behind it.
- Top-1: the most likely next token matches the original BF16 model's 91.3-98.4% of the time, vs 80.8-96.4% for those same unsloth files: 54-73% fewer mismatches (1). MS4.3 98.13% vs Q6_K 97.51%.
- vs EXL3: for the same KL divergence, MS files are 39-59% of the size of EXL3 files (UnstableLlama's EXL3 files and one 6.00 conversion of our own); every MS file is smaller than every EXL3 file measured.
- Memory: every file runs in about 3 GB or less at a 32k context (estimate).
(1) KL divergence from the original BF16 model, mean over four English texts; top-1 = how often the quantized model's most likely next token is the BF16 model's. All 19 of unsloth's Qwen3.5-2B GGUFs (unsloth/Qwen3.5-2B-GGUF, revision f6d5376b), measured on stock llama.cpp.
MS (Mad Shift) was created by Matthew Priester (GitHub, ORCID). If you use these files or build on the format, please credit MS (Mad Shift) by Matthew Priester.
Run it now
MS files need the MS fork of llama.cpp, ms-llama.cpp: stock llama.cpp, Ollama and LM Studio cannot load them.
1. Install the engine (Windows, NVIDIA GPU; once; PowerShell):
$d="$env:LOCALAPPDATA\ms-llama"; $z=if($env:PROCESSOR_ARCHITECTURE -eq "ARM64"){"arm64-cuda"}else{"x64-cuda12"}; ni -ItemType Directory -Force $d|Out-Null; curl.exe -fL -o "$d\ms.zip" "https://github.com/devvexus/ms-llama.cpp/releases/latest/download/ms-llama-win-$z.zip"; if($?){tar -xf "$d\ms.zip" -C $d}
Linux and macOS: build from source, see the ms-llama.cpp README.
2. Run any file (each needs about 3 GB of memory or less). It downloads once, then serves a chat page and an API on http://localhost:8080. Replace <file> with a name from the table:
& "$env:LOCALAPPDATA\ms-llama\llama-server.exe" -hf devvexus/Qwen3.5-2B-MS --hf-file <file> -c 32768 -np 1 --port 8080
-np 1 serves one chat at a time: every extra parallel slot needs its own copy of the model's recurrent state, which can push layers off the GPU and slow generation down a lot. After the first run, you can add --offline so the server starts without checking Hugging Face; that check can take a few minutes. On Windows the first download may print 'failed to create symlink ... switching to degraded mode'; it is harmless: the Hugging Face cache just can't use symlinks, so updated files may take extra disk space.
<file> |
Size |
|---|---|
MS5/Qwen3.5-2B-MS5.gguf |
1441 MB |
MS4.3/Qwen3.5-2B-MS4.3.gguf |
1301 MB |
MS4/Qwen3.5-2B-MS4.gguf |
1262 MB |
MS3.7/Qwen3.5-2B-MS3.7.gguf |
1165 MB |
MS3.5/Qwen3.5-2B-MS3.5.gguf |
1088 MB |
MS3/Qwen3.5-2B-MS3.gguf |
858 MB |
MS2.7/Qwen3.5-2B-MS2.7.gguf |
775 MB |
A .gguf you downloaded yourself: -m <file>.gguf.
Images (optional). The vision encoder, vision/mmproj-Qwen3.5-2B-F16.gguf (0.67 GB, F16), lets the model read images. -hf does not load it by itself: add --mmproj-url with its address, or --mmproj <file> for a downloaded copy. It works with every MS file and adds about 0.9-1.1 GiB of GPU memory when on (nothing when off). The vision encoder is not MS-encoded and was not part of our evaluation; MS-encoding it is future work. For example:
& "$env:LOCALAPPDATA\ms-llama\llama-server.exe" -hf devvexus/Qwen3.5-2B-MS --hf-file <file> -c 32768 -np 1 --port 8080 --mmproj-url https://huggingface.co/devvexus/Qwen3.5-2B-MS/resolve/main/vision/mmproj-Qwen3.5-2B-F16.gguf
Use it in your apps
While the server runs:
- Claude Code:
$env:ANTHROPIC_BASE_URL="http://localhost:8080"; $env:ANTHROPIC_AUTH_TOKEN="local"; claudeClaude Code's own prompt is large (about 24k tokens in our test with Claude Code 2.1.325), so give the server more context for it:-c 49152instead of-c 32768. - OpenAI-compatible apps: base URL
http://localhost:8080/v1, any API key. - Apps that run llama.cpp themselves: point them at
%LOCALAPPDATA%\ms-llama. - Ollama and LM Studio can't load MS files.
Files
| MS file | Size | KLD | unsloth file (size) | its KLD | MS is | Top-1, MS / unsloth |
|---|---|---|---|---|---|---|
| MS5 | 1441 MB | 0.00142 | UD-Q5_K_XL (1467 MB) | 0.00640 | 4.5x closer | 98.36% / 96.41% |
| MS4.3 | 1301 MB | 0.00219 | UD-Q4_K_XL (1340 MB) | 0.01472 | 6.7x closer | 98.13% / 94.87% |
| MS4 | 1262 MB | 0.00270 | Q4_K_M (1281 MB) | 0.02007 | 7.4x closer | 98.02% / 93.94% |
| MS3.7 | 1165 MB | 0.00376 | IQ4_XS (1173 MB) | 0.02756 | 7.3x closer | 97.52% / 92.92% |
| MS3.5 | 1088 MB | 0.00730 | Q3_K_M (1107 MB) | 0.06848 | 9.4x closer | 97.00% / 89.00% |
| MS3 | 858 MB | 0.02793 | UD-IQ2_M (860 MB) | 0.23063 | 8.3x closer | 93.32% / 80.79% |
| MS2.7 | 775 MB | 0.04361 | UD-IQ2_M (860 MB) | 0.23063 | 5.3x closer | 91.26% / 80.79% |
Each quant is in its own folder; the optional vision encoder is vision/mmproj-Qwen3.5-2B-F16.gguf (668 MB, F16, see Images). Sizes in MB (10^6 bytes). Each MS file is compared with the smallest unsloth file of the same or larger size (unsloth/Qwen3.5-2B-GGUF, revision f6d5376b). Neither the MS files nor unsloth's Qwen3.5-2B files contain the model's MTP layer, so file sizes compare directly. Checksums: SHA256SUMS.
Per-text KL divergence and every unsloth file
KL divergence per text and top-1 agreement (how often the quantized model's most likely next token is the BF16 model's):
| MS file | wiki | web | code | math | Top-1 |
|---|---|---|---|---|---|
| MS5 | 0.00154 | 0.00137 | 0.00138 | 0.00141 | 98.36% |
| MS4.3 | 0.00293 | 0.00201 | 0.00249 | 0.00131 | 98.13% |
| MS4 | 0.00323 | 0.00232 | 0.00364 | 0.00160 | 98.02% |
| MS3.7 | 0.00465 | 0.00358 | 0.00414 | 0.00268 | 97.52% |
| MS3.5 | 0.00985 | 0.00727 | 0.00857 | 0.00352 | 97.00% |
| MS3 | 0.03043 | 0.02851 | 0.02987 | 0.02293 | 93.32% |
| MS2.7 | 0.05064 | 0.04520 | 0.04303 | 0.03557 | 91.26% |
All of unsloth's files on the same eval (revision f6d5376b):
| unsloth file | Size (MB) | Mean KLD | wiki | web | code | math | Top-1 |
|---|---|---|---|---|---|---|---|
| UD-Q6_K_XL | 1864 | 0.00169 | 0.00166 | 0.00151 | 0.00255 | 0.00103 | 98.33% |
| Q6_K | 1575 | 0.00277 | 0.00320 | 0.00285 | 0.00299 | 0.00204 | 97.51% |
| UD-Q5_K_XL | 1467 | 0.00640 | 0.00763 | 0.00669 | 0.00684 | 0.00444 | 96.41% |
| Q5_K_M | 1435 | 0.00705 | 0.00790 | 0.00699 | 0.00855 | 0.00477 | 96.32% |
| Q5_K_S | 1385 | 0.00870 | 0.00963 | 0.00866 | 0.01009 | 0.00643 | 95.92% |
| UD-Q4_K_XL | 1340 | 0.01472 | 0.01663 | 0.01482 | 0.01721 | 0.01020 | 94.87% |
| Q4_1 | 1294 | 0.02533 | 0.02827 | 0.02442 | 0.03137 | 0.01725 | 93.37% |
| Q4_K_M | 1281 | 0.02007 | 0.02235 | 0.01963 | 0.02377 | 0.01455 | 93.94% |
| Q4_K_S | 1218 | 0.02402 | 0.02709 | 0.02416 | 0.02804 | 0.01678 | 93.48% |
| Q4_0 | 1215 | 0.03868 | 0.04405 | 0.03945 | 0.04255 | 0.02868 | 91.75% |
| IQ4_NL | 1213 | 0.02714 | 0.03131 | 0.02857 | 0.02965 | 0.01902 | 93.07% |
| IQ4_XS | 1173 | 0.02756 | 0.03171 | 0.02875 | 0.03052 | 0.01925 | 92.92% |
| UD-Q3_K_XL | 1159 | 0.04540 | 0.05203 | 0.04761 | 0.04888 | 0.03308 | 91.16% |
| Q3_K_M | 1107 | 0.06848 | 0.07982 | 0.06987 | 0.07449 | 0.04975 | 89.00% |
| Q3_K_S | 1031 | 0.11840 | 0.14122 | 0.12376 | 0.12050 | 0.08812 | 85.76% |
| UD-Q2_K_XL | 967 | 0.19697 | 0.22487 | 0.20646 | 0.19608 | 0.16048 | 82.04% |
| UD-IQ3_XXS | 932 | 0.13256 | 0.15108 | 0.13215 | 0.14557 | 0.10144 | 85.02% |
| UD-IQ2_M | 860 | 0.23063 | 0.25809 | 0.23265 | 0.24466 | 0.18711 | 80.79% |
| UD-IQ2_XXS | 768 | 0.59367 | 0.64371 | 0.57999 | 0.65169 | 0.49930 | 69.07% |
Medians and the worst chunk of every text, for every file: kld_results.json.
Memory per file
Memory needed to run each file fully on a GPU (or in RAM on CPU), approximate, with the default batch size. The context cache grows with the context length; a smaller -ub lowers the compute buffers.
| Quant | File size | Memory at 8k context | Memory at 32k context | Free memory to have (32k) |
|---|---|---|---|---|
| MS5 | 1441 MB | ~2.6 GB | ~2.9 GB | 4 GB |
| MS4.3 | 1301 MB | ~2.4 GB | ~2.7 GB | 3 GB |
| MS4 | 1262 MB | ~2.4 GB | ~2.7 GB | 3 GB |
| MS3.7 | 1165 MB | ~2.3 GB | ~2.6 GB | 3 GB |
| MS3.5 | 1088 MB | ~2.2 GB | ~2.5 GB | 3 GB |
| MS3 | 858 MB | ~2.0 GB | ~2.3 GB | 3 GB |
| MS2.7 | 775 MB | ~1.9 GB | ~2.2 GB | 3 GB |
Speed
On an NVIDIA RTX Spark N1X laptop GPU (GB10, Blackwell, 6144 cores), Windows 11 on ARM; MS and unsloth files both on ms-llama.cpp, each MS file against the smallest unsloth file of the same or larger size (and MS4.3 against Q6_K):
| MS file | unsloth file | Generation, tokens/s | Prompt processing, tokens/s |
|---|---|---|---|
| MS5 | UD-Q5_K_XL | 119.1 vs 120.9 (0.99x) | 10327 vs 11170 (0.92x) |
| MS4.3 | UD-Q4_K_XL | 126.5 vs 127.5 (0.99x) | 11305 vs 11936 (0.95x) |
| MS4.3 | Q6_K | 126.5 vs 117.5 (1.08x) | 10997 vs 10612 (1.04x) |
| MS4 | Q4_K_M | 127.5 vs 131.0 (0.97x) | 11412 vs 11815 (0.97x) |
| MS3.7 | IQ4_XS | 131.6 vs 136.4 (0.96x) | 11400 vs 12469 (0.91x) |
| MS3.5 | Q3_K_M | 135.4 vs 138.2 (0.98x) | 11406 vs 10191 (1.12x) |
| MS3 | UD-IQ2_M | 159.1 vs 154.4 (1.03x) | 11314 vs 9817 (1.15x) |
| MS2.7 | UD-IQ2_M | 168.2 vs 154.4 (1.09x) | 11359 vs 9905 (1.15x) |
Generation: chat benchmark without MTP (20 chat prompts x 256 tokens, greedy). Prompt processing: llama-bench -p 512 -n 0 -ngl 99, MS and unsloth files alternating in one session (median of 3).
EXL3
EXL3 (exllamav3) is a trellis format. We measured UnstableLlama's Qwen3.5-2B EXL3 files and one 6.00bpw conversion of our own (exllamav3 v0.0.28, default settings) against the same BF16 model:
| File | Size | Mean KLD | Top-1 |
|---|---|---|---|
| 2.10bpw | 1.76 | 0.274546 | 79.08% |
| 3.10bpw | 1.93 | 0.057093 | 90.58% |
| 4.00bpw | 2.09 | 0.016215 | 94.91% |
| 5.00bpw | 2.26 | 0.004471 | 97.19% |
| 6.00bpw, own conversion | 2.43 | 0.001518 | 98.26% |
| 6.00bpw | 2.43 | 0.001559 | 98.19% |
| 8.00bpw | 2.78 | 0.000433 | 98.83% |
Sizes are the file bytes needed for text generation (vision tower excluded), like the MS sizes. The EXL3 files were scored with exllamav3 on an RTX 4080 SUPER against a reference regenerated from the same BF16 file; on that machine the MS and unsloth files agree with this card's values within 1.4 standard errors on every text. exllamav3 with the BF16 weights scores 0.00042 against llama.cpp's BF16 logits (mean of the four texts); the EXL3 values include it. The chart shows the better of the two 6.00bpw files.
How the comparison was made
KL divergence (KLD) and top-1 agreement against the BF16 model, measured with llama-perplexity --kl-divergence (512-token context, 128 chunks per text, the second half of each chunk scored: 255 tokens per chunk, 32,640 per text) on four English texts that were not used to make the files: wiki (Wikipedia articles), web (FineWeb-Edu), code (real code patches from SWE-Gym) and math (MATH test problems and solutions). The headline number is the mean over the four texts; lower KLD is better. Top-1 agreement is how often the quantized model's most likely next token is the BF16 model's. The texts, with their sources and licenses, are published as devvexus/ms-eval-texts (the math text as a script that rebuilds it). Neither the MS files nor unsloth's Qwen3.5-2B files contain the model's MTP layer, so file sizes compare directly.
Why KL divergence: it compares the quantized model's whole next-token distribution with the original model's at every position of held-out text. It is not a task score: it measures how closely a file follows the original model on held-out text.
unsloth's files: unsloth/Qwen3.5-2B-GGUF at revision f6d5376b, measured on stock llama.cpp 36a0e2ffc. MS files: measured on ms-llama.cpp 6085028c3 with GGML_CUDA_MMMS_ACC=f32. Both against the same BF16 reference logits. All numbers are our own measurements.
A few text passages dominate their text's mean for every quant (for example drawing code inside a math solution).
Check it yourself: any English text works. Make reference logits with the BF16 model once, then compare any quant against them (MS files with the fork's build, other GGUFs with the same build or stock llama.cpp):
./build/bin/llama-perplexity -m Qwen3.5-2B-BF16.gguf -f text.txt -c 512 --kl-divergence-base base.kld
./build/bin/llama-perplexity -m Qwen3.5-2B-MS5.gguf -f text.txt -c 512 --kl-divergence-base base.kld --kl-divergence
What MS is
MS (Mad Shift) is a GGUF weight format for llama.cpp. MS uses the bitshift-trellis construction and the 1MAD decoding function described in QTIP (Tseng, Sun, Hou and De Sa, NeurIPS 2024).
Every large matrix is stored trellis-coded after a random Hadamard rotation (undone at run time); a tensor with k bits (k = 1..6) costs k + 0.0625 bits per weight. The fork has CPU and CUDA kernels for it. The files are made with the bitshift-trellis construction described in QTIP + our own calibration-aware, layer-by-layer conversion.
Names: an MSN file is built to beat unsloth's QN-class files at the same or a smaller size; N is a size class, not the average bits per weight.
How these files were made
- Conversion: the bitshift-trellis construction described in QTIP + our own calibration-aware, layer-by-layer conversion, from Qwen's original weights. No tensors are copied from other GGUFs.
- Make your own MS files: these files are made with the MS converter, open source (Apache 2.0): mad-shift-converter. A lightweight MS encoder for ms-llama.cpp's
llama-quantize, built for fast conversion, is in development, so that anyone can make MS files of most models on their own hardware. - Embedding table (shared with the output head): MS-coded; from MS4.3 down, the most frequent tokens get higher precision.
- Vision encoder:
vision/mmproj-Qwen3.5-2B-F16.gguf, converted from Qwen's original weights with llama.cpp'sconvert_hf_to_gguf.py(--mmproj --outtype f16); not MS-encoded.
Limitations
- The vision encoder is not MS-encoded and was not part of our evaluation (see Images).
- Needs the MS fork; GPU (CUDA) and CPU (x86 and ARM) are supported there.
- Accuracy was measured on four English texts and one held-out Chinese text. The split embedding table keeps rarely used tokens (much of the non-English vocabulary) at lower precision, so the lead is smaller on Chinese (see Chinese above).
Citation
Technical report: Priester, M. (2026). MS (Mad Shift): Trellis-Coded GGUF Quantization for llama.cpp (Version 1.0). Zenodo. https://doi.org/10.5281/zenodo.24145285
Please cite MS (Mad Shift). MS uses the bitshift-trellis construction and the 1MAD decoding function described in QTIP; please cite QTIP for them:
@misc{priester2026madshift,
author = {Matthew Priester},
title = {{MS (Mad Shift)}: trellis-coded GGUF quantization for llama.cpp},
year = {2026},
howpublished = {\url{https://github.com/devvexus/ms-llama.cpp}}
}
@inproceedings{tseng2024qtip,
title = {{QTIP}: Quantization with Trellises and Incoherence Processing},
author = {Tseng, Albert and Sun, Qingyao and Hou, David and De Sa, Christopher},
booktitle = {Advances in Neural Information Processing Systems},
year = {2024}
}
License
The model files are licensed under Apache 2.0, like the base model
Qwen/Qwen3.5-2B (Copyright 2026 Alibaba Cloud): see LICENSE and NOTICE.
They are modified (quantized) versions of Qwen3.5-2B. The MS engine, ms-llama.cpp, is MIT-licensed. MS (Mad Shift) was
created by Matthew Priester; please keep the NOTICE file when you redistribute these files. Checksums: SHA256SUMS.
This project is not affiliated with or endorsed by ggml-org/llama.cpp, Qwen/Alibaba, unsloth or NVIDIA. Other product names (Claude Code, Ollama, LM Studio and others) are trademarks of their owners, named only to describe compatibility.
- Downloads last month
- 2,834
We're not able to determine the quantization variants.


