README.md

---
license: apache-2.0
---

This repository contains improved Mixtral-8x7B quantized models in GGUF format for use with `llama.cpp`. The models are fully compatible with the oficial `llama.cpp` release and can be used out-of-the-box.

The table shows a comparison between these models and the current `llama.cpp` quantization approach using Wikitext perplexities for a context length of 512 tokens.
The "Quantization Error" columns in the table are defined as `(PPL(quantized model) - PPL(int8))/PPL(int8)`.
Running the full `fp16` Mixtral8x7b model on the systems I have available takes too long, so I'm comparing against the 8-bit quantized model, where I get `PPL = 4.1049`.
From past experience the 8-bit quantization should be basically equivalent to `fp16`. 

| Quantization | Model file | PPL(llama.cpp) | Quantization Error  | PPL(new quants) | Quantization Error |
|--:|--:|--:|--:|--:|--:|
|Q2_K | mixtral-8x7b-q2k.gguf | 7.4660 | 81.9% | 5.0576 | 23.2% |
|Q3_K_S | mixtral-8x7b-q3k-small.gguf | 4.4601 | 8.65% | 4.3848 | 6.82% |
|Q3_K_M| mixtral-8x7b-q3k-medium.gguf | 4.4194 | 7.66% | 4.2884 | 4.47% |
|Q4_K_S| mixtral-8x7b-q4k-small.gguf | 4.2523 | 3.59% | 4.1764 | 1.74% |
|Q4_K_M| mistral-8x7b-q4k-medium.gguf | 4.2523 | 3.59% | 4.1652 | 1.47% |
|Q5_K_S | mixtral-7b-q5k-small.gguf | 4.1395 | 0.84% | 4.1278 | 0.56% |
|Q4_0 | mixtral-8x7b-q40.gguf | 4.2232 | 2.88% | 4.2001 | 2.32% |
|Q4_1 | mistral-8x7b-q41.gguf | 4.2547 | 3.65% | 4.1713 | 1.62% |
|Q5_0 | mistral-8x7b-q50.gguf | 4.1426 | 0.92% | 4.1335 | 0.70% |