|
--- |
|
base_model: mistral-community/Mixtral-8x22B-v0.1 |
|
license: apache-2.0 |
|
--- |
|
|
|
# Importance-Matrix quantizations of Mixtral-8x22B-v0.1 💫 |
|
|
|
the imatrix.dat file was calcuated over 1000 chunks with wikitext.train.raw( included ) |
|
|
|
Wrote a bit of custom c++ to avoid quantizing certain layers, tested fully compatible with llama.cpp as of 10April2024. |
|
|
|
To put it all asa single file ( this is not needed with llama.cpp as it will autodetect the chunks but can help troubleshooting ollama) |
|
|
|
``` |
|
cat mix4ns-0000* > mix4ns.gguf |
|
|
|
``` |
|
careful this can take 5 minutes or up to 10-15 on slow instances, check progress with ls -la |
|
|
|
# Run with llama.cpp |
|
|
|
``` |
|
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp/ && make -j |
|
|
|
./main -m ~/mix4ns-00001-of-00005.gguf -n 256 -t 64 --temp 0.2 --color -p "How to build a city on mars via aldrin cycler orgbits?" |
|
|
|
``` |
|
# Perplexity benchmarks |
|
|
|
Command I used to run these on 48 core CPU only machine, you can add -ngl 16 to offload 16 layers or more to gpu on your own. |
|
|
|
```./perplexity -m ~/mix4xs.gguf -f wiki.test.raw --chunks 12 -t 48 ``` |
|
|
|
The results are interesting. quantizing from hf-bf16 folder to f16 gguf adds a bit of loss (increases perplexity). |
|
I've noticed on smaller models that going straight from huggingface repo folder to 8bit via using python convert.py --outtype q8_0 produces less perplexity than going hf-f16-q8_0. |
|
What's even more interesting is that quantizing TWICE (hf-q8_0 and then q8_0-imatrix) also produces better perplexity compared to regular f16gguf to imatrix. |
|
|
|
All you need to pay attention to is the final value PPL = 2.2585 in this case that of a regular 8bit |
|
|
|
# NOT ALL 8 BIT ARE CREATED EQUAL, this took 9 hours to convert to 8bit on a 64core cpu 256GB-RAM (8channel DDR5) |
|
|
|
 |
|
|
|
Even though the file is a tiny bit slower, it gets a tiny bit lower perplexity. |
|
It looks like nothing here over 12 chunks, and 2.2584-mix8ns vs 2.2585-mix8 regular q8_0 |
|
but past testing on smaller models and 100+ chunks has shown this difference to be a bit more pronounced |
|
``` |
|
perplexity regular q8_0 (from f16): 126.35 seconds per pass - ETA 6.32 minutes |
|
[1]2.6256,[2]3.1043,[3]3.6463,[4]3.2092,[5]2.6847,[6]2.4791,[7]2.3112,[8]2.2502,[9]2.2858,[10]2.2690,[11]2.2693,[12]2.2585, |
|
Final estimate: PPL = 2.2585 +/- 0.06534 |
|
|
|
perplexity q8_0 (slow convert.py from hf): 96.86 seconds per pass - ETA 4.83 minutes |
|
[1]2.6191,[2]3.1045,[3]3.6551,[4]3.2302,[5]2.6990,[6]2.4908,[7]2.3167,[8]2.2541,[9]2.2877,[10]2.2682,[11]2.2685,[12]2.2584, |
|
Final estimate: PPL = 2.2584 +/- 0.06514 |
|
|
|
perplexity regular iq4_xs (no imatrix): 91.53 seconds per pass |
|
[1]2.6966,[2]3.1749,[3]3.6972,[4]3.2577,[5]2.7905,[6]2.6097,[7]2.4536,[8]2.4001,[9]2.4469,[10]2.4219,[11]2.4366,[12]2.4367, |
|
Final estimate: PPL = 2.4367 +/- 0.07218 |
|
|
|
perplexity regular q4_km (no imatrix): 108.59 seconds per pass |
|
[1]2.6100,[2]3.1304,[3]3.6897,[4]3.3500,[5]2.8118,[6]2.5992,[7]2.4349,[8]2.3816,[9]2.4174,[10]2.3959,[11]2.3988,[12]2.3976, |
|
Final estimate: PPL = 2.3976 +/- 0.07111 |
|
|
|
perplexity EdgeQuant iq4-ns (no imatrix) 84.45 seconds per pass - FILESIZE 77258 MB |
|
[1]2.7195,[2]3.1821,[3]3.7177,[4]3.3017,[5]2.8012,[6]2.6034,[7]2.4318,[8]2.3747,[9]2.4160,[10]2.3931,[11]2.4023,[12]2.4013, |
|
Final estimate: PPL = 2.4013 +/- 0.07116 |
|
|
|
perplexity EdgeQuant iq4-ns (WITH imatrix) 82.76 seconds per pass - FILESIZE 73636 MB ( mix4ns.gguf ) //BEST ONE FOR 80GB CARD |
|
[1]2.7166,[2]3.1720,[3]3.6988,[4]3.3195,[5]2.7949,[6]2.5862,[7]2.4186,[8]2.3621,[9]2.3981,[10]2.3876,[11]2.3971,[12]2.3973, |
|
Final estimate: PPL = 2.3973 +/- 0.07080 |
|
|
|
perplexity 2K (no imatrix) 207.70 seconds per pass - FILESIZE 47564MB (mix2k-noimatrix-but-usable-reference.gguf) |
|
[1]2.9401,[2]3.4224,[3]4.0174,[4]3.8503,[5]3.5607,[6]3.4449,[7]3[9]3.5589,[10]3.6546,[11]3.7810,[12]3.7733, |
|
Final estimate: PPL = 3.7733 +/- 0.13299 |
|
``` |
|
|
|
 |
|
|
|
command to run these was: |
|
``` |
|
./main -m mix4ns.gguf -n 256 -t 48 --temp 0.5 --color -p "How to build a city on mars via shipping through aldrin cycler orbits?" |
|
|
|
``` |
|
|
|
|
