metadata

license: apache-2.0
tags:
  - mixtral
  - conversational
  - finetune

Model Card for Cerebrum-1.0-8x7b-imatrix-GGUF

Quantized from https://huggingface.co/AetherResearch/Cerebrum-1.0-8x7b using llama.cpp commit 46acb3676718b983157058aecf729a2064fc7d34 utilizing an importance matrix.

Quants will be upload with slow german internet so they will appear 1 by 1, stay tuned.

imatrix generated with:

./imatrix -ofreq 4 -b 512 -c 512 -t 14 --chunks 24 -m ../models/Cerebrum-1.0-8x7b-GGUF/cerebrum-1.0-8x7b-Q8_0.gguf -f ./groups_merged.txt

Sadly this means the imatrix is generated from the Q8 instead of the unquantized f16, like it should be, sadly I can't get it to work with the f16 on my machine at the moment. It should still improve the performance of the quants though.