Hugging Face's logo

legraphista
/
internlm2-math-plus-mixtral8x22b-IMat-GGUF

Text Generation
GGUF
English
Chinese
math
quantized
GGUF
imatrix
quantization
imat
static
Model card Files Community
internlm2-math-plus-mixtral8x22b-IMat-GGUF / README.md
legraphista's picture
legraphista
Upload README.md with huggingface_hub
a501ea8 verified
preview code
|
raw
history blame contribute delete
No virus
10.7 kB
metadata
base_model: internlm/internlm2-math-plus-mixtral8x22b
inference: false
language:
  - en
  - zh
library_name: gguf
license: other
pipeline_tag: text-generation
quantized_by: legraphista
tags:
  - math
  - quantized
  - GGUF
  - imatrix
  - quantization
  - imat
  - imatrix
  - static

internlm2-math-plus-mixtral8x22b-IMat-GGUF

Llama.cpp imatrix quantization of internlm/internlm2-math-plus-mixtral8x22b

Original Model: internlm/internlm2-math-plus-mixtral8x22b
Original dtype: BF16 (bfloat16)
Quantized by: llama.cpp b3010
IMatrix dataset: here

Files

IMatrix

Status: ✅ Available
Link: here

Common Quants

Filename Quant type File Size Status Uses IMatrix Is Split
internlm2-math-plus-mixtral8x22b.Q8_0/* Q8_0 149.42GB ✅ Available ⚪ Static ✂ Yes
internlm2-math-plus-mixtral8x22b.Q6_K/* Q6_K 115.54GB ✅ Available ⚪ Static ✂ Yes
internlm2-math-plus-mixtral8x22b.Q4_K/* Q4_K 85.59GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.Q3_K/* Q3_K 67.80GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.Q2_K/* Q2_K 52.11GB ✅ Available 🟢 IMatrix ✂ Yes

All Quants

Filename Quant type File Size Status Uses IMatrix Is Split
internlm2-math-plus-mixtral8x22b.BF16/* BF16 281.25GB ✅ Available ⚪ Static ✂ Yes
internlm2-math-plus-mixtral8x22b.FP16/* F16 281.25GB ✅ Available ⚪ Static ✂ Yes
internlm2-math-plus-mixtral8x22b.Q5_K/* Q5_K 99.97GB ✅ Available ⚪ Static ✂ Yes
internlm2-math-plus-mixtral8x22b.Q5_K_S/* Q5_K_S 96.98GB ✅ Available ⚪ Static ✂ Yes
internlm2-math-plus-mixtral8x22b.Q4_K_S/* Q4_K_S 80.48GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.Q3_K_L/* Q3_K_L 72.59GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.Q3_K_S/* Q3_K_S 61.50GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.Q2_K_S/* Q2_K_S 48.09GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.IQ4_NL/* IQ4_NL 79.78GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.IQ4_XS/* IQ4_XS 75.48GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.IQ3_M/* IQ3_M 64.50GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.IQ3_S/* IQ3_S 61.50GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.IQ3_XS/* IQ3_XS 58.23GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.IQ3_XXS/* IQ3_XXS 54.91GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.IQ2_M/* IQ2_M 46.71GB ✅ Available 🟢 IMatrix ✂ Yes
internlm2-math-plus-mixtral8x22b.IQ2_S.gguf IQ2_S 42.60GB ✅ Available 🟢 IMatrix 📦 No
internlm2-math-plus-mixtral8x22b.IQ2_XS.gguf IQ2_XS 42.00GB ✅ Available 🟢 IMatrix 📦 No
internlm2-math-plus-mixtral8x22b.IQ2_XXS.gguf IQ2_XXS 37.89GB ✅ Available 🟢 IMatrix 📦 No
internlm2-math-plus-mixtral8x22b.IQ1_M.gguf IQ1_M 32.74GB ✅ Available 🟢 IMatrix 📦 No
internlm2-math-plus-mixtral8x22b.IQ1_S.gguf IQ1_S 29.65GB ✅ Available 🟢 IMatrix 📦 No

Downloading using huggingface-cli

If you do not have hugginface-cli installed:

pip install -U "huggingface_hub[cli]"

Download the specific file you want:

huggingface-cli download legraphista/internlm2-math-plus-mixtral8x22b-IMat-GGUF --include "internlm2-math-plus-mixtral8x22b.Q8_0.gguf" --local-dir ./

If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:

huggingface-cli download legraphista/internlm2-math-plus-mixtral8x22b-IMat-GGUF --include "internlm2-math-plus-mixtral8x22b.Q8_0/*" --local-dir internlm2-math-plus-mixtral8x22b.Q8_0
# see FAQ for merging GGUF's

Inference

Simple chat template 

<s>[INST] Can you provide ways to eat combinations of bananas and dragonfruits? [/INST] Sure! Here are some ways to eat bananas and dragonfruits together:
 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey.
 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey. </s><s>[INST] What about solving an 2x + 3 = 7 equation? [/INST]

Chat template with system prompt 

<s>[INST] <<SYS>>
You are a helpful AI.
<</SYS>>

Can you provide ways to eat combinations of bananas and dragonfruits? [/INST] Sure! Here are some ways to eat bananas and dragonfruits together:
 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey.
 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey. </s><s>[INST] What about solving an 2x + 3 = 7 equation? [/INST]

Llama.cpp 

llama.cpp/main -m internlm2-math-plus-mixtral8x22b.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"

FAQ

Why is the IMatrix not applied everywhere?

According to this investigation, it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).

How do I merge a split GGUF?

  1. Make sure you have gguf-split available
  2. Locate your GGUF chunks folder (ex: internlm2-math-plus-mixtral8x22b.Q8_0)
  3. Run gguf-split --merge internlm2-math-plus-mixtral8x22b.Q8_0/internlm2-math-plus-mixtral8x22b.Q8_0-00001-of-XXXXX.gguf internlm2-math-plus-mixtral8x22b.Q8_0.gguf
    • Make sure to point gguf-split to the first chunk of the split.

Got a suggestion? Ping me @legraphista!