aya-23
Collection
4 items
β’
Updated
Llama.cpp imatrix quantization of CohereForAI/aya-23-35B
Original Model: CohereForAI/aya-23-35B
Original dtype: FP16
(float16
)
Quantized by: llama.cpp b2998
IMatrix dataset: here
Status: β
Available
Link: here
Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
---|---|---|---|---|---|
aya-23-35B.Q8_0/* | Q8_0 | 37.18GB | β Available | βͺ No | β Yes |
aya-23-35B.Q6_K.gguf | Q6_K | 28.71GB | β Available | βͺ No | π¦ No |
aya-23-35B.Q4_K.gguf | Q4_K | 21.53GB | β Available | π’ Yes | π¦ No |
aya-23-35B.Q3_K.gguf | Q3_K | 17.62GB | β Available | π’ Yes | π¦ No |
aya-23-35B.Q2_K.gguf | Q2_K | 13.82GB | β Available | π’ Yes | π¦ No |
Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
---|---|---|---|---|---|
aya-23-35B.FP16/* | F16 | 69.97GB | β Available | βͺ No | β Yes |
aya-23-35B.Q5_K.gguf | Q5_K | 25.01GB | β Available | βͺ No | π¦ No |
aya-23-35B.Q5_K_S.gguf | Q5_K_S | 24.34GB | β Available | βͺ No | π¦ No |
aya-23-35B.Q4_K_S.gguf | Q4_K_S | 20.38GB | β Available | π’ Yes | π¦ No |
aya-23-35B.Q3_K_L.gguf | Q3_K_L | 19.15GB | β Available | π’ Yes | π¦ No |
aya-23-35B.Q3_K_S.gguf | Q3_K_S | 15.86GB | β Available | π’ Yes | π¦ No |
aya-23-35B.Q2_K_S.gguf | Q2_K_S | 12.74GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ4_NL.gguf | IQ4_NL | 20.23GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ4_XS.gguf | IQ4_XS | 19.20GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ3_M.gguf | IQ3_M | 16.70GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ3_S.gguf | IQ3_S | 15.86GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ3_XS.gguf | IQ3_XS | 15.09GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ3_XXS.gguf | IQ3_XXS | 13.83GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ2_M.gguf | IQ2_M | 12.68GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ2_S.gguf | IQ2_S | 11.84GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ2_XS.gguf | IQ2_XS | 11.10GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ2_XXS.gguf | IQ2_XXS | 10.19GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ1_M.gguf | IQ1_M | 9.15GB | β Available | π’ Yes | π¦ No |
aya-23-35B.IQ1_S.gguf | IQ1_S | 8.52GB | β Available | π’ Yes | π¦ No |
First, make sure you have hugginface-cli installed:
pip install -U "huggingface_hub[cli]"
Then, you can target the specific file you want:
huggingface-cli download legraphista/aya-23-35B-IMat-GGUF --include "aya-23-35B.Q8_0.gguf" --local-dir ./
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
huggingface-cli download legraphista/aya-23-35B-IMat-GGUF --include "aya-23-35B.Q8_0/*" --local-dir aya-23-35B.Q8_0
# see FAQ for merging GGUF's
According to this investigation, it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).
gguf-split
availablegguf-split
, navigate to https://github.com/ggerganov/llama.cpp/releasesgguf-split
aya-23-35B.Q8_0
)gguf-split --merge aya-23-35B.Q8_0/aya-23-35B.Q8_0-00001-of-XXXXX.gguf aya-23-35B.Q8_0.gguf
gguf-split
to the first chunk of the split.Got a suggestion? Ping me @legraphista!
Base model
CohereForAI/aya-23-35B