AutoCoder
Collection
2 items
β’
Updated
Llama.cpp imatrix quantization of Bin12345/AutoCoder_S_6.7B
Original Model: Bin12345/AutoCoder_S_6.7B
Original dtype: BF16
(bfloat16
)
Quantized by: llama.cpp b3010
IMatrix dataset: here
Status: β
Available
Link: here
Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
---|---|---|---|---|---|
AutoCoder_S_6.7B.Q8_0.gguf | Q8_0 | 7.16GB | β Available | βͺ Static | π¦ No |
AutoCoder_S_6.7B.Q6_K.gguf | Q6_K | 5.53GB | β Available | βͺ Static | π¦ No |
AutoCoder_S_6.7B.Q4_K.gguf | Q4_K | 4.08GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.Q3_K.gguf | Q3_K | 3.30GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.Q2_K.gguf | Q2_K | 2.53GB | β Available | π’ IMatrix | π¦ No |
Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
---|---|---|---|---|---|
AutoCoder_S_6.7B.BF16.gguf | BF16 | 13.48GB | β Available | βͺ Static | π¦ No |
AutoCoder_S_6.7B.FP16.gguf | F16 | 13.48GB | β Available | βͺ Static | π¦ No |
AutoCoder_S_6.7B.Q5_K.gguf | Q5_K | 4.79GB | β Available | βͺ Static | π¦ No |
AutoCoder_S_6.7B.Q5_K_S.gguf | Q5_K_S | 4.65GB | β Available | βͺ Static | π¦ No |
AutoCoder_S_6.7B.Q4_K_S.gguf | Q4_K_S | 3.86GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.Q3_K_L.gguf | Q3_K_L | 3.60GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.Q3_K_S.gguf | Q3_K_S | 2.95GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.Q2_K_S.gguf | Q2_K_S | 2.32GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ4_NL.gguf | IQ4_NL | 3.83GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ4_XS.gguf | IQ4_XS | 3.62GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ3_M.gguf | IQ3_M | 3.12GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ3_S.gguf | IQ3_S | 2.95GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ3_XS.gguf | IQ3_XS | 2.80GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ3_XXS.gguf | IQ3_XXS | 2.59GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ2_M.gguf | IQ2_M | 2.36GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ2_S.gguf | IQ2_S | 2.20GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ2_XS.gguf | IQ2_XS | 2.04GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ2_XXS.gguf | IQ2_XXS | 1.86GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ1_M.gguf | IQ1_M | 1.65GB | β Available | π’ IMatrix | π¦ No |
AutoCoder_S_6.7B.IQ1_S.gguf | IQ1_S | 1.53GB | β Available | π’ IMatrix | π¦ No |
If you do not have hugginface-cli installed:
pip install -U "huggingface_hub[cli]"
Download the specific file you want:
huggingface-cli download legraphista/AutoCoder_S_6.7B-IMat-GGUF --include "AutoCoder_S_6.7B.Q8_0.gguf" --local-dir ./
If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
huggingface-cli download legraphista/AutoCoder_S_6.7B-IMat-GGUF --include "AutoCoder_S_6.7B.Q8_0/*" --local-dir ./
# see FAQ for merging GGUF's
Human: Can you provide ways to eat combinations of bananas and dragonfruits?
Assistant: Sure! Here are some ways to eat bananas and dragonfruits together:
1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey.
2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey.<ο½endβofβsentenceο½>
Human: What about solving an 2x + 3 = 7 equation?
Assistant:
You are a helpful AI.
Human: Can you provide ways to eat combinations of bananas and dragonfruits?
Assistant: Sure! Here are some ways to eat bananas and dragonfruits together:
1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey.
2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey.<ο½endβofβsentenceο½>
Human: What about solving an 2x + 3 = 7 equation?
Assistant:
llama.cpp/main -m AutoCoder_S_6.7B.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"
According to this investigation, it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).
gguf-split
availablegguf-split
, navigate to https://github.com/ggerganov/llama.cpp/releasesgguf-split
AutoCoder_S_6.7B.Q8_0
)gguf-split --merge AutoCoder_S_6.7B.Q8_0/AutoCoder_S_6.7B.Q8_0-00001-of-XXXXX.gguf AutoCoder_S_6.7B.Q8_0.gguf
gguf-split
to the first chunk of the split.Got a suggestion? Ping me @legraphista!
Base model
Bin12345/AutoCoder_S_6.7B