Qwen2-57B-A14B-Instruct-GGUF

Llama.cpp static quantization of Qwen/Qwen2-57B-A14B-Instruct

Original Model: Qwen/Qwen2-57B-A14B-Instruct
Original dtype: BF16 (bfloat16)
Quantized by: https://github.com/ggerganov/llama.cpp/tree/master
IMatrix dataset: here


Files

Common Quants

Filename Quant type File Size Status Uses IMatrix Is Split
Qwen2-57B-A14B-Instruct.Q8_0/* Q8_0 61.02GB βœ… Available βšͺ Static βœ‚ Yes
Qwen2-57B-A14B-Instruct.Q6_K/* Q6_K 47.12GB βœ… Available βšͺ Static βœ‚ Yes
Qwen2-57B-A14B-Instruct.Q4_K.gguf Q4_K 34.85GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.Q3_K.gguf Q3_K 27.51GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.Q2_K.gguf Q2_K 21.06GB βœ… Available βšͺ Static πŸ“¦ No

All Quants

Filename Quant type File Size Status Uses IMatrix Is Split
Qwen2-57B-A14B-Instruct.BF16/* BF16 114.84GB βœ… Available βšͺ Static βœ‚ Yes
Qwen2-57B-A14B-Instruct.FP16/* F16 114.84GB βœ… Available βšͺ Static βœ‚ Yes
Qwen2-57B-A14B-Instruct.Q8_0/* Q8_0 61.02GB βœ… Available βšͺ Static βœ‚ Yes
Qwen2-57B-A14B-Instruct.Q6_K/* Q6_K 47.12GB βœ… Available βšͺ Static βœ‚ Yes
Qwen2-57B-A14B-Instruct.Q5_K.gguf Q5_K 40.80GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.Q5_K_S.gguf Q5_K_S 39.57GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.Q4_K.gguf Q4_K 34.85GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.Q4_K_S.gguf Q4_K_S 32.71GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.IQ4_NL.gguf IQ4_NL 32.72GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.IQ4_XS.gguf IQ4_XS 31.00GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.Q3_K.gguf Q3_K 27.51GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.Q3_K_L.gguf Q3_K_L 29.79GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.Q3_K_S.gguf Q3_K_S 24.91GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.IQ3_M.gguf IQ3_M 25.23GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.IQ3_S.gguf IQ3_S 24.92GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.IQ3_XS.gguf IQ3_XS 23.60GB βœ… Available βšͺ Static πŸ“¦ No
Qwen2-57B-A14B-Instruct.Q2_K.gguf Q2_K 21.06GB βœ… Available βšͺ Static πŸ“¦ No

Downloading using huggingface-cli

If you do not have hugginface-cli installed:

pip install -U "huggingface_hub[cli]"

Download the specific file you want:

huggingface-cli download legraphista/Qwen2-57B-A14B-Instruct-GGUF --include "Qwen2-57B-A14B-Instruct.Q8_0.gguf" --local-dir ./

If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:

huggingface-cli download legraphista/Qwen2-57B-A14B-Instruct-GGUF --include "Qwen2-57B-A14B-Instruct.Q8_0/*" --local-dir ./
# see FAQ for merging GGUF's

Inference

Simple chat template

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{user_prompt}<|im_end|>
<|im_start|>assistant
{assistant_response}<|im_end|>
<|im_start|>user
{next_user_prompt}<|im_end|>

Chat template with system prompt

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_prompt}<|im_end|>
<|im_start|>assistant
{assistant_response}<|im_end|>
<|im_start|>user
{next_user_prompt}<|im_end|>

Llama.cpp

llama.cpp/main -m Qwen2-57B-A14B-Instruct.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"

FAQ

Why is the IMatrix not applied everywhere?

According to this investigation, it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).

How do I merge a split GGUF?

  1. Make sure you have gguf-split available
  2. Locate your GGUF chunks folder (ex: Qwen2-57B-A14B-Instruct.Q8_0)
  3. Run gguf-split --merge Qwen2-57B-A14B-Instruct.Q8_0/Qwen2-57B-A14B-Instruct.Q8_0-00001-of-XXXXX.gguf Qwen2-57B-A14B-Instruct.Q8_0.gguf
    • Make sure to point gguf-split to the first chunk of the split.

Got a suggestion? Ping me @legraphista!

Downloads last month
660
GGUF
Model size
57.4B params
Architecture
qwen2moe

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Examples
Inference API (serverless) has been turned off for this model.

Model tree for legraphista/Qwen2-57B-A14B-Instruct-GGUF

Quantized
(6)
this model