Chinese-Alpaca-2-7B-16K-GGUF

This repository contains the GGUF-v3 models (llama.cpp compatible) for Chinese-Alpaca-2-7B-16K.

Performance

Metric: PPL, lower is better

Quant	original	imatrix (-im)
Q2_K	11.8181 +/- 0.16402	13.6285 +/- 0.19294
Q3_K	9.5596 +/- 0.13369	9.3748 +/- 0.13108
Q4_0	9.6480 +/- 0.13459	-
Q4_K	8.9622 +/- 0.12507	8.9229 +/- 0.12467
Q5_0	8.9274 +/- 0.12485	-
Q5_K	8.8370 +/- 0.12353	8.8221 +/- 0.12348
Q6_K	8.7830 +/- 0.12290	8.7695 +/- 0.12260
Q8_0	8.7644 +/- 0.12261	-
F16	8.7676 +/- 0.12268	-

The model with -im suffix is generated with important matrix, which has generally better performance (not always though).

Others

For Hugging Face version, please see: https://huggingface.co/hfl/chinese-alpaca-2-7b-16k

Please refer to https://github.com/ymcui/Chinese-LLaMA-Alpaca-2/ for more details.