Chinese-LLaMA-2-1.3B-GGUF

This repository contains the GGUF-v3 models (llama.cpp compatible) for Chinese-LLaMA-2-1.3B.

Performance

Metric: PPL, lower is better

Quant	original	imatrix (-im)
Q2_K	21.7422 +/- 0.47160	19.0472 +/- 0.42128
Q3_K	14.8538 +/- 0.32874	14.2966 +/- 0.31684
Q4_0	13.8498 +/- 0.30349	-
Q4_K	16.0222 +/- 0.35555	13.0832 +/- 0.28717
Q5_0	12.5743 +/- 0.27428	-
Q5_K	12.5865 +/- 0.27759	12.6305 +/- 0.27711
Q6_K	12.6588 +/- 0.27682	12.6944 +/- 0.27805
Q8_0	12.7243 +/- 0.27883	-
F16	12.6122 +/- 0.27616	-

The model with -im suffix is generated with important matrix, which has generally better performance (not always though).

Others

For Hugging Face version, please see: https://huggingface.co/hfl/chinese-llama-2-1.3b

Please refer to https://github.com/ymcui/Chinese-LLaMA-Alpaca-2/ for more details.