Hugging Face's logo

second-state
/
Baichuan2-13B-Chat-GGUF

Text Generation
Transformers
GGUF
English
Chinese
baichuan
custom_code
Model card Files Community
Xin Liu commited on
Commit
17c1789
•
1 Parent(s): dbe0df0

Update

Browse files

Signed-off-by: Xin Liu <sam@secondstate.io>

Files changed (1) hide show
  1. README.md +4 -2
README.md CHANGED
@@ -30,7 +30,7 @@ language:
30
 
31
  ## Run with LlamaEdge
32
 
33
- - LlamaEdge version: [v0.2.8](https://github.com/second-state/LlamaEdge/releases/tag/0.2.8) and above
34
 
35
  - Prompt template
36
 
@@ -63,7 +63,7 @@ language:
63
 
64
  | Name | Quant method | Bits | Size | Use case |
65
  | ---- | ---- | ---- | ---- | ----- |
66
- | [Baichuan2-13B-Chat-Q2_K.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q2_K.gguf) | Q2_K | 2 | 6.5 GB| smallest, significant quality loss - not recommended for most purposes |
67
  | [Baichuan2-13B-Chat-Q3_K_L.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q3_K_L.gguf) | Q3_K_L | 3 | 7.67 GB| small, substantial quality loss |
68
  | [Baichuan2-13B-Chat-Q3_K_M.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q3_K_M.gguf) | Q3_K_M | 3 | 7.24 GB| very small, high quality loss |
69
  | [Baichuan2-13B-Chat-Q3_K_S.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q3_K_S.gguf) | Q3_K_S | 3 | 6.77 GB| very small, high quality loss |
@@ -75,3 +75,5 @@ language:
75
  | [Baichuan2-13B-Chat-Q5_K_S.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q5_K_S.gguf) | Q5_K_S | 5 | 9.82 GB| large, low quality loss - recommended |
76
  | [Baichuan2-13B-Chat-Q6_K.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q6_K.gguf) | Q6_K | 6 | 12.1 GB| very large, extremely low quality loss |
77
  | [Baichuan2-13B-Chat-Q8_0.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q8_0.gguf) | Q8_0 | 8 | 14.8 GB| very large, extremely low quality loss - not recommended |
 
 
 
30
 
31
  ## Run with LlamaEdge
32
 
33
+ - LlamaEdge version: [v0.2.11](https://github.com/second-state/LlamaEdge/releases/tag/0.2.11) and above
34
 
35
  - Prompt template
36
 
 
63
 
64
  | Name | Quant method | Bits | Size | Use case |
65
  | ---- | ---- | ---- | ---- | ----- |
66
+ | [Baichuan2-13B-Chat-Q2_K.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q2_K.gguf) | Q2_K | 2 | 5.93 GB| smallest, significant quality loss - not recommended for most purposes |
67
  | [Baichuan2-13B-Chat-Q3_K_L.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q3_K_L.gguf) | Q3_K_L | 3 | 7.67 GB| small, substantial quality loss |
68
  | [Baichuan2-13B-Chat-Q3_K_M.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q3_K_M.gguf) | Q3_K_M | 3 | 7.24 GB| very small, high quality loss |
69
  | [Baichuan2-13B-Chat-Q3_K_S.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q3_K_S.gguf) | Q3_K_S | 3 | 6.77 GB| very small, high quality loss |
 
75
  | [Baichuan2-13B-Chat-Q5_K_S.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q5_K_S.gguf) | Q5_K_S | 5 | 9.82 GB| large, low quality loss - recommended |
76
  | [Baichuan2-13B-Chat-Q6_K.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q6_K.gguf) | Q6_K | 6 | 12.1 GB| very large, extremely low quality loss |
77
  | [Baichuan2-13B-Chat-Q8_0.gguf](https://huggingface.co/second-state/Baichuan2-13B-Chat-GGUF/blob/main/Baichuan2-13B-Chat-Q8_0.gguf) | Q8_0 | 8 | 14.8 GB| very large, extremely low quality loss - not recommended |
78
+
79
+ *Quantized with llama.cpp b1953*