tsumeone commited on
Commit
a114ea1
1 Parent(s): a0689c8

Upload test.txt

Browse files
Files changed (1) hide show
  1. test.txt +16 -0
test.txt ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 3bit quantized version of this: https://huggingface.co/ausboss/llama-30b-supercot
2
+
3
+ GPTQ quantization using https://github.com/0cc4m/GPTQ-for-LLaMa
4
+
5
+ Made at the request of someone that wanted a 3bit version. The file is 17% smaller than 4bit non-groupsize, but the wikitext2 ppl is 12% worse. I don't have a functioning Ooba install so I can't test this myself.
6
+
7
+ Command used to quantize:
8
+ ```python llama.py c:\llama-30b-supercot c4 --wbits 3 --true-sequential --groupsize 128 --save_safetensors 4bit-128g.safetensors```
9
+
10
+ Evaluation & Score (Lower is better):
11
+ * WikiText2: 5.22 (12% worse than 4bit non-groupsize)
12
+ * PTB: 19.63 (11% worse than 4bit non-groupsize)
13
+ * C4: 6.93 (7% worse than 4bit non-groupsize)
14
+
15
+ 4bit non-groupsize version is here: https://huggingface.co/tsumeone/llama-30b-supercot-4bit-cuda
16
+ 4bit 128 groupsize version is here: https://huggingface.co/tsumeone/llama-30b-supercot-4bit-128g-cuda