squeeze-ai-lab commited on
Commit
22b5b77
1 Parent(s): 47edd60

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +27 -0
README.md ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ **SqueezeLLM** is a post-training quantization framework that incorporates a new method called Dense-and-Sparse Quantization to enable efficient LLM serving.
2
+
3
+ **TLDR:** Deploying LLMs is difficult due to their large memory size. This can be addressed with reduced precision quantization.
4
+ But a naive method hurts performance. We address this with a new Dense-and-Sparse Quantization method.
5
+ Dense-and-Sparse splits weight matrices into two components: A dense component that can be heavily quantized without affecting model performance,
6
+ as well as a sparse part that preserves sensitive and outlier parts of the weight matrices With this approach,
7
+ we are able to serve larger models with smaller memory footprint, the same latency, and yet higher accuracy and quality.
8
+ For more details please check out our [paper](https://arxiv.org/pdf/2306.07629.pdf).
9
+
10
+
11
+ ## Model description
12
+
13
+ 3-bit quantized OPT 6.7B model using SqueezeLLM. More details can be found in the [paper](https://arxiv.org/pdf/2306.07629.pdf).
14
+
15
+ * **Base Model:** [OPT 6.7B](https://arxiv.org/abs/2205.01068)
16
+ * **Bitwidth:** 3-bit
17
+ * **Sparsity Level:** 0% (dense-only)
18
+
19
+ ## Links
20
+
21
+ * **Paper**: [https://arxiv.org/pdf/2306.07629.pdf](https://arxiv.org/pdf/2306.07629.pdf)
22
+ * **Code**: [https://github.com/SqueezeAILab/SqueezeLLM](https://github.com/SqueezeAILab/SqueezeLLM)
23
+
24
+
25
+ ---
26
+ license: other
27
+ ---