| **SqueezeLLM** is a post-training quantization framework that incorporates a new method called Dense-and-Sparse Quantization to enable efficient LLM serving. | |
| **TLDR:** Deploying LLMs is difficult due to their large memory size. This can be addressed with reduced precision quantization. | |
| But a naive method hurts performance. We address this with a new Dense-and-Sparse Quantization method. | |
| Dense-and-Sparse splits weight matrices into two components: A dense component that can be heavily quantized without affecting model performance, | |
| as well as a sparse part that preserves sensitive and outlier parts of the weight matrices With this approach, | |
| we are able to serve larger models with smaller memory footprint, the same latency, and yet higher accuracy and quality. | |
| For more details please check out our [paper](https://arxiv.org/pdf/2306.07629.pdf). | |
| ## Model description | |
| 3-bit XGen-7B instruction-tuned model (i.e. finetuned model on public domain instructional data) with 8K sequence length quantized using SqueezeLLM. | |
| More details on the quantization method can be found in the [paper](https://arxiv.org/pdf/2306.07629.pdf). | |
| More detailed model descriptions can be found in the [link](https://huggingface.co/Salesforce/xgen-7b-8k-inst). | |
| * **Base Model:** [XGen-7B-8K-Inst](https://huggingface.co/Salesforce/xgen-7b-8k-inst) (by Salesforce AI Research) | |
| * **Bitwidth:** 3-bit | |
| * **Sparsity Level:** 0% (dense-only) | |
| ## Links | |
| * **Paper**: [https://arxiv.org/pdf/2306.07629.pdf](https://arxiv.org/pdf/2306.07629.pdf) | |
| * **Code**: [https://github.com/SqueezeAILab/SqueezeLLM](https://github.com/SqueezeAILab/SqueezeLLM) | |
| --- | |
| license: other | |
| --- | |