File size: 2,362 Bytes
a554edf b1a112b b2dd352 f4e0807 b1a112b f3b1454 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 a554edf b2dd352 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
---
base_model: mistralai/Mistral-7B-v0.3
language:
- en
license: apache-2.0
model_name: Mistral 7B v0.3
pipeline_tag: text-generation
tags:
- gptq
- 8-bit
inference: false
model_creator: Mistral AI
model_type: mistral
prompt_template: '{prompt}'
quantized_by: iproskurina
base_model_relation: quantized
---
# Mistral 7B v0.3 - GPTQ
- Model creator: [Mistral AI](https://huggingface.co/mistralai)
- Original model: [Mistral 7B v0.3](https://huggingface.co/mistralai/Mistral-7B-v0.3)
The model published in this repo was quantized to 8bit using [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ).
**Quantization details**
**All quantization parameters were taken from [GPTQ paper](https://arxiv.org/abs/2210.17323).**
GPTQ calibration data consisted of 128 random 2048 token segments from the [C4 dataset](https://huggingface.co/datasets/c4).
The grouping size used for quantization is equal to 128.
## How to use this GPTQ model from Python code
### Install the necessary packages
Requires: Transformers 4.33.0 or later, Optimum 1.12.0 or later, and AutoGPTQ 0.4.2 or later.
```shell
pip3 install --upgrade transformers optimum
# If using PyTorch 2.1 + CUDA 12.x:
pip3 install --upgrade auto-gptq
# or, if using PyTorch 2.1 + CUDA 11.x:
pip3 install --upgrade auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/
```
If you are using PyTorch 2.0, you will need to install AutoGPTQ from source. Likewise if you have problems with the pre-built wheels, you should try building from source:
```shell
pip3 uninstall -y auto-gptq
git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
git checkout v0.5.1
pip3 install .
```
### You can then use the following code
```python
from transformers import AutoTokenizer, TextGenerationPipeline,AutoModelForCausalLM
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
pretrained_model_dir = "iproskurina/Mistral-7B-v0.3-GPTQ-8bit-g128"
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True)
model = AutoGPTQForCausalLM.from_quantized(pretrained_model_dir, device="cuda:0", model_basename="model")
pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer)
print(pipeline("auto-gptq is")[0]["generated_text"])
```
|