|
--- |
|
base_model: facebook/opt-350m |
|
inference: false |
|
model_creator: facebook |
|
model_name: opt-350m |
|
model_type: opt |
|
pipeline_tag: text-generation |
|
quantized_by: iproskurina |
|
tags: |
|
- pretrained |
|
license: other |
|
language: |
|
- en |
|
datasets: |
|
- c4 |
|
--- |
|
|
|
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/629a3dbcd496c6dcdebf41cc/t-6kpqFpEYJPT6zmvnm49.png" width="200" /> |
|
|
|
# OPT-350M - GPTQ |
|
|
|
|
|
- Model creator: [Meta AI](https://huggingface.co/facebook) |
|
- Original model: [OPT-350M](https://huggingface.co/facebook/opt-350m) |
|
|
|
The model published in this repo was quantized to 4bit using [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ). |
|
|
|
**Quantization details** |
|
|
|
**All quantization parameters were taken from [GPTQ paper](https://arxiv.org/abs/2210.17323).** |
|
|
|
GPTQ calibration data consisted of 128 random 2048 token segments from the [C4 dataset](https://huggingface.co/datasets/c4). |
|
|
|
The grouping size used for quantization is equal to 128. |
|
|
|
## How to use this GPTQ model from Python code |
|
|
|
### Install the necessary packages |
|
|
|
```shell |
|
pip install accelerate==0.26.1 datasets==2.16.1 dill==0.3.7 gekko==1.0.6 multiprocess==0.70.15 peft==0.7.1 rouge==1.0.1 sentencepiece==0.1.99 |
|
git clone https://github.com/upunaprosk/AutoGPTQ |
|
cd AutoGPTQ |
|
pip install -v . |
|
``` |
|
Recommended transformers version: 4.35.2. |
|
|
|
### You can then use the following code |
|
|
|
```python |
|
|
|
from transformers import AutoTokenizer, TextGenerationPipeline,AutoModelForCausalLM |
|
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig |
|
pretrained_model_dir = "iproskurina/opt-350m-gptq-4bit" |
|
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True) |
|
model = AutoGPTQForCausalLM.from_quantized(pretrained_model_dir, device="cuda:0", model_basename="model") |
|
pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer) |
|
print(pipeline("auto-gptq is")[0]["generated_text"]) |
|
``` |
|
|
|
[**LICENSE**](https://huggingface.co/facebook/opt-350m/blob/main/LICENSE.md) |
