---
base_model: bigscience/bloom-1b7
inference: false
model_creator: bigscience
model_name: bloom-1b7
model_type: bloom
pipeline_tag: text-generation
quantized_by: iproskurina
tags:
- pretrained
license: bigscience-bloom-rail-1.0
language:
- ak
- ar
- as
- bm
- bn
- ca
- code
- en
- es
- eu
- fon
- fr
- gu
- hi
- id
- ig
- ki
- kn
- lg
- ln
- ml
- mr
- ne
- nso
- ny
- or
- pa
- pt
- rn
- rw
- sn
- st
- sw
- ta
- te
- tn
- ts
- tum
- tw
- ur
- vi
- wo
- xh
- yo
- zh
- zhs
- zht
- zu
datasets:
- c4
---



# 🌸 BLOOM 1B7 - GPTQ
- Model creator: [BigScience](https://huggingface.co/bigscience)
- Original model: [BLOOM 1b7](https://huggingface.co/bigscience/bloom-1b7)

The model published in this repo was quantized to 4bit using [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ).

**Quantization details**

**All quantization parameters were taken from [GPTQ paper](https://arxiv.org/abs/2210.17323).**

GPTQ calibration data consisted of 128 random 2048 token segments from the [C4 dataset](https://huggingface.co/datasets/c4).

The grouping size used for quantization is equal to 128.

## How to use this GPTQ model from Python code

### Install the necessary packages

```shell
pip install accelerate==0.26.1 datasets==2.16.1 dill==0.3.7 gekko==1.0.6 multiprocess==0.70.15 peft==0.7.1 rouge==1.0.1 sentencepiece==0.1.99
git clone https://github.com/upunaprosk/AutoGPTQ
cd AutoGPTQ
pip install -v .
```
Recommended transformers version: 4.35.2.

### You can then use the following code

```python

from transformers import AutoTokenizer, TextGenerationPipeline,AutoModelForCausalLM
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
pretrained_model_dir = "iproskurina/bloom-1b7-gptq-4bit"
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True)
model = AutoGPTQForCausalLM.from_quantized(pretrained_model_dir, device="cuda:0", model_basename="model")
pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer)
print(pipeline("auto-gptq is")[0]["generated_text"])
```