Create README.md

README.md

@@ -0,0 +1,104 @@
+---
+base_model: bigscience/bloom-7b1
+inference: false
+model_creator: bigscience
+model_name: bloom-7b1
+model_type: bloom
+pipeline_tag: text-generation
+quantized_by: iproskurina
+tags:
+- pretrained
+license: bigscience-bloom-rail-1.0
+language:
+  - ak
+  - ar
+  - as
+  - bm
+  - bn
+  - ca
+  - code
+  - en
+  - es
+  - eu
+  - fon
+  - fr
+  - gu
+  - hi
+  - id
+  - ig
+  - ki
+  - kn
+  - lg
+  - ln
+  - ml
+  - mr
+  - ne
+  - nso
+  - ny
+  - or
+  - pa
+  - pt
+  - rn
+  - rw
+  - sn
+  - st
+  - sw
+  - ta
+  - te
+  - tn
+  - ts
+  - tum
+  - tw
+  - ur
+  - vi
+  - wo
+  - xh
+  - yo
+  - zh
+  - zhs
+  - zht
+  - zu
+datasets:
+- c4
+---
+
+
+
+# 🌸 BLOOM 7b1 - GPTQ
+- Model creator: [BigScience](https://huggingface.co/bigscience)
+- Original model: [BLOOM 7b1](https://huggingface.co/bigscience/bloom-7b1)
+
+The model published in this repo was quantized to 4bit using [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ).
+
+**Quantization details**
+
+**All quantization parameters were taken from [GPTQ paper](https://arxiv.org/abs/2210.17323).**
+
+GPTQ calibration data consisted of 128 random 2048 token segments from the [C4 dataset](https://huggingface.co/datasets/c4).
+
+The grouping size used for quantization is equal to 128.
+
+## How to use this GPTQ model from Python code
+
+### Install the necessary packages
+
+```shell
+pip install accelerate==0.26.1 datasets==2.16.1 dill==0.3.7 gekko==1.0.6 multiprocess==0.70.15 peft==0.7.1 rouge==1.0.1 sentencepiece==0.1.99
+git clone https://github.com/upunaprosk/AutoGPTQ
+cd AutoGPTQ
+pip install -v .
+```
+Recommended transformers version: 4.35.2.
+
+### You can then use the following code
+
+```python
+
+from transformers import AutoTokenizer, TextGenerationPipeline,AutoModelForCausalLM
+from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
+pretrained_model_dir = "iproskurina/bloom-7b1-gptq-4bit"
+tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True)
+model = AutoGPTQForCausalLM.from_quantized(pretrained_model_dir, device="cuda:0", model_basename="model")
+pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer)
+print(pipeline("auto-gptq is")[0]["generated_text"])
+```