commit quant model

README.md

@@ -1,3 +1,52 @@
 ---
 license: apache-2.0
+language:
+  - ar
+  - hi
+  - id
+pipeline_tag: text-generation
+tags:
+  - multilingual
 ---
+
+# mGPT-quantized
+
+The concept: 8-bit quantized version of [mGPT](https://huggingface.co/ai-forever/mGPT), a 1.3B param model released by AI-Forever / Sberbank AI in April 2022.
+
+On the GPT scale, it is a similar # of parameters to GPT2-XL, but on 60+ languages.
+
+My goal is to evaluate this on Arabic, Hindi, and Indonesian tasks, where there are fewer autoregressive language models in this size range.
+
+For English: use a GPT model or LLaMa2-7B
+
+[AI-Forever](https://huggingface.co/ai-forever) also released a 13B-parameter model, and in August 2023 added 1.3B-param models for about 1/3 of the model languages. If your language is Mongolian, for example, use mGPT-1.3B-mongol and not this one.
+
+## How was the model created?
+
+Quantization of mGPT 1.3B was done using `bitsandbytes` library:
+
+```python
+from transformers import BitsAndBytesConfig, GPT2LMHeadModel
+
+quantization_config = BitsAndBytesConfig(
+    load_in_8bit=True,
+    bnb_8bit_compute_dtype=torch.bfloat16,
+    bnb_8bit_use_double_quant=True,
+    bnb_8bit_quant_type="nf4",
+)
+
+qmodel = GPT2LMHeadModel.from_pretrained(
+    "ai-forever/mGPT",
+    load_in_8bit=True,
+    torch_dtype=torch.bfloat16,
+    quantization_config=quantization_config,
+    device_map="auto"
+)
+
+qmodel.save_pretrained("model_name")
+```
+
+## Future steps
+
+- mGPT could be further quantized (4-bit), but `model.save_pretrained()` currently throws a `NotImplementedError` error.
+- It would be great to load and quantize the 10x larger mGPT-13B, but that would take more resources.

config.json

@@ -0,0 +1,46 @@
+{
+  "_name_or_path": "monsoon-nlp/mGPT-quantized",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "gradient_checkpointing": false,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 2048,
+  "n_embd": 2048,
+  "n_head": 16,
+  "n_inner": null,
+  "n_layer": 24,
+  "n_positions": 2048,
+  "quantization_config": {
+    "bnb_4bit_compute_dtype": "float32",
+    "bnb_4bit_quant_type": "fp4",
+    "bnb_4bit_use_double_quant": false,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": null,
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": false,
+    "load_in_8bit": true,
+    "quant_method": "bitsandbytes"
+  },
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.32.1",
+  "use_cache": true,
+  "vocab_size": 100000
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07db59049df6b63f89851b71726f8f7962c9e1e132c29a08d7cca93f234e0992
+size 1629130141