Upload OpenAssistant/pythia-12b-sft-v8-7k-steps ctranslate fp16 weights

README.md

@@ -0,0 +1,112 @@
+---
+license: apache-2.0
+language:
+  - en
+tags:
+- ctranslate2
+- int8
+- float16
+  - sft
+pipeline_tag: text-generation
+widget:
+  - text: <|prompter|>What is a meme, and what's the history behind this word?<|endoftext|><|assistant|>
+  - text: <|prompter|>What's the Earth total population<|endoftext|><|assistant|>
+  - text: <|prompter|>Write a story about future of AI development<|endoftext|><|assistant|>
+---
+# # Fast-Inference with Ctranslate2
+Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU.
+
+quantized version of [OpenAssistant/pythia-12b-sft-v8-7k-steps](https://huggingface.co/OpenAssistant/pythia-12b-sft-v8-7k-steps)
+```bash
+pip install hf-hub-ctranslate2>=2.0.6 
+```
+Converted on 2023-05-19 using
+```
+ct2-transformers-converter --model OpenAssistant/pythia-12b-sft-v8-7k-steps --output_dir /home/feil_m/tmp-ct2fast-pythia-12b-sft-v8-7k-steps --force --copy_files tokenizer.json README.md tokenizer_config.json generation_config.json special_tokens_map.json .gitattributes --quantization float16
+```
+
+Checkpoint compatible to [ctranslate2>=3.13.0](https://github.com/OpenNMT/CTranslate2) and [hf-hub-ctranslate2>=2.0.6](https://github.com/michaelfeil/hf-hub-ctranslate2)
+- `compute_type=int8_float16` for `device="cuda"` 
+- `compute_type=int8`  for `device="cpu"`
+
+```python
+from hf_hub_ctranslate2 import TranslatorCT2fromHfHub, GeneratorCT2fromHfHub
+from transformers import AutoTokenizer
+
+model_name = "michaelfeil/ct2fast-pythia-12b-sft-v8-7k-steps"
+# use either TranslatorCT2fromHfHub or GeneratorCT2fromHfHub here, depending on model.
+model = GeneratorCT2fromHfHub(
+        # load in int8 on CUDA
+        model_name_or_path=model_name, 
+        device="cuda",
+        compute_type="int8_float16",
+        tokenizer=AutoTokenizer.from_pretrained("OpenAssistant/pythia-12b-sft-v8-7k-steps")
+)
+outputs = model.generate(
+    text=["How do you call a fast Flan-ingo?", "User: How are you doing? Bot:"],
+)
+print(outputs)
+```
+
+# Licence and other remarks:
+This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.
+
+# Original description
+    
+- base model: [OpenAssistant/pythia-12b-pre-v8-12.5k-steps](https://huggingface.co/OpenAssistant/pythia-12b-pre-v8-12.5k-steps)
+- wandb: https://wandb.ai/open-assistant/supervised-finetuning/runs/pcw1ejda
+- [sampling report](https://raw.githubusercontent.com/Open-Assistant/oasst-model-eval/main/sampling_reports/oasst-sft/2023-05-07_OpenAssistant_pythia-12b-sft-v8-7k-steps_sampling_noprefix2.json)
+
+```
+pythia-12b-sft-8:
+  dtype: fp16
+  log_dir: "pythia_log_12b"
+  learning_rate: 6e-6
+  model_name: OpenAssistant/pythia-12b-pre-v8-12.5k-steps
+  output_dir: pythia_model_12b
+  weight_decay: 0.0
+  residual_dropout: 0.0
+  max_length: 2048
+  use_flash_attention: true
+  warmup_steps: 100
+  gradient_checkpointing: true
+  gradient_accumulation_steps: 2
+  per_device_train_batch_size: 4
+  per_device_eval_batch_size: 4
+  eval_steps: 251
+  save_steps: 500
+  num_train_epochs: 8
+  save_total_limit: 4
+  num_train_epochs: 8
+  save_total_limit: 3
+  use_custom_sampler: true
+  sort_by_length: false
+  save_strategy: steps
+  datasets:
+    - oasst_export:
+        lang: "bg,ca,cs,da,de,en,es,fr,hr,hu,it,nl,pl,pt,ro,ru,sl,sr,sv,uk"
+        input_file_path: 2023-05-06_OASST_labels.jsonl.gz
+        val_split: 0.05
+    - vicuna:
+        val_split: 0.05
+        max_val_set: 800
+        fraction: 0.4
+    - dolly15k:
+        val_split: 0.05
+        max_val_set: 300
+    - grade_school_math_instructions:
+        val_split: 0.05
+    - code_alpaca:
+        val_split: 0.05
+        max_val_set: 250
+    - red_pajama:
+        fraction: 0.05
+        max_val_set: 1000
+    - wizardlm_70k:
+        val_split: 0.05
+        max_val_set: 500
+        fraction: 0.4
+    - poem_instructions:
+        fraction: 0.5
+        val_split: 0.025
+```

config.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "4.28.0.dev0"
+}

model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78d2692d1abb075f34f6fc22b99785dd8429a3fbda460b81a8643537975ac029
+size 23683674702

special_tokens_map.json

@@ -0,0 +1,14 @@
+{
+  "additional_special_tokens": [
+    "<|prefix_begin|>",
+    "<|prompter|>",
+    "<|assistant|>",
+    "<|system|>",
+    "<|prefix_end|>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|padding|>",
+  "sep_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,10 @@
+{
+  "add_prefix_space": false,
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "special_tokens_map_file": "/admin/home-hailey/.cache/huggingface/hub/models--EleutherAI--gpt-neox-20b/snapshots/4e49eadb5d14bd22f314ec3f45b69a87b88c7691/special_tokens_map.json",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "unk_token": "<|endoftext|>"
+}