Upload 9 files

README.md

@@ -1,3 +1,109 @@
----
-license: apache-2.0
----
+---
+license: mit
+datasets:
+  - IlyaGusev/ru_turbo_alpaca
+  - IlyaGusev/ru_turbo_alpaca_evol_instruct
+  - IlyaGusev/ru_turbo_saiga
+  - IlyaGusev/ru_sharegpt_cleaned
+  - IlyaGusev/oasst1_ru_main_branch
+  - IlyaGusev/gpt_roleplay_realm
+  - lksy/ru_instruct_gpt4
+language:
+  - ru
+  - en
+library_name: peft
+tags:
+  - Saiga
+  - ruGPT-3.5
+  - 13B
+  - chat
+  - lora
+  - Peft
+  - adapter
+---
+
+# ruGPT-3.5 13B LoRA: Adapter-Only Version
+
+Welcome to the adapter-only version of ruGPT-3.5 13B LoRA. This model is built upon the foundation of [ruGPT-3.5-13B](https://huggingface.co/ai-forever/ruGPT-3.5-13B).
+
+📌 Important: This model was trained using settings identical to [GigaSaiga](https://huggingface.co/IlyaGusev/gigasaiga_lora), but incorporates additional dataset.
+
+🔗 Training code is [here](https://github.com/EvilFreelancer/ruGPT-3.5-13B-lora).
+
+> Note: If you prefer, you can opt to use the ruGPT-3.5 13B fp16 base model.
+
+## Code sample
+
+```python
+from peft import AutoPeftModelForCausalLM
+
+model = AutoPeftModelForCausalLM.from_pretrained("zpbrent/peft-ruGPT-3.5-13B-lora")
+```
+
+## Examples
+
+```text
+User: Привет! Почему трава зелёная?
+ruGPT-3.5: Трава зелёная благодаря хлорофиллу - пигменту растений, который поглощает световые волны определенной длины и преобразует их в энергию химических связей. Хлорофилл также обеспечивает растения питательными веществами из почвы через фотосинтез.
+```
+
+```text
+User: Придумай стишок про нейросети.
+ruGPT-3.5: Нейросеть – это чудо науки,
+Она способна решать любые задачи.
+И если ты хочешь стать программистом,
+То лучше всего изучать нейронные сети.
+```
+
+```text
+User: Высота самой высокой горы на Земле?
+ruGPT-3.5: Самая высокая гора на Земле находится в Гималаях и называется Эверест (8848 метров).
+```
+
+```text
+User: Сколько максимум лет может прожить галопогосская черепаха?
+ruGPT-3.5: Галопогосские черепахи могут жить до 100 лет и более. Они являются одними из самых долгоживущих животных на планете.
+```
+
+## 📚 Training Datasets
+
+The datasets utilized for training this model are consistent with those used for [Saiga-2](https://github.com/IlyaGusev/rulm).
+
+Here's the comprehensive list:
+
+- [ru_turbo_alpaca](https://huggingface.co/datasets/IlyaGusev/ru_turbo_alpaca)
+- [ru_turbo_alpaca_evol_instruct](https://huggingface.co/datasets/IlyaGusev/ru_turbo_alpaca_evol_instruct)
+- [ru_turbo_saiga](https://huggingface.co/datasets/IlyaGusev/ru_turbo_saiga)
+- [ru_sharegpt_cleaned](https://huggingface.co/datasets/IlyaGusev/ru_sharegpt_cleaned)
+- [oasst1_ru_main_branch](https://huggingface.co/datasets/IlyaGusev/oasst1_ru_main_branch)
+- [gpt_roleplay_realm](https://huggingface.co/datasets/IlyaGusev/gpt_roleplay_realm)
+- [ru_instruct_gpt4](https://huggingface.co/datasets/lksy/ru_instruct_gpt4)
+
+## 🛠 Training Procedure
+
+The following `bitsandbytes` quantization config was used during training:
+
+- quant_method: bitsandbytes
+- load_in_8bit: True
+- load_in_4bit: False
+- llm_int8_threshold: 6.0
+- llm_int8_skip_modules: None
+- llm_int8_enable_fp32_cpu_offload: False
+- llm_int8_has_fp16_weight: False
+- bnb_4bit_quant_type: fp4
+- bnb_4bit_use_double_quant: False
+- bnb_4bit_compute_dtype: float32
+
+## ⚙️ Framework Versions
+
+Ensure you have the following framework versions for compatibility:
+
+- PyTorch 2.1.0
+- PEFT 0.5.0
+- bitsandbytes 0.41.1
+- transformers 4.34.0
+
+## Links
+
+- https://t.me/evilfreelancer
+- https://dzen.ru/evilfreelancer

adapter_config.json

@@ -0,0 +1,19 @@
+{
+  "auto_mapping": {"parent_library":"setfit","base_model_class":"SetFitModel"},
+  "base_model_name_or_path": "zpbrent/setfit-minilm-distilled",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "lora_alpha": 16,
+  "lora_dropout": 0.05,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ]
+}

adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1563ed0c38f5e596b21307c2dfbe72227f9745ab85543ec412a33f20d59e45a
+size 52457121

added_tokens.json

@@ -0,0 +1,7 @@
+{
+  "</s>": 3,
+  "<mask>": 5,
+  "<pad>": 0,
+  "<s>": 2,
+  "<|endoftext|>": 1
+}

generation_config.json

@@ -0,0 +1,15 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": 3,
+  "pad_token_id": 0,
+  "transformers_version": "4.34.0",
+  "temperature": 0.2,
+  "top_p": 0.9,
+  "top_k": 30,
+  "do_sample": true,
+  "max_new_tokens": 1536,
+  "num_beams": 1,
+  "repetition_penalty": 1.15,
+  "no_repeat_ngram_size": 15
+}

special_tokens_map.json

@@ -0,0 +1,8 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "pad_token": "<pad>",
+  "sep_token": "<s>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,59 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "errors": "replace",
+  "mask_token": "<mask>",
+  "model_max_length": 2048,
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "sep_token": "<s>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "tokenizer_file": null,
+  "unk_token": "<|endoftext|>"
+}