Upload 2 files

README.md

@@ -1,3 +1,248 @@
----
-license: apache-2.0
----
+---
+base_model:
+- nitky/Oumuamua-7b-base
+- stabilityai/japanese-stablelm-base-gamma-7b
+- tokyotech-llm/Swallow-MS-7b-v0.1
+- DataPilot/ArrowPro-7B-KillerWhale
+- nitky/RP-7b-instruct
+- mistralai/Mistral-7B-Instruct-v0.3
+- mistralai/Mistral-7B-v0.1
+library_name: transformers
+tags:
+- mergekit
+- merge
+language:
+  - ja
+  - en
+pipeline_tag: text-generation
+license: apache-2.0
+---
+# Oumuamua-7b-instruct
+
+This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
+
+## Output example
+
+```
+[INST] <<SYS>>
+あなたは日本語を話す優秀なアシスタントです。回答には必ず日本語で答えてください。また考える過程も出力してください。
+<</SYS>>
+
+道路で泣いている子供がいます。その子供には何があったと考えられますか？ [/INST] 道路で泣いている子供がいる場合、考えられる理由はいくつかあります。
+
+1. 怪我をした: 転んで怪我をしたり、何かにぶつかって痛みを感じている可能性があります。
+2. 迷子になった: 親や保護者とはぐれてしまった、または道に迷って不安になっている可能性があります。
+3. お腹が空いた: 長時間食事をしていない、またはお腹が空いているために泣いている可能性があります。
+4. 疲れた: 長時間歩いていたり、遊び疲れて疲れている可能性があります。
+5. 寂しかった: 一人でいることに寂しさを感じている、または親や友達がいなくて寂しいと感じている可能性があります。
+6. 怖い思いをした: 車や不審者に遭遇した、または何かに驚いた可能性があります。
+7. 体調が悪い: 発熱や頭痛、吐き気などの体調不良を感じている可能性があります。
+8. トイレに行きたい: おむつが濡れていたり、トイレに行きたくて泣いている可能性があります。
+9. 何かを求めている: 注意を引きたかったり、助けを求めている可能性があります。
+
+子供の状況をよく観察し、声をかけて話を聞いてみることが大切です。必要に応じて、保護者や警察、救急車を呼ぶなどの対応をする必要があるかもしれません。
+```
+
+## Test environment
+
+This model was tested using [text-generation-webui](https://github.com/oobabooga/text-generation-webui/tree/main). I use preset `min_p` and `Null preset` with temperature=0.3 for Generation.
+
+
+## Usage
+
+This format must be adhered to strictly, as deviations may result in less optimal outputs from the model.
+
+The template used to construct a prompt for the Instruct model is specified as follows:
+
+```
+<s>[INST] <<SYS>>\n{SYSTEM_PROMPT}\n<</SYS>>\n\n{USER_MESSAGE_1} [/INST] {BOT_MESSAGE_1}</s>[INST] {USER_MESSAGE_2} [/INST] 
+```
+
+
+Please be aware that ``<s>`` and ``</s>`` are special tokens used for the beginning of string (BOS) and end of string (EOS), respectively, while [INST] and [/INST] are considered regular strings.
+
+For the "{SYSTEM_PROMPT}" part, We recommend using "あなたは日本語を話す優秀なアシスタントです。回答には必ず日本語で答えてください。また考える過程も出力してください。"
+
+For the "{USER_MESSAGE_1}" part, We recommend using {instruction}\n{input}
+
+In other words, We recommend the following:
+
+``` 
+<s>[INST] <<SYS>>\nあなたは日本語を話す優秀なアシスタントです。回答には必ず日本語で答えてください。また考える過程も出力してください。\n<</SYS>>\n\n{instruction1}\n{input1} [/INST] {BOT_MESSAGE_1}</s>[INST] {instruction2}\n{input2} [/INST] 
+```
+
+### Use the instruct model
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_name = "nitky/Oumuamua-7b-instruct"
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+device = "cuda"
+
+messages = [
+    {"role": "system", "content": "あなたは日本語を話す優秀なアシスタントです。回答には必ず日本語で答えてください。また考える過程も出力してください。"},
+    {"role": "user", "content": "道路で泣いている子供がいます。その子供には何があったと考えられますか？"}
+]
+
+encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")
+
+model_inputs = encodeds.to(device)
+model.to(device)
+
+generated_ids = model.generate(model_inputs, max_new_tokens=256, do_sample=True, temperature=0.3)
+decoded = tokenizer.batch_decode(generated_ids)
+print(decoded[0])
+```
+
+## Merge Details
+### Merge Method
+
+This model was merged using the [Model Stock](https://arxiv.org/abs/2403.19522) merge method using [nitky/Oumuamua-7b-base](https://huggingface.co/nitky/Oumuamua-7b-base) as a base.
+
+### Models Merged
+
+The following models were included in the merge:
+* [stabilityai/japanese-stablelm-base-gamma-7b](https://huggingface.co/stabilityai/japanese-stablelm-base-gamma-7b)
+* [tokyotech-llm/Swallow-MS-7b-v0.1](https://huggingface.co/tokyotech-llm/Swallow-MS-7b-v0.1)
+* [DataPilot/ArrowPro-7B-KillerWhale](https://huggingface.co/DataPilot/ArrowPro-7B-KillerWhale)
+* [nitky/RP-7b-instruct](https://huggingface.co/nitky/RP-7b-instruct)
+* [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)
+* [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
+
+### Configuration
+
+The following YAML configuration was used to produce this model:
+
+```yaml
+merge_method: task_arithmetic
+base_model: mistralai/Mistral-7B-v0.1
+models:
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight:
+      - filter: embed_tokens
+        value: 1.0
+      - value: 0
+dtype: bfloat16
+tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
+name: Mistral-7B-v0.1-VE-Swallow-MS
+---
+merge_method: task_arithmetic
+base_model: stabilityai/japanese-stablelm-base-gamma-7b
+models:
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight:
+      - filter: embed_tokens
+        value: 1.0
+      - value: 0
+dtype: bfloat16
+tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
+name: japanese-stablelm-base-gamma-7b-VE-Swallow-MS
+---
+merge_method: task_arithmetic
+base_model: DataPilot/ArrowPro-7B-KillerWhale
+models:
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight:
+      - filter: embed_tokens
+        value: 1.0
+      - value: 0
+dtype: bfloat16
+tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
+name: ArrowPro-7B-KillerWhale-VE-Swallow-MS
+---
+merge_method: task_arithmetic
+base_model: nitky/RP-7b-instruct
+models:
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight:
+      - filter: embed_tokens
+        value: 1.0
+      - value: 0
+dtype: bfloat16
+tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
+name: RP-7b-instruct-VE-Swallow-MS
+---
+merge_method: task_arithmetic
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+models:
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight:
+      - filter: embed_tokens
+        value: 1.0
+      - value: 0
+dtype: bfloat16
+tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
+name: Mistral-7B-Instruct-v0.3-VE-Swallow-MS
+---
+merge_method: linear
+models:
+  - model: nitky/Oumuamua-7b-base
+    parameters:
+      weight: 1.0
+  - model: tokyotech-llm/Swallow-MS-7b-instruct-v0.1
+    parameters:
+      weight: 0.8
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight: -0.8
+  - model: Mistral-7B-Instruct-v0.3-VE-Swallow-MS
+    parameters:
+      weight: 0.8
+  - model: Mistral-7B-v0.1-VE-Swallow-MS
+    parameters:
+      weight: -0.8
+dtype: bfloat16
+name: Oumuamua-7b-instruct-alpha
+---
+merge_method: linear
+models:
+  - model: nitky/Oumuamua-7b-base
+    parameters:
+      weight: 1.0
+  - model: ArrowPro-7B-KillerWhale-VE-Swallow-MS # include Mistral-7B-Instruct-v0.3
+    parameters:
+      weight: 0.8
+  - model: japanese-stablelm-base-gamma-7b-VE-Swallow-MS
+    parameters:
+      weight: -0.8
+dtype: bfloat16
+name: Oumuamua-7b-instruct-beta
+---
+merge_method: linear
+models:
+  - model: nitky/Oumuamua-7b-base
+    parameters:
+      weight: 1.0
+  - model: RP-7b-instruct-VE-Swallow-MS
+    parameters:
+      weight: 0.8
+  - model: japanese-stablelm-base-gamma-7b-VE-Swallow-MS
+    parameters:
+      weight: -0.8
+  - model: Mistral-7B-Instruct-v0.3-VE-Swallow-MS
+    parameters:
+      weight: 0.8
+  - model: Mistral-7B-v0.1-VE-Swallow-MS
+    parameters:
+      weight: -0.8
+dtype: bfloat16
+name: Oumuamua-7b-instruct-gamma
+---
+merge_method: model_stock
+base_model: nitky/Oumuamua-7b-base
+models:
+  - model: Oumuamua-7b-instruct-alpha
+  - model: Oumuamua-7b-instruct-beta
+  - model: Oumuamua-7b-instruct-gamma
+dtype: bfloat16
+name: Oumuamua-7b-instruct
+```

mergekit_config.yml

@@ -0,0 +1,127 @@
+merge_method: task_arithmetic
+base_model: mistralai/Mistral-7B-v0.1
+models:
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight:
+      - filter: embed_tokens
+        value: 1.0
+      - value: 0
+dtype: bfloat16
+tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
+name: Mistral-7B-v0.1-VE-Swallow-MS
+---
+merge_method: task_arithmetic
+base_model: stabilityai/japanese-stablelm-base-gamma-7b
+models:
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight:
+      - filter: embed_tokens
+        value: 1.0
+      - value: 0
+dtype: bfloat16
+tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
+name: japanese-stablelm-base-gamma-7b-VE-Swallow-MS
+---
+merge_method: task_arithmetic
+base_model: DataPilot/ArrowPro-7B-KillerWhale
+models:
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight:
+      - filter: embed_tokens
+        value: 1.0
+      - value: 0
+dtype: bfloat16
+tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
+name: ArrowPro-7B-KillerWhale-VE-Swallow-MS
+---
+merge_method: task_arithmetic
+base_model: nitky/RP-7b-instruct
+models:
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight:
+      - filter: embed_tokens
+        value: 1.0
+      - value: 0
+dtype: bfloat16
+tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
+name: RP-7b-instruct-VE-Swallow-MS
+---
+merge_method: task_arithmetic
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+models:
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight:
+      - filter: embed_tokens
+        value: 1.0
+      - value: 0
+dtype: bfloat16
+tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
+name: Mistral-7B-Instruct-v0.3-VE-Swallow-MS
+---
+merge_method: linear
+models:
+  - model: nitky/Oumuamua-7b-base
+    parameters:
+      weight: 1.0
+  - model: tokyotech-llm/Swallow-MS-7b-instruct-v0.1
+    parameters:
+      weight: 0.8
+  - model: tokyotech-llm/Swallow-MS-7b-v0.1
+    parameters:
+      weight: -0.8
+  - model: Mistral-7B-Instruct-v0.3-VE-Swallow-MS
+    parameters:
+      weight: 0.8
+  - model: Mistral-7B-v0.1-VE-Swallow-MS
+    parameters:
+      weight: -0.8
+dtype: bfloat16
+name: Oumuamua-7b-instruct-alpha
+---
+merge_method: linear
+models:
+  - model: nitky/Oumuamua-7b-base
+    parameters:
+      weight: 1.0
+  - model: ArrowPro-7B-KillerWhale-VE-Swallow-MS # include Mistral-7B-Instruct-v0.3
+    parameters:
+      weight: 0.8
+  - model: japanese-stablelm-base-gamma-7b-VE-Swallow-MS
+    parameters:
+      weight: -0.8
+dtype: bfloat16
+name: Oumuamua-7b-instruct-beta
+---
+merge_method: linear
+models:
+  - model: nitky/Oumuamua-7b-base
+    parameters:
+      weight: 1.0
+  - model: RP-7b-instruct-VE-Swallow-MS
+    parameters:
+      weight: 0.8
+  - model: japanese-stablelm-base-gamma-7b-VE-Swallow-MS
+    parameters:
+      weight: -0.8
+  - model: Mistral-7B-Instruct-v0.3-VE-Swallow-MS
+    parameters:
+      weight: 0.8
+  - model: Mistral-7B-v0.1-VE-Swallow-MS
+    parameters:
+      weight: -0.8
+dtype: bfloat16
+name: Oumuamua-7b-instruct-gamma
+---
+merge_method: model_stock
+base_model: nitky/Oumuamua-7b-base
+models:
+  - model: Oumuamua-7b-instruct-alpha
+  - model: Oumuamua-7b-instruct-beta
+  - model: Oumuamua-7b-instruct-gamma
+dtype: bfloat16
+name: Oumuamua-7b-instruct