update

README.md

@@ -0,0 +1,194 @@
+---
+thumbnail: https://github.com/rinnakk/japanese-pretrained-models/blob/master/rinna.png
+license: llama2
+language:
+- ja
+- en
+inference: false
+datasets:
+- databricks/databricks-dolly-15k
+- kunishou/databricks-dolly-15k-ja
+- izumi-lab/llm-japanese-dataset
+---
+
+# `rinna/youri-7b-chat`
+
+![rinna-icon](./rinna.png)
+
+# Overview
+The model is the instruction-tuned version of [`rinna/youri-7b`](https://huggingface.co/rinna/youri-7b). It adopts a chat-style input format.
+
+* **Model architecture**
+
+    A 32-layer, 4096-hidden-size transformer-based language model. Refer to the [llama2 paper](https://arxiv.org/abs/2307.09288) for architecture details.
+
+* **Fine-tuning**
+    
+    The fine-tuning data is the subset of the following datasets.
+    * [Databricks Dolly data](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
+    * [Japanese Databricks Dolly data](https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja)
+    * [Anthropic HH RLHF data](https://huggingface.co/datasets/Anthropic/hh-rlhf) and its Japanese translation
+    * [FLAN Instruction Tuning data](https://github.com/google-research/FLAN) and its Japanese translation
+    * [Izumi lab LLM Japanese dataset](https://github.com/masanorihirano/llm-japanese-dataset/tree/main)
+      * The following sections are used
+        * alt
+        * aozora-txt
+        * CourseraParallel
+        * ParaNatCom
+        * Tab-delimited_Bilingual_Sentence_Pairs
+        * tanaka-corpus
+        * wikinews
+        * wordnet
+        * yasashi-japanese
+      * The [remaining sections](https://github.com/masanorihirano/llm-japanese-dataset/tree/main/datasets-cc-by-sa) contain commonly used evaluation corpora so they are skipped to prevent data leak.
+
+* **Authors**
+    
+    - [Tianyu Zhao](https://huggingface.co/tianyuz)
+    - [Kei Sawada](https://huggingface.co/keisawada)
+
+---
+
+# Benchmarking
+
+Evaluation experiments suggest that rinna's `youri-7b` series outperforms other open-source Japanese LLMs on Japanese tasks according to our runs. 
+
+| Model | Model type | 4-task score | 6-task score | 8-task score |
+| :-- | :-- | :-- | :-- | :-- | 
+| rinna/youri-7b-instruction | SFT | 83.88 | 80.93 | 63.63 |
+| **rinna/youri-7b-chat** | **SFT** | **78.29** | **78.47** | **62.18** |
+| matsuo-lab/weblab-10b-instruction-sft | SFT | 78.75 | 75.05 | 59.11 |
+| rinna/youri-7b | pre-trained | 73.32 | 74.58 | 58.87 |
+| stabilityai/japanese-stablelm-instruct-alpha-7b | SFT | 70.10 | 71.32 | 54.71 |
+| elyza/ELYZA-japanese-Llama-2-7b | pre-trained | 71.72 | 69.28 | 53.17 |
+| elyza/ELYZA-japanese-Llama-2-7b-instruct | SFT | 70.57 | 68.12 | 53.14 |
+| stabilityai/japanese-stablelm-base-alpha-7b | pre-trained | 61.03 | 65.83 | 51.05 |
+| matsuo-lab/weblab-10b | pre-trained | 66.33 | 65.58 | 50.74 |
+| meta/llama2-7b | pre-trained | 56.33 | 54.80 | 42.97 |
+| rinna/japanese-gpt-neox-3.6b | pre-trained | 47.20 | 54.68 | 41.80 |
+| rinna/bilingual-gpt-neox-4b | pre-trained | 46.60 | 52.04 | 40.03 |
+    
+---
+
+# How to use the model
+
+~~~~python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+tokenizer = AutoTokenizer.from_pretrained("rinna/youri-7b-chat")
+model = AutoModelForCausalLM.from_pretrained("rinna/youri-7b-chat")
+
+if torch.cuda.is_available():
+    model = model.to("cuda")
+
+instruction = "次の日本語を英語に翻訳してください。"
+input = "自然言語による指示に基づきタスクが解けるよう学習させることを Instruction tuning と呼びます。"
+
+context = [
+    {
+        "speaker": "設定",
+        "text": instruction
+    },
+    {
+        "speaker": "ユーザー",
+        "text": input
+    }
+]
+prompt = [
+    f"{uttr['speaker']}: {uttr['text']}"
+    for uttr in context
+]
+prompt = "\n".join(prompt)
+prompt = (
+    prompt
+    + "\n"
+    + "システム: "
+)
+token_ids = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
+
+with torch.no_grad():
+    output_ids = model.generate(
+        token_ids.to(model.device),
+        max_new_tokens=200,
+        do_sample=True,
+        temperature=0.5,
+        pad_token_id=tokenizer.pad_token_id,
+        bos_token_id=tokenizer.bos_token_id,
+        eos_token_id=tokenizer.eos_token_id
+    )
+
+output = tokenizer.decode(output_ids.tolist()[0])
+print(output)
+"""
+設定: 次の日本語を英語に翻訳してください。
+ユーザー: 自然言語による指示に基づきタスクが解けるよう学習させることを Instruction tuning と呼びます。
+システム:  Learning to solve tasks based on natural language instructions is called instruction tuning.</s>
+"""
+
+output = output[len(prompt):-len("</s>")].strip()
+input = "大規模言語モデル（だいきぼげんごモデル、英: large language model、LLM）は、多数のパラメータ（数千万から数十億）を持つ人工ニューラルネットワークで構成されるコンピュータ言語モデルで、膨大なラベルなしテキストを使用して自己教師あり学習または半教師あり学習によって訓練が行われる。"
+
+context.extend([
+    {
+        "speaker": "システム",
+        "text": output
+    },
+    {
+        "speaker": "ユーザー",
+        "text": input
+    }
+])
+prompt = [
+    f"{uttr['speaker']}: {uttr['text']}"
+    for uttr in context
+]
+prompt = "\n".join(prompt)
+prompt = (
+    prompt
+    + "\n"
+    + "システム: "
+)
+token_ids = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
+
+with torch.no_grad():
+    output_ids = model.generate(
+        token_ids.to(model.device),
+        max_new_tokens=200,
+        do_sample=True,
+        temperature=0.5,
+        pad_token_id=tokenizer.pad_token_id,
+        bos_token_id=tokenizer.bos_token_id,
+        eos_token_id=tokenizer.eos_token_id
+    )
+
+output = tokenizer.decode(output_ids.tolist()[0])
+print(output)
+"""
+設定: 次の日本語を英語に翻訳してください。
+ユーザー: 自然言語による指示に基づきタスクが解けるよう学習させることを Instruction tuning と呼びます。
+システム: Learning to solve tasks based on natural language instructions is called instruction tuning.
+ユーザー: 大規模言語モデル（だいきぼげんごモデル、英: large language model、LLM）は、多数のパラメータ（数千万から数十億）を持つ人工ニューラルネットワークで構成されるコンピュータ言語モデルで、膨大なラベルなしテ キストを使用して自己教師あり学習または半教師あり学習によって訓練が行われる。
+システム:  Large language models (LLMs) are computer language models consisting of a deep artificial neural network with millions to billions of parameters that are trained by self-supervised learning or semi-supervised learning using vast unlabeled text corpora.</s>
+"""
+~~~~
+
+---
+
+# Tokenization
+The model uses the original llama-2 tokenizer.
+
+---
+
+# How to cite
+~~~
+@misc{RinnaYouri7bChat, 
+    url={https://huggingface.co/rinna/youri-7b-chat}, 
+    title={rinna/youri-7b-chat}, 
+    author={Zhao, Tianyu and Sawada, Kei}
+}
+~~~
+---
+
+# License
+[The llama2 license](https://ai.meta.com/llama/license/)

config.json

@@ -0,0 +1,27 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 4096,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 32,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.34.0",
+  "use_cache": true,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "transformers_version": "4.34.0"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bcf5f583fcd1b32a9a471a77942cc947107719120c13a5300d2380a9407e782d
+size 13476871854

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer_checklist.chk

@@ -0,0 +1 @@
+eeec4125e9c7560836b4873b6f8e3025  tokenizer.model

tokenizer_config.json

@@ -0,0 +1,35 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}