tokyotech-llm
/

Swallow-MS-7b-instruct-v0.1

+---
+language:
+  - en
+  - ja
+library_name: transformers
+pipeline_tag: text-generation
+model_type: mistral
+license: apache-2.0
+---
+# Swallow-MS-7b-v0.1
+Our Swallow-MS-7b-v0.1 model has undergone continuous pre-training from the Mistral-7B-v0.1, primarily with the addition of Japanese language data. **The instruction tuning version will be released soon.**
+# Model Release Updates
+We are excited to share the release schedule for our latest models:
+- **April 25, 2024**: Released the [Swallow-MS-7b-instruct-v1.0](https://huggingface.co/tokyotech-llm/Swallow-MS-7b-instruct-v1.0).
+- **March 11, 2024**: Released the [Swallow-MS-7b-v0.1](https://huggingface.co/tokyotech-llm/Swallow-MS-7b-v0.1)
+![logo](./logo.png)
+This repository provides large language models developed by [TokyoTech-LLM](https://tokyotech-llm.github.io/).
+## Model Details
+* **Model type**: Please refer to Mistral technical report for details on the model architecture.
+* **Language(s)**: Japanese English
+* **Tokenizer**: This model employs a tokenizer that features a broadened vocabulary based on Japanese data. This allows for a more efficient representation of text using fewer tokens, leading to a notably faster inference process.
+* **Contact**: swallow[at]nlp.c.titech.ac.jp
+## Instruct Model Performance
+### MT-Bench JA
+TODO
+## Base Model Performance
+### Japanese tasks
+|Model|Size|JCommonsenseQA|JEMHopQA|NIILC|JSQuAD|XL-Sum|MGSM|WMT20-en-ja|WMT20-ja-en|Average|
+|---------------------------|-------|---------|-------|-------|-------|------|------------|------------|------|-----|
+|   |   |4-shot|4-shot|4-shot|4-shot|1-shot|4-shot|4-shot|4-shot||
+| CyberAgentLM2-7B |7B| 0.2198 | 0.5047 | 0.5066 | 0.7799 | 0.0233 | 0.0600 | 0.2345 | 0.1499 | 0.3098 |
+| Llama 2  |7B| 0.3852 | 0.4240 | 0.3410 | 0.7917 | 0.1905 | 0.0760 | 0.1783 | 0.1738 | 0.3201 |
+| japanese-stablelm-base-beta-7b|7B| 0.3610 | 0.4478 | 0.4432 | 0.8318 | 0.2195 | 0.0720 | 0.1946 | 0.1226 | 0.3366 |
+| japanese-stablelm-base-ja_vocab-beta-7b|7B| 0.2172 | 0.4482 | 0.4309 | 0.8202 | 0.0757 | 0.0520 | 0.1601 | 0.1453 | 0.2937 |
+| ELYZA-japanese-Llama-2-7b|7B| 0.5791 | 0.4703 | 0.4019 | 0.8226 | 0.1312 | 0.0600 | 0.1795 | 0.1289 | 0.3467 |
+| ELYZA-japanese-Llama-2-7b-fast|7B| 0.5308 | 0.4330 | 0.3898 | 0.8131 | 0.1289 | 0.0720 | 0.1678 | 0.1143 | 0.3312 |
+| youri-7b (base) |7B| 0.4620 | 0.4776 | 0.4999 | 0.8506 | 0.1957 | 0.0640 | 0.2671 | **0.1971** | 0.3768 |
+| Swallow-7b |7B| 0.4808 | 0.5078 | 0.5968 | 0.8573 | 0.1830 | 0.1240 | 0.2510 | 0.1511 | 0.3940 |
+| Swallow-7b-plus |7B| 0.5478 | **0.5493** | **0.6030** | 0.8544 | 0.1806 | 0.1360 | 0.2568 | 0.1441 | 0.4090 |
+| Qwen-7B |7B| 0.7712 | 0.4234 | 0.2376 | 0.8594 | 0.1371 | 0.2160 | 0.1689 | 0.1801 | 0.3742 |
+| nekomata-7b |7B| 0.7417 | 0.4928 | 0.5022 | 0.8707 | 0.1676 | 0.1240 | **0.2673** | 0.1815 | 0.4185 |
+| Mistral-7B-v0.1 |7B| 0.7301 | 0.4245 | 0.2722 | 0.8563 | 0.2006 | 0.1760 | 0.1405 | 0.1733 | 0.3717 |
+| japanese-stablelm-base-gamma-7b|7B| 0.7364 | 0.4643 | 0.5568 | **0.8910** | **0.2293** | 0.1680 | 0.2390 | 0.1561 | 0.4301 |
+| Swallow-MS-7b-v0.1 |7B| **0.8570** | 0.4915 | 0.5519 | 0.8802 | 0.1988 | **0.2240** | 0.2494 | 0.1667 | **0.4524** |
+### English tasks
+|Model|Size|OpenBookQA|TriviaQA|HellaSwag|SQuAD2.0|XWINO|GSM8K|Average|
+|---|---|---|---|---|---|---|---|---|
+|   |   |8-shot|8-shot|8-shot|8-shot|8-shot|8-shot||
+| CyberAgentLM2-7B |7B| 0.2860 | 0.3496 | 0.5003 | 0.3510 | 0.8581 | 0.0705 | 0.4026 |
+| Llama 2  |7B| 0.3580 | 0.6265 | 0.5860 | 0.3207 | 0.9049 | 0.1410 | 0.4895 |
+| japanese-stablelm-base-beta-7b|7B| 0.3620 | 0.5903 | 0.5707 | 0.2992 | 0.8994 | 0.1198 | 0.4736 |
+| japanese-stablelm-base-ja_vocab-beta-7b|7B| 0.3520 | 0.5549 | 0.5644 | 0.3079 | 0.8942 | 0.0538 | 0.4545 |
+| ELYZA-japanese-Llama-2-7b|7B| 0.3400 | 0.5875 | 0.5595 | 0.2721 | 0.8989 | 0.1638 | 0.4703 |
+| ELYZA-japanese-Llama-2-7b-fast|7B| 0.3280 | 0.5817 | 0.5530 | 0.2605 | 0.8989 | 0.1425 | 0.4608 |
+| youri-7b (base) |7B| 0.3400 | 0.5257 | 0.5540 | 0.3297 | 0.8938 | 0.0963 | 0.4566 |
+| Swallow-7b |7B| 0.3180 | 0.4836 | 0.5308 | 0.3125 | 0.8817 | 0.1130 | 0.4399 |
+| Swallow-7b-plus |7B| 0.3280 | 0.4558 | 0.5259 | 0.3134 | 0.8929 | 0.1061 | 0.4370 |
+| Qwen-7B |7B| 0.3640 | 0.5695 | 0.5787 | **0.3799** | 0.8933 | **0.4617** | 0.5412 |
+| nekomata-7b |7B| 0.3340 | 0.4371 | 0.5340 | 0.2933 | 0.8766 | 0.1531 | 0.4380 |
+| Mistral-7B-v0.1 |7B| **0.3660** | **0.7050** | **0.6264** | **0.3799** | **0.9157** | 0.3533 | **0.5577** |
+| japanese-stablelm-base-gamma-7b|7B| 0.3240 | 0.5745 | 0.5739 | 0.3546 | 0.8976 | 0.1911 | 0.4860 |
+| Swallow-MS-7b-v0.1 |7B| 0.3440 | 0.5976 | 0.5810 | 0.3364 | 0.9037 | 0.2623 | 0.5042 |
+### Code generation tasks
+|Model|Size|JHumanEval|HumanEval|
+|---|---|---|---|
+|   |   |pass@1|pass@1|
+| CyberAgentLM2-7B |7B|0.0634|0.0756|
+| Llama 2  |7B|0.1152|0.1378|
+| japanese-stablelm-base-beta-7b|7B|0.1018|0.1280|
+| japanese-stablelm-base-ja_vocab-beta-7b|7B|0.0896|0.1122|
+| ELYZA-japanese-Llama-2-7b|7B|0.0287|0.0427|
+| ELYZA-japanese-Llama-2-7b-fast|7B| 0.0000	|0.0037|
+| youri-7b (base) |7B|0.0829|0.0982|
+| Swallow-7b |7B|0.0183|0.0183|
+| Swallow-7b-plus |7B| 0.0061|0.0037|
+| Qwen-7B |7B|0.1701|0.1805|
+| nekomata-7b |7B|0.0988|0.1402|
+| Mistral-7B-v0.1 |7B|**0.2555**|**0.2933**|
+| japanese-stablelm-base-gamma-7b|7B|0.1823|0.1915|
+| Swallow-MS-7b-v0.1 |7B|0.2305|0.2768|
+## Evaluation Benchmarks
+### Japanese evaluation benchmarks
+We used llm-jp-eval(v1.0.0) and JP Language Model Evaluation Harness(commit #9b42d41). The details are as follows:
+- Multiple-choice question answering (JCommonsenseQA [Kurihara+, 2022])
+- Open-ended question answering (JEMHopQA [Ishii+, 2023])
+- Open-ended question answering (NIILC [Sekine, 2003])
+- Machine reading comprehension (JSQuAD [Kurihara+, 2022])
+- Automatic summarization (XL-Sum [Hasan+, 2021])
+- Machine translation (WMT2020 ja-en [Barrault+, 2020])
+- Machine translation (WMT2020 en-ja [Barrault+, 2020])
+- Mathematical reasoning (MGSM [Shi+, 2023])
+### English evaluation benchmarks
+We used the Language Model Evaluation Harness(v.0.3.0). The details are as follows:
+- Multiple-choice question answering (OpenBookQA [Mihaylov+, 2018])
+- Open-ended question answering (TriviaQA [Joshi+, 2017])
+- Machine reading comprehension (SQuAD 2.0 [Rajpurkar+, 2018])
+- Commonsense reasoning (XWINO [Tikhonov & Ryabinin, 2021])
+- Natural language inference (HellaSwag [Zellers+, 2019])
+- Mathematical reasoning (GSM8k [Cobbe+, 2021])
+### Code evaluation benchmarks
+We utilized the Code Generation LM Evaluation Harness [Allal+, 2022] (commit #0261c52). The details are as follows:
+- Code generation (HumanEval [Chen+, 2021])
+- Code generation in Japanese (JHumanEval [Satoh+, 2024])
+## Usage
+First install additional dependencies in [requirements.txt](./requirements.txt):
+```sh
+pip install -r requirements.txt
+```
+### Instruction format Ver1.0
+This format must be adhered to strictly, as deviations may result in less optimal outputs from the model.
+The template used to construct a prompt for the Instruct model is specified as follows:
+```
+<s>[INST] <<SYS>>\n{Instruction}\n<</SYS>>\n\n{USER_MESSAGE_1} [INST] {BOT_MESSAGE_1} </s>[INST] {USER_MESSAGE_2}[/INST]
+```
+Please be aware that  ``<s> `` and  ``</s> `` are special tokens used for the beginning of string (BOS) and end of string (EOS), respectively, while [INST] and [/INST] are considered regular strings.
+### Use the instruct model Ver1.0
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+model_name = "tokyotech-llm/Swallow-MS-7b-instruct-v1.0"
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+device = "cuda"
+messages = [
+    {"role": "system", "content": "あなたは誠実で優秀な日本人のアシスタントです。"},
+    {"role": "user", "content": "東京工業大学の主なキャンパスについて教えてください"}
+]
+encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")
+model_inputs = encodeds.to(device)
+model.to(device)
+generated_ids = model.generate(model_inputs, max_new_tokens=128, do_sample=True)
+decoded = tokenizer.batch_decode(generated_ids)
+print(decoded[0])
+```
+### Use the base model
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+model_name = "tokyotech-llm/Swallow-MS-7b-v0.1"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
+prompt = "東京工業大学の主なキャンパスは、"
+input_ids = tokenizer.encode(
+    prompt,
+    add_special_tokens=False,
+    return_tensors="pt"
+)
+tokens = model.generate(
+    input_ids.to(device=model.device),
+    max_new_tokens=128,
+    temperature=0.99,
+    top_p=0.95,
+    do_sample=True,
+)
+out = tokenizer.decode(tokens[0], skip_special_tokens=True)
+print(out)
+```
+## Training Datasets
+### Continual Pre-Training
+The following datasets were used for continual pre-training.
+- [Algebraic Stack](https://huggingface.co/datasets/EleutherAI/proof-pile-2)
+- [Japanese Wikipedia](https://dumps.wikimedia.org/other/cirrussearch)
+- [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)
+- [Swallow Corpus](https://chokkan.org/temp/tokyotech-llm/swallow-corpus)
+- [The Pile](https://huggingface.co/datasets/EleutherAI/pile)
+### Instruction Tuning
+#### Ver1.0
+The following datasets were used for the instruction tuning.
+- [OpenAssistant Conversations Dataset](https://huggingface.co/datasets/llm-jp/oasst1-21k-ja) was used, where human utterances are included but the responses are not used. Instead, the responses were generated using the [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/datasets/llm-jp/oasst1-21k-jahttps://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) model.
+- [OpenAssistant Conversations Dataset 21k Ja](https://huggingface.co/datasets/llm-jp/oasst1-21k-ja)
+- [OpenAssistant Conversations Dataset 21k En](https://huggingface.co/datasets/llm-jp/oasst1-21k-en)
+- [Databricks Dolly 15k Ja](https://huggingface.co/datasets/llm-jp/databricks-dolly-15k-ja)
+- [Databricks Dolly 15k En](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
+Please note that some of the data had issues with quality or format, so not all of it was used.
+## Risks and Limitations
+The models released here are still in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.
+## Acknowledgements
+We thank Mistral AI for releasing Mistral 7B v0.1 under an open license for others to build on.
+Our project is supported by the [ABCI Large-scale Language Model Building Support Program](https://abci.ai/en/link/llm_support_program.html) of the National Institute of Advanced Industrial Science and Technology.
+## License
+apache-2.0
+## Authors
+Here are the team members:
+- From [Okazaki Laboratory](https://www.nlp.c.titech.ac.jp/index.en.html), the following members:
+  - [Naoaki Okazaki](https://www.chokkan.org/index.ja.html)
+  - [Sakae Mizuki](https://s-mizuki-nlp.github.io/)
+  - [Hiroki Iida](https://meshidenn.github.io/)
+  - [Mengsay Loem](https://loem-ms.github.io/)
+  - [Shota Hirai](https://huggingface.co/Kotemo428)
+  - [Kakeru Hattori](https://aya-se.vercel.app/)
+  - [Masanari Ohi](https://twitter.com/stjohn2007)
+- From [YOKOTA Laboratory](https://www.rio.gsic.titech.ac.jp/en/index.html), the following members:
+  - [Rio Yokota](https://twitter.com/rioyokota)
+  - [Kazuki Fujii](https://twitter.com/okoge_kaz)
+  - [Taishi Nakamura](https://twitter.com/Setuna7777_2)
+  - [Takumi Okamoto](https://www.linkedin.com/in/takumi-okamoto)
+  - [Ishida Shigeki](https://www.wantedly.com/id/reborn27)