Commit
•
0e37c98
1
Parent(s):
0818a52
Upload README.md
Browse files
README.md
CHANGED
@@ -10,10 +10,17 @@ license: apache-2.0
|
|
10 |
|
11 |
# Swallow-MS-7b-v0.1
|
12 |
|
13 |
-
Our Swallow-MS-7b-v0.1 model has undergone
|
14 |
|
|
|
|
|
|
|
|
|
|
|
15 |
![logo](./logo.png)
|
16 |
|
|
|
|
|
17 |
## Model Details
|
18 |
|
19 |
* **Model type**: Please refer to Mistral technical report for details on the model architecture.
|
@@ -21,6 +28,9 @@ Our Swallow-MS-7b-v0.1 model has undergone continuous pre-training from the [Mis
|
|
21 |
* **Tokenizer**: This model employs a tokenizer that features a broadened vocabulary based on Japanese data. This allows for a more efficient representation of text using fewer tokens, leading to a notably faster inference process.
|
22 |
* **Contact**: swallow[at]nlp.c.titech.ac.jp
|
23 |
|
|
|
|
|
|
|
24 |
## Base Model Performance
|
25 |
|
26 |
### Japanese tasks
|
@@ -133,9 +143,9 @@ from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
133 |
import torch
|
134 |
|
135 |
model_name = "tokyotech-llm/Swallow-MS-7b-v0.1"
|
136 |
-
tokenizer = AutoTokenizer.from_pretrained(model_name
|
137 |
|
138 |
-
model = AutoModelForCausalLM.from_pretrained(model_name)
|
139 |
prompt = "東京工業大学の主なキャンパスは、"
|
140 |
input_ids = tokenizer.encode(
|
141 |
prompt,
|
@@ -164,7 +174,7 @@ The following datasets were used for continual pre-training.
|
|
164 |
- [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)
|
165 |
- [Swallow Corpus](https://chokkan.org/temp/tokyotech-llm/swallow-corpus)
|
166 |
- [The Pile](https://huggingface.co/datasets/EleutherAI/pile)
|
167 |
-
|
168 |
## Risks and Limitations
|
169 |
|
170 |
The models released here are still in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.
|
|
|
10 |
|
11 |
# Swallow-MS-7b-v0.1
|
12 |
|
13 |
+
Our Swallow-MS-7b-v0.1 model has undergone continual pre-training from the Mistral-7B-v0.1, primarily with the addition of Japanese language data.
|
14 |
|
15 |
+
# Model Release Updates
|
16 |
+
|
17 |
+
We are excited to share the release schedule for our latest models:
|
18 |
+
- **April 26, 2024**: Released the [Swallow-MS-7b-instruct-v0.1](https://huggingface.co/tokyotech-llm/Swallow-MS-7b-instruct-v0.1)
|
19 |
+
- **March 11, 2024**: Released the [Swallow-MS-7b-v0.1](https://huggingface.co/tokyotech-llm/Swallow-MS-7b-v0.1)
|
20 |
![logo](./logo.png)
|
21 |
|
22 |
+
This repository provides large language models developed by [TokyoTech-LLM](https://tokyotech-llm.github.io/).
|
23 |
+
|
24 |
## Model Details
|
25 |
|
26 |
* **Model type**: Please refer to Mistral technical report for details on the model architecture.
|
|
|
28 |
* **Tokenizer**: This model employs a tokenizer that features a broadened vocabulary based on Japanese data. This allows for a more efficient representation of text using fewer tokens, leading to a notably faster inference process.
|
29 |
* **Contact**: swallow[at]nlp.c.titech.ac.jp
|
30 |
|
31 |
+
## Instruct Model Performance
|
32 |
+
|
33 |
+
|
34 |
## Base Model Performance
|
35 |
|
36 |
### Japanese tasks
|
|
|
143 |
import torch
|
144 |
|
145 |
model_name = "tokyotech-llm/Swallow-MS-7b-v0.1"
|
146 |
+
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
147 |
|
148 |
+
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
|
149 |
prompt = "東京工業大学の主なキャンパスは、"
|
150 |
input_ids = tokenizer.encode(
|
151 |
prompt,
|
|
|
174 |
- [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)
|
175 |
- [Swallow Corpus](https://chokkan.org/temp/tokyotech-llm/swallow-corpus)
|
176 |
- [The Pile](https://huggingface.co/datasets/EleutherAI/pile)
|
177 |
+
|
178 |
## Risks and Limitations
|
179 |
|
180 |
The models released here are still in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.
|