dreamerdeo commited on
Commit
2f79132
1 Parent(s): c4374dd

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +211 -0
README.md ADDED
@@ -0,0 +1,211 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ - zh
5
+ - id
6
+ - th
7
+ - vi
8
+ - ms
9
+ - lo
10
+ datasets:
11
+ - cerebras/SlimPajama-627B
12
+ - Skywork/SkyPile-150B
13
+ - allenai/MADLAD-400
14
+ - cc100
15
+ tags:
16
+ - multilingual
17
+ - sea
18
+ - sailor
19
+ license: apache-2.0
20
+ base_model: Qwen/Qwen1.5-4B
21
+ model-index:
22
+ - name: Sailor-4B
23
+ results:
24
+ - task:
25
+ type: text-generation
26
+ dataset:
27
+ name: XQuAD-Thai
28
+ type: XQuAD-Thai
29
+ metrics:
30
+ - name: EM (3-Shot)
31
+ type: EM (3-Shot)
32
+ value: 46.82
33
+ - name: F1 (3-Shot)
34
+ type: F1 (3-Shot)
35
+ value: 63.34
36
+ - task:
37
+ type: text-generation
38
+ dataset:
39
+ name: TyDiQA-Indonesian
40
+ type: TyDiQA-Indonesian
41
+ metrics:
42
+ - name: EM (3-Shot)
43
+ type: EM (3-Shot)
44
+ value: 53.98
45
+ - name: F1 (3-Shot)
46
+ type: F1 (3-Shot)
47
+ value: 73.48
48
+ - task:
49
+ type: text-generation
50
+ dataset:
51
+ name: XQuAD-Vietnamese
52
+ type: XQuAD-Vietnamese
53
+ metrics:
54
+ - name: EM (3-Shot)
55
+ type: EM (3-Shot)
56
+ value: 47.65
57
+ - name: F1 (3-Shot)
58
+ type: F1 (3-Shot)
59
+ value: 67.09
60
+ - task:
61
+ type: text-generation
62
+ dataset:
63
+ name: XCOPA-Thai
64
+ type: XCOPA-Thai
65
+ metrics:
66
+ - name: EM (3-Shot)
67
+ type: EM (3-Shot)
68
+ value: 53.4
69
+ - task:
70
+ type: text-generation
71
+ dataset:
72
+ name: XCOPA-Indonesian
73
+ type: XCOPA-Indonesian
74
+ metrics:
75
+ - name: EM (3-Shot)
76
+ type: EM (3-Shot)
77
+ value: 69.20
78
+ - task:
79
+ type: text-generation
80
+ dataset:
81
+ name: XCOPA-Vietnamese
82
+ type: XCOPA-Vietnamese
83
+ metrics:
84
+ - name: EM (3-Shot)
85
+ type: EM (3-Shot)
86
+ value: 68.20
87
+ - task:
88
+ type: text-generation
89
+ dataset:
90
+ name: M3Exam-Thai
91
+ type: M3Exam-Thai
92
+ metrics:
93
+ - name: EM (3-Shot)
94
+ type: EM (3-Shot)
95
+ value: 27.88
96
+ - task:
97
+ type: text-generation
98
+ dataset:
99
+ name: M3Exam-Indonesian
100
+ type: M3Exam-Indonesian
101
+ metrics:
102
+ - name: EM (3-Shot)
103
+ type: EM (3-Shot)
104
+ value: 31.27
105
+ - task:
106
+ type: text-generation
107
+ dataset:
108
+ name: M3Exam-Vietnamese
109
+ type: M3Exam-Vietnamese
110
+ metrics:
111
+ - name: EM (3-Shot)
112
+ type: EM (3-Shot)
113
+ value: 40.69
114
+ - task:
115
+ type: text-generation
116
+ dataset:
117
+ name: BELEBELE-Thai
118
+ type: BELEBELE-Thai
119
+ metrics:
120
+ - name: EM (3-Shot)
121
+ type: EM (3-Shot)
122
+ value: 36.11
123
+ - task:
124
+ type: text-generation
125
+ dataset:
126
+ name: BELEBELE-Indonesian
127
+ type: BELEBELE-Indonesian
128
+ metrics:
129
+ - name: EM (3-Shot)
130
+ type: EM (3-Shot)
131
+ value: 41.33
132
+ - task:
133
+ type: text-generation
134
+ dataset:
135
+ name: BELEBELE-Vietnamese
136
+ type: BELEBELE-Vietnamese
137
+ metrics:
138
+ - name: EM (3-Shot)
139
+ type: EM (3-Shot)
140
+ value: 38.89
141
+ ---
142
+
143
+ <div align="center">
144
+ <img src="banner_sailor.jpg" width="700"/>
145
+ </div>
146
+
147
+ Sailor is a suite of Open Language Models tailored for South-East Asia (SEA), focusing on languages such as 🇮🇩Indonesian, 🇹🇭Thai, 🇻🇳Vietnamese, 🇲🇾Malay, and 🇱🇦Lao.
148
+ Developed with careful data curation, Sailor models are designed to understand and generate text across diverse linguistic landscapes of SEA region.
149
+ Built from [Qwen 1.5](https://huggingface.co/collections/Qwen/qwen15-65c0a2f577b1ecb76d786524) , Sailor encompasses models of varying sizes, spanning from 0.5B to 7B versions for different requirements.
150
+ We further fine-tune the base model with open-source datasets to get instruction-tuned models, namedly Sailor-Chat.
151
+ Benchmarking results demonstrate Sailor's proficiency in tasks such as question answering, commonsense reasoning, and other tasks in SEA languages.
152
+
153
+ ## Model Summary
154
+ - **Model Collections:** [Base Model & Chat Model](https://huggingface.co/collections/sail/sailor-65e19a749f978976f1959825)
155
+ - **Project Website:** [sailorllm.github.io](https://sailorllm.github.io/)
156
+ - **Codebase:** [github.com/sail-sg/sailor-llm](https://github.com/sail-sg/sailor-llm)
157
+ - **Technical Report:** Coming Soon
158
+
159
+
160
+ ## Training details
161
+ Sailor is crafted by continually pre-training from language models like the remarkable Qwen 1.5 models, which already has a great performance on SEA languages.
162
+ The pre-training corpus heavily leverages the publicly available corpus, including
163
+ [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B),
164
+ [SkyPile](https://huggingface.co/datasets/Skywork/SkyPile-150B),
165
+ [CC100](https://huggingface.co/datasets/cc100) and [MADLAD-400](https://huggingface.co/datasets/allenai/MADLAD-400).
166
+
167
+ By employing aggressive data deduplication and careful data cleaning on the collected corpus, we have attained a high-quality dataset spanning various languages.
168
+ Through systematic experiments to determine the weights of different languages, Sailor models undergo training from 200B to 400B tokens, tailored to different model sizes.
169
+ The approach boosts their performance on SEA languages while maintaining proficiency in English and Chinese without significant compromise.
170
+ Finally, we continually pre-train the Qwen1.5-0.5B model with 400 Billion tokens, and other models with 200 Billion tokens to obtain the Sailor models.
171
+
172
+ ## Requirements
173
+ The code of Sailor has been in the latest Hugging face transformers and we advise you to install `transformers>=4.37.0`.
174
+
175
+ ## Quickstart
176
+
177
+ Here provides a code snippet to show you how to load the tokenizer and model and how to generate contents.
178
+
179
+ ```python
180
+ from transformers import AutoModelForCausalLM, AutoTokenizer
181
+ device = "cuda" # the device to load the model
182
+
183
+ model = AutoModelForCausalLM.from_pretrained("sail/Sailor-7B", device_map="auto")
184
+ tokenizer = AutoTokenizer.from_pretrained("sail/Sailor-7B")
185
+
186
+ input_message = "Model bahasa adalah model probabilistik"
187
+ ### The given Indonesian input translates to 'A language model is a probabilistic model of.'
188
+
189
+ model_inputs = tokenizer([input_message], return_tensors="pt").to(device)
190
+
191
+ generated_ids = model.generate(
192
+ model_inputs.input_ids,
193
+ max_new_tokens=64
194
+ )
195
+
196
+ generated_ids = [
197
+ output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
198
+ ]
199
+
200
+ response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
201
+ print(response)
202
+ ```
203
+
204
+ # License
205
+
206
+ Sailor is distributed under the terms of the Apache License 2.0.
207
+ No restrict on the research and the commercial use, but should comply with the [Qwen License](https://huggingface.co/Qwen/Qwen1.5-1.8B/blob/main/LICENSE).
208
+
209
+ # Contact Us
210
+
211
+ If you have any questions, please raise an issue or contact us at [doulx@sea.com](mailto:doulx@sea.com) or [liuqian@sea.com](mailto:liuqian@sea.com).