Sailor: Open Language Models for South- E ast A sia (2024)

Abstract: We present Sailor, a family of open language models ranging from 0.5B to 14B parameters, tailored for South-East Asian (SEA) languages. From Qwen1.5, Sailor models accept 200B to 400B tokens during continual pre-training, primarily covering the languages of English, Chinese, Vietnamese, Thai, Indonesian, Malay, and Lao. The training leverages several techniques, including BPE dropout for improving the model robustness, aggressive data cleaning and deduplication, and small proxy models to optimize the data mixture. Experimental results on four typical tasks indicate that Sailor models demonstrate strong performance across different benchmarks, including commonsense reasoning, question answering, reading comprehension and examination. We share our insights to spark a wider interest in developing large language models for multilingual use cases.

## introduction large language models (llms) have seen remarkable improvements recently, driven by the rapid growth of internet data @xcite and advances in pre-training techniques. however, mainstream llms @xcite @xcite primarily rely on english data for training. for example, 89.70% of the training data of llama-2 is english @xcite . consequently, these english-centric llms often struggle to achieve comparable performance across other languages (e.g., thai), due to their inadequate exposure to those languages during pre-training. in this paper, we aim to develop the llms that perform well across the south-east asia (sea) region, encompassing a range of languages that include english, chinese, vietnamese, thai, indonesian, malay, and lao. to cater to varying needs, we release both base model and chat model in five variant size (0.5b, 1.8b, 4b, 7b and 14b) foot_0 , offering greater flexibility. additionally, we open source all of our data cleaning and deduplication pipeline foot_1 that turns out to be extremely important for the quality of llms, especially in the scenario of continual pre-training. besides the open models, we explore several techniques in a fully transparent manner to accelerate the development of multilingual llms, which encompasses three main areas of investigation. first, we employ small-scale models as proxies to optimize hyperparameters for continual pre-training, focusing on learning rates and data mixture ratios from diverse sources. second, we examine the efficacy of various data processing techniques, including the merging of adjacent short examples, as well as document-level and wordlevel code-switching. finally, we address tokenization challenges by investigating the use of bpe dropout @xcite to improve the robustness of llms. with exploring the above techniques, we summarize the key insights for multilingual llm continual pre-training, as illustrated in figure 1: (1) language models struggle with multiple languages, and continual pre-training presents an opportunity to improve specific language capabilities. (2) code-switching techniques can be beneficial in multilingual scenarios, improving the ability to handle language mixing. (3) language models are sensitive to subword segmentation, and techniques like bpe dropout can improve model robustness. (4) even available high-quality multilingual corpora may still require further data deduplication and cleaning. (5) simulation experiments on smaller models can provide insights into performance trends for large-scale experiments. ## continue pre-training for base model a crucial aspect of continual pre-training is meticulous data processing and the selection of a suitable llm as the foundation. this section outlines our data processing pipeline, model selection criteria, and implementation details. ## insights during sailor development, we perform ablation studies on small lms to understand the impact of various strategies foot_8 . we then apply the key insights gained from these studies to improve llm. all techniques are listed in ## conclusion in this paper, we present the sailor family of open language models (apache license 2.0), tailored for south-east asian languages, which exhibit strong performance across various multilingual tasks and benchmarks, fostering advancements in multilingual language models for the sea region.