S ea LLM s - Large Language Models for S outheast A sia (2024)

Abstract: Despite the remarkable achievements of large language models (LLMs) in various tasks, there remains a linguistic bias that favors high-resource languages, such as English, often at the expense of low-resource and regional languages. To address this imbalance, we introduce SeaLLMs, an innovative series of language models that specifically focuses on Southeast Asian (SEA) languages. SeaLLMs are built upon popular English-centric models through continued pre-training with an extended vocabulary, specialized instruction and alignment tuning to better capture the intricacies of regional languages. This allows them to respect and reflect local cultural norms, customs, stylistic preferences, and legal considerations. Our comprehensive evaluation demonstrates that SeaLLM models exhibit superior performance across a wide spectrum of linguistic tasks and assistant-style instruction-following capabilities relative to comparable open-source models. Moreover, they outperform ChatGPT-3.5 in non-Latin languages, such as Thai, Khmer, Lao, and Burmese, by large margins while remaining lightweight and cost-effective to operate.

## introduction the advent of large language models (llms) has radically transformed the field of natural language processing, demonstrating remarkable abilities in text generation, comprehension, and decision-making tasks @xcite @xcite @xcite @xcite . while the proficiencies of these models are extraordinary, the majority of existing llms embody a linguistic hierarchy overwhelmingly dominated by english @xcite @xcite . this dominance undermines the multilingual capability of such models, with particularly prejudicial outcomes for lower-resource and regional languages, where data scarcity and tokenization challenges lead to disproportionately poor model performance. this linguistic disparity not only impedes access to state-ofthe-art ai technologies for non-english-speaking populations but also risks cultural homogenization and the loss of linguistic diversity. while hyperpolyglot models exist @xcite @xcite , they may pay a high cost for high-resource language performance while lacking in multilingual instruction-following abilities. recognizing the urgent need to democratize ai and empower linguistically diverse regions, we introduce seallms foot_0 , a suite of specialized language models optimized for southeast asian languages foot_1 . these languages, while rich and diverse, often lack the extensive dataset support available for more widely spoken languages, resulting in a stark performance gap in existing llm applications. as a long-term continuous effort, as of this writing, seallms come in three versions @xcite . seallm-13b-v1, which was pre-trained from llama-2-13b, eclipses the performance of most available open-source llms in a comprehensive array of tasks including world knowledge assessments, language comprehension, and generative capabilities in sea languages. for english and alike, seallms do not only preserve, but also demonstrate enhanced performance in tasks that were part of the original llama training set. when evaluated on multilingual instructionfollowing tasks with gpt-4 as a judge @xcite , seallm-13b-v1 outperforms chatgpt-3.5 by large margins in less-represented languages such as khmer, lao or burmese. meanwhile, seallm-7b-v2, which was pre-trained from mistral-7b @xcite , demonstrates better performances in math and commonsense reasoning than comparable baselines, surpassing chatgpt-3.5 in reasoning for common sea languages, while being much smaller in sizes. later, seallm-7b-v2.5, which was further pre-trained from gemma-7b @xcite , shows significant improvements in sea languages over seallm-7b-v2. figure 2 illustrates the four-stage training process of seallms. in the first stage, detailed in section 2.3, we conduct continuous pre-training from the foundational models @xcite with an extended vocabulary tailored for sea languages. next, we fine-tune the model in a novel hybrid paradigm with a mixture of multilingual pre-training data and englishdominant instruction fine-tuning data (section 3.2). the following stage subsequently fine-tunes the model on a balanced and custom-built multilingual sft dataset. finally, we conduct self-preferencing alignment optimization using the seallm model itself, without relying on human annotators or more powerful llms (openai, 2023b). ## conclusion in conclusion, our research presents a substantial advance in the development of equitable and culturally aware ai with the creation of seallms, a specialized suite of language models attuned to the linguistic and cultural landscapes of southeast asia. through rigorous pre-training enhancements and culturally tailored fine-tuning processes, seallms have demonstrated exceptional proficiency in language understanding and generation tasks, challenging the performance of dominant players such as chatgpt-3.5, particularly in sea languages. the models' attunement to local norms and legal stipulations-validated by human evaluations-establishes seallms as not only a technical breakthrough but a socially responsive innovation, poised to democratize access to high-quality ai language tools across linguistically diverse regions. this work lays a foundation for further research into language models that respect and uphold the rich tapestry of human languages and cultures, ulti-mately driving the ai community towards a more inclusive future.