S ea LLM s 3: Open Foundation and Chat Multilingual Large Language Models for S outheast A sian Languages (2025)

Abstract: Large Language Models (LLMs) have shown remarkable abilities across various tasks, yet their development has predominantly centered on high-resource languages like English and Chinese, leaving low-resource languages underserved. To address this disparity, we present SeaLLMs 3, the latest iteration of the SeaLLMs model family, tailored for Southeast Asian languages. This region, characterized by its rich linguistic diversity, has lacked adequate language technology support. SeaLLMs 3 aims to bridge this gap by covering a comprehensive range of languages spoken in this region, including English, Chinese, Indonesian, Vietnamese, Thai, Tagalog, Malay, Burmese, Khmer, Lao, Tamil, and Javanese. Leveraging efficient language enhancement techniques and a specially constructed instruction tuning dataset, SeaLLMs 3 significantly reduces training costs while maintaining high performance and versatility. Our model excels in tasks such as world knowledge, mathematical reasoning, translation, and instruction following, achieving state-of-the-art performance among similarly sized models. Additionally, we prioritized safety and reliability by addressing both general and culture-specific considerations and incorporated mechanisms to reduce hallucinations. This work underscores the importance of inclusive AI, showing that advanced LLM capabilities can benefit underserved linguistic and cultural communities.

## introduction large language models (llms) such as @xcite and gemini @xcite have demonstrated remarkable capabilities across a wide array of tasks, ranging from natural language understanding and generation to more specialized domain applications @xcite . these models have proven valuable, offering substantial benefits to the global community, especially through the proliferation of open-source llms such as llama @xcite , mistral @xcite , qwen @xcite , and gemma @xcite . however, the majority of these efforts have been concentrated on high-resource languages such as english and chinese, or well-developed regions like europe @xcite . consequently, the development of llms tailored for low-resource languages or underdeveloped regions has been significantly overlooked, resulting in a lack of inclusivity and equitable distribution of ai advancements across diverse linguistic and cultural communities @xcite . to bridge this gap, we introduced the seallms model @xcite , specifically designed llms for southeast asian languages. southeast asia (sea) is a region with a rich diversity of languages spoken by millions of people, yet it suffers from a significant lack of language technology support @xcite . the initiative of seallms thus aims to make the benefits of llms accessible to speakers of these languages, addressing their unique linguistic and cultural nuances. following this endeavor, several other models have been dedicated to this region as well, such as sea-lion (ai singapore, 2023) and sailor @xcite . however, these models often face significant limitations: they are typically released only as foundational or chat models, offer limited options in terms of model size, and cover a limited number of sea languages. moreover, the relatively scarce availability of language corpora further constrains the amount of training data available, hindering the development and performance of these models. in this work, we introduce seallms 3, the latest iteration of the seallms model family. this version is designed to cover a more diverse array of southeast asian languages, including english, chinese, indonesian, vietnamese, thai, tagalog, malay, burmese, khmer, lao, tamil, and javanese. different from the conventional continuepretraining paradigm @xcite @xcite , we conduct efficient language enhancement by training languagespecific neurons only based on a foundation model @xcite , significantly reducing the overall training cost. moreover, such targeted training also ensures that the performance of high-resource languages can remain unaffected during the enhancement. furthermore, seallms 3 is trained using a specially constructed instruction tuning dataset that encompasses a wide variety of task types and carefully balanced language distributions. this approach ensures that the model can handle the linguistic diversity of the southeast asian region while maintaining high performance and versatility across different applications. as a result, it achieves state-of-the-art performance among models with similar sizes, excelling across a diverse array of tasks such as world knowledge @xcite , mathematical reasoning @xcite , translation (costa-jussà et al., 2022), and instruction following. in the meantime, we pay special attention to the model's reliability and trustworthiness during its development, which are often under-considered in multilingual settings @xcite . in particular, we address both general and culture-specific safety considerations to ensure the models provide contextually appropriate responses. the model is also specifically trained to be aware of its knowledge boundary and refuse what it does not know. this focus on safety and reliability has resulted in seallms 3 exhibiting reduced hallucination and delivering safe, coherent responses, especially in queries closely related to southeast asian culture. we open-source both the foundational and chat models of seallms 3 foot_0 . the foundational model could serve as a base for conducting instruction tuning tailored to specific application requirements. meanwhile, the chat model has already undergone instruction tuning and is ready for direct use of handling a wide range of tasks. ## evaluations we conduct extensive evaluations against models with similar sizes, including sailor-7b / sailor-7b-chat @xcite , gemma-7b / gemma-7b-it @xcite , qwen2-7b / qwen2-7b-chat @xcite , llama-3-8b / llama-3-8b-instruct @xcite , aya-23-8b @xcite , and the previous versions (mainly v2.5) of seallms @xcite . the evaluations can be generally categorized into the following two dimensions: (1) model capability: we assess the model's performance on human exam questions, its ability to follow multiturn instructions, its proficiency in mathematics, and its translation accuracy of different language pairs; (2) model trustworthiness: we evaluate the model's safety and tendency to hallucinate, particularly in the context of southeast asia. 4.1 model capability evaluation 4.1.1 multilingual world knowledge dataset we utilize the m3exam dataset @xcite , comprising real human exam questions collected from different countries and spanning different subjects and educational stages. this dataset effectively tests the model's multilingual world knowledge in a manner more akin to realworld settings. we take the questions in english (en), chinese (zh), indonesian (id), vietnamese (vi), and thai (th). we also employ the translated mmlu @xcite questions for evaluation, which primarily tests the cross-lingual alignment of the model as the required knowledge is still mainly western-focused. for evaluation, we use accuracy score as the metric. ## conclusion in conclusion, seallms 3 represents a significant advancement in the development of large language models for southeast asian languages, addressing the region's unique linguistic and cultural challenges. by adopting an efficient language enhancement approach and constructing a comprehensive instruction tuning dataset, seallms 3 achieves state-of-the-art performance while maintaining cost-effectiveness. our commitment to reliability and safety, particularly in contextually appropriate responses and knowledge boundaries, further strengthens the model's applicability and trustworthiness. the open-sourcing of both foundational and chat models ensures that seallms 3 is accessible for a wide range of applications, fostering further innovation and inclusivity in ai development for southeast asia.