SEA - HELM : S outheast A sian Holistic Evaluation of Language Models (2025)

Abstract: With the rapid emergence of novel capabilities in Large Language Models (LLMs), the need for rigorous multilingual and multiculturalbenchmarks that are integrated has become more pronounced. Though existing LLM benchmarks are capable of evaluating specificcapabilities of LLMs in English as well as in various mid- to low-resource languages, including those in the Southeast Asian (SEA)region, a comprehensive and culturally representative evaluation suite for the SEA languages has not been developed thus far.Here, we present SEA-HELM, a holistic linguistic and cultural LLM evaluation suite that emphasises SEA languages, comprisingfive core pillars: (1) NLP CLASSICS, (2) LLM-SPECIFICS, (3) SEA LINGUISTICS, (4) SEA CULTURE, (5) SAFETY. SEA-HELMcurrently supports Filipino, Indonesian, Tamil, Thai, and Vietnamese. We also introduce the SEA-HELM leaderboard, which allows users to understand models’ multilingual and multicultural performance in a systematic and user-friendly manner. We make the SEA-HELM evaluation code publicly available.

## introduction the proliferation of generative approaches to natural language processing (nlp) through large language models (llms) has rendered many traditional datasets for nlp evaluation compromised @xcite , obsolete or saturated @xcite . while essentially trained to predict the next token in a sequence, llms have shown significant emergent competencies, including summarisation, question answering, translation, coding, and advanced reasoning (brown et al., 2020; yeo 1 formerly known as bhasa @xcite ). et al., 2024). they are also extensively used for new applications, such as chatbots that can hold sustained open-ended conversations @xcite . this advancement has led to a significant disparity between the range of llm capabilities and the datasets and frameworks to evaluate them rigorously. traditional approaches to nlp evaluation, which emphasise on alignment with a predefined ground truth reference, are not sufficient in measuring the complex abilities of llms @xcite @xcite . this gap is further exacerbated in lower-resource languages in southeast asia (sea), owing to a lack of both training and testing data on the internet @xcite . the sea region is home to nearly 700 million speakers across more than 1,000 languages (see appendix h.18). while this region represents almost 10% of the global population and constitutes approximately one-seventh of the world's total languages, most of these languages remain unsupported by major llms, such as mistral @xcite and claude @xcite . the complications arising from a lack of data as well as uneven digital access and representation contribute to the impeded development of llms in those languages in the sea region. this matter is further aggravated by the fact that some sea languages are written in non-latin scripts, which presents a challenge for tokenizers when processing limited data. despite the mentioned obstacles, multilingual llm and benchmark development in sea strive to close the gap and adapt to the current trends in the field. some models now explicitly support sea languages and also claim to provide representation for sea cultural knowledge @xcite @xcite . there have also been many benchmarks claiming to measure llms' multilingual and multicultural capabilities for the sea region @xcite @xcite @xcite @xcite . however, there has yet to be a comprehensive and culturally representative benchmark suite for evaluating llms for sea cultural and linguistic competencies. we have thus developed sea-helm (southeast asian holistic evaluation of language models), foot_0 a systematic, integrated, and continually maintained benchmark suite which aims to measure the sea language and cultural competencies of llms in a targeted and comprehensive manner. sea-helm achieves integration by collating localised evaluation datasets and llm prompts, running tests on models together to enable standardised comparisons, and presenting results aggregated by languages, tasks and models. it is our view that no single metric explains a model's suitability for sea, and thus sea-helm is designed to test a holistic set of competencies, illustrated in figure 1 . specifically, sea-helm is organised into five evaluation pillars: (1) nlp classics; (2) llm-specifics; (3) sea linguistics; (4) sea culture; and (5) safety, which together encompass an extensive range of tasks for each sea language to ensure that a wide range of relevant aspects, from linguistic nuances to cultural representations, are considered and accounted for. the five pillars are also meticulously and rigorously crafted to achieve fair, transparent, and authentic multilingual and multicultural evaluations of llms in the region. we deliberately incorporate community participation by involving native speakers of the sea languages at each stage of the dataset planning and construction to ensure linguistic accuracy and cultural authenticity. we summarise the contributions of sea-helm here: ## sea-helm to address the lack of holistic multilingual and multicultural evaluations for the sea region, we designed and developed sea-helm, which draws its inspiration from helm @xcite . this evaluation suite consists of five core pillars: (1) nlp classics, (2) llm-specifics, (3) sea linguistics, (4) sea culture, and (5) safety, and has been recently integrated with helm. the spread of tasks and languages is detailed in ## conclusion in conclusion, we introduce sea-helm, a holistic evaluation suite that caters to the sea languages and cultures. to achieve a comprehensive evaluation suite, sea-helm was designed around the following five core pillars: (1) nlp classics, (2) llm-specifics, (3) sea linguistics, (4) sea culture, (5) safety. additionally, the sea-helm leaderboard is intended to serve as a comprehensive and regularly maintained resource for ai researchers in academia and industry. our results show that there are still significant gaps between high-resource languages such as english and the mid-to low-resource languages in southeast asia, including several with official status and millions of speakers. however, these results also show that the dedicated fine-tuning of llms with parameter sizes between 7 and 9 billion can significantly narrow the gap with respect to the much larger state-of-the-art models such as gpt-4o and deepseek-r1. thus, by calling attention to the realistic, localised needs of sea languages, we encourage more concentrated efforts on data collection, curation and fine-tuning of dedicated lighter-weight llm solutions for the region.