N usa BERT : Teaching I ndo BERT to be Multilingual and Multicultural (2025)

Abstract: We present NusaBERT, a multilingual model built on IndoBERT and tailored for Indonesia’s diverse languages. By expanding vocabulary and pre-training on a regional corpus, NusaBERT achieves state-of-the-art performance on Indonesian NLU benchmarks, enhancing IndoBERT’s multilingual capability. This study also addresses NusaBERT’s limitations and encourages further research on Indonesia’s underrepresented languages.

## introduction indonesia's exceptional linguistic landscape, encompassing over 700 languages and dialects @xcite , presents a significant challenge for current natural language processing (nlp) techniques, such as pre-trained language models. these techniques often fall short in handling the nation's intricate and multifaceted linguistic tapestry. furthermore, the bilingual nature of indonesian colloquial conversations (mixing indonesian and english) with the majority continuing to also communicate in regional languages as their daily conversational language poses a complex problem to be solved by language models. nonetheless, pre-trained language models have shown remarkable progress in recent years showing their ability to solve a wide range of natural language processing tasks, including the indonesian language. these language models are trained on a large corpus and are fine-tuned to solve specific, downstream tasks. language models such 1 https://github.com/lazarusnlp/nusabert as bert @xcite and gpt @xcite are typically trained on a monolingual corpus and were originally trained on an english corpus. in the studies that followed, languagespecific language models like indobert @xcite and indobart @xcite have been tailored for the indonesian language and regional languages of indonesia like javanese and sundanese. despite the large size discrepancy between the english and indonesian corpus, indobert managed to leverage the contextualized indonesian language model to attain exceptional results on multiple downstream natural language understanding (nlu) tasks. although demonstrating remarkable capabilities across various tasks, these models often perform poorly when applied to languages with unique characteristics like those found in the many regions of indonesia. for instance, indobert faces limitations when addressing the intricacies of codemixing @xcite and the specific needs of low-resource languages @xcite . furthermore, while efforts like xlm-r @xcite and mbert @xcite have aimed to introduce crosslinguality, their focus on achieving state-of-the-art performance in cross-lingual language understanding tasks may not fully address the unique issues faced by language models operating within indonesia's complex multilingual and multicultural environment. @xcite showed that even these large multilingual models fail to outperform classical baselines on extremely low-resource languages. in light of this, we propose nusabert, a model that builds upon indobert and targets the linguistic complexities of low-resource regional languages in indonesia. nusabert leverages the vocabulary expansion technique proposed by phayathaibert @xcite , and aims to achieve state-ofthe-art performance on multilingual benchmarks. ## related works recent years have witnessed significant progress in indonesian nlp research. pre-trained language models like indobert @xcite and indobart @xcite have demonstrated the effectiveness of this approach for various indonesian language tasks. indobert, based on bert @xcite , was specifically trained on a large indonesian text corpus. it achieved state-of-the-art performance on the in-donlu benchmark @xcite , a collection of indonesian-specific nlu tasks like text classification, question answering, and named entity recognition, demonstrating its competence in understanding the nuances of the indonesian language. indobart, based on the bart architecture @xcite , focuses on sequenceto-sequence tasks within the indonesian language. this model has found success in language generation tasks like machine translation and text summarization, highlighting its ability to process and produce natural indonesian text. nusax @xcite , a benchmark for 10 under-resourced indonesian local languages, shows that when indobert and indobart are fine-tuned for these languages, they achieve impressive results in sentiment analysis and machine translation, respectively. afterward, nusawrites @xcite was released and complements nusax by providing a more lexically diverse and culturally relevant dataset on 12 underrepresented local languages. upon fine-tuning different models on these new benchmarks, results show that multilingual models like xlm-r @xcite and mbert @xcite and monolingual models (indobert, indobart, indogpt) fail to outperform classical machine learning models on several extremely low-resource languages. the success of phayathaibert @xcite , a thai language model specifically designed to address the challenge of unassimilated loanwords, offers valuable inspiration for tackling challenges faced by nlp models in indonesia. similar to thai, low-resource regional languages in indonesia are frequently influenced by other languages due to code-mixing. this phenomenon leads to a significant number of unassimilated loanwords, which are words from other languages adopted into the regional language but fully integrated into its grammar. phayathaibert addresses this challenge by incorporating techniques such as vocab-ulary expansion. this technique involves augmenting the model's vocabulary with these loanwords and variations, allowing it to better recognize and understand them within the context of the regional language. similarly, indobertweet @xcite , an extension of indolem's indobert @xcite , tackles the challenge of informal language and social media slang by specifically augmenting its vocabulary with terms commonly found in indonesian twitter data. their vocabulary expansion and subword embedding averaging technique @xcite helped the model better understand and process the nuances of informal communication, which often deviates from standard indonesian grammar and incorporates slang terms. ## nusabert this section introduces the vocabulary expansion method applied to indobert @xcite , the corpus dataset used for training, and the continued pre-training procedure. subsequently, we will evaluate our resultant models on downstream tasks to measure their natural language understanding, multilinguality, and multicultural capabilities. ## conclusion in this study, we introduced nusabert, a multilingual language model specifically tailored to the linguistic diversity of indonesia. basing our model on indobert, we applied vocabulary expansion and continued pre-training on a multilingual corpus that introduces the regional languages of indonesia. nusabert achieves state-of-the-art results when evaluated on indonesian and multilingual nlu benchmarks such as indonlu, nusax, and nusawrites. these findings highlight the effectiveness of our proposed approach in enhancing the multilingual and multicultural capabilities of indobert to address indonesia's unique linguistic framework. we also discussed several limitations of nusabert and how to potentially resolve them. we hope nusabert will enable further research in the under-represented languages of indonesia.