N usa X : Multilingual Parallel Sentiment Dataset for 10 I ndonesian Local Languages (2023)

Abstract: Natural language processing (NLP) has a significant impact on society via technologies such as machine translation and search engines. Despite its success, NLP technology is only widely available for high-resource languages such as English and Chinese, while it remains inaccessible to many languages due to the unavailability of data resources and benchmarks. In this work, we focus on developing resources for languages in Indonesia. Despite being the second most linguistically diverse country, most languages in Indonesia are categorized as endangered and some are even extinct. We develop the first-ever parallel resource for 10 low-resource languages in Indonesia. Our resource includes sentiment and machine translation datasets, and bilingual lexicons. We provide extensive analyses and describe challenges for creating such resources. We hope this work can spark NLP research on Indonesian and other underrepresented languages.

## introduction indonesia is one of the most populous and linguistically-diverse countries in the world, with more than 700 languages spoken across the country @xcite . however, while many of these languages are spoken by millions of people they have received little attention from the nlp community. there are very few public datasets, preventing the global research community from exploring these languages. to this end, we introduce nusax, 1 a high-quality multilingual parallel corpus that covers 10 local languages from indonesia: acehnese, balinese, banjarese, buginese, madurese, minangkabau, javanese, ngaju, sundanese, and toba batak. the nusax dataset was created by translating smsa (purwarianti and crisdayanti, 2019) -an existing indonesian sentiment analysis dataset containing comments and reviews from the indonlu benchmark @xcite ) -using competent bilingual speakers, coupled with additional human-assisted quality assurance. sentiment analysis is one of the most popular nlp tasks, and has been explored in many applications in indonesia, including presidential elections @xcite , product reviews @xcite , stock forecasting @xcite , and covid-19 monitoring @xcite . by translating an existing text, we additionally produce a parallel corpus, which is useful for building and evaluating translation systems. as we translate from a regional high-resource language (indonesian), we ensure that the topics and entities reflected in the data are culturally relevant to the other languages, which is generally not the case when translating an english dataset @xcite . we apply the corpus to two downstream tasks: sentiment analysis and machine translation. we use the new benchmark to assess the performance of existing indonesian language models (lms), multilingual lms, and classical machine learning methods. our contributions are as follows: we work on 10 local languages in indonesia: acehnese, balinese, banjarese, buginese, madurese, minangkabau, javanese, ngaju, sundanese, and toba batak. most of these languages have a population of over 2 million speakers @xcite , but are underrepresented in nlp research. figure 1 shows the taxonomy of these languages and indonesian. geographically, these languages are spoken on different big islands in indonesia, including sumatra, borneo, java, madura, and sulawesi. the languages belong to the austronesian language family under the malayo-polynesian subgroup. while some of the covered languages are written in multiple scripts, we use the latin script in nusax, which has become predominant for all covered languages. indonesian (ind) is the national language of indonesia based on the 1945 constitution of the republic of indonesia (article 36). it is written in latin script, and was developed from literary "classical malay" of the riau-johor sultanate @xcite , with regional variants. its lexical similar-ity to standard malay is over 80%. it has a rich affixation system, including a variety of prefixes, suffixes, circumfixes, and reduplication. most of the affixes in indonesian are derivational @xcite . acehnese (ace) is a language spoken mainly in the aceh province. although it is the de facto language of aceh, language use is shifting to indonesian in urban areas. acehnese has features typical of the mon-khmer languages of mainland southeast asia, a result of its former status as part of the early chamic dialect continuum on the coast of vietnam. in addition to the large number of diphthongs, it has a high percentage of monosyllabic root morphemes. balinese (ban) is a language spoken mainly in the bali province. it has three main dialects: highland balinese, lowland balinese, and nusa penida. since the early 20th century, it has mainly been written in the latin script, but also has its own balinese script. the word order in balinese is svo. balinese has three sociolinguistic registers @xcite . banjarese (bjn) is a language spoken in kalimantan (central, east, south, and west kalimantan provinces). it is dominant in the south kalimantan province and is also growing rapidly in the central and eastern kalimantan provinces. it has two main dialects: kuala and hulu. although it is a malayic language, it has many javanese loanwords, probably acquired during the majapahit period from the late thirteenth century until the fifteenth century @xcite . it has 73% of lexical similarity with indonesian and is written in arabic and latin scripts @xcite . buginese (bug) is a language spoken mainly in the south sulawesi, southeast sulawesi, central sulawesi, and west sulawesi provinces. the word order is svo. verb affixes are used to mark persons. historically, it was written in the buginese script (derived from brahmi script), but is mainly written in latin script now @xcite . buginese employs sentence patterns, pronouns, and other terms to express politeness @xcite . madurese (mad) is a language spoken in the east java province, mainly on madura island, south and west of surabaya city, bawean, kangean, and sapudi islands. it has vowel harmony, gemination, rich affixation, reduplication, and svo basic word order @xcite . minangkabau (min) is a language spoken mainly in west sumatra and other provinces on sumatra island such as bengkulu and riau. although it is classified as malay, it is not intelligible with indonesian. standard minangkabau voice can be characterised as an indonesian-type system, whereas colloquial minangkabau voice is more effectively characterised as a sundic-type system @xcite . javanese (jav) is a language spoken mainly on java island. it is the de facto language of provincial identity in central and eastern java. the number of native javanese speakers is greater than the number of indonesian l1 speakers @xcite . javanese consists of several regional dialects, which differ primarily in pronunciation and vocabulary. javanese has an elaborate system of speech levels related to the relation of the speaker to the interlocutor that depend on social status, age, kinship distance, and familiarity @xcite . it used to be written in javanese script, but since the 20th century has mostly been written in latin script. ngaju (nij) is a language spoken in the central kalimantan province. it is widely used as a language for trade in much of kalimantan, from the barito to the sampit river. it has various affixes and reduplication, and its word order is similar to indonesian. pronouns have enclitic forms to mark possessors in a noun phrase or passive agents @xcite . sundanese (sun) is a language spoken mainly in the banten and west java provinces. it is the de facto language of provincial identity in western java. the main dialects are bogor (krawang), pringan, and cirebon. it has elaborate coding of respect levels. it has been written in latin script since the mid-19th century but was previously written in arabic, javanese, and sundanese scripts. sundanese is a predominantly svo language, and has voice marking and incorporates some (optional) actor-verb agreement, i.e., number and person @xcite . toba batak (bbc) is a language spoken in the north sumatra province. similarly to acehnese, it is slowly being replaced by indonesian in urban and migrant areas. it used to be written in the batak script but is mainly written in latin script now. the batak languages are verb-initial, and have verb systems reminiscent of philippine languages, although they differ from them in many details @xcite . ## data construction our data collection process consists of several steps. first, we take an existing dataset in a high-resource local language (indonesian) as a base for expansion to the other ten languages, and ask human annotators to translate the text. to ensure the quality of the final translation, we run quality assurance with additional human annotators. ## data collection challenges in this section, we discuss challenges faced during data collection. finding annotators we found collecting the nusax dataset challenging. first of all, finding local language-speaking annotators is not easy, and popular platforms such as mturk do not support these languages. instead, we looked for annota-tors through local indonesian networks and forums, such as the inacl forum, local campus forums, or the indonesian polyglot community. we intended to cover as many local languages as possible, but based on the available annotators, only the 10 languages presented in this paper were possible, as we needed at least 2 annotators for each language. searching for annotators online is not easy, due to disparities in internet penetration in different parts of indonesia. hence, we might not reach potential annotators through online communities alone. however, holding an in-person workshop for data collection is also not practical; indonesia is an archipelago and traveling between islands is costly. similar challenges occur in many other regions, including africa and south america. communication with annotators communication between the authors and annotators was done through whatsapp, as the most popular communication tool in indonesian @xcite . annotation was conducted through spreadsheets. we found that some of the annotators use mobile apps instead of a desktop for annotation. their reasons include ease of use, no access to a laptop, and better keyboard support for typing diacritics. in the most extreme case, one annotator printed out the sheet and performed the annotation on paper, then took a picture of the paper and sent it back to us. we found some annotators to be difficult to contact, due to other commitments such as college or work. some of them were not responsive and had to be replaced by new annotators. multilingual parallel corpora several multilingual parallel corpora have been developed to support studies on machine translation such as gcp (imamura and sumita, 2018), leipzig (gold-emerging language benchmarks recently, benchmarks in underrepresented languages have emerged, such as masakhaner (adelani et al., 2021), americasnli (ebrahimi et al., 2022), pmindia (haddow and kirefu, 2020), samanantar (ramesh et al., 2022), and naijasenti (muhammad @xcite . particularly, for indonesian languages, nlp benchmarks have been developed such as indonlu @xcite , indolem @xcite , indonlg @xcite , indonli @xcite , and english-indonesian machine translation @xcite . ## conclusion in this paper, we propose nusax, the first parallel corpus for 10 low-resource indonesian languages. we create a new benchmark for sentiment analysis and machine translation in zero-shot and full-data settings. we present a comprehensive analysis of the language similarity of these languages from both linguistic and empirical perspectives by assessing the cross-lingual transferability of existing indonesian and multilingual pre-trained models. we hope nusax can enable nlp research for under-represented languages, and can be used as a testbed for adaptation or few-shot learning methods that take advantage of similarities between languages. nusax opens up the possibility for future research that focuses on covering more local languages, and additionally, further extension to other tasks and domains. our study on cross-lingual transfer enables further exploration on cross-lingual zero-shot learning for more diverse tasks in local languages. our guidelines and discussion of data collection issues may also motivate future work on more efficient high-quality data collection for extremely low-resource languages.