One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in I ndonesia (2022)

Abstract: NLP research is impeded by a lack of resources and awareness of the challenges presented by underrepresented languages and dialects. Focusing on the languages spoken in Indonesia, the second most linguistically diverse and the fourth most populous nation of the world, we provide an overview of the current state of NLP research for Indonesia’s 700+ languages. We highlight challenges in Indonesian NLP and how these affect the performance of current NLP systems. Finally, we provide general recommendations to help develop NLP technology not only for languages of Indonesia but also other underrepresented languages.

## introduction research in natural language processing (nlp) has traditionally focused on developing models for english and a small set of other languages with large amounts of data (see figure 1 , bottom right). while the lack of data is generally cited as the key reason for the lack of progress in nlp for underrepresented languages @xcite , we argue that another factor relates to the diversity and the lack of understanding of the linguistic characteristics of such languages. through the lens of the languages spoken in indonesia, the world's second-most linguistically diverse country, we seek to illustrate the challenges in applying nlp technology to such a diverse pool of languages. indonesia is the 4th most populous nation globally, with 273 million people spread over 17,508 islands. there are more than 700 languages spoken in indonesia, equal to 10% of the world's languages, second only to papua new guinea @xcite . however, most of these languages are not well documented in the literature; many are not formally taught, and no established standard exists across speakers @xcite , we compile acl anthology to count the distribution of published works that mention languages spoken in indonesia. top: distribution of papers in 20 years. bottom: number of papers per a million speakers. we compare languages spoken in europe, asia, and indonesia. 2020). many of them are decreasing in use, as indonesian (bahasa indonesia), the national language, is more frequently used as the primary language across the country. this process may ultimately result in a monolingual society @xcite . among more than 700 indonesian local languages, many are threatened. 440 languages are listed as endangered and 12 as extinct according to data from ethnologue @xcite illustrated in figure 2 . @xcite found nearly half of a sample of 98 indonesian local languages to be endangered while @xcite observed 71 among 151 indonesian local languages to have less than 100k speakers. left: language vitality. right: speaker count. ## opportunities based on the challenges for indonesian nlp highlighted in the previous section, we formulate proposals for improving the state of indonesian nlp research, as well as of other underrepresented languages. our proposals cover several aspects including metadata documentation; potential research directions; and engagement with communities. ## conclusion in this paper, we highlight challenges in indonesian nlp. indonesia is one of the most populous countries and the second-most linguistically diverse country of the world, with over 700 local languages, yet indonesian nlp is underrepresented and underexplored. based on the observed challenges, we also present recommendations to improve the situation, not only for indonesian but also for other underrepresented languages. marco lui and timothy baldwin. 2012. langid. py: an off-the-shelf language identification tool. in proceedings of the acl 2012 system demonstrations, pages 25-30. edwin lunando and ayu purwarianti. 2013. indonesian social media sentiment analysis with sarcasm detection. in 2013 international conference on advanced computer science and information systems (icacsis), pages 195-198. andrea madotto, zhaojiang lin, genta indra winata, and pascale fung. 2021. few-shot bot: promptbased learning for dialogue systems. arxiv preprint arxiv:2110.08118. putu devi maharani and komang dian puspita candra. 2018. variasi leksikal bahasa bali dialek kuta selatan. mudra jurnal seni budaya, 33(1):76-84. rahmad mahendra, alham fikri aji, samuel louvan, fahrurrozi rahman, and clara vania. 2021. in-donli: a natural language inference dataset for indonesian. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 10511-10527, online and punta cana, dominican republic. association for computational linguistics. rahmad mahendra, septina dian larasati, and ruli manurung. 2008. extending an indonesian semantic analysis-based question answering system with linguistic and world knowledge axioms. in proceedings of the 22nd pacific asia conference on language, information and computation, pages 262-271, the university of the philippines visayas cebu college, cebu city, philippines. de la salle university, manila, philippines. rahmad mahendra, heninggar septiantri, haryo akbarianto wibowo, ruli manurung, and mirna adriani. 2018. cross-lingual and supervised learning approach for indonesian word sense disambiguation task. in proceedings of the 9th global wordnet conference, pages 245-250, nanyang technological university (ntu), singapore. global wordnet association. miftahul mahfuzh, sidik soleman, and ayu purwarianti. 2019. improving joint layer rnn based keyphrase extraction by using syntactical features. in 2019 international conference of advanced informatics: concepts, theory and applications (icaicta), pages 1-6. ieee. ryan mcdonald, joakim nivre, yvonne quirmbach-brundage, yoav goldberg, dipanjan das, kuzman ganchev, keith hall, slav petrov, hao zhang, oscar täckström, claudia bedini, núria bertomeu castelló, and jungmee lee. 2013. universal dependency annotation for multilingual parsing. in proceedings of the 51st annual meeting of the association for computational linguistics (volume 2: short papers), pages 92-97, sofia, bulgaria. association for computational linguistics. figure 5: count of published papers per year on indonesian and european languages from 2000 to 2020.