MC ^2 : Towards Transparent and Culturally-Aware NLP for Minority Languages in C hina (2024)

Abstract: Current large language models demonstrate deficiencies in understanding low-resource languages, particularly the minority languages in China. This limitation stems from the scarcity of available pre-training data. To address this accessibility challenge, we present MC ^2 , a Multilingual Corpus of Minority Languages in China, which is the largest open-source corpus of its kind so far. MC ^2 includes four underrepresented languages: Tibetan, Uyghur, Kazakh, and Mongolian. Notably, we focus on the less common writing systems of Kazakh and Mongolian, i.e., Kazakh Arabic script and traditional Mongolian script, respectively, which have been long neglected in previous corpus construction efforts. Recognizing the prevalence of language contamination within existing corpora, we adopt a quality-centric solution for collecting MC ^2 , prioritizing accuracy while enhancing diversity. Furthermore, we underscore the importance of attending to the multiplicity of writing systems, which is closely related to the cultural awareness of the resulting models. The MC ^2 corpus and related models are made public to the community.

## introduction recently, the rapid development of large language models (llms) has been fueled by the availability of high-quality pre-training data. however, only a handful of high-resource languages, such as english and chinese, have benefitted from the advantages of this progress in llms. despite having a substantial user base, many languages remain excluded from the benefits of these advancements due to a lack of suitable corpora. in this paper, we focus on such underrepresented languages, particularly addressing the minority languages in china, including tibetan, uyghur, kazakh, and mongolian. as illustrated in figure 1 , the four languages, although spoken by tens of millions, face a critical deficiency in linguistic resources, posing obstacles to both academic research and the development of practical ai applications. notably, there is no open-source corpus available for kazakh in arabic script and mongolian in traditional mongolian script -two language variants in china adopting underrepresented writing systems. in the case of tibetan and uyghur, although several multilingual datasets such as oscar @xcite and culturax @xcite cover these languages, their quality falls short. our preliminary study reveals a critical issue of language misidentification. up to 34% of data in the uyghur split of culturax is actually kazakh or arabic texts. this raises concerns for future research in lowresource nlp, as conducting experiments on these significantly contaminated monolingual corpora may yield misleading conclusions. ## data collection and analysis here we describe the procedure of creating mc 2 , the largest open-source corpus so far for four minority languages in china. before collecting mc 2 , we first conduct a preliminary audit in the low-resource language splits of previous multilingual corpora and find severe quality issues such as language contamination. therefore, we propose a quality-centric solution for data collection of low-resource languages, which aims to ensure accuracy while improving the comprehensiveness and coverage of the data. the collection procedure of mc 2 mainly consists of three steps: (1) gathering web crawls, (2) incorporating existing datasets, and (3) deduplicat-ing and filtering. throughout the collection process, we adhere to the protocol of quality-centric collection. we hope to establish a reliable groundwork for subsequent language model training or linguistic research. notably, our framework for data collection is generally language-agnostic and can be easily applied to collecting web corpora for other low-resource languages. ## discussion on underrepresented writing systems many languages adopt distinct writing systems across various regions. for instance, in china, minority languages such as kazakh and mongolian employ scripts that differ from the cyrillic scripts used in kazakhstan and mongolia, as illustrated in to delve into the multiplicity of writing systems, we conduct empirical studies from both the technical and the cultural perspectives. we find that imperfect transliteration between writing systems harms model training. furthermore, corpora in different writing systems of the same language encapsulate unique cultural context specific to their respective language communities. these findings underscore the importance of gathering resources for minority writing systems. ## continual pretraining with mc 2 to demonstrate the practical value of our corpus, we train models with mc 2 and compare their performance with competitive counterparts. instead of training from scratch, we conduct continual pretraining on existing models with mc 2 , which is an effective technique to adapt models to new languages @xcite @xcite . we continually pretrain xlm-roberta-large @xcite with mc 2 , obtaining mc 2 xlmr-large. thanks to the high quality of mc 2 , it performs comparably to cino, a counterpart trained with closedsource data. we also attempt to adapt larger models such as llama @xcite to the four languages, obtaining mc 2 llama-13b. the newly trained model shows the superior ability of in-context learning in these languages. ## conclusion we present mc 2 , an open-source corpus for four minority languages in china. it is also the first corpus focusing on two underrepresented writing systems, i.e., the kazakh arabic script and the traditional mongolian script. to address the severe quality issues in previous low-resource datasets, we adopt a quality-centric strategy to collect mc 2 . we also emphasize the cultural significance of including low-resource writing systems through empirical studies. our data and models are openly accessible to the community to facilitate research and applications on low-resource languages.