CINO : A C hinese Minority Pre-trained Language Model (2022)

Abstract: Multilingual pre-trained language models have shown impressive performance on cross-lingual tasks. It greatly facilitates the applications of natural language processing on low-resource languages. However, there are still some languages that the current multilingual models do not perform well on. In this paper, we propose CINO (Chinese Minority Pre-trained Language Model), a multilingual pre-trained language model for Chinese minority languages. It covers Standard Chinese, Yue Chinese, and six other ethnic minority languages. To evaluate the cross-lingual ability of the multilingual model on ethnic minority languages, we collect documents from Wikipedia and news websites, and construct two text classification datasets, WCM (Wiki-Chinese-Minority) and CMNews (Chinese-Minority-News). We show that CINO notably outperforms the baselines on various classification tasks. The CINO model and the datasets are publicly available at http://cino.hfl-rc.com .

## introduction the multilingual pre-trained language model (mplm) is known for its ability to understand multiple languages, and its surprising zero-shot cross-lingual ability @xcite . the zero-shot cross-lingual transfer ability enables the mplm to be applied on the target languages with limited or even no annotated data by fine-tuning the mplm on the source language with rich annotated data. mplms greatly facilitate transferring the current nlp technologies to low-resource languages and reduce the cost of developing nlp applications for low-resource languages. the existing public mplms such as mbert @xcite , xlm @xcite and xlm-r @xcite handle 100 languages, but there are still some challenges on low-resource languages understanding: recently, more advanced mplms have been proposed, such as ernie-m @xcite , veco @xcite and unicoder @xcite . these models focus on multilingual training objectives, such as leveraging parallel sentences to improve the alignment between different languages, and have improved notably over xlm-r. however, these models have not paid attention to the low-resource languages, so the problem remains unsolved. for the above reasons, it is necessary to develop multilingual pre-trained language models for lowresource and ethnic minority languages. in this paper, we focus on chinese minority languages. in china, standard chinese (mandarin chinese) is the predominant language. besides standard chinese, we consider several most spoken minority languages. these languages are in different language families with varying writing systems, as summarized in although each of the listed minority languages is spoken by at least millions of people, their digital corpus resources are quite limited. for example, in the cc-100 corpus used by xlm-r, the size of the uyghur (ug) corpus is 0.4 gb, which is about 1% of the chinese (simplified) corpus (46.9 gb); also, there are no tibetan (bo) or (traditional) mongolian (mn) corpora in the cc-100. we propose a multilingual pre-trained language model named cino (chinese minority pre-trained language model), which covers standard chinese, yue chinese (cantonese) and six ethnic minority languages. as far as we know, this is the first multilingual pre-trained language model for the chinese minority languages. cino largely has the same structure as xlm-r and has been adapted for minority languages by resizing its vocabulary and adopting a fast masked language modeling objective for the pre-training. the reason for training a multilingual pre-trained model rather than multiple monolingual pre-trained models is threefold. first, a multilingual model is more convenient than multiple monolingual models. second, for low-resource languages, multilingual pre-training leads to better performance than monolingual pre-training @xcite . third, a multilingual pretrained model provides cross-lingual transfer ability, which reduces the data annotation cost for low-resource languages. studies have also shown that pre-training with more languages leads to better cross-lingual performance on low-resource languages @xcite . the public natural language understanding tasks in chinese minority languages are extremely limited. in this work, we construct two multilingual datasets from two data sources to support evaluating the zero-shot cross-lingual ability of mplms on the chinese minority languages: (1) the wcm (wiki-chinese-minority) dataset is a multilingual text classification dataset built from wikipedia cor-pora, with 10 classes, consisting of 63k examples. to evaluate cino from different perspectives, we run experiments on tibetan news classification corpus (tncc), korean news topic classification (ynat), wcm, and cmnews. results show that cino has acquired the ability of minority language understanding and outperforms the existing baselines on the chinese minority languages. to summarize, our contributions are: non-english pre-trained language models and benchmarks. many pre-trained models have been trained on english corpora, or corpora that are heavily biased toward english. to make nlp techniques accessible to people from different cultures, researchers have developed pre-trained models and benchmarks targeting different languages: flaubert and the flue benchmark for french @xcite , klue-bert and the klue benchmark for korean @xcite , in-dobert and the indolem benchmark for indonesian @xcite , and there are chinese-bert-wwm @xcite and arabic bert @xcite . however, there are no pre-trained language models targeting chinese ethnic minority languages. ## cino model in this section, we present the cino model structure and the pre-training methodology. we denote by n the number of pre-training languages, c i the monolingual corpus of the ith language (@xmath0, . . . , n ). let n i be the number of sentences and l i be the mean sequence length in c i . let c i represent the total number of tokens of c i . ## text classification datasets for minority languages multilingual tasks have been used widely to evaluate the cross-lingual transferability of multilingual models @xcite . nevertheless, the pre-existing multilingual datasets hardly cover the chinese ethnic minority languages. for example, tibetan, mongolian and uyghur have never appeared in any task in the xtreme benchmark. to evaluate the cross-lingual transferability of cino, we construct two text classification datasets wcm (wikipedia-chinese-minority) and cmnews (chinese-minority-news). ## discussion on limitations coverage of ethnic minority languages. due to the scarcity of minority language corpora, cino only covers standard chinese and some of the most popular minority languages and dialects. while being spoken by millions of people, some languages, such as the yi language, are omitted in this study since we can not find sufficient data for pre-training. pre-training objectives. in our early trials of multilingual pre-training, we leveraged both monolingual and bilingual parallel data, and combined the mlm objective with a cross-lingual alignment objective, similar to the tlm objective used in @xcite and @xcite . intuitively, parallel data contain more information than monolingual data. however, we have not observed significant improvements over pre-training with only monolingual data and the mlm objective. the performance of cino may be improved if parallel data can be effectively used. languages from different cultures. among the languages in ## conclusion in this paper, we introduce cino, a multilingual pre-trained language model for chinese minority languages. it takes the same structure as xlm-r but with a different vocabulary and is pre-trained with an adapted mlm objective to reduce computational costs. we build multilingual text classification datasets wcm from wikipedia and cmnews from ethnic minority news for zero-shot ability evaluation on the chinese minority languages. we evaluate cino on several text classification tasks. the results show that cino achieves notable improvements over the existing baselines.