Low-Resource Text Classification via Cross-lingual Language Model Fine-tuning (2020)

Abstract: Text classification tends to be difficult when data are inadequate considering the amount of manually labeled text corpora. For low-resource agglutinative languages including Uyghur, Kazakh, and Kyrgyz (UKK languages), in which words are manufactured via stems concatenated with several suffixes and stems are used as the representation of text content, this feature allows infinite derivatives vocabulary that leads to high uncertainty of writing forms and huge redundant features. There are major challenges of low-resource agglutinative text classification the lack of labeled data in a target domain and morphologic diversity of derivations in language structures. It is an effective solution which fine-tuning a pre-trained language model to provide meaningful and favorable-to-use feature extractors for downstream text classification tasks. To this end, we propose a low-resource agglutinative language model fine-tuning AgglutiFiT, specifically, we build a low-noise fine-tuning dataset by morphological analysis and stem extraction, then fine-tune the cross-lingual pre-training model on this dataset. Moreover, we propose an attention-based fine-tuning strategy that better selects relevant semantic and syntactic information from the pre-trained language model and uses those features on downstream text classification tasks. We evaluate our methods on nine Uyghur, Kazakh, and Kyrgyz classification datasets, where they have significantly better performance compared with several strong baselines.

## introduction text classification is the backbone of most natural language processing tasks such as sentiment analysis, classification of news topics, and intent recognition. although deep learning models have reached the most advanced level on many natural language processing(nlp) tasks, these models are trained from scratch, which makes them require larger datasets. still, many low-resource languages lack rich annotated resources that support various tasks in text classification. for ukk languages, words are derived from stem affixes, so there is a huge vocabulary. stems represent of text content and affixes provide semantic and grammatical functions. diversity of morphological structure leads to transcribe speech as they pronounce while writing and suffer from high uncertainty of writing forms on these languages which causes the personalized spelling of words especially less frequent words and terms @xcite . data collected from the internet are noisy and uncertain in terms of coding and spelling @xcite . the main problems in nlp tasks for ukk languages are uncertainty in terms of spelling and coding and annotated datasets inadequate poses a big challenge for classifying short and noisy text data. data augmentation can effectively solve the problem of insufficient marker corpus in low-resource language datasets. s ¸ahin and @xcite present two simple text augmentation techniques using "crops" sentences by removing dependency links, and "rotates" sentences by moving the tree fragments around the root. however, this may not be sufficient for several other tasks such as cross-language text classification due to irregularities across ukk languages in these kinds of scenarios. pre-trained language models such as @xcite or @xcite have become figure 1 : high-level illustration of agglutifit an effective way in nlp and yields state-of-the-art results on many downstream tasks. these models require only unmarked data for training, so they are especially useful when there is very little market data. fully exploring fine-tuning can go a long way toward solving this problem xu et al. (2020). @xcite conduct an empirical study on fine-tuning, although these methods achieve better performance, they did not perform well on ukk low-resource agglutinative languages due to the morphologic diversity of derivations. the significant challenge of using language model fine-tuning on low-resource agglutinative languages is how to capture feature information. to apprehend rich semantic patterns from plain text, zhang et al. (2019a) incorporating knowledge graphs (kgs), which provide rich structured knowledge facts for better language understanding. @xcite propose to incorporate explicit contextual semantics from pre-trained semantic role labeling (sembert) which can provide rich semantics for language representation to promote natural language understanding. ukk languages are a kind of morphologically rich agglutinative languages, in which words are formed by a root (stem) followed by suffixes. these methods are difficult to capture the semantic information of ukk languages. as the stems are the notionally independent word particles with a practical meaning, and affixes provide grammatical functions in ukk languages, morpheme segmentation can enable us to separate stems and remove syntactic suffixes as stop words, and reduce noise and capture rich feature in ukk languages texts in the classification task. in this paper, as depict in figure -1 , we propose a low-resource agglutinative language model finetuning model: agglutif it that is capable of addressing these issues. first, we use xlm -r pretrain a language model on a large cross-lingual corpus. then we build a fine-tuning dataset by stem extraction and morphological analysis as the target task dataset to fine-tune the cross-lingual pre-training model. moreover, we introduce an attention-based fine-tuning strategy that selects relevant semantic and syntactic information from the pre-trained language model and uses discriminative fine-tuning to capture different types of information on different layers. to evaluate our model, we collect and annotate nine corpora for text classification of ukk low-resource agglutinative language, including topic classification, sentiment analysis, intention classification. the experimental results show agglutif it can significantly improve the performance with a small number of labeled examples. the contributions of this paper are summarized as follows: ## related work in the field of natural language processing, low-resource text processing tasks receives increasing attention. we briefly reviewed three related directions: data augmentation, language model pre-training, and fine-tuning. data augmentation data augmentation is that solves the problem of insufficient data by creating composite examples that are generated from but not identical to the original document. wei and zou (2019) present eda, easy data augmentation techniques to improve the performance of text classification task. for a given sentence in the training set, eda randomly chooses and performs one of the following operations: synonym replacement, random insertion, random swap, random deletion. ukk languages have few synonyms for a certain word, so the substitution of synonyms cannot add much data. its words are formed by a root (stem) followed by suffixes, and as the powerful suffixes can reflect semantically and syntactically, random insertion, random swap, random deletion may change the meaning of a sentence and cause the original tags to become invalid. in the text classification, training documents are translated into another language by using an external system and then converted back to the original language to generate composite training examples, this technology known as backtranslation. @xcite work experiments with backtranslation as data augmentation strategies for text classification. the translation service quality of uyghur is not good, and kazakh and kyrgyz do not have mature and robust translation service, so it is difficult to use the three languages in backtranslation. s ¸ahin and @xcite propose an easily adaptable, multilingual text augmentation technique based on dependency trees. it augments the training sets of these low-resource languages which are known to have extensive morphological case-marking systems and relatively free word order including uralic, turkic, slavic, and baltic language families. ## methodology in this section, we will explain our methodology, which is also shown in figure -1. our training consists of four stages. we first pre-train a language model on a large scale cross-lingual text corpus. then the pre-trained model is fine-tuned by the fine-tuning dataset on unsupervised language modeling tasks. the fine-tuning dataset is constructed by means of stem extraction and morpheme analysis on the downstream classification datasets. moreover, we use an attention-based fine-tuning to build our classification model and uses discriminative fine-tuning to capture different types of information on different layers. finally, train the classifier using target task datasets. ## conclusion we propose agglutif it , an effective language model fine-tuning method that can be applied to a low-resource agglutinative language classification tasks. this novel fine-tuning technique that via stem extraction and morphological analysis builds a low-noise fine-tuning dataset as the target task dataset to fine-tune the cross-lingual pre-training model. moreover, we propose an attention-based fine-tuning strategy that better selects relevant semantic and syntactic information from the pre-trained language model to provide meaningful and favorable-to-use feature for downstream text classification tasks. we also use discriminative fine-tuning to capture different types of information on different layers. our method significantly outperformed existing strong baselines on nine low-resource agglutinative language datasets of three representative low-resource agglutinative text classification tasks. we hope that our results will catalyze new developments in low-resource agglutinative languages task for nlp.