Practical Transformer-based Multilingual Text Classification (2021)

Abstract: Transformer-based methods are appealing for multilingual text classification, but common research benchmarks like XNLI (Conneau et al., 2018) do not reflect the data availability and task variety of industry applications. We present an empirical comparison of transformer-based text classification models in a variety of practical monolingual and multilingual pretraining and fine-tuning settings. We evaluate these methods on two distinct tasks in five different languages. Departing from prior work, our results show that multilingual language models can outperform monolingual ones in some downstream tasks and target languages. We additionally show that practical modifications such as task- and domain-adaptive pretraining and data augmentation can improve classification performance without the need for additional labeled data.

## introduction while the development of natural language understanding (nlu) applications often begins with high-resource languages such as english, there is a need to create products that are accessible to speakers of the world's nearly 7,000 languages. only 5% of the world's population is estimated to speak english as a first language. 1 the growth of nlu-centric products within diverse language markets is evidenced by the increase in language support for popular consumer applications such as virtual assistants, web search, and social media platforms. as of mid-2020, google assistant supported 44 languages on smartphones, followed by siri (21 languages) and amazon alexa (8 languages). at the start of 2021, google search and microsoft bing supported 149 and 40 languages respectively. also at this time, twitter officially supported a total of 45 languages with facebook reaching over 100 languages. recent studies @xcite ) have explored tradeoffs of multi versus monolingual model paradigms. however, we observe that existing multilingual text classification benchmarks are designed to measure zero-shot cross-lingual transfer rather than supervised learning @xcite , though the latter is more applicable to industry settings. thus, the goal of this paper is to evaluate multilingual text classification approaches with a focus on real applications. ## multilingual text classification we consider a series of practical components for building multilingual text classification systems. ## data we choose sentiment analysis and hate speech detection as evaluation tasks due to their relevance to industry applications and the availability of multilingual datasets. an overview of the datasets is shown in ## experimental setup preprocessing and tokenization we apply minimal preprocessing to both datasets, replacing urls and twitter usernames @xmath0. at all stages of training, we use the default tokenizers associated with each pretrained @xcite and hateval @xcite . lm (see training we use 80% of each training set for training and the rest for validation. during dapt and tapt, we train using the mlm objective for 10 epochs. during supervised fine-tuning, we train for 5 epochs. we use the default hyperparameters for all pretrained lms and apply dropout of 0.4 to the final classification layer. evaluation we report the test set macroaveraged f1 score for both datasets. (for cls, this is equivalent to accuracy since the classes are balanced.) for reference, prior results on cls and hateval are shown in ## results and analysis we report results for all experiments in cls for languages where extremely highresource monolingual lms are available (en and fr), models perform best in the mono-target setting, in which a monolingual lm is fine-tuned on target language data. this is consistent with prior findings that xlm-r suffers from fixed model capacity and vocabulary dilution @xcite . however, for de and ja, which are not lowresource languages but whose monolingual lm pretraining corpora are relatively limited in size and domain (see hateval on average, xlm-r models perform better on hateval than those fine-tuned from monolingual lms. unlike for cls, this is true even in en, suggesting that for some classification tasks, the lm pretraining corpus is not as important for downstream task performance as xlm-r's larger model capacity and cross-lingual transfer. though scores were much higher for the relabeled en dataset than the original, the effects of lm finetuning, tapt, dapt, and data augmentation were consistent. ## conclusion we conduct an empirical evaluation of transformerbased methods for multilingual text classification in a variety of pretraining and fine-tuning settings. we evaluate our results on two multilingual datasets spanning five languages: cls (sentiment analysis) and hateval (hate speech detection). additionally, we contribute a relabeled version of hate-val to address mislabeled test examples and enable meaningful comparisons in future work. our results and analysis show that practical methods such as task-and domain-adaptive pretraining and data augmentation using machine translations consistently improve model performance without requiring additional labeled data. we further show that multilingual model performance can vary based on task semantics, and that monolingual models are not always guaranteed to outperform massively multilingual models like xlm-r due to its large pretraining corpora and increased capacity. our work points to a number of future directions, including cross-domain and cross-task transfer, low-resource and few-shot learning, and practical alternatives to large multilingual models such as distillation.