Applying Multilingual and Monolingual Transformer-Based Models for Dialect Identification (2020)

Abstract: We study the ability of large fine-tuned transformer models to solve a binary classification task of dialect identification, with a special interest in comparing the performance of multilingual to monolingual ones. The corpus analyzed contains Romanian and Moldavian samples from the news domain, as well as tweets for assessing the performance. We find that the monolingual models are superior to the multilingual ones and the best results are obtained using an SVM ensemble of 5 different transformer-based models. We provide our experimental results and an analysis of the attention mechanisms of the best-performing individual classifiers to explain their decisions. The code we used was released under an open-source license.

## introduction dialect identification is a natural language processing (nlp) task that started receiving more interest in recent years, in part due to vardial, the workshop on nlp for similar languages, varieties and dialects @xcite @xcite and its organized evaluation campaigns @xcite @xcite . this paper presents our solution to the rdi shared task of vardial 2020 @xcite on behalf of team "anumit , i". the problem we focus on is the binary classification of moldavian and romanian samples, training on the moroco @xcite corpus and a small number of collected tweets, and testing afterwards on an additional set of tweets. multiple experiments were done on this corpus @xcite @xcite @xcite ), but we are, to the best of our knowledge, the first ones to apply some of the models in this paper to solve the task. we study the performance of 3 multilingual models, trained on romanian corpora, and 2 other bert-based models trained only on romanian data. the results show clearly that the ones explicitly trained for romanian tasks are superior for the dialect identification task at hand. we present the results of the different individual classifiers and an svm ensemble of them, as well as the words that the romanian-trained ones pay attention to, in order to differentiate between the two dialects. we released all the code under an open-source license: https://github.com/cristianviorelpopa/ transformers-dialect-identification. ## related work considerable progress was made in the nlp field in recent years with the introduction of attention models @xcite and, later on, transformers @xcite , deep neural networks that use an encoder-decoder architecture. a multitude of models @xcite @xcite that make use of the transformer architecture emerged and achieved state-of-the-art performance on a large number of nlp tasks through transfer learning. these are pre-trained on large amounts of textual data in order to encompass general knowledge of the target language(s) to later be fine-tuned on downstream tasks, such as question-answering @xcite . this allows for better results on small datasets, where previously deep learning was not a viable solution. the architecture of the transformer-based models features multiple layers of transformers with multi-head attention, which allow the model to jointly attend to information from different representation subspaces at different positions @xcite . a short time after, multilingual transformer-based models made an appearance. these are trained on multiple languages in order to solve suitable problems, such as cross-lingual sentence classification @xcite . the multilingual bert model (mbert) uses the original bert architecture and training objectives, but is trained on corpora of up to 104 languages, including romanian. the model learns to infer masked tokens in the input sequences (masked language modeling) and ignores the binary classification (next sentence prediction) objective of the initial bert. xlm @xcite is a cross-lingual model employs multiple language modeling objectives, including a modified variant of the bert masked language modeling supporting text streams with an arbitrary number of sentences and subsampling of frequent tokens. xlm-roberta (xlm-r) @xcite is an update of xlm that is additionally trained on 2.5tb of newly cleaned commoncrawl @xcite data. both xlm and xlm-r are trained on 100 languages, including romanian. in addition to the multilingual models, there is also the possibility of training transformer models, such as bert, from scratch using only monolingual data for the specific task. such progress has been made in the case of the romanian language @xcite . these are trained on the opus @xcite , oscar @xcite and romanian wikipedia corpora. the popularity of transformer-based models determined a lot of research to be done for explaining their inside attention mechanisms @xcite @xcite . this growing field of study is called "bertology." ## method our research focuses entirely on pre-trained transformer-based models fine-tuned using the same textual data and similar hyper-parameters. ## experimental results in the results presented further, the metrics we analyze are the area under curve (auc) and macroaveraged f1 score, which requires a decision threshold to clearly distinguish romanian-predicted samthe linguistadors 0.429412 ples from moldavian ones. we choose to study the auc since we believe it is stronger than the f1 score for showcasing the underlying potential of the model, as it is based on all the possible decision thresholds. the news extracts and tweets have distinct enough samples that the performance varies greatly. for this reason, we consider both results relevant and showcase them separately. for the news extracts, we assess the performance on the 5,923 extracts in the validation set, with the threshold being set based on the same samples. for the tweets, we determine the threshold based on the 215 samples in the validation set, while we are interested in the performance of the larger test set with 5,022 samples. the svm ensemble is an exception to this rule, as it is trained on half of the validation tweets, while the other half is used to determine the best threshold. because the performance on the news extracts is very good and the auc quickly approaches 1.0 for all models, we only showcase a specific area of the roc with false positive rate (fpr) between 0.0 and 0.1, as well as true positive rate (tpr) between 0.9 and 1.0. this way, we are able to present a clearer comparison between the models. ## conclusion in this paper, we analyzed a total of 3 multilingual and 2 monolingual transformer-based models finetuned on a task of dialect identification between romanian and moldavian dialects. we present our methodology for pre-processing the data and found that the monolingual models outclass the multilingual ones on data originating from two different sources, with an svm ensemble combining them all that achieves the best macro-f1 score. lastly, we provide a study on the explainability of the best models, focusing on the tokens that these classifiers find the most important in distinguishing the dialects.