Dialect Identification through Adversarial Learning and Knowledge Distillation on R omanian BERT (2021)

Abstract: Dialect identification is a task with applicability in a vast array of domains, ranging from automatic speech recognition to opinion mining. This work presents our architectures used for the VarDial 2021 Romanian Dialect Identification subtask. We introduced a series of solutions based on Romanian or multilingual Transformers, as well as adversarial training techniques. At the same time, we experimented with a knowledge distillation tool in order to check whether a smaller model can maintain the performance of our best approach. Our best solution managed to obtain a weighted F1-score of 0.7324, allowing us to obtain the 2nd place on the leaderboard.

## introduction dialect identification has attracted researchers from both the fields of speech and natural language processing, because of its wide appliance in different tasks such as automatic speech recognition @xcite , machine translation @xcite , or opinion mining @xcite . vardial @xcite ) is an yearly workshop that deals with boosting the research in this direction by creating computational resources for languages that are closely related with each other, varieties in language, and dialects. this year's edition @xcite was composed of four subtasks: (1) dravidian language identification (dli), (2) romanian dialect identification (rdi), (3) social media variety geolocation (smg), and (4) uralic language identification (uli). we chose to participate in the second subtask of the workshop, the rdi subtask. this subtask was also proposed in the previous edition of the workshop @xcite , but this time the participants are given an augmented version of the moldavian and romanian dialectal corpus (moroco) dataset @xcite that contains texts from the news domain. also, as in the previous edition, the test set comes from another domain, so cross-domain algorithms must be employed in order to maximize the results. romanian language resources are in an ongoing process of maturity, and the language is slowly starting to gain the datasets necessary to not be considered under-resourced anymore. some of the recent publicly available romanian corpora include the large romanian sentiment dataset (laroseda) @xcite , the romanian named entity corpus (ronec) (dumitrescu and avram, 2020), and the romanian version of the cross-lingual question answering dataset (xquad) @xcite . moreover, with the rise of transformer-based pretrained language models @xcite , some romanian model versions have also been created @xcite . the speech resources are also starting to catch-up with the introduction of the romanian speech corpus (rsc) @xcite which was recently released for public usage, counting around 100 hours of speech, and with the introduction of a deep neural network architecture based on deep-speech2 @xcite for automatic speech recognition @xcite . in this work, we proposed a series of models based on transformers, pre-trained on the romanian language, and aimed to tackle the dialect identification task. by using different techniques, including adversarial training @xcite , knowledge distillation @xcite , or generative adversarial networks (gans) @xcite , we managed to obtain good scores, allowing us to classify 2 nd in the rdi subtask organized at vardial 2021. the current work is structured as follows. the next section presents the state of the art regarding the romanian-moldavian dialect identification task. section 3 outlines the methods created by us in order to tackle the previously mentioned challenge, while section 4 describes the results and displays the error analysis we conducted. section 5 concludes the work and features some future improvements that can be made to further increase the performance. ## related work the third edition of the vardial evaluation campaign @xcite took place in 2019 and presented another challenging task for the romanian language -moldavian vs. romanian crossdialect topic identification. the task was composed of three smaller subtasks in which the participants had to (1) discriminate between the moldavian (md) and the romanian (ro) dialects, (2) use moldavian samples to classify romanian samples by topic (@xmath0, and (3) use romanian samples to classify moldavian samples by topic (@xmath1. the highest macro f1-score on the first subtask was 89.50%, achieved by the dteam team @xcite with an ensemble model that combines a skip-gram convolutional neural network (cnn) @xcite using the softmax loss and a cnn that was trained using a triplet loss. the highest scores for the cross-dialect subtasks 2 and 3 were obtained by the tearsofjoy team @xcite that used a linear support vector machine (svm) classifier trained on a combination of character and word n-gram features. they obtained a 61.15% f1-score (macro) for the @xmath2.33% f1-score (macro) for the @xmath3. @xcite adopted a non-transformer approach and employed the usage of neural network models with bidirectional long short-term memory cells @xcite , bidirectional gated recurrent units @xcite , as well as a hierarchical attention network @xcite . the fourth edition of vardial @xcite occured in 2020 and came with the rdi task, a binary classification task where participants had to identify the dialect of a given text -either romanian or moldavian. to differentiate from the previous edition, the evaluation set was taken from another domain, namely twitter messages. the highest f1-score (macro) of 78.75% was achieved by the tubingen team (c ¸öltekin, 2020) by using an ensemble of svms trained on word and character n-grams. also, this edition saw the appliance of the romanian bidirectional encoder representations from transformers (bert) by two teams @xcite , with the best performing variant obtaining a 77.51% f1-score (macro). ## conclusions and future work this work presents our approaches for the romanian dialect identification subtask organized by vardial 2021. we proposed a series of systems based on state-of-the-art, transformer-based models, which imply the usage of adversarial techniques for improving the robustness. at the same time, we also experimented with textbrewer, a knowledge distillation tool that allows us to compress a teacher model into a student model such that the performance can be maintained while reducing the size. moreover, by using an ensemble of all the models we experimented with, we managed to improve the overall performance. for future work, we intend to experiment with different variants of adversarial training for increasing the scores obtained by our models.