Exploring the Power of R omanian BERT for Dialect Identification (2020)

Abstract: Dialect identification represents a key aspect for improving a series of tasks, for example, opinion mining, considering that the location of the speaker can greatly influence the attitude towards a subject. In this work, we describe the systems developed by our team for VarDial 2020: Romanian Dialect Identification, a task specifically created for challenging participants to solve the previously mentioned issue. More specifically, we introduce a series of neural systems based on Transformers, that combine a BERT model exclusively pre-trained on the Romanian language with techniques such as adversarial training or character-level embeddings. By using these approaches, we were able to obtain a 0.6475 macro F1 score on the test dataset, thus allowing us to be ranked 5th out of 8 participant teams.

## introduction currently, the romanian language is still considered an under-resourced language, although in the recent years, several datasets were created that tried to mitigate this problem such as the reference corpus of the contemporary romanian language (corola) @xcite , the romanian named entity corpus (ronec) @xcite , the biomedical gold standard corpus (monero) @xcite , the romanian speech corpus (rsc) @xcite , and the romanian wordnet @xcite . with the rise of attention-based language models, the first romanian bidirectional encoder representations from transformer (ro-bert) appeared and it outperformed multilingual bert (m-bert) @xcite on all the evaluation tasks @xcite 1 . one of the most addressed and challenging tasks in natural language processing research is text dialect identification. as a response to this challenge, @xcite introduced the moldavian and romanian dialectal corpus (moroco), a dataset that contains 33,564 samples of text collected from news websites, grouped in two dialects using the top level domain of the websites: romanian and moldavian (".ro" and ".md"). moreover, a shared task, called romanian dialect identification (rdi), was proposed at vardial 2020 @xcite and it aimed to evaluate the performance of each participant system on this corpus. starting from the moroco dataset, the rdi competition introduces the challenge of properly identifying the romanian or moldavian dialect, considering that the test dataset is from a different domain. that is, the validation dataset contains long texts, written in either the romanian or the moldavian dialect, while the test dataset is composed of short entries, based on tweets. therefore, this difference influenced the performance of our models, considering that we were able to obtain a 97.04% macro-f1 score on the validation set, while the evaluation on the test set yielded a 64.75% macro-f1 score. this work is structured as follows. in section 2, we perform an analysis of existing solutions for closely related dialect identification tasks. section 3 outlines our solutions for the dialect identification issue, while section 4 details the performed experiments, experimental setup, and error analysis. finally, we draw conclusions in section 5. ## related work there are various approaches regarding the language dialect identification task. some of them are centered around the romanian language, while others are focused on different ones, such as the arabic or german dialects. however, they are equally important, considering that some techniques can cross the language barrier and be used as universal dialect identification methods. ## methods we focused our approaches around transformers @xcite , considering that they represent state-of-the-art solutions for solving nlp problems. ## conclusion and future work this paper presented our approaches regarding the romanian dialect identification task, organized by vardial 2020. we proposed a series of transformer-based architectures that intended to solve the dialect identification issue. all the solutions employ the usage of ro-bert, a transformer model pre-trained on romanian language corpora. by fine-tuning ro-bert with two different techniques, standard and custom, we were able to achieve good scores on both the moroco-rdi test dataset and the rdi dataset, used for this year's competition. also, by using an adversarial training technique (freelb) on ro-bert, we improved the state-of-the-art score on the moroco-rdi dataset, while the performance ro 3) cum te protejezi împotriva coronavirusului @xmath0 how to protect yourself against the coronavirus @xmath1 md 4) primele @xmath2 @xmath3 @xmath4 @xmath5 @xmath6 @xmath7 the first @xmath8 @xmath9 @xmath10 @xmath11 @xmath12 @xmath13 md for future work, we intend to also experiment with multi-task learning approaches @xcite , considering that, usually, an auxiliary task can help the model to detect additional features that can lead to increased performance. another aspect we plan to test are corola-based word embeddings, which can replace their counterpart in the embedding concatenation experiment.