Naive B ayes-based Experiments in R omanian Dialect Identification (2021)

Abstract: This article describes the experiments and systems developed by the SUKI team for the second edition of the Romanian Dialect Identification (RDI) shared task which was organized as part of the 2021 VarDial Evaluation Campaign. We submitted two runs to the shared task and our second submission was the overall best submission by a noticeable margin. Our best submission used a character n-gram based naive Bayes classifier with adaptive language models. We describe our experiments on the development set leading to both submissions.

## introduction romanian dialect identification (rdi) 2021 was a shared task focusing on discriminating between two written language varieties, romanian and moldavian. these are considered dialects of the romanian language (ron) by the iso-639-3 standard. 1 together with the uralic language identification (uli) shared task it was one of the newest additions to a series of shared tasks in language or dialect identification organized as parts of the vardial workshop series @xcite @xcite @xcite @xcite . since the 2015 edition of vardial, we have participated in many of the language identification shared tasks as the suki team. in the beginning, we were developing the original heli method @xcite , then we added language model adaptation to it @xcite , and in this article we aim to improve on the results of our implementation of a naive bayes based classifier @xcite using character n-grams with and without language model adaptation @xcite . in this article, we first give a quick overview of the related work on language identification, romanian dialect identification in particular, blacklist classifiers, and language model adaptation in section 2. in section 3, we provide information about the data set and the evaluation measures used in the rdi shared task. then, in section 4, we proceed to describe our experiments in building a competitive romanian dialect identification system using the provided training data. in the experiment section, we also describe the methods used in sufficient detail for the reader to reproduce the systems. sections 5 and 6 provide the results we attained in the shared task and the conclusions we were able to make from them and from our experiments on the training data. ## related work dialect identification is part of a broader task of language identification. automatic language identification of digital texts has been investigated for over 50 years. the first published experiments where a computer program has identified languages were conducted by @xcite . identification of a long, monolingual, text is very simple if the languages to be identified are clearly different from one another. however, the task of language identification includes many challenges which increase its difficulty. for example, the texts can be very short, the languages to be discriminated can be very similar, or the amount of training data can be very small. when these challanges occur together, they can make language identification extremely difficult. the methods used and the challenges faced so far in language identification research were compiled in an extensive survey by @xcite . ## shared task setup the task of the rdi 2021 was to identify the dialect, either romanian or moldavian from a line of text. the participants were provided with two sets of data, one for training and one for development. an augmented moroco data set @xcite was used as the training data. the provided train.txt file included a total of 39,487 samples of texts from the news domain, which is 5,923 samples more than the moroco data set currently available on github. 2 the development data provided were tweets which in the rdi 2020 were used as the target development data (215 tweets) and the test data (5,022 tweets). the data set was published by @xcite as moroco-tweets and are available on github. 3 the participants were informed that the test set would also consist of tweets making the development set in-domain and the training set out-ofdomain in relation to the test set. the task was closed, so only the data provided by the organizers was to be used in preparation of the participating systems. one of the motivations of the task was to evaluate cross-domain language identification systems but, nevertheless, the use of the development data for training the classifiers was not prohibited. the task description was very similar to the one in 2020. however, last year there were only 215 tweets available for development as opposed to the 5,237 tweets available for the 2021 edition. in both the training and the development data, the named entities had been replaced with @xmath0 tags. the participants were informed that this would also be the case with the forthcoming test data. the development data included 2,612 lines of moldavian and 2,625 lines of romanian. the test set was provided in the beginning of the general evaluation period of the vardial evaluation campaign. the exact division of the samples between the two dialects in the test set was not a priori known to the participants. the test set contained 2,665 samples of moldavian and 2,617 samples of romanian which means that there was slightly larger difference between the two dialects than in the development set. as in rdi 2020, the evaluation measure used was the macro f 1 score. it gives both dialects equal value despite the possibility of there being an unequal number of samples in each dialect. ## experiments on the development data when planning to participate in the rdi shared task, we contemplated evaluating two new additional methods combined with our existing language identifiers. the first additional method considered was active language model adaptation based on the length of the text to be identified (mystery text). however, the initial experiments did not produce any meaningful results with the data set at hand and consequently we did not use the adaptation to the text length in the submissions and are not reporting the experiments here. the other additional method under investigation was the use of so called blacklists in language identification to which we return in section 4.2. for all the experiments, as well as for the submission runs, we removed the @xmath1. from the beginning, we suspected that using the in-domain development set would most probably be far more beneficial than using the out-of-domain training set. the language identification methods we were using have a number of parameters which are set using a development set. in light of this, we divided the development set in two parts. we used the first 1,306 lines of moldavian and the first 1,313 lines of romanian for the dev-dev and the rest was used for dev-test. as the task was very similar to the rdi 2020 task, we continued our experiments from the conclusions we had arrived in 2020 @xcite . we had experimented with three types of generative classifiers using character n-grams: the simple scoring, the sum of relative frequencies and the product of relative frequencies. as the results with the two former were clearly inferior to the product of relative frequencies, we begun our experiments using this naive bayesian classifier. ## submissions and results we ended up making two submissions to the shared task. since it might not be obvious from section 4 which systems were finally used to generate the submitted results, we will now give short descriptions of these systems. ## conclusions the results would seem to indicate that the tweets in the actual test data were clearly more out-ofdomain when compared with the development data than our dev-dev set was from the dev-test set. the adaptive version, which gave lower scores between dev-dev and dev-test than the blacklist classifier was clearly superior with the actual test data. we believe, that if the shared task instructions would have explicitly prohibited using the development data as training material, the results would have been even more interesting. as the development data was available for actually training the classifier as well, it proved to be useful to just forget the actual training data and train the classifier using only the smaller development data available. the best submission of the 2020 edition managed to point this out as it used a far smaller development set as training material, still winning the shared task. unfortunately, the two top teams from the rdi 2020 (c ¸öltekin, 2020; popa and s , tefȃnescu, 2020) were not participating in rdi 2021 and the scores of their systems remain a mystery this time. if we compare our rdi 2021 results with the results of the best two teams from 2020, we notice that they are very close.