Dialect Identification under Domain Shift: Experiments with Discriminating R omanian and M oldavian (2020)

Abstract: This paper describes a set of experiments for discriminating between two closely related language varieties, Moldavian and Romanian, under a substantial domain shift. The experiments were conducted as part of the Romanian dialect identification task in the VarDial 2020 evaluation campaign. Our best system based on linear SVM classifier obtained the first position in the shared task with an F1 score of 0.79, supporting the earlier results showing (unexpected) success of machine learning systems in this task. The additional experiments reported in this paper also show that adapting to the test set is useful when the training data comes from another domain. However, the benefit of adaptation becomes doubtful even when a small amount of data from the target domain is available.

## introduction language identification can be performed with near-perfect accuracy from a short text in many settings @xcite , for a recent survey of the solutions). however, automatic discrimination of texts from closely related languages or dialects remains to be a challenging task. the successful discrimination of texts between closely related language varieties may improve language identification for practical applications, as well as providing further insights into the differences between these linguistic varieties. recent vardial evaluation campaigns have featured discrimination challenges between related languages @xcite @xcite . the present study is conducted within the scope of the vardial 2020 @xcite romanian dialect identification (rdi) task. romanian and moldavian are two closely related language varieties spoken in romania and the republic of moldavia respectively. the languages, particularly in written form, are very similar -to the extent that discrimination by human annotators is barely above chance levels @xcite . however, as evidenced by the last year's evaluation campaign @xcite , the machine learning methods applied to the task seem to be more successful @xcite @xcite . the moroco corpus @xcite used in last year's rdi shared task consists of texts from online news. although the data is fairly balanced with respect to the topics, and some of the obvious non-linguistic cues (e.g., named entities) are removed from the data, the data may still contain some unintended non-dialectal cues (e.g., style differences between the newspapers in two countries). a natural question that arises is whether the machine learning methods tap into such non-obvious cues not relevant to linguistic differences, or the data contains a strong signal for identifying the linguistic variation. to this end, the present shared task includes data from two different domains (or genres). the source domain is the moroco corpus of newspaper text, and the target domain texts consist of a newly collected data set gathered from twitter. the systems used in the current study are based on ensembles of linear svm models with a simple adaptation mechanism that retrains the model(s) with the data augmented by the test instances that are classified by a base classifier with high confidence. besides the adaptation to the test set at prediction time, we also experiment with a training set selection method based on reverse-prediction. this method is based on training a classifier on a small target data (development set) and selecting the training instances which the classifier predicts with high confidence. in the remainder of this paper, we describe the system, present the results, and provide a discussion with brief conclusions. ## system description the main task at hand is predicting the language variety (romanian or moldavian) under domain shift. the participants were provided with a large annotated corpus from the source domain (newspaper text) and a small annotated development set from the target domain. the evaluation is based on the performance of the systems on the target domain. all experiments reported in this study are performed using linear svm classifiers with sparse character and word n-gram features. overlapping character and word n-gram features (for all 'n' from 1 to a maximum value, determined during tuning) are combined into a single feature set, and weighted using bm25 @xcite . the input is tokenized using a simple regular expression tokenizer that treats any contiguous alphabetic or non-space character sequence as a token. except for (optional) case normalization (treated as a hyperparameter), and filtering based on low document frequency (also another hyperparameter), no preprocessing or filtering is performed. the same system has been used with minor differences for discriminating similar languages in the earlier vardial evaluation campaigns (c ¸öltekin @xcite @xcite , and obtained top or near-top results. the detailed description of the approach can be found in these papers. for some of the experiments, we use an ensemble of linear svm classifiers described above trained on different, non-overlapping parts of the data. the predictions of the individual classifiers are combined using weighted majority voting where the distances from the decision boundary are used as weights. another interesting aspect of the system is an adaptation technique used during prediction. the adaptation method is similar to the adaptation method used in a few systems in earlier vardial shared tasks @xcite @xcite . the method relies on a base classifier trained on the training data. during testing, the test instances for which the base classifier is confident in its decisions are added to the training set and the classifier is retrained with this augmented training set. one of the differences between the source and the target domain is the average length of the documents. on average, the source domain newspaper texts are naturally longer than the tweets (target domain). although the feature weighting method we use (bm25) counteracts the sensitivity to document size to some extent, we split the source domain documents to sentences. in all of the shared task submissions, the source domain documents were split before training the models. another aspect of some of our systems is filtering large source domain data based on what we call 'reverse-prediction'. with the assumption that the features relevant for the target domain can be captured well by a classifier trained on the small target domain development set, we first tune and train a classifier on the development set. we use this classifier to predict the labels of the large training set. we select the predictions with high confidence that match the gold-standard labels. in this paper, we consider the test instances with a distance of more that 1.0 from the decision boundary as confident predictions. the intuition is that, despite reduced data size, the selected training instances may include features better tuned to the target domain. ## general discussion this paper presented results from (ensembles of) linear svm classifiers on the task of cross-domain discrimination of moldavian and romanian. our systems obtained top positions on the official competition. the results indicate that linear svms are (still) one of the best solutions in certain settings. furthermore, the cross-domain success of the system supports findings of @xcite , that better-thanhuman achievement of the machine learning models is based on dialect differences, and not due to some correlated hidden variable in the data set. the in-domain experiments indicate that if the training sizes are similar, the discrimination is better on the twitter data in comparison to the newspaper text. the in-domain experiment on twitter corpus yield better discrimination than the in-domain experiment with with news sentences despite smaller training set size. this probably indicates that non-standard texts contain more cues to dialectal differences, since they do not necessarily follow common literary and stylistic traditions expected from more formal texts. the adaptation method based on test set augmentation was found to be useful when training and test domains are different. the benefit of the test set adaptation method is not clear when the training and tests are from the same domain, even when the training set is very small. the training data selection experiments resulted in worse results than expected. a potential reason for the failure is the fact that most training instances selected by the procedure contain the features that already occur in the target development set, hence, not providing additional information expected from a large data set. although the training set selection method as used in this study seems to have failed, methods involving relevant, yet diverse instance may be helpful in adapting to a new domain.