The Curious Case of Logistic Regression for I talian Languages and Dialects Identification (2022)

Abstract: Automatic Language Identification represents an important task for improving many real-world applications such as opinion mining and machine translation. In the case of closely-related languages such as regional dialects, this task is often challenging. In this paper, we propose an extensive evaluation of different approaches for the identification of Italian dialects and languages, spanning from classical machine learning models to more complex neural architectures and state-of-the-art pre-trained language models. Surprisingly, shallow machine learning models managed to outperform huge pre-trained language models in this specific task. This work was developed in the context of the Identification of Languages and Dialects of Italy (ITDI) task organised at VarDial 2022 Evaluation Campaign. Our best submission managed to achieve a weighted F1-score of 0.6880, ranking 5th out of 9 final submissions.

## introduction dialect classification represents a key task in the improvement of many other downstream tasks such as opinion mining and machine translation, where the enrichment of text with geographical information can potentially result in improved performances for real-world applications @xcite . as a result, the interest in the study of language variation has been steadily growing in the last few years, as highlighted by the increasing number of publications and events related to the topic @xcite @xcite @xcite @xcite . however, little has been done so far by researchers in the context of automatic dialect and language recognition for the italian language. in this context, the identification of languages and dialects of italy (itdi) task of vardial 2022 equal contribution. evaluation campaign @xcite aims to bridge this gap, facilitating the development of models capable of properly classifying 11 regional languages and dialects from italy's mainland and islands. figure 1 shows the geographical origin of these different dialects and languages. in this paper, we present the results of an extensive evaluation of three different approaches for the automatic identification of the given dialects. after an introductory literature review ( §2), we proceed with a more in-depth discussion on the details of the itdi task and the dataset provided by the organizers ( §3). then, we introduce the proposed architectures ( §4) and the experimental results for each one of them ( §5). we also provide some additional analysis of the models on classification errors and feature space visualization ( §6). finally, we include some concluding remarks on the shared tasks and possible limitations and routes for improvement of our work ( §7). ## related works dialect identification represents a well-known task in the literature, for which the first contributions can be traced back to more than fifty years ago @xcite ). an extensive and complete review of the field can be found in @xcite . however, language identification still represents a non-trivial task in the case of closely-related languages and dialects. although deep neural models nowadays yield state of the art performances in many nlp tasks, shallow machine learning models have shown to be still highly competitive in discriminating between similar languages. some examples are linear svm and naïve bayes classifiers @xcite and logistic regression @xcite . also the use of convolutional neural networks is still popular in this type of task. in particular, cnn-based approaches achieved competitive results in both vardial 2019 evaluation campaign @xcite and vardial 2020 evaluation campaign @xcite . the introduction of transformers @xcite has represented a breakthrough in many nlp tasks, and language identification is no exception. models based on this architecture achieved state-ofthe-art performance in many practical applications. a recent example is again vardial 2020 evaluation campaign, where the use of a fine-tuned version of bert previously trained on three publicly available romanian corpora @xcite reached a weighted f 1 score of 96.25% on the mo-roco dataset @xcite in the romanian vs moldavian identification task. however, the literature regarding automatic italian languages and dialects identification is still relatively underdeveloped. some recent work has been done to encourage the study of the diachronic evolution of italian language and the differences between its dialects @xcite , but no prior work has focused specifically on contemporary italian dialects identification. the language varieties evaluated in this task are 11, both from northern italy (piedmontese, venetian, emilian-romagnol, ligurian, friulian, ladin, and lombard), southern italy (neapolitan and tarantino) and islands (sardinian and sicilian). in the following chapters, varieties' names will be abbreviated coherently with @xcite . this is the first edition of the task.the task is closed, therefore, participants are not allowed to use external data to train their models (except for off-the-shelf pre-trained language models). the training dataset is provided by the organizers and consists of 265 016 selected wikipedia articles from march 1st 2022 dumps, comprehensive of all the 11 varieties evaluated in the task. the development set consists of 6799 annotated sentences that cover only 7 out of the 11 varieties evaluated in the shared tasks (there are no development samples for emilian, neapolitan, ladin, and tarantino). the test set, on the other hand, consists of 11 090 samples, and covers only 8 out of the 11 varieties (piedmontese, sicilian and sardinian are not represented). the composition of the test set was disclosed only after the end of the competition. ## analysis 6.1 error analysis in this section, we present a more fine-grained analysis of the incorrect predictions for our bestperforming model, logistic regression. firstly, we investigate the most confounded dialects and languages on the development set. the resulting confusion matrix is reported in figure 4 . it is possible to observe how the greatest source of confusion for the models is represented by two pairs of dialect, lombard-venetian and sardinian-sicilian. in fact, 6.5% of venetian sentences (81 sentences) are classified as lombard, and 7.9% of sardinian sentences (38 sentences) are labeled as sicilian. this, together with the trade-off between the performances on the exact same two pairs of dialects (observed during the fine-tuning of the model), corroborates the hypothesis of an intrinsic difficulty in the discrimination between the two pairs of dialects. we speculate that this phenomenon might origin in a consistent number of shared lexical features, mainly due to geographical and cultural factors. furthermore, this behaviour is observed also for cnn and bert models (as shown in the confusion matrices included in appendix e), confirming its model-agnostic nature. finally, we leverage the simple and explainable nature of logistic regression to investigate which features contribute the most to wrong classifications (which will be referred to as confounding features). in our analysis, we extract for each wrongly classified sample all the the confounding features with a contribution to the wrong class ψ @xmath0.2, that is all the features that increased the odds of the wrong class by more than 20%. as expected, most of these features are either italian words (for example no, perché, non, con, chi) or words shared between the confounded dialects (for example cossa, lu, me, vegnir). in particular, we further investigate the distribution of these words in the training and validation dataset. the result of this analysis shows a considerable discrepancy in the distributions for most of the studied features, reported for some of them in figure 5 . we therefore speculate that the difference across in-domain and out-domain vocabulary distribution is one of the main issues that cause misclassification of the model. ## conclusion this paper presented the findings of our team at the vardial 2022 itdi shared tasks. the logistic regression model achieved the best results, outperforming the other two models and ranking within lmo scn vec lij fur sc pms (a) tf-idf embeddings. lij scn lmo pms vec fur sc (b) cnn embeddings. lmo scn vec lij fur sc pms (c) bert embeddings. the top 5 submissions. although cnn and bert approaches have not yielded remarkable results, the experiments produced valuable insights. in particular, we observed no notable difference in the model performance of character-based and wordbased cnn, of which the vast vocabulary size is more costly in terms of training time. on the other hand, bert models performed weakly in this cross-domain language identification task, generalising less than linear models. in the future, models' performances could be increased by calibrating different class weights on a validation set comprehensive of all the dialects and languages, and also a more extensive hyperparameters fine-tuning for the neural models could be carried out. this could, eventually, increase the cross-domain adaptability of our models.