Neural Networks for Cross-domain Language Identification. Phlyers @ V ardial 2022 (2022)

Abstract: We present our contribution to the Identification of Languages and Dialects of Italy shared task (ITDI) proposed in the VarDial Evaluation Campaign 2022, which asked participants to automatically identify the language of a text associated to one of the language varieties of Italy. The method that yielded the best results in our experiments was a Deep Feedforward Neural Network (DNN) trained on character ngram counts, which provided a better performance compared to Naive Bayes methods and Convolutional Neural Networks (CNN). The system was among the best methods proposed for the ITDI shared task. The analysis of the results suggests that simple DNNs could be more efficient than CNNs to perform language identification of close varieties.

## introduction in this paper, we present the submissions of team phlyers to the identification of languages and dialects of italy (itdi) shared task of the vardial evaluation campaign 2022 @xcite . the campaign is part of a conference series, the workshop on nlp for similar languages, varieties and dialects (vardial), which has reached its ninth edition, six of which have included several shared tasks @xcite @xcite @xcite . the shared tasks involve the categorization of text documents according to their language variety, typically across different domains. language identification has received attention in the literature because it is important in the context of machine translation and categorization of social media posts, and several approaches to perform it have been proposed @xcite @xcite @xcite . most of the vardial shared tasks invite participants to . develop language identification systems in contexts characterized by minimal diversification of the languages involved and low-resource settings, often with lack of data for the domain of interest. in the next sections, we briefly describe our submissions for the itdi shared task. 1 ## itdi the itdi task involves the classification of sentences from eleven different language varieties from italy. five of these varieties -piedmontese (pms), lombard (lmo), ligurian (lij), emilian-romagnol (eml), venetian (vec) -are part of a northern group composed of gallo-italic and venetan varieties; they are represented in yellow in the carta dei dialetti italiani, the reference map drawn by @xcite using a set of isoglosses that define the boundaries of certain morpho-phonological properties. two of the varieties, neapolitan (nap) and tarantino (roa-tara), are part of the southern group, in pink. @xcite is the only representative of the extreme southern group, in purple. friulian (fur) and ladin (lld) are part of the northeastern rhaeto-romance group, although pellegrini keeps them separate (in orange and dark green). finally, sardinian (sc) is represented in brown. training data is provided in the form of wikipedia dumps containing a total of 233k sentences, while evaluation data is provided in the form of approximately 7k short sentences for seven out of the eleven languages. the test set contains sentences from a subset of the given language varieties, and the classifier is evaluated on sentence level. an inspection of the development data clearly shows that the sentences are not taken from wikipedia articles, but from other sources, like literary texts or folktales (see the fact that the training and validation/testing data come from different domains implies that the task is essentially a cross-domain classification task. ## methods the state-of-the-art methods for language identification are typically inspired by support vector machines (svm) models @xcite @xcite @xcite @xcite and multinomial naïve bayes (nb) models @xcite @xcite @xcite , that are trained on features derived from word and character ngrams. deep learning methods have also been successfully applied to language identification tasks @xcite @xcite @xcite , and in particular several of the most recent vardial shared tasks have been addressed using transformer models @xcite @xcite @xcite . while last year we decided to use convolutional neural networks (cnns) to address the shared tasks @xcite , this year we decided to focus on deep feedforward neural networks (dnns), since they represent an alternative approach to language identification. the reason for this shift of focus is that while cnns have been the most popular neural architecture used for language identification @xcite @xcite , following their success in tasks like image classification and sequence processing, language identification is quite different from such tasks. while in domains like image classification and sequence processing hard-coding features is not straightforward, in language identification the cues for discriminating among classes are usually words or orthographic/morpheme sequences, which can be directly extracted and used as input features for a simple dnn in the form of word and character ngrams of different size. a cnn instead performs feature extraction indirectly, using fixed-size filters applied to input sequences that have to be of the same length (which is rarely the case for texts), and therefore is less flexible. 2 for these reasons, comparing these two different approaches can be informative to decide whether cnns provide any advantage over regular dnns for language identification. ## evaluation this section summarizes our contributions to the itdi shared task and the evaluation of our models. ## results for our first submission, we simply re-trained the dnn excluding piedmontese and tarantino, and submitted the predictions on the test set obtained team model weighted f 1 score suki nb + language adaptation 0.901 baseline svm char-ngram tfidf 0.773 phlyers dnn 0.694 ethz logistic regression 0.688 baseline svm unigram tfidf 0.490 ethz bert 0.576 baseline fasttext 0.132 in this way. the second and third submission were similar, but we re-trained the network changing the way in which the 'silver' languages were represented: instead of the wikipedia sentences, we used the label/sentences from the test set for which the predicted label was associated with a high likelihood, following a language model adaptation strategy similar to the one proposed by @xcite . the main difference is that instead of adding the new predictions, we used them to directly replace the training data for the 'silver' languages, with the aim of obtaining a better representation. we used different likelihood threshold to filter the predictions (>0.90 @xmath0.95, after transforming the logits into probabilities). on average, the number of predictions per class that were included was quite high, between 75-80%, which seemed a good balance in the trade-off between number of sentences and confidence associated with them. the overall results of the itdi shared task are summarized in the best system by far was the suki system @xcite , a naïve bayes-like classifier which performs language adaptation. one of the baselines provided by the organizers, a svm trained on character ngrams, provided the second the organizers provided us with the results per class, in a comparison of the predictions of our three submissions, in ## discussion comparing our class results with those of the other teams, the main weakness of our approach turned out not to be the underprediction of tarantino (roatara), with which all the systems struggled, but that of ligurian ('lij'), which was heavily weighted in the evaluation of the systems, since it was the most common language. an inspection of our evaluation results showed that ligurian was not among the languages for which we were expecting underprediction. moreover, ligurian is a gallo-italic language like lombard and emilian-romagnol, but both languages were associated with high f 1 scores in testing, and therefore cannot be responsible for this misclassification. since the organizers provided us with the gold labels, we were able to further investigate the behavior of our model by examining the confusion matrix (figure 6 ). some of the patterns were expected: most of the tarantino (roa-tara) sentences were classified as neapolitan (nap) or sicilian (scn), the other two southern varieties of the sample, and many of the predictions involving venetian (vec) were instead sentences from ladin (lld), which is spoken in the same region. one pattern is instead very peculiar. sicilian (scn) and sardinian (sc) were the main responsible for the underprediction of ligurian (lij), a result which was unexpected, given that the three languages belong to distinct groups, they were all represented in the evaluation set, and were well discriminated in the evaluation phase. from a linguistic viewpoint, this outcome has an explanation: while ligurian is a gallo-italic language, even classical works like the carta dei dialetti italiani by @xcite noticed that there are at least two broad phenomena that the language shares with varieties spoken far from the gallo-italic area: the preservation of many wordfinal vowels, including -u, and the palatalization of @xcite clusters. this means that even though ligurian is clearly a northern italy language, an analysis limited to some of its phonological sequences or its morphology could well mistake it for languages spoken outside of the area. in particular, the first phenomenon was the main responsible for the mistakes in this specific case. it is worth mentioning that the author of the translation published a second version of the text in which the orthographic conventions are different, and u is replaced by o, which is the case also in the sentences of the evaluation dataset. this variation in orthographic conventions explains why this ambiguity did not emerge in our evaluation phase. there are two reasons why the ambiguity could have affected our results more than those of the other teams. first, in our preprocessing we did not remove proper names from the test sentences because in the evaluation phase they did not seem to affect the results, but clearly having a name like pinocchiu being strongly associated with southern varieties (the only varieties in which the sequence cchiu was present in the training data) heavily affected the performance of our classifier. second, our classifier was not able to learn that the letter ae was unambiguously associated with northern varieties (only ligurian and emilian-romagnol had it), a cue that should have corrected the mistake. ## conclusion while in some of the previous vardial evaluation campaigns neural networks yielded the best performance in language identification tasks, @xcite , it was not the case with this shared task, where traditional shallow models like naïve bayes and support vector machines performed better, and the dnn model we devised failed to capture important cues like the presence of ae in the text. even though we were not able to present neural models that reach state-of-the-art performance, we target prediction text source lij nap dumandò u pinocchiu cun anscêtae e affannu pinocchio (version by cino peripateta) lij sc e ti rendime a mae, e femmu paxe pinocchio (version by cino peripateta) lij scn mi suin mariunettu pinocchio (version by cino peripateta) second, dnns turned out to be more efficient than cnns to handle language identification. they do not suffer from overfitting in the same way that cnns do @xcite , they are more flexible, and they yield a better performance. we hope that our results will encourage the exploration of neural architectures for low-resource language identification and more research in the automatic classification of languages varieties in italy.