Comparing the Performance of CNN s and Shallow Models for Language Identification (2021)

Abstract: In this work we compare the performance of convolutional neural networks and shallow models on three out of the four language identification shared tasks proposed in the VarDial Evaluation Campaign 2021. In our experiments, convolutional neural networks and shallow models yielded comparable performance in the Romanian Dialect Identification (RDI) and the Dravidian Language Identification (DLI) shared tasks, after the training data was augmented, while an ensemble of support vector machines and Naïve Bayes models was the best performing model in the Uralic Language Identification (ULI) task. While the deep learning models did not achieve state-of-the-art performance at the tasks and tended to overfit the data, the ensemble method was one of two methods that beat the existing baseline for the first track of the ULI shared task.

## introduction in this paper, we present the submissions of team phlyers to the vardial evaluation campaign 2021 @xcite . the campaign is part of a conference series, the workshop on nlp for similar languages, varieties and dialects (vardial), which has reached its eighth edition, five of which have included several shared tasks @xcite @xcite . the shared tasks typically involve the categorization of texts according to their language or their dialect. this task, known as language identification, is a classical nlp problem @xcite @xcite , because of its importance for information retrieval, machine translation, and more recently categorization of social media posts @xcite @xcite . in the next sections, we briefly describe the three tasks that we participated in. ## methods previous methods used for language identification typically involve svms @xcite @xcite @xcite and multinomial nb models applied to word and character ngrams @xcite @xcite . deep learning methods based on cnns and lstms have also been successfully applied to language identification tasks @xcite @xcite , and the last two editions of vardial also showed successful applications of bert models @xcite @xcite . for the current tasks, we employed nb and svm models trained on character ngrams as baselines, and compared their performance with that of a cnn. for the cnn, we decided to use the character-based model for text classification that was developed by @xcite , and that was successfully adopted by @xcite to distinguish between standard romanian and moldavian news texts. the architecture of the cnn is summarized in ## results this section summarizes our contributions to the three shared tasks, the evaluation of our models, and their performance on the test datasets. ## conclusion the last editions of the vardial evaluation campaign @xcite have seen an increased use of deep learning techniques for language identification, which in several cases yielded the best performance at the tasks (tudoreanu, 2019; bernier-colborne et al., 2019). in this work, we tried to compare the performance of cnns and shallow models for three out of the four tasks at vardial 2021. while for the uli task developing a cnn turned out to be challenging, for both the rdi and the dli task cnns yielded performances which were in line with the baselines established by the more classic shallow models, even though the final results showed that they are prone to overfitting. it is interesting to note that shuffling the words in the training data improved the accuracy of our cnn classifiers, in particular for the rdi task. the procedure essentially introduces noise in the data, because the order of the words in the sentences will be ungrammatical after they are shuffled, so why it improves the performance of the classifier is not clear. one possibility is that it introduces the network to word combinations that would be possible in the language (for instance, but switching a subject and an object, or by juxtaposing words separated by modifiers), increasing the diversification of the training data. another possibility is that since shuffling words does not affect character sequences within words, but at word boundaries, shuffling has the effect of preventing the network from focusing on sequences with spaces in the middle, which could be less meaningful than sequences within words to learn the lexicon and the morphology associated to each language variety. in the current experiments, this strategy had the effect of reducing overfitting. this outcome will require more investigation in the future. while data augmentation is popular in image classification @xcite , it has so far had limited application in nlp @xcite @xcite . our experiments on the vardial 2021 shared tasks suggest that data augmentation can play an important role in adapting neural models to the task of language identification.