Comparing Approaches to D ravidian Language Identification (2021)

Abstract: This paper describes the submissions by team HWR to the Dravidian Language Identification (DLI) shared task organized at VarDial 2021 workshop. The DLI training set includes 16,674 YouTube comments written in Roman script containing code-mixed text with English and one of the three South Dravidian languages: Kannada, Malayalam, and Tamil. We submitted results generated using two models, a Naive Bayes classifier with adaptive language models, which has shown to obtain competitive performance in many language and dialect identification tasks, and a transformer-based model which is widely regarded as the state-of-the-art in a number of NLP tasks. Our first submission was sent in the closed submission track using only the training set provided by the shared task organisers, whereas the second submission is considered to be open as it used a pretrained model trained with external data. Our team attained shared second position in the shared task with the submission based on Naive Bayes. Our results reinforce the idea that deep learning methods are not as competitive in language identification related tasks as they are in many other text classification tasks.

## introduction discriminating between similar languages (e.g. bulgarian and macedonian or croatian and serbian), language varieties (e.g. brazilian and european portuguese), and dialects is one of the main challenges in automatic language identification (li) in texts. this issue has been addressed in a couple recent surveys @xcite and evaluation papers @xcite . furthermore, it has been the topic of a number of competitions such as tweetlid @xcite and many shared tasks organized in past vardial workshops @xcite @xcite . in this paper, we revisit the challenge of discriminating between similar languages on a challenging code-mixed data set containing three south dravidian languages: kannada, malayalam, and tamil. in addition, there were code-mixed comments on other languages as well. the data set contains 22,164 youtube comments divided into 16,674 instances for training and 4,590 instances for testing, each containing a mix of languages. the data set was released by the organizers of the dravidian language identification (dli) shared task part of the vardial evaluation campaign 2020 @xcite . 1 the goal of the task is to train computational models able to identify the language of each comment in a test set presented in section 3. the dli debuted as the first competition on similar language identification using code-mixed data associated with the vardial workshops. very similar multilingual language identification shared tasks were organized as part of forum for information retrieval evaluation (fire) meetings in @xcite @xcite . 2 a similar competition to the dli are also the shared tasks on language identification in code-switched data @xcite . these shared tasks addressed intrasentential language identification with word level labeling whereas the dli shared task is language identification shared task with predictions at the document level. we take this opportunity to evaluate the performance of two models for this task, a naive bayes classifier using adaptive language models and a transformers-based system described in detail in section 4. ## related work the task of automatic language identification of texts has been under continuous research since 1960s as is witnessed, for example, by the works of @xcite , @xcite , @xcite , @xcite , and @xcite . the problem of discriminating between similar languages, language varieties, and dialects is a particularly challenging one and it has also been addressed by a number of studies such as @xcite ; @xcite ; @xcite , and the aforementioned shared tasks at the vardial workshop. in addition to addressing the issue of discriminating between similar languages, @xcite provide an extensive overview of the history and methods used in language identification of texts in general. ## methods and experiments with the development data in our development environment, we experimented with several different methods: simple scoring, sum of relative frequencies, the product of relative frequencies (naive bayes, nb), nb with language model adaptation, heli, language set identification, and transformers. ## submissions and results in this section, we summarize the systems we used in our submission and provide the results obtained on the test set. ## discussion we did not experiment with the nb identifier using non-alphabetic characters nor with the original casing of the alphabetic characters. in light of the results for the heli method in in rdi, the use of non-alphabetic characters and original casing proved out to be quite important @xcite and might have very well cost us the small difference between the second and the first positions in the dli shared task. even though the difference in performance between the nb model and the transformers was only 3 percentage points in the test set, the fact that the transformers did not outperform the simple naive bayes classifier deserves special attention. one of the reasons to the inferior performance of the pretrained models is probably that the comments contained code-mixed sentences kind of which the pretrained language models like bert and xlm-r had not seen before. ## conclusion we present the submissions by team hwr to the dravidian language identification (dli) shared task at vardial 2021. the dli shared task featured a challenging dataset including youtube comments written in roman script containing code-mixed text with english and one of the three south dravidian languages: kannada, malayalam, and tamil. our two systems, obtained competitive performance with the nb system achieving 2 nd position in the competition. our results are in line with the general trend of deep learning methods not being overtly competitive in language identification tasks as discussed in @xcite .