Optimizing a Supervised Classifier for a Difficult Language Identification Problem (2021)

Abstract: This paper describes the system developed by the Laboratoire d’analyse statistique des textes for the Dravidian Language Identification (DLI) shared task of VarDial 2021. This task is particularly difficult because the materials consists of short YouTube comments, written in Roman script, from three closely related Dravidian languages, and a fourth category consisting of several other languages in varying proportions, all mixed with English. The proposed system is made up of a logistic regression model which uses as only features n-grams of characters with a maximum length of 5. After its optimization both in terms of the feature weighting and the classifier parameters, it ranked first in the challenge. The additional analyses carried out underline the importance of optimization, especially when the measure of effectiveness is the Macro-F1.

## introduction identifying the language in which something is written is a prerequisite for many nlp systems, for example in information retrieval and machine translation, but also for applications as common as spell checkers. this task therefore captured the attention of researchers many years ago and, as highlighted by @xcite , it was quickly considered to be very largely solved. this conclusion, obtained in ideal situations (only one language per text, reduced number of potential languages, relatively long documents) was far too optimistic as shown by the series of vardial evaluation campaigns @xcite . the dravidian language identification (dli) shared task of vardial 2021 @xcite ) is undoubtedly a perfect example of a particularly complex situation even if the number of languages to be distinguished is small since it focuses on three dravidian languages. the major problems it poses are: another difficulty is that all the instances to be classified are youtube comments. they are therefore very similar in nature, structure and by the author's aims regardless of the language in which they are expressed. for example, many of them relate only to the number of views (326k likes in 1 hour, tamil) or to a portion of the video that is considered by the author to be particularly interesting (2.08 to 2.10 gun rotating style semma super, tamil). in a previous edition of vardial @xcite , i obtained good results with a relatively simple system, based mainly on character n-grams and a supervised classifier @xcite . the present study was aimed at determining whether this approach could be competitive in the dli shared task. this report describes the system proposed by the laboratoire d'analyse statistique des textes (last), simpler than the one used for vardial 2017, but better optimized. it ranked first out of four systems, but it should be noted that the differences between the systems were small. the remainder of this report first presents the data provided by the task organizers and describes the proposed system. the following section analyzes the performance obtained by comparing it to other participating systems and assesses the importance of each component and parameter of the system to its efficiency. it must be mentioned that the last also participated in this year extension of the uli 2020 task @xcite . i decided to not discuss this participation here partly because the system used is the same as that proposed for the dli 2021 task and because it was not able to end up with better scores than the organizers' baseline. this is a classic criterion for eliminating underperforming systems. ## data provided for the challenge the learning materials for this shared task consist of 16,674 youtube comments from three south dravidian subgroup language: tamil (iso 639-3: tam), malayalam (iso 639-3: mal), and kannada (iso 639-3: kan). all these comments were written in roman script and most of them contain some code switching: dravidian languages grammar with english lexicon or english grammar with dravidian languages lexicons. the task was made even more complex by adding a fourth category of comments, also in roman script, labeled otherlanguage, whose contains was not made clear to the participants. the average length of the comments is 8.5 tokens and 58 characters. a quarter of the comments contain 6 tokens or less. an important feature of the materials is the presence of a strong imbalance in class frequencies, with two thirds of the instances belonging to the tamil category and one quarter belonging to the malayalam category (see the test materials consisted of 4,588 comments in one of these four categories. as shown in ## developed system in the closed submission type in which i participated, only materials provided by the organizers could be used. the main consequence was to make the automatic identification of words written in english more or less difficult and any syntactic analysis impossible. i thus chose to base the proposed system on character n-grams alone, an approach frequently used for this kind of task @xcite . the system was optimized during the training phase by means of a 5-fold cross-validation procedure, the folds of which being stratified according to the categories. during this phase, the official measure to rank the challenge participants was unknown. due to the strong imbalance of the categories (see this cross-validation step led to design the system as described below. all n-grams observed less than twice in the materials were discarded. ## analyses and results the organizers of the challenge decided to use as performance measure the weighted average f1score (wa-f1), which weights the f1-score for each class by its support and is very close to the micro-f1. when it is important that the systems be the most effective in the most populated categories, the choice of the wa-f1 is evident. its main weakness is to give very little importance to rare categories, as it is the case with kannada and, to a lesser extent, with "other-language". as the proposed system has been optimized for the macro-f1, the results will be presented by means of both measures as did the task organizers in another study on these same languages @xcite . it is important to note, however, that when some classes are very rare, a small change in effectiveness on them can have a big impact on the macro-f1 while it will hardly change the wa-f1. ## discussion and conclusion given the difficulty of the dli 2021 task, the level of performance achieved by the system is appreciable. identifying the other-language category was particularly difficult because it may be thought that it is not homogeneous but composed of different languages in varying proportions. it is not even certain that all the other languages present in the test set were also present in the learning set. the additional analyzes highlighted the positive impact of a series of decisions made during the development of the system, most notably a maximum length of n-grams greater than 3, a logarithmic weighting of the frequency, an l2 normalization and optimising liblinear parameters in cross validation. the designed system completely neglects an important dimension of the task: code-switching. the main reason is that since a system participating in the closed submission type was not allowed to use any data or linguistic resources other than those provided by the organizers (see the submission types section in https://sites.google.com/ view/vardial2021/evaluation-campaign ), trying to identify the english words might have required a lot of effort while the simple use of an english word lists would have made this task much simpler. taking code-switching into account is certainly a development path. a question that could be interesting to address in the future is to try to determine if certain parameters are generalizable to the previous vardial shared tasks (see @xcite for an extensive survey). bm25 seems a good choice. one might think that logarithmic weighting and l2 normalization are too. setting the n-gram length at 5 based on the present study is certainly much more questionable. a priori, the language to be identified should have a significant impact, but on the other hand, it's still about character n-grams with the same supervised classifier.