Using Ensemble Learning in Language Variety Identification (2023)

Abstract: The present work describes the solutions pro- posed by the UnibucNLP team to address the closed format of the DSL-TL task featured in the tenth VarDial Evaluation Campaign. The DSL-TL organizers provided approximately 11 thousand sentences written in three different languages and manually tagged with one of 9 classes. Out of these, 3 tags are considered common label and the remaining 6 tags are variety-specific. The DSL-TL task features 2 subtasks: Track 1 - a three-way and Track 2 - a two-way classification per language. In Track 2 only the variety-specific labels are used for scoring, whereas in Track 1 the common label is considered as well. Our team participated in both tracks, with three ensemble-based sub- missions for each. The meta-learner used for Track 1 is XGBoost and for Track 2, Logis- tic Regression. With each submission, we are gradually increasing the complexity of the en- semble, starting with two shallow, string-kernel based methods. To the first ensemble, we add a convolutional neural network for our second submission. The third ensemble submitted adds a fine-tuned BERT model to the second one. In Track 1, ensemble three is our highest ranked, with an F1 − score of 53.18%; 5.36% less than the leader. Surprisingly, in Track 2 the en- semble of shallow methods surpasses the other two, more complex ensembles, achieving an F 1 − score of 69.35%.

## introduction discriminating between similar languages using a manually annotated data set of true labels @xcite was included on the list of shared tasks in the tenth vardial evaluation campaign @xcite , under the dsl-tl acronym. the topic of discriminating among language varieties and similar languages has been addressed in previous vardial editions @xcite @xcite . however, we find the dsl-tl task compelling as it introduces qualitative humanannotations from multiple sources. in dsl-tl organizers provide a set of sentences coming from news reports foot_0 written in either english, spanish or portuguese and split in a train, development and test subsets. the test split represents a collection of unlabelled sentences, with labels being subject to further submissions from participants. the examples in the train and development sets are tagged with one of nine labels, namely en-gb, en-us, en, es-es, es-ar, es, pt-pt, pt-br and pt. six of the labels provided, aside from the language itself, also specify the language variety, marked with the initials of the country (i.e. gb -great britain, us -usa, es -spain, ar -argentine, pt -portugal, br -brazil). these are referenced to, by the organizers, as variety-specific labels. the remaining three tags, i.e. en, es and pt, are considered common labels. based on this terminology, the task features two subtasks: the dsl-tl task is presented in both the open and closed formats for each of the two aforementioned tracks. three submissions are allowed for each pair (subtask, f ormat), which amounts to a total of maximum 12 different sets of predictions that can be submitted by each team. our team chose the closed format and participated in both tracks, with three submissions for each subtask. all the models submitted are powered by ensemble learning. for track 1, the metalearning is based on extreme gradient boosting (xgboost) @xcite , while for track 2 we employ multinomial logistic regression @xcite as our meta-classifier. the same subset of individual learners is used for each set of ensembles submitted , independent of its meta-learner (i.e. xgboost or logistic regression) and destination (i.e. subtask). with each submission, we gradually increase the complexity of the models plugged into the aforementioned ensembles. we start by combining the powers of two shallow methods, namely support vector machines (svm) @xcite and kernel ridge regression (krr) @xcite , both using string kernels -a feature extraction technique that proved useful in previous endeavours of identifying language varieties (ionescu and @xcite . for our second submission, we augment the ensemble of shallow models with a character-level convolutional neural network (char-cnn) @xcite , which adds depth to the ensemble and a new way of regarding the data (i.e. at the character level). the third ensemble submitted for each track contains the two string kernel based shallow methods, the char-cnn and also a fine-tuned bert @xcite as individual learners. we fine-tuned and evaluated all of the individual models and meta-learners previously mentioned using the development set provided by @xcite . our final submissions include the development subset in the training routine. moreover, our preference for only submitting ensemble models reflect the best results obtained locally with models trained on the training split and tested on the development data. the rest of the present paper is structured as follows. in section 2 we present related work in the space of language varieties identification. we describe in detail our approach for the dsl-tl task in section 3. the experiments conducted and the empirical results obtained are discussed across section 4. a set of conclusions will be drawn in section 5. ## related work usually modeled as a text classification problem and tackled using supervised learning approaches @xcite , language identification (li) research dates from the mid-60's @xcite , with periodic updates until the early 2000s @xcite @xcite . initially focused on dissimilar languages, li has reached a peak when @xcite achieved a nearly-perfect outcome using character n-grams based models to discriminate among different languages in samples collected online. in the last decade, language identification research has regained momentum, with social media becoming a rich and resourceful source of data. user-generated content @xcite and free-form short texts @xcite can be counted among the reasons why the research in the area of language identification was resumed. new challenges have arisen -e.g. mixing two or more different languages in social media content @xcite . moreover, the idea of discriminating among similar languages or language varieties started gathering an entire community around it, especially in the vardial evaluation campaign @xcite @xcite . the problem of discriminating among similar languages has been tackled, to date, using a variety of ml-powered techniques practicing both shallow @xcite , as well as deep-learning @xcite with an accuracy surpassing a 95% threshold. for language varieties on the other hand, we can observe fluctuations in performance as shown in the vardial reports to date @xcite @xcite . for instance, @xcite applies a common approach to three different language varieties: european vs brazilian portuguese, castilian vs argentine spanish and british vs american english. the same model achieves an accuracy above 90% for the first 2 varieties and just below 53% for the third. in the arabic dialect identification task @xcite , the highest ranking systems were based either on ensemble learning or on single svms trained on character and word-level n-grams @xcite and achieved accuracies of around 50%. recent shared tasks @xcite @xcite @xcite continued the work in the space of language varieties, with multiple different languages targeted over the years. among these, we count german @xcite , chinese @xcite and italian jauhiainen et al. (2022) dialects, dutch vs flemish (çöl-tekin and rama, 2017), romanian vs moldavian @xcite , etc. performance was consistent with the results in earlier campaigns (2014 -2016) -the highest ranked results varied greatly from task to task, with n-gram based shallow models often outperforming other approaches. these works show that language identification is not a resolved problem, as we still see a struggle in performance in automatically identifying certain dialects and language varieties. among the most recent works on language identification, we should mention the one on which the current dsl-tl shared task is based. @xcite introduce dsl-tl as the first humanannotated multilingual data set for language variety classification. dsl-tl uses instances from dslcc @xcite ) -an extensive collection of samples for li, introduced and enhanced in prior vardial evaluation campaigns @xcite @xcite . dsl-tl also uses news reports from @xcite . the authors label the data from multiple human sources using a crowdsourcing platform. moreover, alongside the qualitative data set, the authors train multiple models on these samples. the approaches used count naive bayes, adaptive naive bayes and deep learning based methods such as mbert, xlm-r, and xlm-r-ld and are employed as baselines in the shared task referred in the present paper. intriguing perhaps, the authors observe similar performance across the shallow and deep learning based methods. additionally, in some cases, the shallow methods even surpass the deep ones -an observation consistent with prior findings @xcite . analyzing the baselines introduced by @xcite , we consider appropriate to tackle the classification problem posed by the dsl-tl task from both angles. thus, as previously mentioned, we are combining shallow and deep learning techniques in our ensemble-powered solutions. our choice is encouraged by prior research in the space of li, which shows good results obtained by stacking ensembles @xcite . moreover, we choose most of the individual learners used based on their prior impact in language identification tasks: svm with string kernels @xcite , cnns @xcite and bert @xcite . from our perspective, prior success in li is an indication that these meth-ods have a high chance of being suitable for the dsl-tl use-case as well. additionally, each of the two choices of meta-learners were also used before in language variety identification: logistic regression @xcite and xgboost @xcite . ## methods our team submitted three distinct ensemble-based systems for each of the two tracks of the dsl-tl task. the choice of architecture for the metalearner represents the one difference between the ensembles submitted for each track. for the first subtask, we use an xgboost-based meta-learner, whereas for the second one, we rely on logistic regression. as mentioned in both section 1 and section 2, we gradually increase the complexity of the ensemble used in each submission. figure 1 displays the prediction pipeline of the third and most complex system submitted, which is similar with a system that we used in a previous vardial geo-location challenge @xcite . from left to right, also in figure 1 , we can infer how the other pipelines are composed: the first system submitted only uses two shallow models (i.e. svm and krr) and the second submission adds a charlevel cnn to the first system. in the continuation of this section, we briefly describe each individual machine learning technique used in the ensembles submitted, as well as the meta-learners. ## conclusions in this work we propose six ensemble models to address the problem of language-variety identification in news reports. to tackle the two tracks proposed by the dsl-tl task, we employ two similar sets of ensembles which differ only in the choice of meta-learner: xgboost for the 9-way classification in the first track, and logistic regression for the 6-way classification in the second one. by the definition of track 2, our logistic regression based systems are evaluated only on the variety-specific labels provided. however, we have trained these lr powered ensembles also on the common labels, in hopes that the model will learn additional useful representations. for each set of ensembles submitted, we follow a similar strategy: increase the number of models and individual models' complexity for each run. thus, our first submission only combines predictions from krr and svm -two shallow models. in the second ensemble we add a cnn working at character level, and in the third one, we augment the second ensemble with a fine tuned multilingual bert model. for the 9-way classification, our best performing model achieves a macro f1-score of 53.18%, 5% less than the top scoring submission. overall, our model ranks fourth out of 9 total submissions and surpasses two of the four strong baselines proposed by the organizers. in the variety-specific, 6-way classification of track 2, most of the models submitted by participants (including ours) fall behind the proposed baselines. interestingly, our best performing submission in this case is the ensemble of shallow models, which obtains a score of 69.35%, surpassing the other 2, more complex ensembles, that we submitted. given the final results, we conclude that in future similar endeavours we should not underestimate the power of shallow models, as they consistently seem to achieve good results in language identification setups. moreover, we intend on performing a closer analysis of the baselines proposed in @xcite -the paper that introduces dsl-tl, try to replicate and perhaps enhance the already impressive methods that the authors used for this task.