A Report on the V ar D ial Evaluation Campaign 2020 (2020)

Abstract: This paper presents the results of the VarDial Evaluation Campaign 2020 organized as part of the seventh workshop on Natural Language Processing (NLP) for Similar Languages, Varieties and Dialects (VarDial), co-located with COLING 2020. The campaign included three shared tasks each focusing on a different challenge of language and dialect identification: Romanian Dialect Identification (RDI), Social Media Variety Geolocation (SMG), and Uralic Language Identification (ULI). The campaign attracted 30 teams who enrolled to participate in one or multiple shared tasks and 14 of them submitted runs across the three shared tasks. Finally, 11 papers describing participating systems are published in the VarDial proceedings and referred to in this report.

## introduction the vardial evaluation campaign 2020 1 is the most recent iteration of a series of evaluation campaigns featuring multiple shared tasks organized together with the workshop on natural language processing (nlp) for similar languages, varieties and dialects (vardial). it follows three editions organized in 2017 @xcite featuring four shared tasks and in 2018 @xcite and 2019 @xcite featuring five shared tasks. co-located with international nlp conferences such as coling, eacl, and naacl, vardial is a forum for researchers interested in diatopic language variation from a computational perspective. since its first edition in 2014, vardial hosted shared tasks on various topics such as morphosyntactic tagging, cross-lingual dependency parsing, and language and dialect identification. most shared tasks organized at vardial have addressed dialect and language identification on newspaper texts, social media posts, speech transcriptions, and many other genres and domains @xcite . a large number of languages and dialects from different families have been included in the vardial shared tasks: national language varieties of chinese, english, french, spanish, and portuguese, pairs or groups of similar languages such as bosnian, croatian, and serbian and malay and indonesian, and dialects of languages such as arabic and german. some of the datasets made available in these tasks, such as the archimob for swiss german dialects and the multilingual dslcc, have been used outside these competitions evidencing the interest of the nlp community in the topic @xcite @xcite ). 2 in this paper, we present the results and the main findings of the vardial evaluation campaign 2020. three tasks addressing different aspects of language and dialect identification have been organized this year. the romanian dialect identification (rdi) shared task is described in section 4, the social media variety geolocation (smg) task is presented in section 5, and finally the uralic language identification (uli) shared task is described in section 6. we include references to the 11 system description papers written by the participants of the campaign in ## participating teams a total of 30 teams enrolled to participate in this year's vardial evaluation campaign and 14 of them submitted results to one or more shared tasks. in the rdi task attracted 8 teams followed by the smg task with 7 teams who submitted runs to one or more of its three language tracks: bcms, de-at, and ch. this is a similar number of teams that have participated in the most popular shared tasks from the vardial evaluation campaign 2019. only the nrc team submitted results to the uli shared task, which is rather unusual, as tasks in past vardial evaluation campaign have all received a very good number of submissions. it should be noted that the 2020 campaign run from april 20 to july 30 during the early stages of the covid-19 pandemic. lock downs and restrictive measures in many countries during this period have impacted universities and research centers worldwide causing significant disruption. we believe that this situation is very likely to have discouraged more teams to participate in this year's evaluation campaign. the chosen training and test corpora allowed us to evaluate participants on a challenging cross-genre binary dialect identification task: romanian (ro) versus moldavian (md). however, participants were provided with development data comprising both news articles and tweets. the number of samples in the training, the development and the test sets are listed in ## uralic language identification uli the first edition of the uli shared task was a language identification task focusing on differentiating between minority uralic languages and distinguishing them from a large number of other languages. we define minority uralic languages as those languages that are not official state languages in the countries where they are spoken. this definition excludes estonian, finnish and hungarian. the remaining uralic languages are all endangered. they can also be characterized as extremely diverse, at least when it comes to their use and current situation. some of the languages in the shared task are extinct, while others have very young and varying orthographies that are still becoming established. nevertheless, the task also includes languages that are widely used in the modern society and have a large online presence. most of the uralic languages spoken in russia are written using the cyrillic alphabet, often with additional individual characters that differ from the character set used for russian. since the uralic languages form a large and old language family, the varieties in the task are generally far apart from one another. at the same time, the task also contains closely related uralic languages from individual branches, which share a large percentage of their vocabulary and features. the shared task included a total of 178 languages, of which 29 were uralic minority languages. the 29 endangered uralic languages were considered relevant and the 148 languages non-relevant. the uli task consisted of three tracks using the same training and testing data. the tracks differed from each other in how they were scored. the motivation behind including the non-relevant languages in the shared task was to simulate the situation we faced when we were automatically searching for minority uralic languages on the internet during the finno-ugric languages and the internet project @xcite . the different ways of scoring the tracks was also designed to highlight the inherent difficulties of such a search. the third track did not especially focus on the relevant languages, the second track focused on the relevant languages as a group, and the first track forced the participants to consider even the most rare of the relevant languages. the first track, uli-rle (relevant languages as equals), considered all the relevant languages equal in value and the aim was to maximize their average f-score. this is important when one is interested in finding rare languages on, for example, the internet. the f-score was calculated as a macro-averaged f1 score over the relevant languages in the training set. the second track, uli-rss (relevant sentences as equals), considered each sentence in the test set that was written in or was predicted to be in a relevant language as equals. when compared with the first track, this track gave less importance to the very rare languages as their precision was not as important when the resulting f-score was calculated. the resulting f-score was calculated as a micro-f1 over the sentences in the test set for both the sentences in the relevant languages and the ones that were predicted to be in relevant languages. in the first two tracks, there was no difference between the non-relevant languages. all the nonrelevant languages could have been labeled as english in the submissions and it would not have changed the resulting f1-scores. the third track, uli-178 (all 178 languages as equals), however, did not focus on the 29 relevant languages, but instead the target was to maximize the average f-score over all the 178 languages present in the training set. the uli shared task, and especially this track, was the language identification shared task with the largest number of languages used so far. the f-score was calculated as a macro-f1 score over all the languages in the training set. ## conclusion in this paper we present the results and findings of the shared tasks organized as part of the vardial evaluation campaign 2020. three shared tasks were organized this year: romanian dialect identification (rdi), social media variety geolocation (smg), and uralic language identification (uli). each of these tasks tackled an important challenge in language and dialect identification on different languages and dialects. furthermore, in these tasks we provided participants with new datasets that will be made freely available to the research community after the competitions. a total of 14 teams submitted runs across the three shared tasks. we included short descriptions for each team's systems in this report and references to all 11 system description papers in