Findings of the V ar D ial Evaluation Campaign 2021 (2021)

Abstract: This paper describes the results of the shared tasks organized as part of the VarDial Evaluation Campaign 2021. The campaign was part of the eighth workshop on Natural Language Processing (NLP) for Similar Languages, Varieties and Dialects (VarDial), co-located with EACL 2021. Four separate shared tasks were included this year: Dravidian Language Identification (DLI), Romanian Dialect Identification (RDI), Social Media Variety Geolocation (SMG), and Uralic Language Identification (ULI). DLI was organized for the first time and the other three continued a series of tasks from previous evaluation campaigns.

## introduction the computational processing of similar languages, varieties and dialects is a vibrant area of research discussed in a recent survey @xcite . co-located with international conferences, the workshop series on nlp for similar languages, varieties and dialects (vardial) has become the main workshop on this topic reaching its eighth edition in 2021. since its first edition, vardial has included well-attended shared tasks on topics such as language and dialect identification, morphosyntactic tagging, and cross-lingual parsing. these shared tasks became part of the vardial evaluation campaigns featuring multiple shared tasks organized yearly with the workshop @xcite @xcite . together with vardial 2021, we organized the fifth edition of the vardial evaluation campaign. 1 the vardial evaluation campaign 2021 featured four shared tasks addressing different aspects of language and dialect identification. in this paper, we present the results and main findings of 1 https://sites.google.com/view/ vardial2021/evaluation-campaign these shared tasks. section 4 presents the dravidian language identification (dli) shared task included for the first time at vardial 2021. the romanian dialect identification (rdi) shared task is described in section 5 and the social media variety geolocation (smg) task is presented in section 6. these two tasks are task re-runs from vardial 2020 with augmented datasets prepared for vardial 2021. finally, the uralic language identification (uli) shared task, described in section 7, is an open leaderboard shared task that ran between vardial 2020 and 2021. we include references to the 8 system description papers written by the participants of the campaign in ## participating teams a total of nine teams submitted runs to one or more shared tasks in this year's vardial evaluation campaign. in ## dravidian language identification (dli) the dli task is based on three datasets from youtube comments @xcite . in the 2021 (dli) shared task, participants have to train a model on comments written in roman script. our corpora contains all the three types of code-mixed sentences: inter-sentential switch, intra-sentential switch and tag switching. all comments were written in roman script (non-native script) with either one of the south dravidian (tamil, malayalam, and kannada) grammar with english lexicon or english grammar with south dravidian lexicons @xcite . the comments were written in the latin script with different types of code-mixing. the language tag of the comment were given. the challenge of the task was to identify the language of the given comment. it was a challenging task, since tamil, malayalam and kannada are closely related languages, some of the words being common in all these languages. the participants had to train a system to identify the language of each comment. our dataset size is 16,672 comments for training and 4,588 for testing. there were three language tags such as tamil, malayalam and kannada. a new category not in intended language was added to include comments written in a language other than the dravidian languages. a sample comment from our dataset provided is displayed below. the original sentence was annotated in tamil and it contains the english word movie. the corresponding english gloss is you will see what is the movie. ## uralic language identification (uli) the uli shared task focuses on 29 rare uralic languages and especially the difficulties of finding such languages amongst a huge amount of textual material in more common languages. in uli, the 29 rare uralic languages are considered relevant. in addition to them, there are 149 non-relevant languages. the shared task was first organized as part of the vardial evaluation campaign 2020 @xcite . due to low participation, we decided to keep the shared task open even after the campaign was over. only the nrc team had submitted results and they were all well below the baseline. we constructed a leaderboard page with the best results updated as soon as they were evaluated. the uli 2021 shared task contained three separate subtasks: uli-rle, uli-rss, and uli-178. in uli-rle (relevant languages as equals), the defining measure was the macro f 1 score calculated for the relevant languages present in the training set. in uli-rss (relevant sentences as equals), the measure used was the micro f 1 score calculated for sentences either written in or predicted to be written in the relevant languages. in uli-178 (all 178 languages as equals), the macro f 1 score was calculated as an average over all the 178 languages part of the training set repertoire. we were accepting submissions until the end of the evaluation phase of the vardial evaluation campaign 2021 on february 2, 2021. participants who submitted results were all invited to submit a system description paper to appear in the proceedings of vardial 2021. ## conclusion this paper presented the results and findings of the four shared tasks organized as part of the var-dial evaluation campaign 2021: dravidian language identification (dli), romanian dialect identification (rdi), social media variety geolocation (smg), and uralic language identification (uli). each of these tasks addressed an important challenge in language and dialect identification providing participants with either new or augmented versions of existing datasets that are made freely available to the community. we included short descriptions for each team's systems in this report and references to all 8 system description papers published in the vardial workshop proceedings in