S em E val-2020 Task 12: Multilingual Offensive Language Identification in Social Media ( O ffens E val 2020) (2020)

Abstract: We present the results and the main findings of SemEval-2020 Task 12 on Multilingual Offensive Language Identification in Social Media (OffensEval-2020). The task included three subtasks corresponding to the hierarchical taxonomy of the OLID schema from OffensEval-2019, and it was offered in five languages: Arabic, Danish, English, Greek, and Turkish. OffensEval-2020 was one of the most popular tasks at SemEval-2020, attracting a large number of participants across all subtasks and languages: a total of 528 teams signed up to participate in the task, 145 teams submitted official runs on the test data, and 70 teams submitted system description papers.

## introduction offensive language is ubiquitous in social media platforms such as facebook, twitter, and reddit, and it comes in many forms. given the multitude of terms and definitions related to offensive language used in the literature, several recent studies have investigated the common aspects of different abusive language detection tasks @xcite . one such example is semeval-2019 task 6: offenseval 1 @xcite , which is the precursor to the present shared task. offenseval-2019 used the offensive language identification dataset (olid), which contains over 14,000 english tweets annotated using a hierarchical three-level annotation schema that takes both the target and the type of offensive content into account @xcite . the assumption behind this annotation schema is that the target of offensive messages is an important variable that allows us to discriminate between, e.g., hate speech, which often consists of insults targeted toward a group, and cyberbullying, which typically targets individuals. a number of recently organized related shared tasks followed similar hierarchical models. examples include hasoc-2019 @xcite for english, german, and hindi, @xcite for english and spanish, germeval-2019 for german @xcite , and trac-2020 @xcite for english, bengali, and hindi. offenseval-2019 attracted nearly 800 team registrations and received 115 official submissions, which demonstrates the interest of the research community in this topic. therefore, we organized a follow-up, offenseval-2020 2 (semeval-2020 task 12), which is described in this report, building on the success of offenseval-2019 with several improvements. in particular, we used the same three-level taxonomy to annotate new datasets in five languages, where each level in this taxonomy corresponds to a subtask in the competition: this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . the contributions of offenseval-2020 can be summarized as follows: overall, offenseval-2020 was a very successful task. the huge interest demonstrated last year continued this year, with 528 teams signing up to participate in the task, and 145 of them submitting official runs on the test dataset. furthermore, offenseval-2020 received 70 system description papers, which is an all-time record for a semeval task. the remainder of this paper is organized as follows: section 2 describes the annotation schema. section 3 presents the five datasets that we used in the competition. sections 4-9 present the results and discuss the approaches taken by the participating systems for each of the five languages. finally, section 10 concludes and suggests some possible directions for future work. training test language ind grp oth total ind grp oth total english 120 330 22 176 7 043 149 549 580 190 80 850 english for english, we provided two datasets: olid from offenseval-2019 @xcite , and solid, which is a new dataset we created for the task @xcite . solid is an abbreviation for semi-supervised offensive language identification dataset, and it contains 9,089,140 english tweets, which makes it the largest dataset of its kind. for solid, we collected random tweets using the 20 most common english stopwords such as the, of, and, to, etc. then, we labeled the collected tweets in a semi-supervised manner using democratic co-training, with olid as a seed dataset. for the co-training, we used four models with different inductive biases: pmi (turney and littman, 2003), fasttext @xcite , lstm @xcite , and bert @xcite . we selected the off tweets for the test set using this semi-supervised process and we then annotated them manually for all subtasks. we further added 2,500 not tweets using this process without further annotation. we computed a fleiss' κ inter-annotator agreement (iaa) on a small subset of instances that were predicted to be off, and obtained 0.988 for level a (almost perfect agreement), 0.818 for level b (substantial agreement), and 0.630 for level c (moderate agreement). the annotation for level c was more challenging as it is 3-way and also as sometimes there could be different types of targets mentioned in the offensive tweet, but the annotators were forced to choose only one label. arabic the arabic dataset consists of 10,000 tweets collected in april-may 2019 using the twitter api with the language filter set to arabic: lang:ar. in order to increase the chance of having offensive content, only tweets with two or more vocative particles (ya in arabic) were considered for annotation; the vocative particle is used mainly to direct the speech to a person or to a group, and it is widely observed in offensive communications in almost all arabic dialects. this yielded 20% offensive tweets in the final dataset. the tweets were manually annotated (for level a only) by a native speaker familiar with several arabic dialects. a random subsample of offensive and non-offensive tweets were doubly annotated and the fleiss κ iaa was found to be 0.92. more details can be found in @xcite . ## task participation a total of 528 teams signed up to participate in the task, and 145 of them submitted results: 6 teams made submissions for all five languages, 19 did so for four languages, 11 worked on three languages, 13 on two languages, and 96 focused on just one language. tables 13 , 14 , and 15 show a summary of which team participated in which task. a total of 70 teams submitted system description papers, which are listed in representation the vast majority of teams used some kind of pre-trained embeddings such as contextualized transformers @xcite and @xcite embeddings. the most popular transformers were bert @xcite , roberta @xcite , and the multi-lingual mbert (devlin et al., 2019). 4 many teams also used context-independent embeddings from word2vec @xcite or glove @xcite , including language-specific embeddings such as mazajak (farha and magdy, 2019) for arabic. some teams used other techniques: word n-grams, character n-grams, lexicons for sentiment analysis, and lexicon of offensive words. other representations included emoji priors extracted from the weakly supervised solid dataset for english, and sentiment analysis using nltk @xcite , vader (hutto and gilbert, 2014), and flair @xcite . machine learning models in terms of machine learning models, most teams used some kind of pretrained transformers: typically bert, but roberta, xlm-roberta @xcite , al-bert @xcite , and gpt-2 @xcite were also popular. other popular models included cnns @xcite , rnns @xcite , and grus @xcite . older models such as svms @xcite were also used, typically as part of ensembles. ## english track a total of 87 teams made submissions for the english track (23 of them participated in the 2019 edition of the task): 27 teams participated in all three english subtasks, 18 teams participated in two english subtasks, and 42 focused on one english subtask only. pre-processing and normalization most teams performed some kind of pre-processing (67 teams) or text normalization (26 teams), which are typical steps when working with tweets. text normalization included various text transformations such as converting emojis to plain text, foot_2 segmenting hashtags, 6 general tweet text normalization @xcite , abbreviation expansion, bad word replacement, error correction, lowercasing, stemming, and/or lemmatization. other techniques included the removal of @user mentions, urls, hashtags, emojis, emails, dates, numbers, punctuation, consecutive character repetitions, offensive words, and/or stop words. additional data most teams found the weakly supervised solid dataset useful, and 58 teams ended up using it in their systems. another six teams gave it a try, but could not benefit from it, and the remaining teams only used the manually annotated training data. some teams used additional datasets from hasoc-2019 @xcite , the kaggle competitions on detecting insults in social commentary 7 and toxic comment classification foot_5 , the trac-2018 shared task on aggression identification @xcite , the wikipedia detox dataset @xcite , and the datasets from @xcite and @xcite , as well as some lexicons such as hurtlex @xcite and hatebase. 9 finally, one team created their own dataset. ## arabic track a total of 108 teams registered to participate in the arabic track, and ultimately 53 teams entered the competition with at least one valid submission. among them, ten teams participated in the arabic track only, while the rest participated in other languages in addition to arabic. this was the second shared task for arabic after the one at the 4th workshop on open-source arabic corpora and processing tools @xcite , which had different settings and less participating teams. pre-processing and normalization most teams performed some kind of pre-processing or text normalization, e.g., hamza shapes, alif maqsoura, taa marbouta, diacritics, non-arabic characters, etc., and only one team replaced emojis with their textual counter-parts. ## danish track a total of 72 teams registered to participate in the danish track, and 39 of them actually made official submissions on the test dataset. this is the first shared task on offensive language identification to include danish, and the dataset provided to the offenseval-2020 participants is an extended version of the one from (sigurbergsson and derczynski, 2020). pre-processing and normalization many teams used the pre-processing included in the relevant embedding model, e.g., bpe @xcite and wordpiece. other pre-processing techniques included emoji normalization, spelling correction, sentiment tagging, lexical and regex-based term and phrase flagging, and hashtag segmentation. ## greek track a total of 71 teams registered to participate in the greek track, and ultimately 37 of them made an official submission on the test dataset. this is the first shared task on offensive language identification to include greek, and the dataset provided to the offenseval-2020 participants is an extended version of the one from @xcite . pre-processing and normalization the participants experimented with various pre-processing and text normalization techniques, similarly to what was done for the other languages above. one team further reported replacement of emojis with their textual equivalent. ## turkish track a total of 86 teams registered to participate in the turkish track, and ultimately 46 of them made an official submission on the test dataset. all teams except for one participated in at least one other track. this is the first shared task on offensive language identification to include turkish, and the dataset provided to the offenseval-2020 participants is an extended version of the one from (c ¸öltekin, 2020). ## conclusion and future work we presented the results of offenseval-2020, which featured datasets in five languages: arabic, danish, english, greek, and turkish. for english, we had three subtasks, representing the three levels of the olid hierarchy. for the other four languages, we had a subtask for the top-level of the olid hierarchy only. a total of 528 teams signed up to participate in offenseval-2020, and 145 of them actually submitted results across all languages and subtasks. out of the 145 participating teams, 96 teams participated in one language only, 13 teams participated in two languages, 11 in three languages, 19 in four languages, and 6 teams submitted systems for all five languages. the official submissions per language ranged from 37 (for greek) to 81 (for english). finally, 70 of the 145 participating teams submitted system description papers, which is an all-time record. the wide participation in the task allowed us to compare a number of approaches across different languages and datasets. similarly to offenseval-2019, we observed that the best systems for all languages and subtasks used large-scale bert-style pre-trained transformers such as bert, roberta, and mbert. unlike 2019, however, the multi-lingual nature of this year's data enabled cross-language approaches, which proved quite effective and were used by some of the top-ranked systems. in future work, we plan to extend the task in several ways. first, we want to offer subtasks b and c for all five languages from offenseval-2020. we further plan to add some additional languages, especially under-represented ones. other interesting aspects to explore are code-mixing, e.g., mixing arabic script and latin alphabet in the same arabic message, and code-switching, e.g., mixing arabic and english words and phrases in the same message. last but not least, we plan to cover a wider variety of social media platforms.