Galileo at S em E val-2020 Task 12: Multi-lingual Learning for Offensive Language Identification Using Pre-trained Language Models (2020)

Abstract: This paper describes Galileo’s performance in SemEval-2020 Task 12 on detecting and categorizing offensive language in social media. For Offensive Language Identification, we proposed a multi-lingual method using Pre-trained Language Models, ERNIE and XLM-R. For offensive language categorization, we proposed a knowledge distillation method trained on soft labels generated by several supervised models. Our team participated in all three sub-tasks. In Sub-task A - Offensive Language Identification, we ranked first in terms of average F1 scores in all languages. We are also the only team which ranked among the top three across all languages. We also took the first place in Sub-task B - Automatic Categorization of Offense Types and Sub-task C - Offence Target Identification.

## introduction due to the growing number of internet users, cyber-violence emerged with offensive language pervasive across social media. with anonymity as a "privilege", netizens hide behind the screens, behaving in a manner most of them would not otherwise in reality. thus, government organizations, online communities, and technology companies are all striving for ways to detect aggressive language in social media and help build a more friendly online environment. manual filtering is very time consuming and it can cause post-traumatic stress disorder-like symptoms to human annotators. one of the most common strategies @xcite @xcite to tackle the problem is to train systems capable of recognizing offensive content, which can then be deleted or set aside human moderation. semeval 2020 task-12 @xcite is the second edition of offenseval @xcite . in this competition, organizers offers 5 languages datasets including arabic @xcite , danish (sigurbergsson and derczynski, 2020), english @xcite , turkish (c ¸öltekin, 2020) and greek @xcite . in sub-task a, the participants need to predict whether a post uses offensives language. besides, the organizers provide other two sub-tasks which mainly focus on english, to predict the type and target of offensive language. participating in all 3 sub-tasks, we proposed several methods based on pre-training language models including ernie and xlm-r. in sub-task a, we scored 0.9199, 0.851, 0.8258, 0.802, 0.8989 in english, greek, turkish, danish and arabic respectively. we ranked first in average f1 scores, and ranked in top three across all languages. in sub-task b and sub-task c, we also took the first place with 0.7462 and 0.7145. in the following sections, we will elaborate the methods, dataset and experiments of our system. ## dataset we used datasets of offenseval 2019 and offenseval 2020 as our training data. in offenseval 2019, the organizers provide a dataset containing english tweets annotated using a hierarchical three-level annotation. in offenseval 2020, the organizers did not provide additional data in english for training. they provided training data for four other languages, turkish, danish, greek and arabic. in addition, they provide a large amount of weakly labeled data generated by several supervised models. ## conclusion in this paper, we presented our approach on detecting and categorizing offensive language in social media. we proposed a multi-lingual learning method to detect offensive language and a knowledge