G ru P a T o at S em E val-2020 Task 12: Retraining m BERT on Social Media and Fine-tuned Offensive Language Models (2020)

Abstract: We introduce an approach to multilingual Offensive Language Detection based on the mBERT transformer model. We download extra training data from Twitter in English, Danish, and Turkish, and use it to re-train the model. We then fine-tuned the model on the provided training data and, in some configurations, implement transfer learning approach exploiting the typological relatedness between English and Danish. Our systems obtained good results across the three languages (.9036 for EN, .7619 for DA, and .7789 for TR).

## introduction the growth of social media has seen the spread of two different but connected phenomena: on the one hand, they helped to create a more open and connected world, and, on the other hand, they contributed to the spread of offensive and abusive behaviors. although the use of "bad language" is intimately connected with freedom of speech, the phenomenon has become so pervasive that developing natural language processing (nlp) systems that automatically and efficiently detect and classify offensive on-line content is a pressing need @xcite . semeval-2020 task 12: offenseval 2 @xcite ) is a follow-up edition of semeval-2019 task 6: offenseval @xcite and it addresses the problem of offensive language detection in twitter messages by focusing on two pending issues: multilingualism and hierarchical tagset annotation. the multilingualism issue is targeted by providing for the first time data in 5 different languages, namely english, danish, greek, turkish, and arabic, by applying a shared definition of offensive language. the languages cover different values of the typological spectrum in terms of type (fusional vs. agglutinative), language family (indo-european vs. altaic vs. afro-african), genus (germanic vs. greek vs. turkic vs. semitic), subject-object-verb 1 order (svo vs. no dominant order vs. sov vs. vso) @xcite , as well as writing systems. the multilingual aspect poses two additional challenges: (i.) availability of nlp tools and language resources as some of the proposed languages are considered low-resourced (e.g., danish and greek) @xcite ; and (ii.) differences in the perceived offensiveness of a message. in particular, given that offensiveness is a highly subjective category, seeing that a message is always "offensive for someone" @xcite , different communities of speakers may have different perceptions of what is offensive or not. the use of a shared definition is a way of mitigating the potential differences across communities, but this aspect cannot be ignored in the development of a system for offensive language detection. the hierarchical annotation tagset is reflected in three sub-tasks, namely: sub-task a is proposed for all languages, while sub-tasks b and c are proposed for english only. manually annotated training data are available for all languages. no language has an official development dataset. english has a special place in this edition: the organizers provided only automatically annotated material, i.e., silver data, together with the manually annotated training and test data from the 2019 edition. the availability of silver data calls for innovative ways for using such data, such as a full-fledged re-training an existing pre-trained language model rather than directly employing them in a supervised system. recently, @xcite have conducted the first systematic comparison of these three families of models against four different datasets of offensive/abusive language. feature selection and pre-processing were kept to a minimum, while they conducted fine-tuning of the hyper-parameters (i.e., sequence length, drop out, and class weights). results confirm bert as the best performing model. however, improvements (or decrements) across model (per dataset) are minimal. the fluctuating behavior of neural network models is further confirmed with performances being lower than those of a linear model (i.e., an svm) in two datasets. ## system overview the system we propose builds on top of recent work in the use of pre-trained language models à la bert @xcite . this family of recently proposed models are based on learning a language model in an unsupervised fashion from a large amount of data (pre-training), and on a subsequent step to train the model to solve a specific task on annotated data (fine-tuning). bert, in particular, employs bidirectional transformer-based encoders and a masking task for the pre-training, learning to predict randomly removed words in context, and therefore learning contextual word representations. we differentiate with respect to the standard fine-tuning approach by adding a retraining step. it is undisputed that bert and bert-like models are the new state of the art in nlp, however, these models are trained on a massive amount of what could be labelled "standard" natural language data, such as news articles, wikipedia pages, and books. none of these models is somehow "ready to be used" for social media data. 2 in our perspective, retraining bert will have two beneficial effects: first, it improves the tuning of the model towards social media language variety, and, second, it reduces efforts in pre-processing and cleaning of the data for fine-tuning. a further aspect we took into account is multilingualism. we aimed at developing a unifying approach that could be easily applied across the different languages. the lack of monolingual bert models for all the languages in the task foot_1 guided us to select multilingual bert (mbert) @xcite . mbert consists of 12 stacked transformers, with a hidden layer size of 768 and 12 self-attention heads, like its monolingual english counterpart, bert base . mbert is pretrained on the concatenation of monolingual wikipedia pages of 104 languages with a shared word piece vocabulary and it does not make use of any special marker to signal the input language, nor does it have any mechanism that explicitly indicates that translation equivalent pairs should have similar representations. figure 1 graphically illustrates our approach. for each language, we collect potentially offensive tweets and use them to retrain the mbert model by applying the masked language model (mlm) objective. this will provide us with new "shifted" mbert models along three dimensions: (i.) language variety (i.e. social media); (ii.) language (i.e. english, danish, turkish), and (iii.) polarity (i.e., offensive-oriented model). after retraining, the new model is fine-tuned and applied to the test data. we added a linear classifier on top of the pooled output for the [cls] token to generate the predictions. we differentiate the general architecture only with respect to the language specific data used in the re-training and the fine-tuning steps. we developed our system for sub-task a: offensive language identification in three languages, namely english @xcite , danish (sigurbergsson and derczynski, 2020), and turkish (c ¸öltekin, 2020). code, additional training data, and models are publicly available at https:// github.com/davidecolla/offenseval2020 . the following paragraphs describes the process of collecting the additional data per language that we used to retrain mbert. ## experimental setup we used the mbert pre-trained model available via the huggingface transformers library. 4 after retraining mbert for each language, we fine-tuned the models using the training data made available by the task organizers. for english, we used the training set of offenseval 2019. in all fine-tuned settings, we used a standard learning rate of 2e -5, a batch size of 32, and 4 training epochs. pre-processing steps are reported in the appendix. we ran an internal evaluation to verify whether the proposed system works and selected the best retrained model (at least for english and danish). evaluations were conducted using the offenseval 2019 test data for english, while we split the offenseval 2 training data for danish and turkish retaining 90% of the data for fine-tuning and 10% for test. on the basis of the results (see we ran our experiments on a machine with the following configuration: nvidia k40 gpu, intel xeon e5-2680 v3 processor and 64gb of ram @xcite . the training of the mbert architecture on the largest data collection, namely the e3, took eight hour for each epoch, whilst the fine-tuning, performed on the same machine, ran for two hours for each epoch. ## results and discussion tables 2a-2c depict the confusion matrices between the predictions and the gold standard data from the task organizers. it clearly appears that the classifiers are asymmetrically biased, confirming the observation based on the scores. a qualitative error analysis on the output of the classifiers across the three languages has shown some common patterns on the reasons for the misclassifications. we have observed that false positives (not @xmath0.g. en to suck, da latterlige [ridiculous], tr boktan [shitty]) or terms carrying negative polarity, such as en ignorant, da tosse [fool]. as for the false negatives (off @xmath1, we observe two trends: the first, messages contain strong offensive lexical cues that are misspelled (e.g., en stoopid), or difficult to find in common lexicons of abusive terms (e.g., en twat), or idiomatic expressions (e.g., tr kapak olsun [lit. "get a cover"]). the second concerns the presence of ambiguous words (e.g. en jerk, in @user wings over and it's not even a question (sweet chili & jamaican jerk hanger) 5 or implicitly offensive messages (da ned med svensken! [down with the swedish] 6 ; tr s ¸imdi sana anlatsam anlamıcan o yüzden bos ¸ver[if i'll explain this you, you'll not understand it] 7 ), presence of irony (tr @user as ¸ırı komikmis ¸kardes ¸ilk esprin mi [it's too funny bro, is this your first joke?] 8 ), or harsh criticism (tr türkc ¸e pop gibisin sesin güzel konus ¸maların bos ¸güzellik [you sound like turkish pop, your voice is beautiful] 9 ). ## conclusion the approach of our system is based on the combination of re-training and fine-tuning mbert. the re-training step has been added to bias mbert against three aspects: language variety, language, and polarity. the bias in the classifier is sensitive to the collection of the additional training materials. results of the systems across the three languages show that the quality of the retraining data is sensitive to the quality of the language resources and strategies used to retrieve them. on the fine-tuning step, this aspect appears to impact mainly the recall for the off class, as shown by the en results compared to da and tr. among the phenomena we detected as sources of noise in the classification, their explicitness appears 5 instance id: a2825 6 instance id: 1695 7 instance id: 32854 8 instance id: 38605 9 instance id: 43122 to play an important role. recent work has focused on this aspect @xcite by proposing more fine-grained levels of annotations. in future work, we will focus on the hurdles of figurative and idiomatic language usage in offensive messages, following the approach in @xcite , by enriching hurtlex with multi-word expressions (mwes) automatically extracted from corpora in multiple languages.