PRHLT - UPV at S em E val-2020 Task 12: BERT for Multilingual Offensive Language Detection (2020)

Abstract: The present paper describes the system submitted by the PRHLT-UPV team for the task 12 of SemEval-2020: OffensEval 2020. The official title of the task is Multilingual Offensive Language Identification in Social Media, and aims to identify offensive language in texts. The languages included in the task are English, Arabic, Danish, Greek and Turkish. We propose a model based on the BERT architecture for the analysis of texts in English. The approach leverages knowledge within a pre-trained model and performs fine-tuning for the particular task. In the analysis of the other languages the Multilingual BERT is used, which has been pre-trained for a large number of languages. In the experiments, the proposed method for English texts is compared with other approaches to analyze the relevance of the architecture used. Furthermore, simple models for the other languages are evaluated to compare them with the proposed one. The experimental results show that the model based on BERT outperforms other approaches. The main contribution of this work lies in this study, despite not obtaining the first positions in most cases of the competition ranking.

## introduction bert, the bidirectional encoder representations for transformers @xcite , is a model producing context representations that leverage on language model pre-training. it is based on transformers @xcite , which are models that process words in relation to all other words in a sentence, rather than word by word in order. that is, as opposed to directional models, which read the text input sequentially (forward and/or backward), the transformer reads the entire sequence of words at once. this characteristic allows the model to learn the context of a word based on all of its surroundings. hence, bert models can consider the full context of a word by looking at the words that come before and after it. this is very useful to understand the intent behind sentences. therefore, unlike other kinds of models, with bert models it is possible to effectively capture the general meaning of a sentence by detecting relevant words and their relationship to others. bert is a deep, bidirectional, unsupervised language representation, pre-trained using only plain text corpus. the pre-training has been performed by using two strategies on a large corpus of unlabelled text which includes the entire wikipedia and a book corpus. one training strategy is masked language model, where the model attempts to predict the original value of some masked words in a sequence, based on the context provided by the non-masked words in the sequence. the other training strategy is next sentence prediction, where the model learns to predict if given a pair of sentences, the second one is the subsequent sentence in the original document. pre-trained bert can be fine-tuned to many natural language processing (nlp) tasks by adding an additional output layer on a supervised dataset for a target task. therefore, it is eliminated the need for engineering a specific architecture for a task. this approach has advanced the state-of-the-art performances in many natural language processing tasks ranging from sequence classification to question answering. multilingual bert (mbert) foot_0 @xcite ) is a language model pre-trained on wikipedia text from 104 languages in the same way as bert for english. therefore, not only is a contextual model, but the training does not requires supervision. that is, no alignment among the languages is done, but rather in the model the tokens from different languages share an embedding space and a single encoder. there are no cross-lingual objectives specifically designed nor any cross-lingual data, like parallel corpora. however, mbert produces a representation that seems to generalize well from a cross-lingual perspective for a variety of downstream tasks. some studies indicate that this model has surprising cross-lingual abilities @xcite . in this paper, we describe a proposed system that is based on bert for the multilingual offensive language identification in social media (offenseval 2020 foot_1 ) task @xcite . offenseval 2020 is a shared task organized at semeval 2020 as extension of the previous offenseval task 2019 @xcite . the general objective of offenseval is the identification of offensive language in online social media. this is a relevant topic nowadays, since many users take advantage of the perceived anonymity of this kind of communication to incite offensive behaviors. basically, the principal aim is to determine whether a text is offensive or not. moreover, other characteristics taken into account, dividing the task in the next three subtasks: bert is used as a text feature extractor, using the text representation obtained for the classification of the english texts. in order to perform the classification, a simple feed forward neural network is applied on the top of bert to detect whether the original text is offensive or not. the rest of the languages are processed in the same way, but using the mbert model instead. the paper is organized as follows. section 2 presents general ideas of related works. then, sections 3 and 4 describes the dataset used in the task and the proposed system, respectively. experiments and results are then discussed in section 4. finally, we present our conclusions with a summary of our findings in section 5. ## related work related tasks to abusive language analysis, and particularly the offensive language detection, have attracted significant attention during last years to prevent this kind of online behaviour. this is evidenced by different works @xcite @xcite , and the organization of different workshops and shared tasks @xcite @xcite @xcite . in general, models in offenseval 2019 used different approaches, from traditional machine learning such as support vector machines and logistic regression, to deep learning such as convolutional neural networks and recurrent neural networks, some of them including attention mechanisms. moreover, some system employed bert and reached top-performance in the competition @xcite . in the present work, we propose a system based on bert and mbert, analyzing its parameters. ## dataset a multilingual dataset with five languages is provided for the task. therefore, a corpus of annotated texts have been released for each of the following languages: english @xcite , arabic @xcite , danish (sigurbergsson and derczynski, 2020), greek @xcite and turkish (c ¸öltekin, 2020). the tagset matches the offensive language identification dataset (olid) @xcite that was used in offenseval 2019. then, the task is divided into three subtasks according to the tag hierarchy. so that, the 3 subtasks are developed for english, while for the rest of the 4 languages, only the first one is developed. the subtask a aims to discriminate between offensive and non-offensive text. therefore, every text is assigned one of the two following labels: offensive (off) and not offensive (not). on the one hand, offensive texts include insults, threats, and text containing swear words or any form of untargeted profanity, which is unacceptable language. on the other hand, texts that do not contain offense or profanity are considered as not offensive. the objective of sub-task b is to predict the type of offense, just for those texts labeled as offensive in the subtask a. for this subtask, two labels corresponding to the following categories have been defined: targeted insult (tin) and untargeted (unt). the first label corresponds to texts containing an insult or threat and the second one corresponds to texts containing untargeted profanity and swearing. in sub-task c, the goal is to detect the target of offense, only for those texts labeled as targeted insult in the subtask b. in this case, the following three labels have been defined: individual (ind), group (grp) and other (oth). respectively, the first and second labels are for individuals and for groups of people considered as a unity due to a common characteristic. the third label is for the offensive texts that do not belong to any of the previous two labels, such as an organization or an event. then, there are 5 possible label combinations for english according the annotation, while for the other languages there are only 2 possible labels. ## experiments and results the experiments are carried out with the 10-fold cross validation stratified technique. the measure is macro f1-score, according to the one used for the ranking of the systems in the competition, for each language and subtask. due to the large size of the english corpus, we select a sample subset for the analysis of the subtask a. thus, the experimental results presented in this paper have been obtained with a subset of the original texts in the case of the subtask a for english. we followed two strategies to randomly take 1,000,000 texts. firstly, the texts were selected keeping, in the subset, the same class proportion in the original corpus. the other strategy was to take the same amount of texts from both classes. the best results were obtained with the second strategy and are those shown in the tables that are discussed later. we use random, a python library, with seed 4 for the selection, taking the necessary number of elements per class. in the second case we select 500,000 texts from each class.