SINAI at S em E val-2020 Task 12: Offensive Language Identification Exploring Transfer Learning Models (2020)

Abstract: This paper describes the participation of SINAI team at Task 12: OffensEval 2: Multilingual Offensive Language Identification in Social Media. In particular, the participation in Sub-task A in English which consists of identifying tweets as offensive or not offensive. We preprocess the dataset according to the language characteristics used on social media. Then, we select a small set from the training set provided by the organizers and fine-tune different Transformerbased models in order to test their effectiveness. Our team ranks 20th out of 85 participants in Subtask-A using the XLNet model.

## introduction in recent years the way of sharing information has suffered a change by the worldwide proliferation of use of social media. however, the freedom of this use often leads many users to communicate using offensive language. offensive language is defined as the text which uses hurtful, derogatory or obscene terms made by one person to another person @xcite . one of the strategies used to deal with offensive behavior on social media is to monitor or report this type of comment by users. however, this strategy is not entirely feasible due to the huge amount of data generated daily by users. therefore, it is necessary to develop automatic systems capable of identifying this type of content on the web. in this paper, we present the systems we developed as part of our participation in semeval-2020 task 12: offenseval 2 -multilingual offensive language identification in social media @xcite . in particular, the participation in sub-task a in english which consists of identifying tweets as offensive or not offensive. the rest of the paper is structured as follows. in section 2 some previous related studies are introduced. in section 3, we describe the pre-trained models we have studied to address the task. in section 4 we explain the data used in our methods, we present the details of the proposed systems and we discuss the analysis and evaluation results for our system. finally, we conclude in section 5 with remarks. ## background in recent years, the textual analysis of offensive detection has attracted the attention of several researchers in the field of natural language processing (nlp) leading to various academic events and shared tasks in different languages such as the first, second and third editions of the workshop on abusive language @xcite ; the first and second edition of germeval shared task on the identification of offensive language @xcite ; the shared task on aggression identification included in the first workshop on trolling, aggression and cyberbullying @xcite ; the mex-a3t track at iberlef 2019 on authorship and aggressiveness analysis @xcite ; the poleval 2019 shared task 6 on automatic cyberbullying detection in polish twitter @xcite ; the first edition of the hasoc track at fire 2019 on hs and offensive content identification in indo-european languages @xcite ; and finally the subtask 6 on identifying and categorizing offensive language in social media (offenseval) @xcite . these competitions have led to very promising systems to combat offensive language using different techniques. for instance, in offenseval last year's edition participants generally used systems based on deep learning and traditional machine learning. the top teams used ensembles and state-of-the-art deep learning models such as pre-trained models based on transfer learning. transfer learning is a recently emerging technique to enhance learning in a new task through the transfer of knowledge from a related task that has already been learned @xcite . it is widely used to develop models to solve problems where the availability of large data is limited in order to train, validate, and evaluate the systems. for instance, the top-performing team @xcite in sub-task a used different systems (a linear model, an lstm and bert) obtaining the best result with bert-base-uncased model. @xcite ranked sixth using an ensemble of cnn and blstm-bgru together with twitter word2vec embeddings and token/hashtag normalization. @xcite trained a variety of models such as cnn, mlp, svm or rnn combined the best of them in ensembles, but this team proved again that bert is a powerful tool for offensive text classification as it reached the best result. over the last two years, other transformer-based models based on bert such as roberta, xlnet or distilbert have been presented to improve bert on either its prediction metrics or computational speed. in this paper, we focus on explore the effectiveness of some of these models for offensive language detection adapting a small version of the training set provided by the organizers. ## system overview in this section, we describe the pre-trained models we have explored to address the task of offensive language identification. they are based on transformer which is an encoder-decoder architecture based on the attention mechanism @xcite . xlnet @xcite is a generalized autoregressive pre-training method that enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order. this allows the model to learn bidirectional relationships and consequently better handles dependencies and relations between words. xlnet outperforms bert on 20 tasks, including question answering, natural language inference, sentiment analysis, and document ranking. roberta (robustly optimized bert) @xcite ) approach arises to address some of bert's limitations improving training procedure since it was significantly undertrained. in order to do that, the following modifications were carried out: training the model longer, with bigger batches, over more data; removing the next sentence prediction objective from bert's pre-training; training on longer sequences; and introduces dynamic masking. distilbert @xcite is a general-purpose pre-trained version of bert, 40% smaller, 60% faster, that retains 97% of the language understanding capabilities. distilbert uses a technique called distillation, which is a compression technique in which a small model is trained to reproduce the behavior of a larger model @xcite , in this case the large model is bert. the idea is that once a large neural network has been trained, its full output distributions can be approximated using a smaller network. distilbert was trained on very large batches leveraging gradient accumulation, with dynamic masking and removed the next sentence prediction objective. ## conclusions the use of offensive language is a growing concern on social media due to the serious consequences it can have for users. to help combat this problem, automatic systems based on nlp can be very effective. for instance, transfer learning techniques have been proven to be especially effective for several tasks including classification. the nature of informal language which involves the use of abbreviations, dialect words, colloquial expressions or emojis, together with the limited length of the publications in case of twitter, make the task even more challenging. in this paper, we show that the results obtained by the pretrained models we have explored are promising using an adequate preprocessing in tweets. generally, it is not easy to have labeled data because of the high cost and effort involved. for this reason, transfer learning techniques are a valuable resource for carrying out these types of tasks.