I 2 C at S em E val-2020 Task 12: Simple but Effective Approaches to Offensive Speech Detection in T witter (2020)

Abstract: This paper describes the systems developed for I2C Group to participate on Subtasks A and B in English, and Subtask A in Turkish and Arabic in OffensEval (Task 12 of SemEval 2020). In our experiments we compare three architectures we have developed, two based on Transformer and the other based on classical machine learning algorithms. In this paper, the proposed architectures are described, and the results obtained by our systems are presented.

## introduction social media is, currently, one of the most popular mass media. much of the world's population communicates and expresses their opinions through these media. unfortunately, the use of offensive language is very frequent in social networks, having detrimental effects for its users. in @xcite , the authors categorize the hate language as abusive language, aggression, cyberbullying, hatefulness, insults, personal attacks, provocation, racism, sexism, threats or toxicity. all of them have in common the use of offensive language, being a major threat to social media platforms. in recent years, automatic detection of offensive language on social media, using machine learning algorithms, has aroused great interest among researchers. in @xcite , authors present a survey on automatic hate speech detection using nlp, and a critical overview on how the automatic detection of hate speech in text has evolved over the last years is described in @xcite . the early works related to offensive language detection employing the use of features like bag of words, word and character n-grams in conjunction with classical machine learning classifiers @xcite . nlp approaches have the drawback of being dependent on the language used in the text so, recently, other machine learning approaches such as neural networks or deep learning are being used @xcite . most of the researches to detect offensive language focus on the english language. however, there are not many studies in other languages. in this sense, the proposal of 2020 edition of offenseval is very novel, also offering corpus labeled in turkish, arabic, danish and greek. in this paper, we present our participation in offenseval (task 12 of semeval 2020) as i2c team. we focus our efforts on subtasks a and b for english, but we also have worked on subtask a in arabic and turkish. ## data description and preprocess we participated in subtask a (offensive language identification) in english, turkish and arabic, and in subtask b (automatic categorization of offense types) in english @xcite . ## methodology and experiments for all the subtasks, the methodology behind our approach was oriented to find the best combination of algorithms and training dataset to elaborate our models. we decided to test bert ( devlin et al. 2018 to obtain the best model for each subtask, 3 architectures were tested. in the first (#architecture 1), each tweet was preprocessed (hashtags, urls, @users, emojis, ... were removed) using the "bag of word" model with tf-idf for giving weight to the word. stop words were also removed and all words were transformed to lowercase, and lemmatization was used to reduce the vocabulary of words. in this approach, the generated vector is passed as input to the machine learning method. the algorithms applied were lineal regression, svm (rbf kernel), xgboost, randomforest and k-neighbors. we added the hate words existence on each tweet, lexical characteristics and entities using the spacy foot_1 tool. the number of times that any of these hate words appears in the tweet indicates the weight of the characteristic itself. due to the large number of characteristics, a selection of them was made. to make this selection we used a method which eliminates all the characteristics except for the best n of them, where n was 20, 50 and 100. in our case, the chi 2 test was used as selection criterion for the best characteristics. for this architecture we used, for all the experiments, balanced and imbalanced datasets with 20,000 random tweets (training 80%, test 20%). for the balanced dataset, 10,000 random tweets were selected from each of the classes in the original file, and for the imbalanced dataset, the same proportion as the original files had, that is 89% and 11% for subtask a, and 91% and 9% for subtask b. the second tested architecture (#architecture 2) is based on generating the features with a set of transformers @xcite and using them to feed the same machine learning methods mentioned above (figure 1 ). we decided to test the models bert, transfomerxl, xlm, xlmnet, distilbert, roberta y xlm-roberta. we did not use pre-processing, in order to benefit from the representation's capabilities of transformers. we tokenized the tweets with a specific tokenizer provided by transformer for each model. the third approach tested (#architecture 3) is based only in transformers. we decided to test bert, roberta, distilbert, albert and xlm-roberta pretrained models from transformers. again, we did not use pre-processing, in order to benefit from the representation's capabilities of transformers. we tokenized the tweets with a specific tokenizer provided by transformer for each model. for the finetuning of bert and others, we used the default parameters of their respective repository but trained on 2 epochs and a maximum sequence-length of 128. because the original file was huge and imbalanced, for #architecture 2 and #architecture 3 we carried out two tests: one with a random data set with the same proportion of tweets of each class as the original file supplied by the organizers, and the other with a balanced data set (both datasets with 20,000 tweets) and a maximum sequence length of 128, epoch. 4 results in this section, we describe the systems developed for the semeval20 subtask a in english, turkish and arabic and subtask b in english. subtasks a and b in english. in first place, the corpus of tweets provided by the organizers was preprocessed. using #architecture 1 described in the previous section, we tested the algorithms logistic regression, randomforest, xgboost, svm and k-neighbours. results are shown in tables 1, 2, 3 and 4 (best result marked in bold). before testing #architecture 2, we decided to test how transformers behave with balanced and imbalanced datasets. in the same way as #architecture 1, two datasets of 20,000 tweets each were built. for the balanced dataset, 10,000 tweets from each class were randomly selected, and for the imbalanced dataset, we selected 20,000 tweets with the same ratio of classes as the initial dataset. the results are shown in subtask a in turkish. taken into account the results obtained with transformers for subtask a in english, for this subtask, it was decided to test only bert multilingual-cased pretrained model. file supplied by the organization was imbalanced and the number of instances of the minority class required a prior study regarding the need for sampling. several sampling techniques were tested as it is shown in ## conclusions in this paper, we have presented i2c's contribution to semeval20 for subtasks a in english, arabic and turkish and subtask b in english. in order to achieve optimal results in the challenge suggested by the organization, three architectures were developed and tested: the first one was based on classical machine learning methods for text classification tasks, the second used a mixed architecture for features generation to embed all the tweets and feed machine learning classical models, and in the last one we fine-tuned a set of pretrained models from transformers to classify the tweets. among the three architectures, the one based on transformers only obtained the best results, in particular, by bert using the bert-base-uncased pretrained model for the two tasks in english, and bert using the bert-basemultilingual pretrained model for tasks in arabic and turkish. we consider that more research should be put into multilingual solutions. our experiments were done using the default parameters so there is room for improvement with many adjustments that we plan to consider for future works.