Ferryman at S em E val-2020 Task 12: BERT -Based Model with Advanced Improvement Methods for Multilingual Offensive Language Identification (2020)

Abstract: Indiscriminately posting offensive remarks on social media may promote the occurrence of negative events such as violence, crime, and hatred. This paper examines different approaches and models for solving offensive tweet classification, which is a part of the OffensEval 2020 competition. The dataset is Offensive Language Identification Dataset (OLID), which draws 14,200 annotated English Tweet comments. The main challenge of data preprocessing is the unbalanced class distribution, abbreviation, and emoji. To overcome these issues, methods such as hashtag segmentation, abbreviation replacement, and emoji replacement have been adopted for data preprocessing approaches. The main task can be divided into three sub-tasks, and are solved by Term Frequency–Inverse Document Frequency(TF-IDF), Bidirectional Encoder Representation from Transformer (BERT), and Multi-dropout respectively. Meanwhile, we applied different learning rates for different languages and tasks based on BERT and non-BERTmodels in order to obtain better results. Our team Ferryman ranked the 18th, 8th, and 21st with F1-score of 0.91152 on the English Sub-task A, Sub-task B, and Sub-task C, respectively. Furthermore, our team also ranked in the top 20 on the Sub-task A of other languages.

## introduction with the continuous development of society, online social network (osn) and microblog sites have attracted internet users more than any other types of websites. twitter, facebook, and instagram offer a growing variety of services that attract users from different cultures, religions, and interests around the world. the number of osn users are getting larger, the variety of users and the contents uploaded are growing rapidly. the strong inclusiveness and free atmosphere of social networks provide a platform for all kinds of users to communicate, share, and discuss. due to the argument caused by different backgrounds, cultures, and beliefs of users, malicious comments have been emerging all these years. the unrestrained release of offensive remarks on social media have negative impacts on the development of the whole society and may lead to outbreaks such as violence, crime, and hatred. the free and inclusive nature of osn is easy for the widespread of negative effects among large number of users, since the comments and news are almost uncontrollable. therefore, the detection and protection of offensive content on social media is particularly important. offensive comments are mostly related to religion, race, and gender, offending, insulting, or threatening others through a series of derogatory words. this is the main part of the objectionable content, which is widely distributed in social media. many users have abused the power given by social networks to express their opinions excessively. due to the diversity of language and expression on the internet, the regulation of offensive content in osn is extremely difficult to solve. therefore, mining, processing, and automatic detection the information from the offensive content cannot be effectively achieved. the main task of this competition is to identify whether tweets in multiple languages on social media are offensive. this task has the following four challenges. 1. the given dataset is insufficient to train complex * all the corresponding to yanru zhang. models. 2. different language features cause irregular sentences and out-of-speech vocabulary. 3. alack of vocabulary leads to low detection accuracy of offensive content and difficulty in emoji recognition. 4. the distribution of target categories is uneven, and the test set and training set are inconsistent. to achieve the goal of this task, we have adopted the term frequency-inverse document frequency (tf-idf) vectorizer, bidirectional encoder representation from transformers (bert), and multi-dropout in our model. meanwhile, we applied different learning rates to tasks in different languages based on bert and non-bert models in order to get better results. in the rest of this paper, we organize the content as follows: the related works of the hostile content will be described in section 2.the data description, preprocessing details, and the methodology of our model are introduced in section 3. the experimental results are discussed in section 4. the work will be summarize at the end of this paper. ## related work the field of detecting offensive content have attracted increasingly interests in the recent years. many previous works in this domain are strongly related with hate speech detection @xcite and abusive languages of online statement @xcite . offensive comments detecting are tacked with several different approaches, such as feature engineering-based method, using specific bag-of-words, systems combined with several models. using the bag-of-words to detect hate speech is an effective way @xcite . other scholars utilize the word lists containing offenses as words or phrases and get outstanding experimental result @xcite . as for the feature engineering-based approaches such as svm, regression models, convolutional neural networks (cnn), and etc., there are some other attempts. such as the system based on a k-max pooling cnn model @xcite and the cnn model based on word2vec embedding @xcite . for system combined with several models in detecting offensive tweet, alessandro seganti et al. constructed a system which system combines several models (lstm, transformer, openai's gpt, random forest, svm) with various embeddings (custom, elmo, fasttext, universal encoder) together with additional linguistic features (number of blacklisted words, special characters, etc.) @xcite . we also adapt ensemble strategy in our work. it is worth mentioning that the latest natural language processing (nlp) model bidirectional encoder representations from transformers (bert) @xcite has attracted much attention. this model was trained on a large text corpus such as wikipedia, which can be applied to various nlp tasks without changing its core archirtecture. zhenghao @xcite bert to capture linguistic, syntactic and semantic features @xcite . andraz pelicon et al. used a fine-tuned bert model to achieve offensive language identification @xcite . the existing implementation of bert and the exceptional performance of this technique inspired our mind. there are also several other superior methods such as the syntactic features method for identifying the targets and intensity of hate speech @xcite or approaches the task as topic modelling problem using latent dirichlet allocation @xcite . we take these works as reference while constructing our own model. ## result for this task, our team compares two main methods tf-idf vectorizer and bert to deal with the dataset. the result is shown in table1, and it demonstrates that although tf-idf vectorizer is light and flexible, its performance is far less than bert. this is because tf-idf vectorizer only uses the number of occurrences of a word in a file set to map the importance of the word, while the actual tweet comments are implied in sentences, and sometimes even cross sentence boundaries. in contrast, bert can make fully use of the entire sentence for recognition, since it receives training from a huge corpus of different sources, which can better reduce the impact of noise. ## conclusion in this work, we evaluate the performance of the models and methods we use in the three tasks of semeval-2020 task 6: identifying and categorizing offensive language in social media. we have used the radam optimizer, which makes the model have a very good performance for different learning rates. at the same time, we have used the last five layers of @xcite tokens to make the model more able to capture higher-order semantic features. methods and models like bert, tfidf all show good results in the given dataset and ranked the 18th, 8th, and 21st with f1-score of 0:91152 on the english sub-task a, sub-task b, and sub-task c, respectively. in addition, we also achieved the top 20th rankings on the sub-task a of other languages.