UJNLP at S em E val-2020 Task 12: Detecting Offensive Language Using Bidirectional Transformers (2020)

Abstract: In this paper, we built several pre-trained models to participate SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media. In the common task of Offensive Language Identification in Social Media, pre-trained models such as Bidirectional Encoder Representation from Transformer (BERT) have achieved good results. We preprocess the dataset by the language habits of users in social network. Considering the data imbalance in OffensEval, we screened the newly provided machine annotation samples to construct a new dataset. We use the dataset to fine-tune the Robustly Optimized BERT Pretraining Approach (RoBERTa). For the English subtask B, we adopted the method of adding Auxiliary Sentences (AS) to transform the single-sentence classification task into a relationship recognition task between sentences. Our team UJNLP wins the ranking 16th of 85 in English subtask A (Offensive language identification).

## introduction with the explosive growth of data generated by online social network users, the malicious content mixed in the information published by users has brought great challenges to the detection of hate speech and offensive language. due to the difference between social media and traditional media, users are allowed to post information at will. the emergence of large amounts of data invalidates manual review, resulting in a large number of methods that use machine learning for automatic classification. therefore, semeval 2020 released offenseval2 @xcite , which compared to offenseval @xcite increased the size of the dataset and added multilingual. multilingual data sets have been added in offenseval2, includes arabic @xcite , danish (sigurbergsson and derczynski, 2020), english @xcite , greek @xcite . this task of english is divided into three subtasks: offensive language recognition, automatic classification of attack types, and attack target recognition. our method for this task of english is based on bert and roberta. the organizer provided a large number of samples marked by the machine and included the confidence of each sample. because deep neural networks rely on large-scale datasets, the results obtained when using only the olid dataset are not ideal. due to equipment limitations and considering the error of machine labeling samples, we did not use all data for training. based on the olid dataset @xcite , we use the data with a confidence higher than threshold to expand the dataset to make the number of positive and negative samples equal to avoid the problems caused by the balance of dataset. for subtask b, we add auxiliary sentences(as) to transform the single-sentence classification task into a problem of relationship recognition between sentences. we compared the effect of bert and roberta with the nsp task removed, and bert is better for the relationship between sentences. we compare bert and roberta models. finally concluded that the roberta model can achieve better results on large-scale datasets. due to time constraints, we only participated in and submitted the english subtask a. our team ujnlp wins the 16th place (out of 85) in english subtask a. after the competition, we tested the effect of the method on subtask b and subtask c. ## related work in recent years, many researchers and research institutions have done a lot of research on insulting language, hate speech and other offensive language. the english task of offenseval2 consists of three subtasks. subtask a aims to detect offensive language as not offensive (not) or offensive (off). subtask b aims to classify the offensive language as targeting specific entities (tin) or not targeting specific entities (unt). subtask c aims to determine whether the goal of the offensive position is individual (ind), group (grp) or unknown (oth). in addition to research in the offenseval, other researchers also studied chinese @xcite and slovenian @xcite and german @xcite . one of the benefits of transfer learning is that it can learn effectively from limited labeled data. the bidirectional encoder representation from transformer (bert) model proposed by google ai language team @xcite is pre-trained using a large number of corpora from different sources . nuli used linear model, lstm and bert, and finally chose the best bert @xcite . they also pointed out that the bert model performed best in subtask a, and achieved the first place in subtask a of semeval-2019 task 6. in subtasks b and c, the dataset distribution is less smooth and the amount of data is less, so the effect is not as good as a. in contrast to other models, bert uses a two-way representation to take advantage of the left and right context and deepen the understanding of the sentence by capturing long-term dependencies between the parts of the sentence @xcite . @xcite believe that it is very important to preprocess the words. only in this way can the words form a sentence and conform to the normal grammatical structure. @xcite first detects words before training the model. if it is an unknown word, it first uses a word segmentation tool to process it, and then performs error correction operations if it is not in the dictionary. in addition, they also adopted a deep learning algorithm based on attention, which combines bilstm and emoji attention. it has been proved that this processing method is very effective. ## conclusion in this paper, we introduced the results of the ujnlp team participating in semeval-2020 task 12. due to time limitation, we only submitted the results of subtask a. outside of the competition, we tested the effect of our scheme on subtask b and subtask c. we noticed that the task in offenseval2019 has a class imbalance problem, which will have a significant impact on system performance. in order to offset the impact of various imbalances, we screened the data marked by the machine combined with the olid dataset to build a balanced dataset. we use roberta as the basis and use the dataset for fine-tuning. for subtask b, we add auxiliary sentences to convert single sentence classification tasks into sentence relationship recognition tasks, which to achieve good results on bert. our model ranked 16th in subtask a and achieved a result of macro-f1 of 0.9128. the current research results show that subtask a has achieved better results, while subtask b and subtask c still have problems to be solved. in the future, we will further use the proposed model for research on subtasks b and c.