U o B at S em E val-2020 Task 12: Boosting BERT with Corpus Level Information (2020)

Abstract: Pre-trained language model word representation, such as BERT, have been extremely successful in several Natural Language Processing tasks significantly improving on the state-of-the-art. This can largely be attributed to their ability to better capture semantic information contained within a sentence. Several tasks, however, can benefit from information available at a corpus level, such as Term Frequency-Inverse Document Frequency (TF-IDF). In this work we test the effectiveness of integrating this information with BERT on the task of identifying abuse on social media and show that integrating this information with BERT does indeed significantly improve performance. We participate in Sub-Task A (abuse detection) wherein we achieve a score within two points of the top performing team and in Sub-Task B (target detection) wherein we are ranked 4 of the 44 participating teams.

## introduction offensive language is pervasive in social media in this day and age. offensive language is so common and it is often used as emphasis instead of its semantic meaning, because of this, it can be hard to identify truly offensive content. individuals frequently take advantage of the perceived anonymity of computer-mediated communication, using this to engage in behaviour that many of them would not consider in real life. online communities, social media platforms, and technology companies have been investing heavily in ways to cope with offensive language to prevent abusive behavior in social media. one of the most effective strategies for tackling this problem is to use computational methods to identify offense, aggression, and hate speech in user-generated content (e.g. posts, comments, microblogs, etc.). the semeval 2020 task on abuse detection @xcite aims to study both the target and the type of offensive language that has not been covered by previous works on various offenses such as hate speech detection and cyberbullying. in this paper, we focus on the first two sub-tasks: sub-task a, that focuses on offensive language or profanity detection, a binary classification problem wherein the objective is to determine if a tweet is offensive or not. sub-task b, which focuses on the identification of target presence, also a binary classification problem, wherein we are required to determine if a tweet is targeted at someone or something or not. ## related work the offensive language identification dataset (olid) was designed by @xcite for the 2019 version of this task as there was no prior work on this task before then. olid is made up of 14,100 tweets that were annotated using experienced annotators but suffered from limited size, especially class imbalance. to get around this, offenseval 2020 made use of the semi-supervised offensive language identification dataset (solid) @xcite . ## system overview for sub-task a, we test three models: a standard neural network that uses tf-idf features, bert and the ensemble of these two. for sub-task b we use noun counts, bert and the ensemble of both. ## experimental setup for each of the sub-tasks we participate in, we split our training set into a training set and development set in a 4:1 ratio. our test set is the evaluation set for semeval-2019. we submit the best version of these experiments to semeval-2020. also, in each case, we first experiment with bert, then by adding additional parameters to bert and finally by use of cost-weights. all ensemble models were created at the embedding layer by appending additional features to bert embeddings before then using a fully connected layer for classification. ## results and analysis we present our overall rankings on each of the two sub-tasks in as described in section 1.1, we hypothesise that deep learning models, especially those that use pretrained embeddings and so are trained on a small number of epochs, can benefit from corpus level count information. so as to test this hypothesis we create three different models: a simple neural network (snn), consisting of a single layer, that is fed with tf-idf features, bert and an ensemble of the two. the ensemble model is constructed by removing the classification layer from bert and the snn model, concatenating their output and passing this concatenated vector through a new fully connected layer for classification. we perform our experiments using 10% of the training data as our training set, and last year's semeval test set as our test set. we pick the best performing model and use that to generate results for our submission. unfortunately we were unable to train on more than 10% of the training set. our experiments show that corpus level count information captured by tf-idf can indeed boost the performance of bert. our analysis of the training and test data using the wilcoxon signed-rank test as described by tayyar @xcite shows that the training and development sets are different enough to warrant the use of cost-weighting (section 3.5). to this end we introduce cost weighting to each of the three models described above and the results of these experiments are presented in we observe that adding the optimal cost weights to poorly represented classes significantly improves the performance of all models. the ensemble model, however, still outperforms either of snn or bert, despite a large increase in performance for bert after adding cost weights. as mentioned we were only able to train our bert and ensemble models with 10% of the training data. the performance of our models can be further improved given gpu resources as shown in ## conclusion we show that incorporating corpus level information does help improve the performance of bert. we achieve competitive results using just 10% of the available dataset and would like to test the limits by training with the full dataset. our experiments also show that noun counts do help boost the performance of bert, but not as much as cost-weighting.