UHH - LT at S em E val-2020 Task 12: Fine-Tuning of Pre-Trained Transformer Networks for Offensive Language Detection (2020)

Abstract: Fine-tuning of pre-trained transformer networks such as BERT yield state-of-the-art results for text classification tasks. Typically, fine-tuning is performed on task-specific training datasets in a supervised manner. One can also fine-tune in unsupervised manner beforehand by further pre-training the masked language modeling (MLM) task. Hereby, in-domain data for unsupervised MLM resembling the actual classification target dataset allows for domain adaptation of the model. In this paper, we compare current pre-trained transformer networks with and without MLM fine-tuning on their performance for offensive language detection. Our MLM fine-tuned RoBERTa-based classifier officially ranks 1st in the SemEval 2020 Shared Task 12 for the English language. Further experiments with the ALBERT model even surpass this result.

## offensive language detection the automatic detection of hate-speech, cyber-bullying, or aggressive and offensive language became a vividly studied task in natural language processing (nlp) in recent years. the offensive language detection (old) shared task 6 of 2019's international workshop on semantic evaluation (semeval) @xcite attracted submissions from more than 100 teams. the offensive language identification dataset (olid) used in this shared task comprises three hierarchical classification sub-tasks: a) offensive language detection, b) categorization of offensive language, and c) offensive language target identification @xcite . for task a, 14,100 twitter tweets were manually labeled as either offensive (off) or not offensive (not). task b distinguishes the 4,640 offensive tweets from task a into targeted (tin) or general, untargeted (unt) offensive language. task c, finally, separates targeted insults into the three categories: groups (grp), individuals (ind), and others (oth). offenseval 2020, the semeval 2020 offensive language detection shared task, does not provide an own manually labeled training set for the english language @xcite . instead, a large 'weakly labeled' dataset was published by the organizers, containing roughly nine million tweets. each tweet has been automatically classified by an ensemble of five different supervised classifiers trained on the olid dataset. the weakly labeled dataset contains the raw tweet (with user mentions replaced by a special 'user' token) along with the average label probability and the variance of the five classifier predictions. since there is no way that such weak labels themselves carry more useful information to a machine learning system than the original dataset on which the five classifiers were trained, we decided not to use any of the weakly labeled information. instead, for our classification systems, we rely on the 2019 olid dataset only. however, the offenseval 2020 dataset is an ample source to build models using unsupervised learning, particularly for domain-adaptation of a pre-trained language model such as bert @xcite or its successors which are based on the transformer neural network architecture. unfortunately, training a transformer-based language model in an unsupervised manner is incredibly resource-consuming, making it impractical to learn from large datasets without access to larger gpu clusters or tpu hardware. regarding this, the contribution of our paper is two-fold: offensive language detection: nowadays, a number of public datasets are available to train machine classifiers for detecting english offensive language. unfortunately, underlying data sources, category definitions, data sampling strategies, and annotation guidelines differ to a large extent between these datasets. hence, results of different datasets are hardly comparable, and training sets usually cannot be combined to obtain a more robust classification system. @xcite , and fortuna and nunes (2018) conducted insightful surveys on this rapidly growing field. mandl et al. (2019), struß et al. (2019), and basile et al. (2019) recently organized shared tasks on the topic. although winning systems can achieve striking prediction accuracy, old is far from being a solved problem. prediction performance usually drops severely if the target data comprises different characteristics than the training data. gröndahl et al. (2018), for instance, show that many machine learning architectures can be fooled easily just by adding the word "love" to an offensive tweet to make it appear as non-offensive. struß et al. (2019) highlight that linguistic information alone is not enough in many cases to decide whether a tweet is hateful or not. also context information, e.g. about tweeting users themselves (ribeiro et al., @xcite , or mentioned users in tweets @xcite ) can be a useful feature for automatic old. pre-trained language models for text classification: transfer learning with deep neural networks, in general, has proven to be superior over supervised learning for text classification, especially for small training data situations. this is illustrated exemplarily in our last year's approach @xcite to the old semeval shared task which employed unsupervised pre-training of a recurrent neural network architecture with a topic cluster prediction task. practically all winners of the aforementioned shared task competitions employ some form of a fine-tuned bidirectional transformer-based language model, a neural network architecture for which @xcite published with bert the seminal work. this architecture has been proven highly successful for transfer learning. a base model is pretrained with a mlm task and a next-sentence prediction (nsp) task in an unsupervised manner on very large datasets. the knowledge about language regularities and semantic coherence encoded in the network during this step can then be employed successfully in later training steps of fine-tuning the network weights to the actual classification task. for instance, @xcite al. (2019a) 2019 ) test a wide range of bert fine-tuning methods for text classification and develop best practice recommendations. since bert, a number of successor models improving the network architecture, the pre-training strategy, or the pre-training dataset have been published. a selection of these models will be evaluated in section 3. ## fine-tuning transformer networks we investigate two questions regarding the fine-tuning of pre-trained transformer networks for old. first, which pre-trained model performs best on the 2020 old shared task? second, we investigate how much language model fine-tuning on in-domain data prior to classification fine-tuning improves the performance of the best model. ## results regarding the ensembles of model variants, we see in for the offenseval 2020 shared task, we decided to submit the results form the mlm pre-trained roberta ensemble. 4 figure 2 shows the corresponding confusion matrices for the submitted predictions. one observation that can be revealed from the matrices is that the predictions for tasks b and c are considerably biased towards the majority class. for task a, however, we see more false positive cases for the offensive class which is underrepresented in the training data. a qualitative look into a sample of false predictions (cp. fig. 1 ) reveals tweets wrongly predicted as offensive (false positives), some of which seem inconsistently annotated in the gold standard. parts of expressions in examples 1, and 2 qualified as offensive in many other training examples. examples 3, and 4 contain some negative words that may have triggered a higher offensiveness prediction score. for the false negative samples, it is not really obvious why the models missed the correct class, since except for example 6, they actually contain strong offensive vocabulary. ## conclusion after last year's semeval shared task on offensive language detection was already dominated by the then newly published bert model, for the year 2020 competition we were successful in fine-tuning bert's successor models to create the best performing system. the predictions obtained from fine-tuning of the albert model on the olid dataset achieved 92.25 % macro f1-score as the best overall result (including our post-submission experiments) on the official test set of the shared task a for the english language. we also found the 'weak labels' distributed along with 9 million tweets by the shared task organizers not useful for training our classifiers. however, the tweets themselves provided useful indomain data for unsupervised pre-training. with 92.04 % macro-f1, our predictions based on further language model pre-training of the roberta model on ca. 440.000 tweets, before fine-tuning on last year's olid dataset achieved the first rank in the official semeval competition for sub-task a, and also high ranks @xcite for the other two sub-tasks. we conclude that further pre-training of a transformer model with in-domain data is useful for offensive language detection. however, for tasks b and c, our models are clearly biased towards the majority class resulting in somewhat lower ranks. hence, taking the high class imbalance of the olid dataset better into account could further improve our results.