GUIR at S em E val-2020 Task 12: Domain-Tuned Contextualized Models for Offensive Language Detection (2020)

Abstract: Offensive language detection is an important and challenging task in natural language processing. We present our submissions to the OffensEval 2020 shared task, which includes three English sub-tasks: identifying the presence of offensive language (Sub-task A), identifying the presence of target in offensive language (Sub-task B), and identifying the categories of the target (Sub-task C). Our experiments explore using a domain-tuned contextualized language model (namely, BERT) for this task. We also experiment with different components and configurations (e.g., a multi-view SVM) stacked upon BERT models for specific sub-tasks. Our submissions achieve F1 scores of 91.7% in Sub-task A, 66.5% in Sub-task B, and 63.2% in Sub-task C. We perform an ablation study which reveals that domain tuning considerably improves the classification performance. Furthermore, error analysis shows common misclassification errors made by our model and outlines research directions for future.

## introduction the rapid development of user-generated content in social media has given millions of people the ability to easily share their ideas with each other. while users can publicly communicate their beliefs with others, their published content may be offensive to other individuals or groups. since offensive speech can jeopardize others' ability to express themselves, many social media platforms restrict the type of acceptable content on their platforms. manually detecting such content is expensive and time-consuming. thus, automatic detection of these behaviors in social media has attracted researchers' attention. although this topic has been explored in prior works @xcite @xcite , the task of offensive language detection still remains challenging. the offenseval 2020 shared task @xcite aims to encourage continued work in this area, and we progress the study of offensive language detection through our participation in this shared task utilizing contextualized language models. offenseval 2020 evaluates various aspects of offensive language following the scheme of the olid dataset @xcite , including identifying the presence of offensive language (sub-task a), identifying whether the offensive language is targeted (sub-task b), and identifying whether the target of the offensive language is an individual, group, or something else (sub-task c). the task extends the prior work by introducing a new data collection from twitter that spans multiple languages. our experiments explore various versions of the bert model (bidirectional encoder representations from transformers @xcite ) tuned for offensive language identification. as previous studies have mentioned, dealing with social media content can be challenging because it is often short and noisy @xcite . to alleviate this problem, we first fine-tune a bert model on a large amount of unlabelled data gathered from twitter. we pre-process the tweets by splitting hashtags and replacing emoji with textual descriptions. we also explore an ensemble learning method (i.e., a multi-view svm model akin to @xcite ) which combines different word n-gram features of the input text as views and predict the output based on the combination of these views. we report competitive performance of these models, which achieve macro f1 scores of 91.7%, 66.5%, and 63.2% on sub-tasks a, b and, c, ranking 5th, 6th, and 11th for each sub-task out of 85, 43, 39 participated teams respectively. our contributions are as follows: 1) we present variations of bert model tuned for different aspects of offensive language detection; 2) we report competitive results of our models with detailed comparisons; 3) we perform an ablation study to examine the effect of different components in the proposed systems; and 4) we conduct an error analysis to provide insights into how our models perform the classification, and where to focus in future work. ## related work the research of automatic offensive language detection has gained attention in the past decade. most prior work in the domain employs the supervised learning paradigm @xcite . traditional models often made use of rule-based methods, such as template-based strategy @xcite or pre-defined black-lists @xcite . aside of methods, a widely used category of features are surface-level features, e.g., n-gram features @xcite . these features are often highly predictive, and can be easily combined with other approaches. others explored the combination of n-gram features with part-of-speech @xcite as well as dependency information @xcite for offensive language detection. these approaches leverage prior linguistic knowledge in order to generate features. however, the generated features are usually derived from pre-existing natural language processing systems, which could lead to the propagation of errors in the models @xcite . these features are usually combined with classical machine learning classifiers such as svm @xcite and logistic regression @xcite . others have explored using a multi-view learning paradigm @xcite with an ensemble classifier @xcite . deep learning techniques have also been shown to be effective for offensive language detection @xcite @xcite , with approaches such as convolution neural networks (cnn) @xcite and long short-term memory (lstm) networks @xcite . more recently, pre-trained transformer-based networks, such as bert @xcite have shown great advantages in learning context-sensitive word representations. in offenseval 2019, bert-based and ensemble methods were the most effective approaches @xcite @xcite . ## methodology in our experiments, we utilize a pre-trained contextualized language model, namely bert @xcite , to identify the offensive language. also, we explore techniques for pre-processing, contextualized language modeling, and model outputs ensembling. pre-processing. given the unique conventions of language on twitter, we explore several tokenization pre-processing techniques to enable the downstream models to encode information more effectively. since the length of a tweet is limited to 280 characters, emoji are often used to convey emotions and tones efficiently. to address differences in user preferences among users, we replace emoji with a textual description of the icons in order to shorten the domain gap between the corpus and the tweets. we use the the mapping of emoji to english descriptions from the open source python package emoji. 1 hashtags are another common convention in tweets. they are used to describe topics related to certain tweets. they often consist of several words concatenated together, such as #voteredsaveamerica and #trumptrain. since hashtags do not contain whitespaces at word boundaries, additional logic is required for segmentation. we utilize the open source wordsegment foot_1 library to obtain the boundaries and further construct the original textual tokens. the offenseval 2020 dataset replaces all usernames with a @user placeholder. this can result in some long strings of redundant and repetitive placeholders because tweets are often prefixed with numerous users. we tokenize using the nltk tweet tokenizer @xcite and drop @user tokens if repeated more than three times consecutively to avoid redundant information (similar to @xcite ). furthermore, the token url, which is the artificial placeholder for any url encountered in tweets, is also replaced with http to match the vocabulary in the pre-trained embeddings. contextualized language modeling. we utilize the bert contextualized language model @xcite . since there are language differences between the formal text that bert is trained on (i.e., wikipedia and books) and social media posts (i.e., tweets), we first tune the model to the particular domain, akin to the domain pre-training approach described in @xcite . this is accomplished by taking the original model and continuing to train the masked language model and next sentence prediction objectives using a large amount of unlabeled tweets. note that we do not extend the vocabulary; we rely on the model's original wordpiece tokens. we then fine-tune the model for identifying offensive language utilizing labeled training data. since the task is sequence classification, we utilize the classification mechanism of the model (i.e., a linear layer on top of bert's classification token). we train the model minimizing the cross-entropy loss, as compared to the gold training labels. we train a separate model for each sub-task during experiments. model outputs ensembling. as mentioned in section 2, ensemble approaches are often beneficial for offensive language detection and related tasks. in this work, we extend the multi-view svm approach from @xcite with the addition of features from the contextualized language model classifier. specifically, linear svm classifiers (view-classifiers) using various n-gram ranges foot_2 are first trained for each sub-task in addition to the bert-based classifier. then, the outputs of the view-classifiers (probability output from svm and sigmoid output from bert) are concatenated as a feature vector for a final linear svm classifier (the meta-classifier). for the svm view-classifiers, we explore using both l1 and l2 regularization. ## experiment in this section, we present settings, results and analysis for our experiments. we first give a brief introduction of the dataset used for training and evaluation. then we show our experimental settings, and perform a comprehensive analysis including experimental results analysis, ablation analysis, and error analysis foot_3 over our models' results. @xcite introduced the offensive language identification dataset (olid), a large-scale dataset of english tweets constructed by searching for specific keywords that may include offensive words on twitter. they developed a hierarchical annotation schema that determines: 1) if the tweet is offensive (off) or non-offensive (not); 2) if an off tweet is targeted (tin), or untargeted (unt); and 3) if a tin tweet is targeted toward individual (ind), group (grp), or others (oth). we refer the readers to @xcite for more details about the dataset characteristics.