XD at S em E val-2020 Task 12: Ensemble Approach to Offensive Language Identification in Social Media Using Transformer Encoders (2020)

Abstract: This paper presents six document classification models using the latest transformer encoders and a high-performing ensemble model for a task of offensive language identification in social media. For the individual models, deep transformer layers are applied to perform multi-head attentions. For the ensemble model, the utterance representations taken from those individual models are concatenated and fed into a linear decoder to make the final decisions. Our ensemble model outperforms the individual models and shows up to 8.6% improvement over the individual models on the development set. On the test set, it achieves macro-F1 of 90.9% and becomes one of the high performing systems among 85 participants in the sub-task A of this shared task. Our analysis shows that although the ensemble model significantly improves the accuracy on the development set, the improvement is not as evident on the test set.

## introduction with the development of it, social media has become more and more popular for people to express their views and exchange ideas publicly. however, some people may take advantage of the anonymity in social media platform to express their comments rudely, and attack other people verbally with offensive language. to keep a healthy online environment for the adolescences @xcite and to filter offensive messages for the users @xcite , it is necessary and significant for technology companies to develop an efficient and effective computational methods to identify offensive language automatically. transformer-based contextualized embedding approaches such as bert @xcite , xlnet @xcite , roberta @xcite , albert @xcite or electra @xcite have re-established the state-of-the-art for many natural language classification tasks especially the glue dataset @xcite . their pre-trained models were pre-trained on different large datasets, for example, bert was pre-trained on the bookcorpus @xcite and english wikipedia, and roberta was pre-trained on cc-news @xcite , openwebtext @xcite , and stories (trinh and le, 2018) which enable their models to learn different language features. this paper presents six transformer-based offensive language identification models that learn different features from the target utterance. to combine the distinctive learned language features, we introduce an ensemble strategy which concatenates the representations of the individual models and feed them into the linear decoder to make binary classification (section 4.2). it largely improves the performance over the baseline on our dev set (section 4.4). ## related work offensive language in twitter @xcite , facebook @xcite has been widely studied. in addition, different aspects of offensive language have been studied, like the type and target of offensive posts @xcite , cyberbullying @xcite , aggression @xcite , toxic comments @xcite and hate speech @xcite @xcite . many deep learning approaches have been used to address the task. the convolutional neural networks (cnns), long short-term memory networks (lstms) and fasttext were applied on the hate speech detection task @xcite . gamback and sikdar (2017) used four convolutional neural network (cnn) models with random word vectors, word2vec word vectors, character n-gram, and concatenation of word2vec word embeddings and character n-grams as feature embeddings separately to categorize each tweet into four classes: racism, sexism, both (racism and sexism) and non-hate-speech. ## data description the datasets we use are offensive language identification dataset (olid) @xcite and semi-supervised offensive language identification dataset (solid) @xcite . given a tweet, the task is to predict whether the content involves offensive language. id tweet label 09 @user buy more icecream!!! not 71 @user that's because you are an old man. off (a) examples from olid. id tweet avg_conf conf_std 167 @user pre-ordered your book, received in july, started last night and cannot put it down! 0.215 0.188 524 a combination of innocence and corruption 0.691 0.142 (b) examples from solid. olid is a collection of 14,100 english tweets annotated as off or not. it is divided into a training set of 13,240 tweets and test set of 860 tweets @xcite . solid is a collection of about 9 million english tweets labeled in a semi-supervised manner @xcite . the data are annotated with avg_conf and conf_std predicted by several supervised models @xcite . the test set provided by organizers this year has 3887 tweets.