NLPD ove at S em E val-2020 Task 12: Improving Offensive Language Detection with Cross-lingual Transfer (2020)

Abstract: This paper describes our approach to the task of identifying offensive languages in a multilingual setting. We investigate two data augmentation strategies: using additional semi-supervised labels with different thresholds and cross-lingual transfer with data selection. Leveraging the semi-supervised dataset resulted in performance improvements compared to the baseline trained solely with the manually-annotated dataset. We propose a new metric, Translation Embedding Distance, to measure the transferability of instances for cross-lingual data selection. We also introduce various preprocessing steps tailored for social media text along with methods to fine-tune the pre-trained multilingual BERT (mBERT) for offensive language identification. Our multilingual systems achieved competitive results in Greek, Danish, and Turkish at OffensEval 2020.

## introduction online social media has become one of the most important means of communication. unfortunately, the discourse is often laden with abusive language that can have damaging effects on social media users @xcite . as online abuse grows as a serious social problem, many social media platforms are employing human moderators to track inappropriate contents and comments. however, such human moderation comes at the cost of additional operation and the mental suffering of the workers @xcite . in response, the automatic identification of offensive language has been a rising area of interest to both academia and industry. computational methods presented in several prior studies @xcite @xcite have shown promising results on how we can automatically detect a large number of offensive contents with minimal human intervention. as an effort to promote this line of research, offenseval was organized at semeval-2019 @xcite . the organizers released a dataset, offensive language identification dataset (olid), that contains english tweets with human annotations over three sub-tasks: 1) whether a tweet is offensive or not, 2) whether an offensive tweet is targeted or not, and if so 3) whether the target is an individual or a group. the second edition of offenseval, offenseval 2020 : multilingual offensive language identification in social media, holds the same task but extends the olid to multiple languages (english, arabic, danish, greek, and turkish) @xcite . this paper describes our system, nlpdove, submitted to offenseval 2020 . we present two different data augmentation strategies. first, for english, we augment the training set with reliable samples filtered out from the provided semi-supervised dataset. second, we use samples from languages other than the target language to enhance performance in the target language, namely cross-lingual transfer. specifically, we propose a new metric, translation embedding distance(ted), to measure the data transferability of each example in the data. we use the metric to effectively select additional training data samples from other available transfer languages. we also introduce various preprocessing steps to extract meaningful information from user-generated texts despite their inherently noisy nature. for english, we also used the olid from offenseval 2019 @xcite . we randomly split the provided data into train and validation set by 9:1. multilingual pre-trained language model, particularly multilingual bert (mbert) @xcite . our final submitted model is an ensemble of multiple mbert models with different pooling mechanisms, representation layers and randoms seeds. our submission achieved macro f1 of 0.85 in greek (1st/38), 0.79 in danish (3rd/40), 0.80 in turkish (5th/47), 0.91 in english (15th/86), and 0.80 in arabic (20th/54). ## data the most noticeable update in offenseval 2020 is the extension of the task to four additional languages other than english: arabic @xcite , danish (sigurbergsson and derczynski, 2020), greek @xcite , and turkish (c ¸öltekin, 2020). this change facilitates the research of offensive language identification in multiple languages, and also enables researchers to explore the feasibility of cross-lingual approaches of the task. one thing to note is that unlike english tweets which had annotations for all three sub-tasks, the four new languages do not have all of them; they are only annotated on the first sub-task, which tells whether they are offensive or not. in the case of english, in addition to the manually annotated tweets from the previous workshop @xcite , offenseval 2020 released a large semi-supervised dataset where samples are labeled by a set of trained models, not by human annotators @xcite . therefore, the labels are not binary but given as floating-point numbers between 0 and 1 (i.e. the average score of multiple models), with their standard deviation indicating the confidence level among classifiers. for further details regarding the data, we refer to the original paper @xcite . the summary statistics of the dataset are presented in ## method there are three important components in our system: training data augmentation, extensive preprocessing, and model ensembling. the following subsections describe each part in detail. ## related work computational methods to identify offensive language have been studied in many different aspects @xcite . researchers have focused on a variety of categories of offensive language such as sexism, racism, and hate speech in general. @xcite identified a racial bias in hate speech detection datasets and proposed better annotation methods to address the problem. park and fung (2017) pointed out the gender bias in abusive language detection models and suggested a de-biasing algorithm. @xcite introduced a more fine-grained hate speech identification task than binary classification, and presented an architecture that leverages conditional variational autoencoder (cvae). large-scale pre-trained models like bert have achieved state-of-the-art results in various nlp tasks @xcite . @xcite , k et al. (2020), and @xcite explored the multilinguality of mbert and confirmed the model's effectiveness in cross-lingual transfer. in the task of offensive language detection, @xcite has adopted bert to solve the problem. in our work, we also use mbert but improve the method's ability under a multilingual setting by leveraging cross-lingual transfer. ## conclusion in this paper, we describe nlpdove's approach to semeval-2020 task 12: multilingual offensive language identification in social media. the system yielded significant results in the competition for multiple languages, confirming the strength of large-scale pre-trained language models and the importance of various preprocessing schemes and model ensembling. we examine two data augmentation strategies and propose a new metric, ted, to quantify cross-lingual transferability. in qualitative analysis, we inspect factors why data selection with ted increased performance. we also present the limitations of our system, mainly failing to detect the speaker's intention expressed subtly in the context, hoping to offer valuable insights and elicit future work in the field.