Hybrid Emoji-Based Masked Language Models for Zero-Shot Abusive Language Detection (2020)

Abstract: Recent studies have demonstrated the effectiveness of cross-lingual language model pre-training on different NLP tasks, such as natural language inference and machine translation. In our work, we test this approach on social media data, which are particularly challenging to process within this framework, since the limited length of the textual messages and the irregularity of the language make it harder to learn meaningful encodings. More specifically, we propose a hybrid emoji-based Masked Language Model (MLM) to leverage the common information conveyed by emojis across different languages and improve the learned cross-lingual representation of short text messages, with the goal to perform zero- shot abusive language detection. We compare the results obtained with the original MLM to the ones obtained by our method, showing improved performance on German, Italian and Spanish.

## introduction the extensive use of large-scale self-supervised pretraining has greatly contributed to recent progress in many natural language processing (nlp) tasks @xcite @xcite . in this context, masked language modelling objectives represent one of the main novelties of these approaches, where some tokens of an input sequence are randomly masked, and the objective is to predict these masked positions taking the corrupted sequence as input. still, little attention has been devoted to the adaptation of these techniques to tasks dealing with social media data, probably because they are characterized by a very domain-specific language, with high variability and instability. nevertheless, all these challenges make social media data an interesting testbed for novel deep-learning architectures, around the research question: how could the masking mechanism be adapted to target social media language? in this paper, we address the above issue by adapting a novel architecture for cross-lingual models called xlm @xcite to zero-shot abusive language detection, a task that has gained increasing importance given the recent surge in abusive online behavior and the need to develop reliable and efficient methods to detect it. in particular, we evaluate two methods to pre-train bilingual language models, one similar to the original xlm masked model, and the other based on a novel hybrid emoji-based masked model. we then evaluate them on zero-shot abusive language detection for italian, german and spanish, showing that, although our results are below the state-of-the-art in a monolingual setting, the proposed solutions to adapt xlm to social media data are beneficial and can be effectively extended to other languages. in the following, section 2 discusses the related work. section 3 describes our approach to train cross-lingual models for social media data classification, while section 4 presents the experimental setup. section 5 reports on the evaluation results, while section 6 summarizes our findings. ## related work the focus of this paper is the abusive language detection task, which has been widely explored in the last years thanks to numerous datasets, approaches and shared tasks @xcite @xcite @xcite @xcite covering different languages. an increasing number of approaches has been proposed to detect this kind of messages (for a survey on the task, see @xcite and @xcite ). abusive language detection is usually framed as a supervised learning problem, built using a combination of manually crafted features such as n-grams @xcite , syntactic features @xcite , and linguistic features @xcite , to more recent neural networks @xcite @xcite . @xcite address a comparative study of various learning models on the hate and abusive speech on twitter dataset @xcite , while @xcite build the offensive language identification dataset and experiment with svms, bilstm and cnn both on the binary abusive language classification and on a more fine-grained categorization. our work deals with the same task, addressed from a cross-lingual perspective. in recent years, some proposals have been made to tackle abusive language detection in a cross-lingual framework @xcite @xcite , with some attempts at zero-shot learning @xcite . most systems, however, rely on pretrained models and do not investigate the potential of indomain data for pretraining. additionally, as regards masked language models, we are not aware of any work in the literature modifying masking mechanisms for this task. ## evaluation we report the experiment results for each language in tables 2, 3, 4. for all languages, training is performed using only english data. results for german ( we evaluate mlm and he-mlm also for zeroshot italian hate speech classification, comparing the configurations with and @xmath1. for english, the best performing model is he-mlm with emoji tokens, while on italian the he-mlm model with no tokens is better in terms of macro averaged f1. when comparing configurations, we observe that the mlm model with emoji tokens has better f1 score than the mlm one in the non hate speech class, while the mlm model has improved performance on the hate class. this results in the mlm model having better macro average f1 for italian, while the mlm model with emoji tokens shows higher average f1 on english. when considering the hybrid emoji-based models, he-mlm achieves a higher f1 for the hate speech class in english and for the non hate class in italian. this results in the he-mlm model having a higher macro averaged f1 in the italian language, while the he-mlm model with emoji tokens is better on english. as a final test, we evaluate the performance of the model trained on english and spanish ( on all the runs, the classifier achieves a lower performance on german than on the other two languages, while the results on italian and spanish are comparable. this confirms the findings in @xcite suggesting that, even when using the same classification framework, experimental setting and amount of training data, offensive speech detection on german achieves lower performance than on other languages. this may have two possible reasons: on the one hand, german may have inherent characteristics that make it more challenging to classify for abusive language detection, for example the presence of compound words makes hashtag splitting more error-prone. on the other hand, the germeval dataset was built by sampling data from specific users and avoiding keyword-based queries, so to obtain the highest possible variability in the offensive language. this led to the creation of a very challenging dataset, where lexical overlap between training and test data is limited and where hate speech is not associated with specific topics or keywords, as suggested in @xcite . ## conclusions in this paper, we present a novel zero-shot framework for multilingual abusive language detection. we compare two cross-lingual language models, i.e., standard mlm and a hybrid version of mlm based on emojis (he-mlm), highlighting that the latter shows some advantages over the mlm model when used on social media data: first of all, when using emojis, the pre-training step is aimed at predicting tokens that are inherently more relevant for the final abusive language detection task whenever possible, as opposed to random tokens. secondly, emojis convey similar meaning in the languages that we consider, serving as a common trait between languages during pre-training. we also @xmath2. the proposed methods represent a novel contribution with respect to social media data processing and abusive language detection. our aim is not to create a system comparable with monolingual state-of-the-art solutions, but to investigate the possibility to use an unsupervised approach for zeroshot cross lingual abusive language detection. as a first step in this direction, we focused on four european languages, for which similar data were available. the only existing work dealing with zero-shot abusive language detection, presented in @xcite , only focuses on a language pair and, while obtaining promising results, relies on the english and spanish corpora annotated for hateval 2019 following the same guidelines and focusing on hate against immigrants and women. our approach aims to be more robust, comparing datasets annotated for different shared tasks which may adopt slightly different guidelines. in the near future, we plan to further extend the social media-specific datasets we are collecting to pre-train he-mlm, since 5 million tweets we used for each language correspond to a small-sized corpus compared to standard pre-trained language models. then, to investigate whether our results can be generalised also when dealing with typologically different languages, we will test our approach on additional abusive language datasets covering other languages @xcite .