Multilingual and Multitarget Hate Speech Detection in Tweets (2019)

Abstract: Social media networks have become a space where users are free to relate their opinions and sentiments which may lead to a large spreading of hatred or abusive messages which have to be moderated. This paper proposes a supervised approach to hate speech detection from a multilingual perspective. We focus in particular on hateful messages towards two different targets (immigrants and women) in English tweets, as well as sexist messages in both English and French. Several models have been developed ranging from feature-engineering approaches to neural ones. Our experiments show very encouraging results on both languages.

## motivation social media networks such as facebook, twitter, blogs and forums, have become a space where users are free to relate events, personal experiences, but also opinions and sentiments about products, events or other people. this may lead to a large spreading of hatred or abusive messages which have to be moderated. in particular, theses messages may express threats, harassment, intimidation or "disparage a person or a group on the basis of some characteristic such as race, color, ethnicity, gender, sexual orientation, nationality, religion, or other characteristic" @xcite . although some countries, such as the united states, where hate speech is protected under the first amendment as freedom of expression @xcite , many other countries, such as france, have laws prohibiting it, laws that extend to the internet and social media. for instance, since the french law of 27 january 2017 related to equality and citizenship, penalties due to discrimination are doubled (sexism is now considered as an aggravating factor). gender equality has also been declared "major national cause" for the five-year period mandate of french president emmanuel macron foot_0 . in this context, it is important to automatically detect hateful messages on social platforms and possibly to prevent the widespreading of gender/racial stereotypes, especially towards young people. from a computational point of view, hate speech detection is casted as a binary classification task : given a message, classify it as conveying a hateful content or not. most studies focus on offensive contents in general while others on specific type of hate (like racism or hate speech against the lgbt community) relying on feature-based engineering or neural approaches (see @xcite ) for a comprehensive survey). data are mainly tweets written in english, although some recent studies attempt to detect hate speech in spanish @xcite , german @xcite , italian @xcite , slovene @xcite and @xcite , the latter focusing on benevolent sexist messages containing expressions such as for a girl or like a man. other studies propose to tackle hate speech from a multilingual perspective (e.g., english and spanish at ibereval2018 @xcite ), but do not consider any cross-language experiments, as participants' models are trained and tested on each language separately. finally, concerning french, hate speech detection only focuses on racist @xcite or abusive messages @xcite . in this paper, we focus on (1) automatic hate speech detection towards two different targets -immigrants and women -and (2) automatic sexism detection from a multilingual perspective, namely in english and french tweets. for english, the data consists of tweets annotated as conveying hate speech against both immigrants and women, as part of hateeval@semeval2019 @xcite (henceforth hs). for french, the data consists also of tweets, but annotated only for sexism (henceforth sexism). the main contributions of this paper are the following : ## related work both sexism and racism can be expressed at different linguistic granularity levels going from lexical to discursive @xcite ) : e.g, women are often designated through their relationship with men or motherhood or by physical characteristics. sexism can also be hostile or benevolent where messages are subjectively positive expressed in the form of a compliment @xcite . basically, sexism may be expressed explicitly or implicitly (see the following tweets from our french data) using different pragmatic devices, including : most of the classifiers employed in hate speech detection still rely on supervised learning, and when creating a new classifier, one may manually design and encode different types of features from the data instances which will then be directly fed to the classical algorithms (naive bayes, logistic regression, random forest, svm) or use deep learning methods that will automatically learn abstract features from data instances. within the automatic misogyny identification shared task at ibereval 2018, the best results were obtained with support vector machine models with different feature configurations. there are also a few notable neural networks techniques deployed in order to detect hate speech in tweets that outperform the existing models : in @xcite the authors used three methods (convolutional neural network (cnn), long short-term memory and fasttext) combined with either random or glove word embeddings. in @xcite the authors implemented two deep neural network models (cnn + gated recurrent unit layer and cnn + modified cnn layers for feature extraction) in order to classify social media text as racist, sexist, or non-hateful. for most of the harassment and hate speech classification tasks, the most used information is depicted by the surface-level features (e.g. bag of words), the majority of authors choosing to include n-grams in the feature sets due to their high prediction rate. due to the noise present in the data (especially on social media), many authors choose to combine the n-grams with a large section of additional features : linguistic features that take into consideration the pos information, dependency relations (long-distance relationship in between words), or word embeddings, which have the advantage of having similar vector representations for different, but semantically similar words. since the task of hate speech detection and sentiment analysis are closely related, several approaches incorporate the latter as a supplementary classification step, assuming that generally negative sentiment relates to a hateful message @xcite . hate speech detection is a particularly difficult task mostly because in different contexts, the meaning of a message might change as it can be highly dependent on knowledge about the world. because of this, in @xcite , the authors present an approach in which they use automatic reasoning over aspects of the world. as it might be difficult to obtain knowledge about the world, the information about an utterance (meta information) may be used in order to refine unsatisfactory classification. for example, in @xcite , by using the users gender information the results were significantly improved, as the authors found that it is more likely for men to post hate speech messages foot_1 . this idea was further developed in @xcite where the authors introduced demographic aware information (age, gender and location) in order to tackle racism and confirm an important increase in performance. another important feature is based on the assumption that a user known for posting hateful messages is more likely to do so again in the future, thus by using the number of profane words in the users previous messages the detection performance improves @xcite . as far as we know, no work have addressed neither sexism detection in french, nor multitarget hate speech detection. ## data our data come from two corpora. the first one, hs-iw, is an already existing corpus containing english tweets annotated for hate speech against immigrants and women, as part of the hateval task at semeval2019. the second corpus, sexism, is new and contains french tweets collected between october 2017 and may 2018 with specific keywords such as #balancetonporc, #sexisme, names of politician women and men, insults, etc. the tweets have been labelled as sexist or non sexist by 3 annotators (2 female and 1 male annotators 3 ). 329 tweets have been labelled by all annotators and the inter-annotator agreement is 0.89 (cohen's kappa). for these tweets, the final labels have been assigned according to a majority vote. automatically labelling tweets as hateful/not hateful or sexist/not sexist is a challenging task because the language of tweets is full of grammatically and/or syntactic errors, it lacks conversational context, might consist of only one or a few words and because they can be indirectly hateful (through the use of sarcasm or irony) it makes the task of text-based feature extraction difficult. for both corpora, several models have been built, all tested using 10-cross-validation to better compare our results in cross-lingual experiments. in the next sections, we detail our models and then give our results. ## multilingual hate speech detection we also experimented with multilingual embeddings : glove bilingual word embeddings foot_3 obtained as described in @xcite as well as french and english fasttext word vectors mapped into the same embedding space following the alignment approach presented in @xcite . for the experiments we used the same bilstm model described in section 4.1, firstly by using the hs-iw english corpus for training and the sexism french corpus for testing, and secondly by using jointly the two corpora (hs-iw and 30% of the original sexism corpus) for training and testing on the remaining sexism corpus. the multilingual experiments results are somewhat comparable to the results obtained when training and testing on the french data (cf. ## conclusion this paper proposed several models that can be used in order to identify messages that convey hate and proved the portability of these systems for the task of detecting sexist messages in french. as far as we know, this is the first work on sexism detection in french on twitter data, this study serving as a first step towards improving the task. in our future work we plan on studying ways to retrieve contextual information, and as the results seemed promising, we also plan on experimenting more in a multilingual embedding space.