On the Identification and Forecasting of Hate Speech in Inceldom (2023)

Abstract: Spotting hate speech in social media posts is crucial to increase the civility of the Web and has been thoroughly explored in the NLP community. For the first time, we introduce a multilingual corpus for the analysis and identification of hate speech in the domain of inceldom, built from incel Web forums in English and Italian, including expert annotation at the post level for two kinds of hate speech: misogyny and racism. This resource paves the way for the development of mono- and cross-lingual models for (a) the identification of hateful (misogynous and racist) posts and (b) the forecasting of the amount of hateful responses that a post is likely to trigger. Our experiments aim at improving the performance of Transformer-based models using masked language modeling pre-training and dataset merging. The results show that these strategies boost the models’ performance in all settings (binary classification, multi-label classification and forecasting), especially in the cross-lingual scenarios.

## introduction hate speech can be generally defined as "language that is used to express hatred towards a targeted group or is intended to be derogatory, to humiliate, or to insult the members of the group" @xcite . detecting hate speech can be challenging as there is a lack of consensus on its definition, while the use of offensive neologisms makes the task even more arduous @xcite . this is even more critical in environments frequented by incels, short for involuntary celibates, which pertain to the so-called manosphere @xcite and mainly comprise men unsuccessful in finding a sexual partner or significant other. some of these individuals tend to engage in the spread of various forms of hate speech -in particular racism and misogyny-and recurrently adopt novel lexicon in doing so @xcite . such dynamic jargon causes models trained on hate speech to fail to recognize incel-specific instances of hate speech. our contributions are the following: (i) corpora. we introduce two unsupervised corpora on the inceldom domain, one in english and one in italian. a subset of each corpus includes manual annotations for different kinds of hate speech (cf. section 3). 1 the raw data can be used for domain adaptation and language modeling, among other applications. the annotation allows addressing three tasks. binary: determine whether a post p conveys hate speech or not. multi-label: determine whether p is misogynous and/or racist. forecasting: given an original post p ′ (the first post in a thread), forecast the amount of hateful posts that it is likely to trigger in future responses. our experiments show that mlm pre-training is effective, particularly in cross-lingual scenarios, resulting in a 17-point absolute improvement in terms of f 1 -measure in the binary task, and a 34and 18-point increase in the misogyny and racism detection tasks, respectively. combining italian and english datasets leads to a large performance increase of 22 points in terms of f 1 -measure, for the best mlm pre-trained model. in the forecasting setting, our regression model effectively predicts the number of hateful responses a post may generate in the following replies, surpassing the mean squared error (mse) baseline by 37%. ## related work corpora built from incel platforms are rare and not necessarily applicable to the use-case of this study, either due to the source of the data only being partially compatible with the linguistic domain presently tackled @xcite or because of the criteria according to which it was annotated @xcite . most studies have focused on the linguistic properties of incel corpora, predominantly adopting qualitative approaches. for example, tranchese and sugiura (2021) compared incel discourse from reddit forums to the language used in pornography and highlighted its misogynistic implications. @xcite conducted a cross-platform study on incel profiling, by collecting 6.5k youtube videos shared by users in incel forums within reddit, while also examining the youtube recommendation algorithm. their findings show that incel activity on youtube is increasing, stirring towards the dissemination of incel views. @xcite adopted a mixed approach, mainly focusing on text profiling, with their discourse analysis suggesting that incel language is not as coherent as previously assumed, while also employing a multichannel cnn, using 50k incels.me messages, 50k neutral texts composed of 40k paragraphs from random english wikipedia articles, and 10k random english tweets. past studies have relied on the pushshift reddit api to build a corpus within the linguistic domain of inceldom @xcite . @xcite build a dataset from english tweets which can be used to train models to identify and categorize offensive posts, with information on whether the target is a group or individual. recently, more hate speech studies turn towards a new approach: forecasting. @xcite extract politeness strategies and rhetorical prompts to predict whether a conversation will turn uncivil. @xcite predict the intensity of hate that a tweet might carry through its reply chain by exploiting tweet threads and their semantic and propagating structures. @xcite , compiled a dataset of 4.5k tweets and their reply threads, confirming that longitudinal patterns of hate intensity among reply threads are diverse, with no significant correlation with the source tweet. their approach differs from ours in that they calculate hate intensity for chunks of a thread, not for the whole thread at once. @xcite proposed a model for toxicity triggering prediction by integrating text-based features as well as features that are related to shifts in sentiment, topic flow, and discussion context, proving that toxicity triggers contain detectable features. @xcite proposed a model that uses a post's semantic, propagation structure, and temporal features to predict hateful propagation in social media. ## incel corpora we performed a modern diachronic study (partington, 2010) on incel forums, shedding light on the way the language of inceldom evolves. we consider two forums: incels.is, foot_2 in english, and il forum dei brutti, foot_3 in italian. studying such niche communities, as opposed to those hosted for example on reddit, allows us to study a language which is representative of the incel speech community. this is because moderation is more lax, foot_4 which allows users to express themselves more genuinely. the study, discussed at length in appendix a, shows that excessively outdated resources might not be entirely representative of the discourse currently produced by the speech communities being scrutinized. more worthy of notice is that incel language differs from general internet language, especially when hate speech is expressed. such findings show that building new corpora from scratch is a worthwhile effort, as having an accurate representation of current language is a priority. we retrieved dumps of posts from the two forums. the metadata for each post includes: author id, the position of the post in the thread, url, timestamp and both post and thread unique ids. we refer to the unsupervised dataset obtained from the dump of the incels.is forum as ifu-22-en please identify whether each post is categorized as misogynous, racist, or falls into another category: a post is deemed misogynous if it: a post is considered racist if it: figure 1 : guidelines for the corpus annotation, derived from @xcite for misogyny and @xcite for racism. we annotated a subset of the posts from both collections with two independent binary labels: one for misogyny and one for racism. we refer to the resulting datasets as ifs-en and ifs-it, which stand for incel forum supervised in english (with 5,203 instances) and italian (with 500 instances). ifs-en was initially sampled with two constraints: 50% of the posts had to include at least one term characteristic of incel jargon foot_5 and instances had to be longer than five words. the former constraint sought to balance the occurrence of instances with and without incel jargon to prevent models from overly relying on it, while the second aimed at excluding instances which would not be useful during training. for ifs-it, only a 5-word minimum length constraint was applied. figure 1 shows the annotation guidelines. with relation to english, a pilot annotation was first carried out by three annotators on a subset of 50 instances. all annotators have a c2 cefr level of english and are experts in the subject, with a strong foundation in linguistics and gender studies, as well as knowledge of nlp and data annotation. the obtained cohen's kappa inter-annotator agreement (iaa) @xcite was of 0.77, considered substantial (with 0.81 being the threshold for almost perfect). the rest of the instances were annotated by a single annotator. as for italian, two annotators, native speakers of italian and with the same background as above, obtained an iaa of 0.69 over 50 instances. as the iaa was deemed acceptable, the 450 other instances were all labeled by a single annotator. we split ifs-en into training, development and testing partitions with a ratio of 70/15/15, while we use ifs-it only for cross-lingual testing. ## mlm pre-training dataset source lan davidson @xcite hatebase.org en hatexplain @xcite twitter+gab en stormfront @xcite stormfront.org en hateval @xcite twitter en hsd fb @xcite facebook it hsd tw @xcite twitter it to improve the models' understanding of the incel language by training them on the mlm task, producing what we refer to as in-domain incel bert and incel mbert versions. in the monolingual scenario, three samples from the ifu-22-en unsupervised dataset are used, considering randomly-selected splits of 10k, 100k, and 1m posts. we adopt a similar approach in the crosslingual scenario, where we consider (i) the same english subsamples alone; (ii) subsamples of 10k, 100k, and 627k instances in italian from ifu-22-it (the full corpus contains 627k instances); and (iii) 50-50% splits from both ifu-22-en and ifu-22-it of 10k, 100k, and 1m instances. none of the instances used for mlm pre-training include data from ifs-en and ifs-it. in all cases, mlm pre-training is carried out by tokenizing posts with autotokenizer foot_6 and masking tokens with a probability of 15%. we use a batch size of 32 and train the models for one epoch on a single tesla p100 gpu with 16 gb of vram. in order to assess the impact of the mlm pretraining, we perform preliminary experiments on the binary classification task: hate speech or not. we fine-tune each model version using ifs-en tr for training and ifs-en de for development. 8 we then test on ifs-en te in the monolingual scenario and on ifs-it in the cross-lingual scenario. our baseline for monolingual scenarios is bert, while we use mbert in cross-lingual ones. in the cross-lingual scenario, mlm also has a positive impact, but the improvement is not linear with the amount of data. when performing mlm with monolingual data, be it in english or italian, the testing performance on both languages is better than vanilla mbert, when using 10k instances, but drops with additional monolingual training material. using a bilingual combination of mlm material produces the best model when using 1m instances. this configuration boosts the performance: (i) by 39 points on italian, with respect to adding 1m of all-english instances (0.503 vs 0.112) and (ii) by 1 point on the english one (0.845 vs 0.835). with respect to the mbert baseline, training on 1m bilingual instances provides a performance boost of 17 points (0.503 vs. 0.333). going forward, we use the best post-mlm models: incel bert trained on 1m english instances in monolingual experiments and incel mbert trained on 1m bilingual instances in cross-lingual ones. ## downstream tasks this section discusses our three experimental settings: (i) binary hate speech classification, (ii) multi-label misogyny and racism classification, and (iii) hate speech forecasting. in all settings we tokenize input sentences with autotokenizer, padding to a maximum of 256 tokens, including [cls] tokens, and returning attention masks. all models are trained with a batch size of 16, using the adamw optimizer with @xmath0. both classification tasks are evaluated on the basis of f 1 -measure. the forecasting (regression) task is evaluated using mean squared error (mse) and mean absolute error (mae). ## conclusions in this paper, we have explored the creation of models for the automatic identification of hate speech in incel forums: binary hate speech identification, multi-label misogyny and racism identification, and forecasting of the level of hate that the first post of a thread is likely to trigger. our experimentation on the three problems, in monolingual and cross-lingual scenarios, shows that (i) pre-training on the masked language modeling task to make bert-based models more aware of incel language is a key factor to aspire to produce good predictions; (ii) the inclusion of super-vised material extracted from sources external to incel forums can help boost models further, also across languages; and (iii) it is feasible to forecast the amount of hate that an original post will likely trigger prior to any replies, although further improvements are still required. in future work, we plan to delve further into forecasting by implementing temporal and propagation features (e.g., meng et al. (2023); dahiya et al. (2021); lin et al. (2021); almerekhi et al. (2020); jaki et al. (2019) ). based on @xcite , we also plan to expand language coverage, with german- @xcite and spanishlanguage (basile et al., 2019) hate speech datasets being two of the most prominent candidates due to their similarity to english and italian, respectively.