Improving Hate Speech Type and Target Detection with Hateful Metaphor Features (2021)

Abstract: We study the usefulness of hateful metaphorsas features for the identification of the type and target of hate speech in Dutch Facebook comments. For this purpose, all hateful metaphors in the Dutch LiLaH corpus were annotated and interpreted in line with Conceptual Metaphor Theory and Critical Metaphor Analysis. We provide SVM and BERT/RoBERTa results, and investigate the effect of different metaphor information encoding methods on hate speech type and target detection accuracy. The results of the conducted experiments show that hateful metaphor features improve model performance for the both tasks. To our knowledge, it is the first time that the effectiveness of hateful metaphors as an information source for hatespeech classification is investigated.

## introduction in this paper, the usefulness of hateful metaphors used as features for detecting the type and target of dutch online hate speech comments is investigated. although both hate speech and metaphor detection have been researched widely (e.g., @xcite @xcite , and figurative language used in hateful content has been identified as one of the main challenges in (implicit) hate speech detection @xcite , the question whether detecting (hateful) metaphors and using them as features improves hate speech detection models has remained unstudied in previous research. therefore, it is the goal of the present paper to address this question. in order to achieve this goal, we used the dutch lilah 1 corpus which consists facebook comments on online newspaper articles related to either migrants or the lgbt community. the comments were annotated for the type of hate speech and the target of hate speech, and for "hateful metaphors", 1 https://lilah.eu/ i.e., metaphors that express hate towards a specific target (e.g., "het parlement is een circus!"; the parliament is a circus). we investigate whether features based on these manual annotations can improve natural language processing (nlp) models that predict the type (e.g., violence, offense) and target (e.g., migrants, lgbt, journalist) of hateful content. our experimental setup is therefore different from the commonly-used one in the sense that we are focusing only on the fine-grained hate speech categories and not on classification of hateful and non-hateful content. we hypothesize that hateful metaphors contain valuable information for type and target classification, especially in cases of implicit hate speech, and can therefore improve classification accuracy when used as features. prior to the classification experiments, a linguistic analysis of the annotated metaphors is conducted in the framework of conceptual metaphor theory and critical metaphor analysis. we would like to warn that for clarity of exposition, randomly chosen examples of hate speech from our corpus will be provided in this paper, and that some readers could find those offensive. ## related research hate speech detection hate speech -frequently defined as a form of communication that disparages a person or a group on the basis of some characteristic such as race, color, ethnicity, gender, sexual orientation, nationality, religion, or other @xcite -has been extensively researched in the field of nlp. pretrained language models such as bidirectional encoder representations from transformers (bert) and robustly optimized bert pretraining approach (roberta) @xcite provide the best results for hate speech detection, including type and target classification @xcite @xcite , while shallow machine learning models (e.g., support vector ma-chines (svm)) can achieve a near state-of-the-art performance @xcite . examples of successful machine learning models include the winning teams of both subtasks a (binary hate speech detection) and b (binary target classification) of task 5 of semeval 2019: multilingual detection of hate speech against women and immigrants on twitter @xcite . these teams all used svm-based approaches for both languages provided (english and spanish) with the exception of the winner of task b for spanish, who used various other classifiers and combined them by means of majority voting. for english, the winning teams obtained an f1-score of 65% for task a and an emr score of 57% for task b. examples of effective neural approaches can be found in offenseval 2020 @xcite . this shared task consisted of three subtasks: (a) offensive language identification, (b) categorization of offensive types and (c) target identification for multiple languages. for english, each of the top 10 teams for all three tasks used pretrained language models such as bert and roberta. the highest macro f1-scores obtained for task a, b, and c were 92%, 75% and 71%, respectively. figurative and implicit language in hate speech in their hate speech detection survey, @xcite highlight current challenges in hate speech detection. one of the main challenges mentioned is the use of figurative and implicit language such as sarcasm and metaphors, which can lead to classification errors, as evidenced by their experiments. an svm classifier with tf-idf weighted character n-gram features was used to perform hate speech detection on the stormfront, trac, hateval and hatebasetwitter datasets @xcite @xcite . an error analysis of the misclassified instances showed that sarcastic and metaphorical posts were the main causes of misclassifications, next to too little context (posts containing fewer than 6 tokens) and aggressive statements occurring in posts that were not annotated as "hateful". similar findings were observed by @xcite . an ensemble of machine learning and deep learning models was used for multi-class classification of toxic online comments and an error analysis of the incorrect predictions showed that metaphors can lead to classification errors because the models require significant world knowledge to process them. to address the problem of implicit language in hate speech, more recent studies have used datasets that distinguish between implicit and explicit hate speech, such as abuseeval v1.0 @xcite . this dataset was created by annotating the olid/offenseval dataset @xcite for implicitness/explicitness. the authors of abuseeval v1.0 provide results with bert for binary classification (abusive, non-abusive) and multi-class classification (non-abusive, implicit abuse, explicit abuse) for the same train/test split and show that the binary classification task (71.6% macro f1-score) becomes substantially more complex when distinguishing between implicit and explicit abusive language (61.4% macro f1-score). additionally, they show that the results for implicit hate speech detection (24% precision, 23% recall) are substantially lower than for explicit hate speech detection (64% precision, 51% recall). ## conclusion in this paper, we investigated the usefulness of hateful metaphors as predictive features for two less studied hate speech detection subtasks (namely type and target prediction) and analyzed the annotated hateful metaphors in our corpus in line with conceptual metaphor theory and critical metaphor analysis. performances of svm, bertje and robbert were provided for both type and target prediction tasks and these models were then enriched with the hateful metaphor features in various ways to show their usefulness. the results show that the target svm baseline improved by 2.4%. conversely, bertje and robbert could not be improved with additional features for this task. regarding the type prediction task, an improvement up to 4.6% was observed for the svm baseline, whereas the already high-performing bertje and robbert baselines improved by 1.9% f1-score each. from the qualitative analysis that was conducted, it was observed that these improvements contained a large number of implicit forms of hate speech, which is considered to be one of the main challenges of hate speech detection at the moment. this paper is a starting point for further research into the new area of (hateful) metaphors as predictive features for the hate speech classification tasks. further research may include investigating whether the same results achieved with an upperbound baseline in this paper (provided by our manually annotated features) can also be obtained when using labels predicted by models that have been trained to detect hateful metaphors. other future research directions could include investigating more feature encoding methods and conducting ablation studies when combining multiple ways to encode hateful metaphors. in addition, it was observed that the svm model cen be improved more strongly than bertje and robbert, which suggest that the latter models already contain metaphorical information due to pretraining. whether this is indeed the case is yet another subject worth investigating in future studies.