Exploring Stylometric and Emotion-Based Features for Multilingual Cross-Domain Hate Speech Detection (2021)

Abstract: In this paper, we describe experiments designed to evaluate the impact of stylometric and emotion-based features on hate speech detection: the task of classifying textual content into hate or non-hate speech classes. Our experiments are conducted for three languages – English, Slovene, and Dutch – both in in-domain and cross-domain setups, and aim to investigate hate speech using features that model two linguistic phenomena: the writing style of hateful social media content operationalized as function word usage on the one hand, and emotion expression in hateful messages on the other hand. The results of experiments with features that model different combinations of these phenomena support our hypothesis that stylometric and emotion-based features are robust indicators of hate speech. Their contribution remains persistent with respect to domain and language variation. We show that the combination of features that model the targeted phenomena outperforms words and character n-gram features under cross-domain conditions, and provides a significant boost to deep learning models, which currently obtain the best results, when combined with them in an ensemble.

## introduction hate speech is commonly defined as communication that disparages a person or a group on the basis of some characteristic such as race, color, ethnicity, gender, sexual orientation, nationality, religion, or other characteristics @xcite . the exact definition of hate speech, however, remains a disputed topic, as it is a subjective and multi-interpretable concept @xcite . the lack of a consensus on its definition poses a challenge to hate speech annotation. annotating hateful content remains prone to personal bias and is culture-dependent, which often results in low inter-annotator agreement and therefore scarcity of high quality training data for developing supervised hate speech detection systems @xcite @xcite . hate speech online presents additional challenges for natural language processing (nlp): offensive vocabulary and keywords evolve fast due to their relatedness with the hate speech triggering events @xcite , moreover, users may adapt their lexical choices as a countermeasure against identification or introduce minor misspellings to bypass filtering systems @xcite . therefore, we intend to investigate more abstract features, less susceptible to specific vocabulary, topic or corpus bias, which we examine in in-domain and crossdomain settings: training and testing on social media datasets belonging to same/different domains, for three languages: english, slovene, and dutch. our hypothesis is that the style and emotional dimension of hateful textual content may provide useful cues for its detection. we investigate this through a binary hate speech classification task using features that model such information, i.e., function words and emotion-based features. the latter are operationalized in terms of the types of emotions expressed and the frequency of emotionconveying words in the data. function word usage is one of the most important and revealing aspects of style in written language, as shown by numerous studies in stylometric analysis for authorship attribution @xcite @xcite . while stylometric characteristics have been implicitly included in some hate speech detection studies (e.g., in bag-of-words or character-level models), their impact on the task has not been studied. we propose the hypothesis that stylometric characteristics of hateful writing are distinctive enough to contribute to the hate speech detection task. in other words, hate speech acts as a specific text type with an associated writing style. on the other hand, we are motivated by psychological and sociological studies, which correlate toxic behaviour online with the emotional profile of the user @xcite . however, unlike previous research that used sentiment information for detecting unacceptable content @xcite @xcite , we test whether we are able to capture some of these phenomena by going beyond the sentiment level (positive / negative / neutral) to a more fine-grained emotion level. we compare the performance of stylometric and emotion-based features with commonly used features for hate speech detection: words, character n-grams, and their combination, and with more recent deep learning models that currently provide the state-of-the-art results for the hate speech detection task @xcite : convolutional neural networks (cnn), long shortterm memory networks (lstm), and bidirectional encoder representations from transformers (bert). the results of these experiments indicate that the combination of stylometric and emotion-based features performs better than words and character ngrams under cross-domain conditions, and allows to further improve the results of deep learning models when combined with them in an ensemble. in summary, the contributions of the research presented here are the following: (i) evaluating the contribution of stylometric and emotion-based features to hate speech detection, (ii) examining how robust and persistent their contribution is with respect to domain and language variation, (iii) comparing their performance with commonly used features for the hate speech detection task: words and character n-grams, and with the state-of-the-art deep learning models. ## results and discussion the main goal of this paper is to identify and analyze specific linguistic phenomena with respect to their role in hate speech detection. in particular, we focused on function word usage (as an expression of style of the hateful content) and emotion (as personality and psychological state indicators with respect to the usage of emotion terms in written messages). first, we perform a separate analysis of the contribution of the phenomena we target, and then compare the performance of combining features that encode these phenomena with the commonly used features for the hate speech detection task: words, character n-grams, and their combination (tf weighting scheme and the liblinear svm classifier with optimized parameters), and with more recent deep learning models that achieve state-ofthe-art results for hate speech detection @xcite : cnn, lstm, and bert (see section 2.2). sections 3.1 and 3.2 show the results of these experiments in in-domain and cross-domain settings, respectively. ## conclusions the goal of this work was to evaluate and quantify the role of stylometric and emotion-based features in the hate speech detection task. we showed that stylometric and emotional dimensions of hateful content provide useful cues for its detection, as evidenced by the positive impact of stylometric and emotion-based features in various in-domain experiments for all the considered languages. their contribution remains persistent with respect to domain variations. under cross-domain conditions, our approach that combines features that capture the targeted phenomena performs better than commonly used features for hate speech detection such as words, character n-grams, and their combination. finally, we showed that in cross-domain settings our approach that incorporates stylometric and emotion-based features significantly contributes to the recent deep learning models when combined through a majority-voting ensemble, which allows to achieve the highest results for the languages addressed in this work. a manual error analysis showed that this improvement is brought by the ability of stylometric and emotion-based features to capture implicit and longer instances of hate speech. the consistent and substantial improvement in hate speech detection brought by including stylometric and emotion-based features in the different setups and for different languages explored indicates that their usage is a robust indicator of the hateful content. the importance of stylometric features points in the direction of the existence of a linguistic register for hate speech messages with specific stylistic properties and a negative emotional load. focusing on these features in text representation leads to more cross-domain robustness.