Understanding and Interpreting the Impact of User Context in Hate Speech Detection (2021)

Abstract: As hate speech spreads on social media and online communities, research continues to work on its automatic detection. Recently, recognition performance has been increasing thanks to advances in deep learning and the integration of user features. This work investigates the effects that such features can have on a detection model. Unlike previous research, we show that simple performance comparison does not expose the full impact of including contextual- and user information. By leveraging explainability techniques, we show (1) that user features play a role in the model’s decision and (2) how they affect the feature space learned by the model. Besides revealing that—and also illustrating why—user features are the reason for performance gains, we show how such techniques can be combined to better understand the model and to detect unintended bias.

## introduction communication and information exchange between people is taking place on online platforms at a continuously increasing rate. while these means allow everyone to express themselves freely at any time, they are massively contributing to the spread of negative phenomenons such as online harassment and abusive behavior. among those, which are all to discourage, online hate speech has attracted the attention of many researchers due to its deleterious effects @xcite @xcite . the extremely large volume of online content and the high speed at which new one is generated exclude immediately the chance of content moderation being done manually. this realization has naturally captured the attention of the machine learning (ml) field, seeking to craft automatic and scalable solutions @xcite @xcite . methods for detecting hate speech and similar abusive behavior have been thus on the rise, consistently improving in terms of performance and generalization @xcite . however, even the current state of the art still faces limitations in accuracy and is yet not ready to be deployed in practice. hate speech recognition remains an extremely difficult task @xcite , in particular when the expression of hate is implicit and hidden behind figures of speech and sarcasm. alongside language features, recent works have considered utilizing user features as an additional source of knowledge to provide detection models with context information @xcite . as a general trend, models incorporating context exhibit improved performance compared to their pure textbased counterparts @xcite . nevertheless, the effect, which these additional features have on the model, has not been interpreted or understood yet. so far, models have mostly been compared only in terms of performance metrics. the goal of this work is to shed light on the impact generated by including user features-or more in general context-into hate speech detection methods. our methodology heavily relies on a combination of modern techniques coming from the field of explainable artificial intelligence (xai). we show that adding user and social context to models is the reason for performance gains. we also explore the model's learned features space to understand how such features are leveraged for detection. at the same time, we discover that models incorporating user features suffer less from bias in the text. unfortunately, those same models contain a new type of bias that originates from adding user information. ## experimental setup in this section, we describe in detail the different datasets and detection models that we include in our interpretability-driven analysis. ## proposed analysis we now describe our methodology in detail. recall that our models differ precisely on the usage of user features. as we will see shortly, their comparison beyond accuracy measurements sheds light on the different model properties and hence on the potential impact of incorporating context features. ## conclusion and future work in our work, we investigated the effects of user features in hate speech detection. in previous studies, this was done by comparing models based on performance metric. we have shown that post-hoc explainability techniques provide a much deeper understanding of the models' behavior. in our case, when applied to two models that differ specifically on the usage of context features, the in-depth comparison reveals the impact that such additional features can have. the two utilized techniques-shapley values estimation and learned feature space explorationconvey different kinds of information. the first one quantifies how each feature plays a role but does not tell us what is happening in the background. the second one illustrates the model's perception of the tweets but does not provide any quantitative information for the prediction. furthermore, we have seen that artificially crafting and modifying a tweet can be useful to examine the models' behavior in particular scenarios. in concrete exam-ples, the two approaches worked as bias detectors present in the text as well as in the user features. we believe that analyzing detection models is vital for understanding how certain features shape the way data is processed. accuracy alone is by no means a sufficient metric to decide which model to prefer. our work shows that even models that perform significantly better can potentially lead to new types of bias. we urge researchers in the field to compare recognition approaches beyond accuracy to avoid potential harm to affected users. data scarcity is still a main issue faced by current researchers, especially when it comes to context features. we believe that larger and more complete datasets will improve our understanding of how certain features interact and will help future research in advancing both in accuracy and bias mitigation. figure 7 shows the feature space learned by our text model on davidson. overall, the distribution looks similar as the one of waseem visualized in figure 4d . we can notice that hate tweets are extremely sparse and mixed with the offensive ones. this is reflected by the poor model performance on the hate class, possibly caused by the conceptual overlap that these two classes have. on the other hand, non-harmful tweets are mostly concentrated in one area of the plot, confirming the satisfactory f1 scored achieved.