Unraveling the Dynamics of Semi-Supervised Hate Speech Detection: The Impact of Unlabeled Data Characteristics and Pseudo-Labeling Strategies (2024)

Abstract: Despite advances in machine learning based hate speech detection, the need for larges amounts of labeled training data for state-of-the-art approaches remains a challenge for their application. Semi-supervised learning addresses this problem by leveraging unlabeled data and thus reducing the amount of annotated data required. Underlying this approach is the assumption that labeled and unlabeled data follow similar distributions. This assumption however may not always hold, with consequences for real world applications. We address this problem by investigating the dynamics of pseudo-labeling, a commonly employed form of semi-supervised learning, in the context of hate speech detection. Concretely we analysed the influence of data characteristics and of two strategies for selecting pseudo-labeled samples: threshold- and ratio-based. The results show that the influence of data characteristics on the pseudo-labeling performances depends on other factors, such as pseudo-label selection strategies or model biases. Furthermore, the effectiveness of pseudo-labeling in classification performance is determined by the interaction between the number, hate ratio and accuracy of the selected pseudo-labels. Analysis of the results suggests an advantage of the threshold-based approach when labeled and unlabeled data arise from the same domain, whilst the ratio-based approach may be recommended in the opposite situation.

## introduction topic shifts in online hate speech arising from changing social media trends or news poses a challenge for hate speech detection systems (florio 1 florian.ludwig@zitis.bund.de ana.alvespinto@zitis.bund.de klara.dolos@zitis.bund.de torsten.zesch@fernuni-hagen.de et al., 2020). in order to keep the pace and follow such dynamic changes developers of such systems need to adapt their models to the continuously changing contexts and linguistic patterns @xcite . since these models rely on large amounts of annotated training data @xcite the dynamic nature of abusive language in online discourses complicates the application of state-of-the-art deep learning models. gathering high quality training data is time-consuming and often requires human expertise to be involved in the annotation process @xcite . semisupervised learning address these challenges by training models with a small amount of data annotated (labeled) for the specific use case together with a large amount of unlabeled data. these approaches improve model performance over purely supervised learning approaches by using information that is present in the unlabeled data (van engelen and hoos, 2020), and are therefore being actively explored in dynamic domains such as automatic hate speech detection, where data efficiency is crucial. since unlabeled data seems to be easy to obtain, recent research in the field of semi-supervised hate speech detection focuses on the learning algorithms themselves rather than the training data. the underlying assumption is that the labeled and unlabeled data share the same characteristics and therefore follow the same data distribution. this assumption however does not hold in real world scenarios where the high pace of change of online hate speech is accompanied by changes in the characteristics of associated data. therefore, we investigate the influence of data characteristics on semi-supervised model performances. as we investigate pseudo-labeling based semi-supervised learning @xcite @xcite we are especially interested in the different benefits regarding model performance of two common pseudo-label selection strategies. in summary, the contributions of this work are: ## related work various approaches for automatic hate speech detection have been proposed in recent years (jahan and oussalah, 2023), reaching from lexical @xcite to traditional machine learning @xcite to deep learning based approaches @xcite @xcite . due to the high demand for labeled data of current approaches @xcite , semi-supervised training methods have emerged as an active line of research in the context of hate speech detection @xcite @xcite . for instance zia et al. investigated the use of self-training to improve hate speech detection performance in multilingual settings. similarly, @xcite used self-training to enhance hate speech detection models, having reported an improvement of 7% relative to supervised baselines. whilst imbalanced class ratios and the complexities in the detection of implicit hate speech were identified as challenges in the training process, no thorough examination of their impact on the self-training performances was conducted. in a previous study by the same authors (alsafari and sadaoui, 2021a), an ensemble of different classification models was trained on a seed hate speech dataset to predict pseudo-labels for a large unlabeled dataset. the authors evaluated various ways to combine predictions from multiple models within the ensemble in order to obtain reliable pseudo-labels. while these works applied pseudolabeling and other semi-supervised learning techniques to improve hate speech classifiers, they did not analyze how these approaches are affected by typical challenges in the hate speech detection domain. in our work, we thoroughly investigate how data properties, specific to the hate speech domain, and their interaction with other components, such as pseudo-label selection strategies, affect the performance of pseudo-labeling-based approaches. the influence of different data and pseudo-label characteristics has also been studied in other areas. wei et al. reported on the negative effect of imbalanced pseudo-labels on model performance. furthermore, they reported improvements over other pseudo-labeling based approaches by applying an iterative re-balancing framework for pseudo-labels, indicating the importance of a balanced class ratio in the pseudo-labels. the influence of the accuracy of pseudo-labels was investigated in turn by @xcite in the task of sentiment analysis. the authors found that the accuracy of the pseudo-labels strongly affects model performance. in relation to these works, our work focuses on the specific domain of hate speech detection with its unique challenges. more over, in contrast to previous works we analyse how the interaction of multiple components, such as data and pseudo-label characteristics, model biases and pseudo-label selection strategy affects the performance of the investigated approaches. based on our findings, we further provide recommendations for real-world applications of semi-supervised learning in the domain of hate speech detection. ## results and discussion this section starts by presenting and discussing the results of the supervised reference models, as well as the prediction confidences and pseudo-label accuracies of the baseline metric model for the unlabeled portion of the base dataset. afterwards we present the performances of the semi-supervised learning approaches with respect to different characteristics of the unlabeled data, and discuss these results in face of the characteristics of the corresponding selected pseudo-labels, the distributions of the predicted hate speech probability and of the annotated toxicity values of the selected hate samples. the section finalises with a summary of the main observations/results. the model tends to make more predictions in favor of the normal class. moreover, these predictions have a higher degree of accuracy than the hate speech class. ## recommendations for real-world applications our findings suggest, that the threshold-based approach should be applied if the characteristics of unlabeled data favor the effects of the teacher model bias, leading a larger number of confident pseudolabels. this is typically the case when labeled and unlabeled data arise from the same domain, e.g., when they share the same target groups of hate speech. the ratio-based approach provided bet-ter results in opposite scenarios. especially when domain adaptation is needed due to a lack of labeled data in the target domain, the ratio-based approach should be considered. prediction confidences can be analyzed, for example, by computing a histogram, which can be a valuable tool for deciding which selection strategy to use. when a large number of confident pseudo-labels are obtained, the threshold-based selection strategy should be preferred, otherwise the ratio-based strategy. additionally, given the good model performances achieved for (nearly) balanced data, it is recommended to include a reasonable amount of hate speech in the unlabeled data. public real-world or synthetic hate speech datasets can be used to this end. although these datasets may be annotated with different annotation schemes, the "hate" labels contained in these datasets may be similar to the labeled data in the specific use case, and therefore already more "informative" to the model than randomly crawled data, which typically contain a very small amount of hate speech @xcite . ## conclusion in this work, we investigated two pseudo-labeling based approaches for semi-supervised training of hate speech detection models and therefore contributed to the understanding of the complex interaction between data properties, model biases, and pseudo-label selection strategies. we showed that selection of pseudo-labels is determinant to the final performance of the approaches. in view of real-world applications, the results suggest an advantage of threshold-based pseudo-label selection strategies over ratio-based selection strategies when labeled and unlabeled hate speech data arise from the same domain, since a larger number of confident pseudo-labels can be expected in this scenario. in turn, ratio-based selection strategies are preferable when labeled and unlabeled data arise from different domains. these results show the need for further exploration and investigation of alternative pseudo-label selection strategies as well as other families of semi-supervised learning algorithms.