Label-aware Hard Negative Sampling Strategies with Momentum Contrastive Learning for Implicit Hate Speech Detection (2024)

Abstract: Detecting implicit hate speech that is not directly hateful remains a challenge. Recent research has attempted to detect implicit hate speech by applying contrastive learning to pre-trained language models such as BERT and RoBERTa, but the proposed models still do not have a significant advantage over cross-entropy loss-based learning. We found that contrastive learning based on randomly sampled batch data does not encourage the model to learn hard negative samples. In this work, we propose Label-aware Hard Negative sampling strategies (LAHN) that encourage the model to learn detailed features from hard negative samples, instead of naive negative samples in random batch, using momentum-integrated contrastive learning. LAHN outperforms the existing models for implicit hate speech detection both in- and cross-datasets. The code is available at https://github.com/Hanyang-HCC-Lab/LAHN

## introduction online hate speech has become one of the major social problems as it leads to discrimination against certain groups and social conflict @xcite . hate speech can be either explicit, which directly uses hateful language, or implicit, which metaphorically implies hateful language @xcite . while explicit hate speech can be addressed relatively easily through the use of automated filters @xcite @xcite , dealing with implicit hate speech is highly challenging. for example, identity term bias @xcite , where identity terms (e.g., black, jew) frequently appear in hateful contexts, can over-bias the model and cause false positives in implicit hate speech detection. given the characteristics of implicit hate speech, it will be important to encourage blacks have contributed nothing to humanity. jews recognize that they can sometimes be victims of racism themselves. ## method as shown in figure 2-(a) , the existing method performs contrastive learning on all other samples with equal weight, including those of the same class, for a given anchor in a mini-batch. this characteristic makes it difficult for the model to focus on distinguishing between the anchor and hard negatives, which are semantically similar and embedded close to the anchor. on the other hand, our lahn first uses label information to identify negatives with labels different from the anchor's, which should be embedded in opposite directions. next, the momentum encoder calculates the probability that the negative is in the same class as the anchor, and this probability is used to multiply the similarity value between the negative and the anchor. finally, lahn extracts the top-k hard negatives based on the calculated values and performs compact contrastive learning for the given anchor (figure 2-(b) ). through this, lahn better captures the detailed differences between implicit hate speech and non-hate speech by focusing on hard negatives. figure 3 illustrates the overview of our lahn. random mini-batch samples are used as the input to the main and momentum encoders, and the output embedding of the momentum encoder is inserted into the momentum queue. hard negative sampling for each anchor is performed after the momentum queue is filled with at least a quarter of the size. as in moco @xcite , we use an exponential moving average (ema) to slowly update the momentum encoder based on the weights of the main encoder. algorithm 1 presents the pseudocode implementation. ## conclusion and future work in this paper, we propose lahn that incorporates the momentum queue to extract hard negatives, which are more likely to be confused by the model as compared to anchors. we demonstrated the effectiveness of lahn in both in-dataset and crossdataset performance evaluations, compared to existing methods. in the future, we will focus on finding new sampling metrics that are more advanced than similarity-based hard negative sampling for implicit hate speech detection and finding ways to exploit better semantic features from hard negatives.