C on P rompt: Pre-training a Language Model with Machine-Generated Data for Implicit Hate Speech Detection (2023)

Abstract: Implicit hate speech detection is a challenging task in text classification since no explicit cues (e.g., swear words) exist in the text. While some pre-trained language models have been developed for hate speech detection, they are not specialized in implicit hate speech. Recently, an implicit hate speech dataset with a massive number of samples has been proposed by controlling machine generation. We propose a pre-training approach, ConPrompt, to fully leverage such machine-generated data. Specifically, given a machine-generated statement, we use example statements of its origin prompt as positive samples for contrastive learning. Through pre-training with ConPrompt, we present ToxiGen-ConPrompt, a pre-trained language model for implicit hate speech detection. We conduct extensive experiments on several implicit hate speech datasets and show the superior generalization ability of ToxiGen-ConPrompt compared to other pre-trained models. Additionally, we empirically show that ConPrompt is effective in mitigating identity term bias, demonstrating that it not only makes a model more generalizable but also reduces unintended bias. We analyze the representation quality of ToxiGen-ConPrompt and show its ability to consider target group and toxicity, which are desirable features in terms of implicit hate speeches.

## introduction warning: this paper contains content that can be offensive and upsetting. implicit hate speech is a disparaging statement targeting a certain group without explicit cues such as swear words. for example, we as a society should not take care of those with mental illness is an example of implicit hate speech targeting figure 1 : the generalization issue of existing pretrained language models (hatebert and fbert) in hate speech domain when adapted to implicit hate speech detection task. the performance of the pretrained models severely drops on cross-dataset evaluation, although all datasets target the same task (implicit hate speech detection). each model is fine-tuned on implicit hate corpus (ihc) dataset. mentally disabled group. since there is a lack of explicit cues, it is difficult to detect implicit hate speech using methods such as lexicon-based approaches @xcite . training pre-trained language models on implicit hate speech dataset showed satisfactory performance on in-dataset evaluation (e.g., a model trained on the training set of dataset a is evaluated on the test set of dataset a) @xcite . however, the trained models fail to generalize to other implicit hate speech datasets @xcite . in other words, the performance of the models drops consistently on cross-dataset evaluation (e.g., the model trained on the training set of dataset a is evaluated on the test set of dataset b of the same task). one possible way to improve the generalization ability is further pre-training models on the relevant large corpus. however, existing models pre-trained on abusive or hate speech corpus are not specialized in implicit hate speech. for example, the existing pre-trained language models in hate speech domain such as hatebert @xcite and fbert @xcite suffer from pernegative instance examples the mentally disabled cant really feel pain -we as a society should not take care of those with mental @xmath0: mentally disabled people / toxicity: toxic -society would be better off if the mentally ill were not allowed to reproduce 1 2 society would be better off if the mentally ill were not allowed to reproduce target: mentally disabled people / toxicity: toxic target: mentally disabled people / toxicity: toxic target: mentally disabled people / toxicity: non-toxic target: asian people / toxicity: toxic ## conprompt conprompt aims at making the representation between a generated statement and the example statements in its origin prompt similar. since example statements exhibit some desired properties and the generated statement resembles it, pulling a generated statement and its origin example statements would enable a model to learn such desired properties between them. simcse chooses one positive sample per anchor sample. here, we use a machine-generated state-ment g i as an anchor sample. as a positive sample for the anchor g i , we propose to use an example statement of the origin prompt. we denote p (g i ) as a function that returns a set of example statements (i.e., a prompt) which g i originated from. given a prompt p (g i ) = {s 1 , ..., s m } which consists of m example statements, we randomly select one example statement s pos i as a positive sample for g i . when we simply follow simcse, the resulting objective for an anchor is: for n samples in a mini-batch, our objective is: we also use masked language model (mlm) objective l mlm on machine-generated statements since @xcite showed that incorporating mlm objective was beneficial for the performance improvement on transfer task. following @xcite , after we choose 15% of tokens, we mask 80% of the tokens, replace 10% of the tokens with random tokens, and leave 10% of the tokens unchanged. our final objective for pre-training is: toxigen-conprompt we present toxigen-conprompt, a pre-trained bert using toxi-gen as a dataset and the proposed conprompt as a pre-training approach. before pre-training, we discovered that email information, urls, user or channel mentions are included in the machinegenerated statements in toxigen. since this can cause harm to society regarding a privacy issue, we anonymize them following @xcite . then, we use the anonymized dataset as a pre-training source for toxigen-conprompt. more details on the process are described in the ethics statement section. we use bert-base-uncased foot_3 as an initial model and further pre-train the model on toxigen. we use cosine similarity to calculate the similarity and use τ = 0.03. we use train subset of toxi-gen foot_4 for pre-training, which consists of 250,934 machine-generated statements and 23,322 humancurated prompts with 522 distinct example statements foot_5 . for pre-training toxigen-conprompt, we set the learning rate as 5e-5, max sequence length as 64, batch size as 256, and train for 5 epochs. we use the representation of [cls] from bert as h(•). we utilize 4 nvidia rtx3090 gpus with batch size 64 per device. dataset we follow most of the dataset settings in @xcite . we use the following datasets: implicithatecorpus (ihc) @xcite , dynahate (dh) @xcite , and social bias inference corpus-hate (sbic-h). considering the definition of hate speech, instead of using social bias inference cor-pus (sbic) utilized in @xcite , we use the subset of it (i.e., sbic-h). among the sbic dataset, we set an offensive-labeled sample with target group as a hate class and set a nonoffensive sample as a non-hate class. we do not use the samples that are labeled as offensive without target group. this dataset setup is in line with @xcite . further information regarding the datasets can be found in appendix c. baseline pre-trained language models for a fair comparison between pre-trained models, we use the pre-trained models that are based on bert-base-uncased. as baselines, we experimented with three existing pre-trained models with different pre-training sources: 1) bert; 2) hate-bert; 3) fbert. please refer to appendix d for the details. fine-tuning setup we fine-tune each pre-trained model on a dataset using the cross-entropy loss with binary labels (i.e., hate or non-hate), which is a general fine-tuning approach. 7 for a thorough comparison of the generalization ability between the pre-trained models, we conduct two types of experiments: 1) full fine-tuning and 2) probing. in the full fine-tuning experiment, we fine-tune each pre-trained language model (encoder) with a classifier on it. though it is a common practice to finetune both the encoder and classifier, fine-tuning often leads to catastrophic forgetting @xcite , which makes the comparison between the pre-trained models indirect. thus, we also conduct the probing experiment where we freeze the encoder (i.e., pre-trained models) and solely train the classifier with one linear layer to enable a more direct comparison of the pre-trained representations, similarly to the method in aghajanyan et al. @xcite . the implementation details of fine-tuning can be found in appendix e. ## discussion we propose a pre-training approach that leverages machine-generated data. the use of machinegenerated data to pre-train a model requires careful consideration given the unpredictable nature of machines. we emphasize the importance of validating machine-generated data before using it for pre-training a model. the dataset (i.e., toxigen) we leveraged to pretrain toxigen-conprompt was thoroughly validated in @xcite , including humanvalidation. notably, the human validation conducted in their work showed that about 90.5% of the machine-generated statements were considered to be written by humans, which demonstrates the high quality of the machine-generated data. fur-thermore, 98.2% of the statements in toxigen were considered to be implicit, which is a proportion higher than that of many other hate speech datasets. you can refer to @xcite for the detailed validation results of toxigen. the high quality of toxigen confirms the suitability as a pre-training source. leveraging machine-generated data led to the superior performance of toxigen-conprompt compared to other pre-trained models. we remark that other existing pre-trained models were pre-trained on human-generated data. similarly, there have been recent works that demonstrated the superiority of models trained on machinegenerated data over those trained on humangenerated data @xcite . for instance, @xcite trained a commonsense model using a machine-generated knowledge graph and empirically showed the effectiveness of using machine-generated data over humangenerated data. therefore, we believe that developing methodologies for leveraging machinegenerated data is a promising direction that can benefit our society more than it poses harm when utilized with cautious care such as validating machinegenerated data before utilizing it. we hope our approach can serve as a significant step in this direction, particularly in terms of implicit hate speech detection. ## conclusions we have proposed a pre-training strategy, con-prompt to fully leverage machine-generated data. given a machine-generated sentence, we have cast the idea of utilizing examples from a prompt as positive samples for contrastive learning. we have presented toxigen-conprompt, a pretrained language model pre-trained on a machinegenerated implicit hate speech dataset. toxigen-conprompt outperforms various pre-trained hate speech language models, including hatebert and fbert on cross-dataset evaluation and shows its generalization ability on implicit hate speech detection. in addition, toxigen-conprompt is effective in reducing the identity term bias. we have demonstrated that toxigen-conprompt learns desirable features of target group and toxicity in terms of implicit hate speech.