Delving into Qualitative Implications of Synthetic Data for Hate Speech Detection (2024)

Abstract: The use of synthetic data for training models for a variety of NLP tasks is now widespread. However, previous work reports mixed results with regards to its effectiveness on highly subjective tasks such as hate speech detection. In this paper, we present an in-depth qualitative analysis of the potential and specific pitfalls of synthetic data for hate speech detection in English, with 3,500 manually annotated examples. We show that, across different models, synthetic data created through paraphrasing gold texts can improve out-of-distribution robustness from a computational standpoint. However, this comes at a cost: synthetic data fails to reliably reflect the characteristics of real-world data on a number of linguistic dimensions, it results in drastically different class distributions, and it heavily reduces the representation of both specific identity groups and intersectional hate.

## introduction recent advancements in generative large language models (llms), with models having the potential to quickly produce large amounts of textual data, have resulted in a number of works on synthetic data generation in the nlp community @xcite @xcite . indeed, synthetic data may mitigate issues related to data scarcity, minimizing the need to collect real data and, in some cases, even to manually annotate it. beside the advantages in terms of effort, synthetic data could comply better with privacy regulations, replacing real data with realistic data that can be freely shared. in the light of this potential, recent works have tried to identify the settings and tasks in which data augmentation could be successfully employed @xcite . for example, @xcite showed that classifiers trained with real data generally outperform those trained using only synthetic data, especially when the task is subjective, whereas @xcite found that synthetic data can increase cross-dataset performance. beside extrinsic evaluations, however, little attention has been paid to the advantages and risks of employing synthetic data in sensitive tasks like hate speech detection, with the few existing efforts reporting mixed results in terms of performance @xcite . in this paper, we address a scenario in which one may need to perform hate speech detection on unseen data, and they would like to exploit the potential of both generative llms and existing hate speech datasets: what advantages can synthetic data offer in this respect? what are the risks associated with using llms for this type of application? could generated data amplify bias or harm? as a first exploration in this direction, we focus on hate speech detection in english, i.e., a high-resource language for which several llms and hate speech datasets are already available @xcite . in this work, we augment an existing english hate speech dataset, with the goal of evaluating whether, on unseen data from a different distribution, training on paraphrased data is better than using original hate speech data. we couple this evaluation with a thorough manual qualitative analysis of the generated data, assessing fluency, grammaticality and 'artificiality'. given that biases may affect specific targets of hate differently @xcite , we also devote particular attention to a per-target analysis, showing the effects of the usage of llms to produce synthetic data on target identity distribution, and subsequently its impact on fairness. since generated data is increasingly being used even for sensitive applications @xcite , it is important that also the nlp community critically addresses the impact of synthetic data including ethical risks, along the lines of similar discussions in other research communities @xcite . our work is an initial contribution in this direction. ## related work in the context of hate speech detection, data augmentation and synthetic data have been proposed as means to mitigate many issues in datasets @xcite @xcite . those include dataset decay or obsolescence and their impact on reproducibility @xcite , the over-reliance of models on specific lexical items such as identity mentions @xcite @xcite , and the psychological impact on annotators @xcite . the representation of minority identity groups is another issue in hate speech detection literature, with targets that have been covered extensively such as race and gender-related hate @xcite @xcite , while other phenomena and targets have received less attention, such as religious hate @xcite or hate against the lgbtqia+ community @xcite . synthetic data-based methods have been found effective for a number of nlp tasks @xcite , with models trained on synthetic data achieving similar or, in some cases, better performance than models trained on gold data @xcite . @xcite categorize synthetic data into two separate categories, based on how derivative the data is with respect to a real-world dataset. generated data refers both to an ideally 'novel' output foot_0 that is produced by a generative model, while augmented data refers to any real-world data instance that was modified in some way, e.g., via perturbations such as synonym replacement or random word deletion @xcite . given that previous work has shown that direct generation may not work well in all scenarios for hate speech detection and in general on subjective tasks @xcite , and since this is not a low-resource scenario, we frame our synthetic data creation process as a sort of middle ground between these, along the line of @xcite : para-phrasing original real-world data rather than generating text sequences ex-novo. our rationale for this choice is potentially preserving similar content to the original real-world data, while conceivably increasing the lexical variety of the data, which are typical desiderata in synthetic data approaches for this kind of task @xcite . ## gold data most work frames synthetic data creation as a data augmentation task in a low-resource setup, i.e., by starting from a small sample of gold data. for hate speech detection, however, there exist datasets in many languages, also in the light of the shared tasks that have been organized over the years @xcite . this makes the low-resource assumption unrealistic for languages such as english. since our study focuses on english, we devise an experimental setup that allows us to leverage existing hate speech resources by casting data augmentation as paraphrasing rather than as zero-shot generation. this allows us also to potentially mitigate effects related to model alignment, with llms often being programmatically blocked in generating hateful messages from scratch. for investigating the effects that synthetic data can have on hate speech detection, we choose measuring hate speech (mhs; @xcite as our corpus to paraphrase, since it covers different target identity categories. mhs includes posts from three online platforms (i.e., twitter, youtube, and reddit) and contains annotations not only regarding the presence of hate speech, but also about any target identities that are mentioned in the text, regardless of whether it contains hate speech or not. since we focus on the binary classification of hate speech, we use the hate speech label rather than the continuous hate speech scores. the hate speech label in the mhs dataset can take on three values (0: non hateful, 1: unclear, 2: hateful). given that the dataset is released in disaggregated form, we manually average all the annotations for a given post, mapping the post to the hateful label if the average score for hate speech of that text is above 1, and to non hateful if it is lower. we also aggregate the information of target identities, assigning the presence of a specific target identity if at least half of the original annotators for a given post marked that identity as present. after this process, 35,242 annotated posts remained, of which 9,046 annotated as containing hate speech (∼26%). we select 10% of the dataset as test set (3,524), 5% of the remaining examples as development data (1,586 examples) and the remaining texts as the training data (30,132 posts). we use two more datasets for testing the out-ofdistribution generalization of classifiers. first, we test our models on the multi-domain agreement dataset (mda; @xcite , which contains twitter posts annotated for offensive content spanning across three main topics: the black lives matter movement, covid-19, and the 2020 us elections. for mda, we use the default test data split (3,057 examples). second, we test our models on the hatecheck dataset @xcite , which contains 3,727 adversarial examples tailored at finding weaknesses of hate speech detection models. ## synthetic data generation to be able to analyze the extrinsic impact on performance and the intrinsic characteristics of synthetic data for hate speech detection, we first artificially create training data. previous works focusing on synthetic data for hate speech and abusive content-related tasks have experimented with task-specific decoding @xcite , pipelines including humans in the loop for validating generated texts @xcite , or fine-tuning generative large language models on real world data @xcite . however, the growing performance of generative llms makes it possible to use them in numerous tasks without any fine-tuning @xcite . because of this, we opt for a synthetic data creation setup in which we prompt llms to paraphrase the original texts. we expect the output text to i) be similar to the original social media post, ii) reflect the same hate speech label, and iii) preserve roughly the same meaning and topic. we analyze these aspects in our human evaluation in section 6. our synthetic data creation pipeline consists of two steps. first, we prompt the models (section 4.1) to obtain synthetic versions of the real data in the mhs corpus, creating one artificial counterpart for each example in the dataset. after extracting the paraphrased text from the model output, we perform two additional filtering steps on the synthetic sequences (section 4.2). ## extrinsic evaluation we analyze the extrinsic impact of synthetic data by fine-tuning classifiers on both artificial and original data. this analysis contextualizes the main contribution of this work, namely the intrinsic evaluation of synthetic data (section 6), and it is aimed at addressing the following question: what is the quantitative usefulness of synthetic data for the downstream task of hate speech detection? for our experiments, we use three pre-trained classifiers: roberta large, roberta base @xcite , and deberta v3 base @xcite . we compare the performance of a model trained on original gold data with the performance of the same model trained on synthetic data only and with that of a model trained on both synthetic and gold data, in order to assess how effectively the synthetic data can mimic the gold training data. for brevity, we report roberta large results in this section, since our findings are reflected across all classifiers. 3 while the classifiers are always trained on data (original or paraphrased) from mhs, they are tested on all datasets detailed in section 3, in order to assess both their in-distribution and their out-of-distribution performance. the metrics we use for evaluating classifiers are macro-f 1 and minority class (hate) f 1 . details of the model implementations are reported in appendix a. the amount of training data for synthetic setups reported in our results show that models trained on synthetic data alone ( test data mhs mda hatecheck m-f1 hate f1 m-f1 m-f1 original gold data (mhs) .811 ±.004 .718 ±.008 .507 ±.027 .386 ±.026 gen. model filter llama-2 chat 7b + gold data no .809 ±.005 .719 ±.007 .583 ±.014 .558 ±.029 yes .813 ±.003 .723 ±.005 .531 ±.006 .451 ±.010 mistral 7b instruct + gold data no .812 ±.002 .723 ±.003 .587 ±.009 .609 ±.014 yes .816 ±.002 .728 ±.003 .529 ±.007 .464 ±.011 mixtral 8x7b instruct + gold data no .811 ±.003 .723 ±.005 .593 ±.012 .619 ±.010 yes .813 ±.003 .723 ±.004 .527 ±.008 .455 ±.025 in models trained on a mixture of synthetic and gold data ( better out-of-distribution performance with synthetic data could also be explained by models trained on original gold data and synthetic data potentially learning different types of shortcuts. with the original gold datasets often being constructed using keyword sampling, specific terms are often relied upon by models for classification of hate speech @xcite . on the other hand, with synthetic datasets, models could instead learn different shortcuts (which could potentially be more unpredictable, as we will discuss in section 6), while becoming more robust to the more 'traditional' shortcuts, which are often a consequence of overfitting the original gold training data. this possible explanation could also account for the slightly lower performance on the same-distribution data when using synthetic training examples, as models trained on synthetic instances would not overfit the original data as much anymore. ## intrinsic evaluation our experiments suggest that synthetic data can be useful in making models more robust to outof-distribution scenarios (cf. the qualitative analysis was carried out by two annotators, one male and one female, both with expertise in online language use, hate speech, and llm-generated text. the human evaluation focuses on three aspects: these aspects can, in fact, have a number of implications on real-world usage of synthetic data for hate speech detection. for instance, if synthetic data is not realistic, it may introduce spurious correlations between certain tokens and labels, making models overfit to lexical items that rarely occur in real-world data @xcite . on the other hand, label preservation is important because the data augmentation process assumes that the label of the original text will be preserved. indeed, data augmentation gives the opportunity to modify existing data in order to obtain more training samples without further manual annotation. however, if a large fraction of the labels changes after augmentation, it might not always be worth it, as classifiers trained on wrongly-labeled synthetic data could have unpredictable performance. finally, in the frequent cases in which the targets of hate represented in a dataset have been carefully balanced to ensure a fair representation of different groups, changing this distribution through the augmentation process may not be desirable. moreover, training a classifier on synthetic data in which specific targets of hate have been neglected would potentially affect classifier fairness, hurting already marginalized communities @xcite . we conduct the human annotation in two steps: the manually annotated data for this work is publicly available at https://github.com/dhfbk/ delving . annotation details are reported in the following subsections. ## conclusion in this work, we have carried out an assessment of synthetic data beyond the mainstream classifier performance evaluation, with the goal of linking classifier performance with an intrinsic qualitative analysis. our aim is to understand the potential risks and drawbacks of using synthetic data for a delicate task such as hate speech detection. while from mere classifier performance synthetic data shows to be helpful in out-of-distribution scenarios, our qualitative analysis proves that we should not take for granted the preservation of key features of gold data in synthetic data. first, synthetic data might introduce spurious correlations due to the language used by models, as it is easily spotted by expert humans. in addition, we showed that the preservation of hate speech labels during the augmentation process should not be automatically assumed, even when the data still appears to be useful for training a classifier. finally, llm-generated paraphrases of gold data show a drastically different identity category distribution compared with the original data. overall, our analysis shows that while classifier performance might show synthetic data to be potentially useful, it can hide potential risks we may often be unaware of.