T oxi C raft: A Novel Framework for Synthetic Generation of Harmful Information (2024)

Abstract: In different NLP tasks, detecting harmful content is crucial for online environments, especially with the growing influence of social media. However, previous research has two main issues: 1) a lack of data in low-resource settings, and 2) inconsistent definitions and criteria for judging harmful content, requiring classification models to be robust to spurious features and diverse. We propose Toxicraft, a novel framework for synthesizing datasets of harmful information to address these weaknesses. With only a small amount of seed data, our framework can generate a wide variety of synthetic, yet remarkably realistic, examples of toxic information. Experimentation across various datasets showcases a notable enhancement in detection model robustness and adaptability, surpassing or close to the gold labels.

## introduction the 21st-century digital realm presents vast connectivity and information exchange opportunities alongside the challenge of widespread harmful content like cyberbullying, hate speech, and misinformation, impacting individuals and communities negatively. as such, the development of effective mechanisms for detecting and mitigating harmful content is of paramount importance @xcite @xcite . the emergence of the transformer-based model @xcite led to the development of complex models capable of identifying toxic content with remarkable accuracy. however, the effectiveness of these models hinges on the quality and diversity of the datasets used for their training @xcite . traditional datasets, figure 1 : harmful detection approaches often curated manually, tend to lack the diversity required to cover the multifaceted nature of harmful content. consequently, while these models excel at recognizing explicit instances of toxicity, they struggle when faced with subtler forms @xcite . many datasets containing harmful content are typically sourced from social media platforms like twitter or online forums @xcite @xcite . however, these datasets often exhibit significant class imbalances, particularly concerning specific types of toxic language. training a model to detect specific harmful content or opting for smaller models due to resource or latency constraints requires substantial amounts of human-labeled data. unfortunately, such data are scarce in downstream tasks and expensive to annotate @xcite . additionally, smaller or low-resource datasets face challenges like limited linguistic diversity and a higher risk of overfitting @xcite . privacy concerns also emerge when using social media data obtained without user consent for research purposes @xcite . furthermore, dataset decay is also a notable issue, especially with online social media, including abusive content, which are often deleted over time @xcite 1 . recent advancements in large language models (llms) such as the gpt series @xcite have prompted researchers to explore their potential for generating synthetic data @xcite @xcite . while llmbased data augmentation often improves model performance, findings are mixed regarding whether synthetic data generated by llms can consistently match the effectiveness of models trained on realworld, human-annotated data @xcite . llms have also shown promise in generating synthetic data for training harmful content detection models. however, work in this area remains limited @xcite and much of the existing research is relatively outdated @xcite @xcite @xcite . moreover, the effectiveness of llms in this domain is hindered by their difficulty with subjective tasks and their tendency to amplify biases inherent in human-annotated training data. these biases stem from inconsistent standards used in defining 'gold' labels, and llms often struggle to capture the complex nuances and diversity of human communication in highly subjective tasks. to address these challenges, we introduce the toxicraft framework, which enhances robustness and reduces bias in synthetic data generation. the toxicraft framework is specifically designed to produce higher-quality synthetic data that better supports model training. an example of the process is illustrated in figure 1 . in figure 1 , (a ) is direct training where a smaller model is trained on a limited dataset, that may affect model quality. (b) shows the use of large language models to do harmful detaction. (c) illustrates zero-shot synthesis, where use llms to generate a synthetic dataset via zero-shot prompts, used to train smaller models without additional data. our toxicraft method (d) employs carefully crafted prompts and attributes, utilizing gpt-4 as the backbone to generate a synthetic dataset for training more refined smaller models. a more detailed explanation of our approach can be found in section 3. to summarise, our work 1 twitter's policy only allows the display of post ids rather than the text itself, making content inaccessible over time, often due to deletion by either the user or twitter, particularly in cases involving harmful information. ## methodology in this section, we commence by defining the problem statement and subsequently introduce the tox-icraft framework. toxicraft is designed as a versatile and effective few-shot learning framework that operates through dataset generation. toxicraft framework are illustrated in figure 2 . ## ablation study in this section, we present the results of our ablation study, which focuses on three key aspects: the choice of model, the impact of different modules, and the robustness of our framework under different settings. these experiments were conducted on the ahsd dataset, htpo dataset, with 200 seed data and bert as the downstream model. ## managing risks of generating hateful content the use of large language models (llms) to generate hateful content presents significant ethical concerns, particularly around amplifying harmful biases. to mitigate these risks, we implemented strict content review processes to filter inappropriate outputs and limited the scope of generation to controlled experimental environments. we carefully ensured that synthetic data remained focused on research objectives. all experiments adhered to ethical guidelines, underwent reviews, and complied with responsible ai standards. the datasets generated were made available with clear warnings to promote transparency and encourage responsible use in future research. we will actively collaborate with the community to monitor, responding swiftly to any indications of misuse. ## conclusion in this study, we introduced toxicraft, a framework that effectively enhances data augmentation for harmful content detection tasks in low-resource settings. toxicraft utilizes large language models to synthetically expand seed datasets, overcoming the diversity and volume limitations of conventional augmentation methods. our extensive evaluations demonstrate that toxicraft significantly improves model robustness, outperforming or closed to baselines trained with gold label datasets. this work contributes to the ongoing efforts to develop dataefficient and adaptable models for sensitive content moderation, and sets a foundation for future research on leveraging synthetic data generation in various domains.