Can LLM s Recognize Toxicity? A Structured Investigation Framework and Toxicity Metric (2024)

Abstract: In the pursuit of developing Large Language Models (LLMs) that adhere to societal standards, it is imperative to detect the toxicity in the generated text. The majority of existing toxicity metrics rely on encoder models trained on specific toxicity datasets, which are susceptible to out-of-distribution (OOD) problems and depend on the dataset’s definition of toxicity. In this paper, we introduce a robust metric grounded on LLMs to flexibly measure toxicity according to the given definition. We first analyze the toxicity factors, followed by an examination of the intrinsic toxic attributes of LLMs to ascertain their suitability as evaluators. Finally, we evaluate the performance of our metric with detailed analysis. Our empirical results demonstrate outstanding performance in measuring toxicity within verified factors, improving on conventional metrics by 12 points in the F1 score. Our findings also indicate that upstream toxicity significantly influences downstream metrics, suggesting that LLMs are unsuitable for toxicity evaluations within unverified factors.

## introduction large language models (llms) are prone to exhibiting biases and generating offensive content, as the bias deeply ingrained in their training data @xcite . existing research @xcite @xcite addresses such issues, focusing on representative identity terms or triggerprompts reflecting fine-grained toxic categories @xcite @xcite . these studies utilize typical toxicity metrics such as toxicity classifier scores and perspec-tiveapi 1 to determine whether a llm's reaction is toxic. however, typical metrics are vulnerable to domain shift and perturbations shown in figure 1 and appendix a.2. such phenomenon aligns 1 https://perspectiveapi.com/ figure 1 : an example of latte's toxicity detection results compared to traditional detectors. the blue box represents metrics that are trained on each criteria or dataset, whereas latte is a metric that detects toxicity using only the toxicity definition without any further training procedure. with the results that the use of those approaches shows significant susceptibility in the distribution shifts @xcite . as those metrics are trained on a particular dataset, they struggle to identify instances that deviate from the predefined notion of toxicity within the dataset (ood of toxicity) @xcite @xcite . recently some researchers adopt llms to evaluate utterances by themselves and make llms selfdebiasing to improve policy compliance @xcite . however, these methodologies tend to blindly trust llms and often overlook the adverse impact of upstream bias on the self-debiasing process. additionally, it remains unclear in what aspects these evaluation methods are better than traditional metrics, and what prompt factors influence the evaluation performance. in this paper, we propose latte (llms as toxicity evaluator) to address the variability in definitions of toxicity depending on context and to mitigate the negative impact of upstream bias. first, provided that the definition of toxicity can vary dynamically depending on diverse contexts, it is essential to ensure that the metric is flexible enough to adapt to diverse contexts. that is, the metric should function effectively even when toxicity definitions change, ideally without having to train on a new dataset. to achieve this, we take advantage of llm's zero-shot capabilities with our proposed evaluation prompt. second, the methodology using foundational neural models should not be indiscriminately applied as value prompts, which are commonly used in reasoning methods, as upstream bias has a significant impact on downstream tasks @xcite . therefore, we identify the safe domains for each llm by a structured toxicity investigation, and apply our method only within such domains. as shown in figure 2 , we first define toxicity factors and set up a benchmark tailored to each factor. next, we propose a toxicity investigation process incorporating the concept of neutrality to identify the safe domains where llms are not inherently toxic and maintain a neutral stance with respect to each factor. when llms are used as toxicity evaluators in unsafe domains, we demonstrate that their evaluation performance cannot be trusted. therefore, it is essential to identify safe domains. once finding a safe domain, we compare the conventional measurements to latte, our proposed evaluation metric. experimental results reveal that latte demonstrates superior performance by more than 4 points in accuracy and 12 points in f1 score, compared to existing metrics in evaluation datasets. in addition, our metric are robust to changes in definition of toxicity and format perturbations. lastly, we show that the neutrality of upstream llms does contribute to performances in downstream metrics, and our latte approach can be adaptable to diverse llms. ## definition of toxicity: three factors in this work, we define toxicity using three distinct factors -demeaning content, partiality, and ethical preference. to articulate the notion of toxicity within our research, we refer to elements of nonmaleficence, fairness, and justice from the seven ai trustworthy factors @xcite . recently, llms' non-maleficence is comprehensively analyzed by @xcite . besides, a large amount of researches such as perspective api 1 , toxigen @xcite , and hate speech detection evaluate the toxicity whether they intend to insult or defame. therefore, we define the factor that represents such offensive and profanities as demeaning. from the viewpoint of fairness, those that unilaterally oppose a specific group or stance can also be perceived as toxic. smith et al. ( 2022 ) deal with fairness based on demographic terms in detail, and @xcite address argumentative and contentious questions as sensitive questions. in addition, such a factor and the demeaning factor do not always coincide with each other @xcite , and stereotypes aren't always negatively assumed @xcite . as a result, such elements are collectively defined as partiality. furthermore, the concept of justice is technically interpreted in @xcite . from an ethical perspective, individual values are segmented into virtue, deontology, and utilitarianism. even if the utterances are not explicitly demeaning or partial, those that conflicts with an individual's ethical values can cause discomfort. we hence categorize such instances as "weak toxic." we collectively defined such factors as ethical preference. ## experiment in section §4.1, we introduce datasets for investigating the model's inherent toxicity. in section §4.2, we set up datasets for experimenting the feasibility of llms being used as evaluators. former datasets are primarily designed for detecting toxic utterances, whereas the purpose of detecting toxicity is auxiliary for latter datasets. all the examples of investigation dataset are in appendix c. for metrics, we utilize the task performance accuracy and f1 score. @xcite is a representative english dataset covering a diverse set of harms, containing context-dependent harms, enhancements to existing demeaning datasets such as @xcite , bold @xcite , toxigen @xcite . in this work, we utilize the demeaning category. hatespeech @xcite is a dataset focusing on english texts, analyzing an incitement of emotion and violence in hate speech instances, reflecting numerous toxicity datasets such as civil comments @xcite , social bias inference corpus @xcite , and contextual abuse dataset @xcite . @xcite is constructed for the question answering (qa) task, and is comprised of 11 stereotype categories. followed by sampling 100 examples from each group, a total of 1,100 samples are converted into a multiple choice-qa format to test llms' demographic bias. square @xcite comprises of sensitive questions and responses based on korean culture. in our work, we obtain test sets from inclusive-opinion and ethically-aware categories, for measuring the awareness of demographic content and argumentative contents. in our work, we designate those test sets as square demographic. @xcite utilizes natural language scenarios to generate numerous situations, encompassing interpersonal dynamics and daily events. in our experimental design, we use the deontology, virtue, utilitarianism test scenarios. 2022 ) is a short text paraphrased dataset of toxic and neutral utterances, filtered from the jigsaw, reddit, and twitter datasets. they define toxicity as the use of profanity. in our definition, it covers the demeaning factors. we utilize toxic and the neutral utterances. prosocial dialogue @xcite is a conversational dataset incorporating prosocial norms. we define need-caution utterances as toxic utterances and casual utterances as non-toxic utterances. in our definition, it covers both demeaning factor and demographic-oriented partiality factor. square contentious other than square demographic, we use contentious-unacceptable utterance pairs for investigating how neutrality is important in the downstream stage. all the details of evaluation datasets are in appendix e. ## conclusion despite the rapid advancements in the ai field, metrics related to toxicity remain in a state of stagnation. recently, there has been an increase in research that evaluates semantic areas such as naturalness by using llms. however, such methodologies should not be applied without caution in the field of toxicity. in our research, we propose a toxicity investigation framework and an evalua-tion metric that considers diverse llm factors. we find that current llms are reliable within confined factors. recent research @xcite , demonstrates that llms tend to behave more like children than adults. it implies that we have to provide detailed information regarding toxicity and relevant contexts for controlling llms. to provide such a detailed information during the period of llms' proliferation, we need to discuss the minimum standard for toxicity, not just keywordbased and trigger-prompts approach, in order to steer llms to be prosocial.