Don’t Go To Extremes: Revealing the Excessive Sensitivity and Calibration Limitations of LLM s in Implicit Hate Speech Detection (2024)

Abstract: The fairness and trustworthiness of Large Language Models (LLMs) are receiving increasing attention. Implicit hate speech, which employs indirect language to convey hateful intentions, occupies a significant portion of practice. However, the extent to which LLMs effectively address this issue remains insufficiently examined. This paper delves into the capability of LLMs to detect implicit hate speech and express confidence in their responses. Our evaluation meticulously considers various prompt patterns and mainstream uncertainty estimation methods. Our findings highlight that LLMs exhibit two extremes: (1) LLMs display excessive sensitivity towards groups or topics that may cause fairness issues, resulting in misclassifying benign statements as hate speech. (2) LLMs’ confidence scores for each method excessively concentrate on a fixed range, remaining unchanged regardless of the dataset’s complexity. Consequently, the calibration performance is heavily reliant on primary classification accuracy. These discoveries unveil new limitations of LLMs, underscoring the need for caution when optimizing models to ensure they do not veer towards extremes. This serves as a reminder to carefully consider sensitivity and confidence in the pursuit of model fairness.

## introduction the fairness and trustworthiness of large language models (llms) have drawn widespread attention @xcite . several evaluations @xcite have been conducted to determine whether llms exhibit bias or discrimination against individuals or groups in terms of protected attributes like race, gender, or religion. these evaluations primarily focus on explicit harmful statements containing specific hateful words. however, a significant proportion of online hate speech is characterized by implicit ex-pressions, utilizing indirect language to convey malicious intentions @xcite . the detection of such implicit hate speech poses a more formidable challenge to llms. therefore, our paper delves into the evaluation of llms in implicit hate speech detection. although some works have touched upon the issue of llms in implicit hate speech, they have only roughly evaluated whether llms can identify implicit hate speech and the quality of generated explanations @xcite . they overlook many key questions that pique our curiosity: in this paper, we evaluate the performance of llms in implicit hate speech detection, examining both primary classification and uncertainty calibration. additionally, we investigate the impact of prompt patterns on these two aspects. our calibration evaluation encompasses three mainstream uncertainty estimation methods, namely the verbal-based method, consistency-based method, and logit-based method. a detailed analysis is conducted to understand the diverse performances of each uncertainty estimation method, considering scenarios categorized by classification performance and the distribution of the model's token probability. our experimental evaluations are conducted on three distinct implicit hate speech detection datasets using llama-2-7b (chat) @xcite , mixtral-8x7b @xcite , and gpt-3.5-turbo @xcite . we find that llms exhibit extreme behavior in both classification and calibration tasks, leading to excessive sensitivity and poor calibration: ## evaluation design to study the ability of llms in implicit hate speech detection, we design the evaluation encompassing both the primary classification and uncertainty calibration. we also investigate the impact of prompt patterns on both of these tasks. ## experiment settings models we conduct experiments with three kind of llms, llama-2-7b (chat), mixtral-8x7b, and gpt-3.5-turbo. datasets our experiments use three implicit hate speech detection datasets: latent hatred @xcite , sbic (v2) @xcite , and tox-igen @xcite . see appendix a.2 for more details of the data preprocess. metrics our evaluation encompasses both primary classification and uncertainty calibration. in assessing task classification performance, we utilize precision, recall, and f1 scores to evaluate the predicted answers. meanwhile, we employ three metrics for calibration performance: the area under the receiver-operator characteristic curve (auroc) @xcite quantifies the probability that the model assigns a higher uncertainty score to an incorrect prediction than to a correct one. the expected calibration error (ece) @xcite ) is calculated as the mean squared discrepancy between the average accuracy and confidence for each bin, with the magnitude of each deviation scaled by the fraction of samples falling into the respective bin. the brier score (bs) @xcite measures the mean squared difference between the confidences and the actual outcomes. better calibration is represented with a higher auc and lower values for ece or bs. experiment setting we present six demonstrations (i.e., examples) in the prompt for few-shot in-context learning, organized in a balanced class and random order. the verbalized confidence are set to 60%, 70%, 90%, 70%, 60%, 90% in the demonstrations. we adjust the parameters to report a better outcome for each individual model. a consistent temperature setting of 1 is applied to all three models. greedy decoding is employed for llama-2-7b and gpt-3.5-turbo, while top-p sampling with a value of 0.9 is opted for mixtral-8x7b. llama-2-7b and mixtral-8x7b exhibit oversensitivity: the majority of points for llama-2-7b and mixtral-8x7b fall above the equilibrium line in both the latent hatred and sbic datasets. this suggests that these models classify statements without hateful intent as hate speech, indicating an excessive level of sensitivity. ## results of primary classification the over-sensitivity arises from the presence of sensitive groups and topics within benign expressions. llms display excessive sensitivity on latent hatred and sbic datasets, while it does not exhibit such over-sensitivity on the toxigen dataset. that is because examples of the negative class in the dataset latent hatred and sbic contain sensitive groups or sensitive topics. for example, the statement "vandals target black women maryland home" is a description of a news event that involves sensitive groups of "black women" and sensitive topics "vandals" and "target" but does not contain any hateful emotions. however, examples of the negative class in the dataset toxigen don't contain sensitive topics ("muslims in the 19th century migrated from europe and the middle east to the u.s.") or sensitive groups ("i felt so much better after confiding that in someone and realizing i'm not alone"). the presence of sensitive groups and topics confuses llms, leading to misjudgment of benign expressions. gpt-3.5 is positioned closer to the upper-right corner, indicating an overall better performance. it also demonstrates a relatively balanced performance between recall and precision, albeit with a slight reversal compared to llama-2-7b and mixtral-8x7b. this indicates that gpt-3.5 did not exhibit excessive sensitivity, although there is still room for improvement in its ability to detect implicit intentions. different prompt patterns exhibit varying degrees of over-sensitivity. in the case of llama-2-7b, the most notable imbalance is observed in the cloze prompt pattern across all three datasets, with biases ranging from 12% to 33%. the target pattern shows biases of 24% and 14% on the latent hatred and sbic datasets, respectively, while the vanilla qa pattern exhibits a bias of 17% on the latent hatred dataset. only the cot pattern demonstrates a relatively balanced performance across all three datasets for llama-2-7b. regarding mixtral-8x7b, all prompt patterns exhibit significant biases on the latent hatred dataset, ranging from 20% to 50%. on the sbic dataset, prompt patterns choice qa, cloze, and vanilla qa all exceed 40%, and on the toxigen dataset, both cloze and target biases surpass 10%. although different prompt patterns result in varied performances, they consistently demonstrate similar trends on the same model. we also present the f1 score in appendix a.4. we find that when the f1 score achieves its best performance, there can be a significant imbalance between precision and recall. this cautions us against relying solely on f1 and overlooking the issue of imbalance between precision and recall. ## results of confidence calibration the calibration results in the uncertainty estimation method that yields the best calibration performance is highlighted in each scenario. we find that the logit-based confidence achieves the highest auc across all the datasets and llms. however, the uncertainty estimation method achieving the highest ece/bs varies depending on the scenario. we also find that the confidence distribution exhibits extremes, whether it's overly conservative or overly confident (fig. 5 , 6, 7. the verbal-based confidence estimation method typically produces figure 2 : the best-performing uncertainty estimation method in different scenarios categorized by the model's output token logit and primary classification performance. logit-based confidence scores achieve the best auc in all scenarios, while the ece for each method varies across scenarios. a relatively conservative confidence score. the consistency-based method usually demonstrates a very high confidence score. the distribution of logit-based confidence varies depending on the model, with gpt-3.5-turbo and llama-2-7b tending to generate high logits, while mixtral-8x7b tends to produce conservative logits. our subsequent analysis reveals that the calibration performance significantly depends on the primary classification performance due to the highly concentrated confidence distribution. ## future work the insights we've gained for future work include two aspects: firstly, future optimizations should aim to avoid misjudging benign expressions, especially those containing sensitive groups. recent efforts aim to ensure that llms do not generate harmful output toward sensitive groups. however, it is unknown whether llms might veer towards another extreme. imagine if all discussions about black people were filtered out on social media platforms; that would be unfair as well. secondly, future optimization should aim to avoid excessive concentration of confidence and should take into account varying levels of task complexity. we unveil factors influencing calibration: calibration performance significantly relies on the primary task performance due to extremely concentrated confidence. some future research pathways and ideas include: (1) integrating correction for oversensitivity during rlhf. in addition to identifying harmful content during rlhf, humans should guide llms to recognize challenging benign expressions involving sensitive groups but not malicious ones. (2) improving the reflection of confidence in logits during llm training or inference. optimization of the decoding process, sft loss, or post-processing of logits can be explored to prevent the concentration of logits within a fixed range. (3) considering tasks of varying complexity in llm calibration optimization. developing uncertainty estimation methods that are adaptable to different task complexities is imperative. ## related work implicit hate speech detection hate speech inflicts significant harm on specific communities. researchers have developed various hate speech detection models @xcite . recently, there has been a growing interest among researchers in addressing implicit hate @xcite @xcite . certain studies have employed llms to generate explanations with step-by-step reasoning for detecting implicit hate speech @xcite . while huang et al. (2023 ) explored the quality of explanations generated by chatgpt for detecting implicit hateful tweets, their research exclusively focused on implicit instances, neglecting non-implicit statements. in our exploration, we assess the performance of llms in detecting implicit hate speech, taking into account various aspects with thoughtful consideration. we investigate whether there is an imbalance in the predictions of positive and negative classes when llms are employed for hate speech detection. additionally, we thoughtfully consider various prompt patterns in our analysis. ## conclusion the ability of llms in implicit hate speech detection is insufficiently examined. in this paper, we explore the performance of llms in identifying implicit hate speech and the calibration of three mainstream uncertainty estimation methods. we find that llms have fallen into two extremes, excessively focusing on sensitive groups and exhibiting extreme confidence score distributions. these extremes result in over-sensitivity and poor calibration respectively. these discoveries unveil new limitations of llms, underscoring the need for caution when optimizing models to ensure they do not veer towards extremes.