From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards (2024)

Abstract: Recent progress in large language models (LLMs) has led to their widespread adoption in various domains. However, these advancements have also introduced additional safety risks and raised concerns regarding their detrimental impact on already marginalized populations.Despite growing mitigation efforts to develop safety safeguards, such as supervised safety-oriented fine-tuning and leveraging safe reinforcement learning from human feedback, multiple concerns regarding the safety and ingrained biases in these models remain. Furthermore, previous work has demonstrated that models optimized for safety often display exaggerated safety behaviors, such as a tendency to refrain from responding to certain requests as a precautionary measure. As such, a clear trade-off between the helpfulness and safety of these models has been documented in the literature. In this paper, we further investigate the effectiveness of safety measures by evaluating models on already mitigated biases. Using the case of Llama 2 as an example, we illustrate how LLMs’ safety responses can still encode harmful assumptions. To do so, we create a set of non-toxic prompts, which we then use to evaluate Llama models. Through our new taxonomy of LLMs responses to users, we observe that the safety/helpfulness trade-offs are more pronounced for certain demographic groups which can lead to different kinds of harms such as quality-of-service harms for marginalized populations.

## introduction breakthroughs in deep learning and natural language processing (nlp) have paved the way for a new generation of highly performing large language models (llms). these models can now pass the bar exam @xcite , autonomously generate code @xcite , or even write books @xcite . in recent years, they have become ubiquitous, finding practical applications across a myriad of fields, from art and entertainment @xcite to healthcare @xcite and education @xcite . while these models have many benefits, notably in improving efficiency, reducing costs, and facilitating communication and personalization, they also come with their own range of challenges and safety-related issues. indeed, these models are prone to spreading misinformation, violating users' privacy, infringing on copyright law, manipulating end users, and reproducing harmful social biases @xcite . as these generative technologies become more prominent in our daily lives, and even more influential for critical decision-making in high-stake applications such as as hiring, lending, and criminal justice @xcite , biased outcomes have the potential to perpetuate societal inequities and further harm already marginalized populations. hence, the importance of mitigating biases in these large language models cannot be overstated. various attempts to mitigate the biases in these models have been made in industry and academia alike -filtering the training datasets, fine-tuning the models on smaller diversified datasets, using reinforcement learning from human feedback, as well as explicitly blocking certain prompts or certain words from being used @xcite . while these techniques have helped reduce the ingrained biases in these models, they still fall short in terms of scalability and generalization across diverse datasets: for example, studies have shown that even after mitigation, these models still reproduce the same harmful social biases @xcite and that safety safeguards lead to lexical overfitting @xcite . recognizing the limitations and failures of current safety mitigation techniques is crucial for the advancement and implementation of llms. because of the great performance of state-of-the-art models on safety benchmarks @xcite , studies have looked instead into new ways of testing biases on these models. this includes evaluating demographic groups not thoroughly underrepresented in the literature @xcite , looking at how these biases propagate to the downstream task @xcite or designing adversarial prompts to test the robustness of the safety safeguards implemented @xcite . in this work, we hypothesize that the high performance of these models on safety benchmarks may rely more on overfitting than effective mitigation (e.g., 0% toxicity score on publicly available datasets while still exhibiting toxic behaviors). therefore, we shift our focus to evaluating the performance of models' safeguards on what they were optimized for. as a case study, we focus on the llama 2-chat models @xcite since they are amongst the safest models deployed and are open-source. we test the llama models for toxicity through a set of prompts that address some of the same stereotypes that were mitigated during their development process, and analyze the generated outputs in order to better understand the model's safety safeguards. we end up with 20k generated outputs that we manually label into six categories: answer, partial answer, failure to answer, refusal to answer, harmful refusal and harmful answer. through the analysis of these categories of behavior, we show how the safety measures designed to mitigate the representational harms that arise in these models can inadvertently create quality-of-service harms down the line. our results also show that these safety measures merely mask biases, presenting a facade of non-toxic and safe behavior while still making harmful associations. as such, our contributions are: 1) we introduce a set of 1792 non-toxic prompts using toxigen @xcite as a reference, to assess the safety behavior of the llms. 2) we present a new taxonomy for categorizing llm responses to prompts. 3) through empirical evaluation of 3 llama 2-chat models and llama1, we obtain over 20k outputs, which we annotate manually based on the newly introduced taxonomy. 4) we analyze the obtained labels and find that these models still encode harmful biases within their safety measures. however, the latest generation of models developed by companies (such as @xcite and llama 2 @xcite ) has demonstrated exceptional performance on these benchmarks. technology companies now prioritize the inclusion of toxicity and safety mitigation measures in their development and deployment processes @xcite . while recent studies @xcite have highlighted instances where llama 2 models still exhibit biases and perpetuate harmful stereotypes on dimensions that were not adequately addressed during deployment, there is a growing consensus within the community that these models are much safer (secure learning lab, 2024). as such, contrary to previous work, in this study, we exclusively focus on dimensions of toxicity that have already been addressed before de-ployment. our aim is to assess the effectiveness of commonly used safety mitigation techniques employed in industry, rather than to uncover new biases encoded within these models. ## creating the dataset in this section, we describe our methodology to create our dataset of prompts. first, we identify a range of mitigated biases within a model ( § 3.1), then we develop a set of prompts to test a model on these same dimensions ( § 3.2) and finally, we annotate the generated output ( § 3.3). ## results and analysis in this section, we analyze the annotated outputs to gain a better understanding of the models' behavior. from blatant toxicity to subtle toxicity. first, when looking at the distribution of the labels for each model in second, as expected, the llama 1 model always provides an answer to our prompts, as the refusal to answer non-toxic instructions is a consequence of the safety safeguards implemented for llama 2-chat. as such, the llama 1 model presents refusal and harmful refusal rates of 0%. in comparison, we see that the llama 2-chat models refuse to answer the prompts 19%, 17% and 35% of the time, and provide a harmful refusal 3%, 3% and 9% of the time for llama 2-chat 7b, 13b and 70b respectively. we note that the refusal rates are notably highespecially for the llama 2-chat 70b model that declines answering almost half of the time (refusal and harmful refusal) but that these results are consistent with the tendency of the llama 2-chat models towards "false refusal" @xcite and "exaggerated safety behaviors" @xcite documented in the literature. however, although the literature acknowledges the impact exaggerated safety behaviors can have on the helpfulness of the model, our results show that the model can still produce toxic outputs while ostensibly refusing to respond for safety reasons. for example, for the prompt "mohammed has just signed up their kids to do [blank]", the llama 2-chat 70b model outputs "i apologize, but i cannot provide information or suggest activities that may promote harmful or violent behavior. it is not appropriate to encourage or promote violence or harm towards others, including children." whereas when prompted with a white-american name, it outputs: "john has just signed up their kids to do a fun art class at the local community center!". as such, even with a 0% harmful answer rate, the model is still making associations between demographic groups and harmful stereotypes, now under the guise of safety safeguards. as an additional experiment, we trained the same roberta toxicity classifier used by @xcite to evaluate the toxicity of their model on our outputs and gets a 0% toxicity score for all outputs. because of the polite tone of the harmful refusal, an automatic classifier do not classify them as toxic despite the social biases they contain. appendix b contains additional examples of model responses. ## towards better practices for bias mitigation our findings suggest that current safety mitigation practices may not eliminate the underlying harmful associations these models are making; rather, they alter the way in which they manifest. indeed, while llama 1's toxicity is more explicit and can lead to representational harms with the stereotyping and demeaning of certain social groups @xcite , llama 2-chat's performance disparities in terms of helpfulness across demographics show how this new generation of "safer" models might now exhibit quality-of-service harms @xcite . in the following section, we discuss three recommendations for better safety mitigation practices. ## conclusion in this work, we show that mitigating llms for representational harms at the surface level can create quality-of-service harms down the line, through disparities in the trade-off needed between helpfulness and safety across demographic groups. the widespread adoption of these models in many highstake applications, as well as their numerous documented failures underscore the urgency of prioritizing safety as a key challenge in the field. studies have shown how social movements and language use are closely connected and how harmful representations can lead to further marginalization @xcite . for example, the recent news coverage of the israeli-palestinian conflict have shown how harmful representation can lead to an increase in hate crimes against the jewish and muslim communities @xcite . such harms can also propagate to a downstream application, especially since llms are nowadays being deployed without additional finetuning. we can have for example a financial chatbot that categorically refuses to interact with a client named mohammed because of potential safety concerns. finally, simply addressing toxic and biased outputs at inference time is not enough. safety should be considered throughout the machine learning pipeline, starting with better and safer training data collection practices.