Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset (2025)

Abstract: In an effort to mitigate the harms of large language models (LLMs), learning from human feedback (LHF) has been used to steer LLMs towards outputs that are intended to be both less harmful and more helpful. Despite the widespread adoption of LHF in practice, the quality of this feedback and its effectiveness as a safety mitigation technique remain unclear. This study addresses these issues by auditing the widely-used Helpful and Harmless (HH) dataset by Anthropic. Our work includes: (1) a thorough investigation of the dataset’s content through both manual and automated evaluation; (2) experiments demonstrating the dataset’s impact on models’ safety; and (3) an analysis of the 100 most influential papers citing this dataset. Through our audit, we showcase how conceptualization failures and quality issues identified in the HH dataset can create additional harms by leading to disparate safety behaviors across demographic groups. Our findings highlight the need for more nuanced, context-sensitive approaches to safety mitigation in LLMs.

## introduction learning from human feedback (lhf) has gained popularity in recent years as a strategy to mitigate the harms of large language models (llms) while preserving their utility. lhf consists of having human annotators assign relative preferences between model outputs, then training a model to generate outputs which are more likely to be preferred by the annotators. although the first instances of lhf were in robotics @xcite ) and text summarization @xcite , it has been adopted as a harm reduction strategy by companies like anthropic @xcite , google * these authors contributed equally to the work @xcite and @xcite . among the guiding principles for aligning llms, askell et al. ( 2021 ) define an ai system as aligned if it is helpful, honest, and harmless (hhh). helpfulness refers to the llm's capacity to perform a task or answer a question. honesty pertains to factual accuracy and truthfulness. harmlessness encompasses a range of considerations. following the adoption of the hhh principles, recent studies have shifted towards a more integrated approach, recognizing these challenges as part of a broader "safety bundle" of misaligned behaviors @xcite @xcite . anthropic's helpful and harmless dataset @xcite serves as a prototypical example of this trend-a human preference dataset designed for safety mitigation, cited over 1000 times and used to train more than 200 models. * this presents a paradigm shift, as earlier research in responsible natural language processing (nlp) focused on distinct methods for identifying, quantifying, and mitigating issues such as stereotypical bias @xcite , toxicity @xcite , and privacy leakage @xcite . despite the widespread adoption of the hh dataset as a benchmark for model alignment, it is not clear to what extent this dataset truly succeeds at making llms more harmless and more helpful. in fact, recent work shows that models trained with lhf with hh preferences are more likely to exhibit safety failures associated with these preferences and the trade-offs they create (i.e. a safer model is less likely to respond to a query, making it less helpful, and vice versa) @xcite @xcite or to showcase superficial safety alignment @xcite -meaning that the model's alignment predominantly improves the style of its outputs rather than influencing its underlying knowledge. furthermore, while various work has looked at the shortcomings of open-source corpora @xcite @xcite , to the best of our knowledge, none has investigated the content of preference datasets. in this work, we conduct a comprehensive audit to examine how the hh dataset embodies the hhh principles and explore the connection between the conceptualization of these principles and the safety failures reported in the literature. we investigate the following three research questions by evaluating the quality of the dataset and its effectiveness for safety mitigation: (rq1) what's in the hh dataset? we provide a thorough audit of the hh dataset through an exploratory analysis as well as a human evaluation and show its various shortcomings, including a failure to conceptualize harmlessness and quality issues ( §3); (rq2) what can models learn from the hh dataset? we train models using different variations of the hh dataset and evaluate them for safety @xcite . our analysis reveals that using lhf with the hh dataset can lead to disparate exaggerated safety behaviors across demographic groups ( §4); and (rq3) how did the hh dataset impact the community? we conduct a survey of the 100 most influential papers that reference the original study and offer insights into how the dataset has been adopted by the broader research community as well as how some inherent limitations of the dataset are framed as an inevitable trade-off between helpfulness and harmlessness ( §5). through this multidimensional audit, we showcase the limitations of the "safety bundle" introduced by the hhh principles and operationalized by the hh dataset. in light of these findings, we highlight the need for more nuanced and contextsensitive approaches for safety mitigation as well as a shift in how these considerations are addressed by the community. ## related work learning from hh preferences. lhf is a policy optimization method in which the objectives are defined by a set of human preferences @xcite . while multiple human preference datasets have emerged fol-lowing the growing popularity of this technique, only a few focus on llm safety @xcite @xcite . among them, the hh dataset was the first open-source dataset to conceptualize the helpful and harmless objectives now integrated into the training of new generative models, including google deepmind's gemini @xcite ) and meta's llama 2 models @xcite . the goal of performing lhf with helpful and harmless preferences is to make llms both more competent (helpful) and reduce the chances of generating harmful content (harmless). despite the growing popularity of lhf with hh, recent work has shown how training using these preferences could cause a llm to exhibit exaggerated safety behaviors and refuse to answer benign prompts @xcite since harmlessness and helpfulness are expressed as competing objectives during training. this phenomenon is often referred to as the "helpfulness-harmlessness trade-off" or "safety trade-off." previous work has also shown how this training can increase gender bias @xcite and make llms express stronger political opinions @xcite . because of these limitations, the effectiveness of training with hh preferences is not clear. in this work, we audit the most popular open-source large scale dataset for human hh preferences @xcite in an effort to better understand how effective these preferences are at improving models' safety. ## discussion our audit of the hh dataset across various dimensions highlights how while the dataset might appear to make llms more harmless, it might also paradoxically create additional harms by leading a model to make spurious associations between certain demographic groups and harmfulness as seen in section 4. this might explain some of the safety failures documented in the literature @xcite . in this section, we provide recommendations to alleviate these issues. recontextualizing safety. our work demonstrates problems with the hh dataset which groups disparate issues -such as private data leakage, information on how to commit a crime, or islamophobia -under the umbrella term of harmfulness. we suggest that bundling such vastly different underlying sociotechnical issues must be done carefully if at all, or else it risks driving further algorithmic harms ( shelby 2019 ) describe this issue as the "portability trap," i.e. a failure to recognize how a solution designed for one specific context might be harmful when applied to another context. for example, a language model that refuses to respond to instructions containing the word killing is annoying at worst, whereas a model that refuses to respond to prompts containing certain identity terms can perpetuate existing inequalities. as such, we recommend a recontextualization of safety in the sense that we believe that each mitigated issue should be linked to real-life harms and rooted in the appropriate social context. instead of a catch-all dataset that addresses a wide range of issues, each challenge could be addressed individually, with the appropriate nuance, expertise and sensitivity in mind. redefining the helpfulness/harmlessness trade-off. the widespread popularity of the hh dataset, being the first large-scale open-source dataset to operationalize the hh principles, has driven greater adoption of lhf as a mitigation strategy and reinforced the notion of safety tradeoff. by framing these two principles as competing objectives, lhf with hh fuels the narrative that safety must come at the cost of utility, rather than contribute to it. we posit that the apparent contradiction between these principles arises primarily because ethical considerations are often deferred until the final stages of development, due to the escalating costs associated with pre-training large language models. this creates a disconnect between the tasks models are trained for -e.g. nextword prediction on text sourced from the internet -and the tasks they are evaluated on -interacting in respectful, non-offensive conversations with diverse end-users. researchers and practitioners can draw on decades of research in safety engineering @xcite to understand how to better integrate social and ethical considerations throughout the ai system lifecycle. these considerations should be viewed as integral to the overall quality of the system, rather than optional or auxiliary, standalone dimensions. utility and safety are not inherently contradictory; instead, a good system is one that functions appropriately for everyone. ## conclusion in this work, we conduct a comprehensive data audit that includes (1) a detailed analysis of the hh dataset's content, ( 2 ) experiments with models trained on various iterations of the dataset, and (3) a survey on the use of the dataset by the community and we uncover various limitations of the hh dataset, and more generally of learning with hh preferences. these limitations include quality issues as well as a failure to conceptualize the concept of harmlessness, i.e. the "safety bundle." following our findings, we highlight the importance of adopting a sociotechnical approach to harm reduction, and shifting the narrative around the safety trade-off.