H elp S teer: Multi-attribute Helpfulness Dataset for S teer LM (2024)

Abstract: Existing open-source helpfulness preference datasets do not specify what makes some responses more helpful and others less so. Models trained on these datasets can incidentally learn to model dataset artifacts (e.g. preferring longer but unhelpful responses only due to their length). To alleviate this problem, we collect HelpSteer, a multi-attribute helpfulness dataset annotated for the various aspects that make responses helpful. Specifically, our 37k-sample dataset has annotations for correctness, coherence, complexity, and verbosity in addition to overall helpfulness of responses. Training Llama 2 70B using the HelpSteer dataset with SteerLM technique produces a model that scores 7.54 on MT Bench, which is currently the highest score for open models that do not require training data from more powerful models (e.g. GPT-4). We release this dataset with CC-BY-4.0 license at https://huggingface.co/datasets/nvidia/HelpSteer

## introduction helpfulness and safety are the twin prime objectives for aligning domain-general models to follow user instructions @xcite @xcite . while various studies @xcite @xcite have thoroughly defined the aspects of safety, the criteria for what constitutes a helpful model remains opaque. recent trends have suggested a shift from asking this question altogether and settling on the notion that model responses should align with user or annotator preferences. while using blackbox user/annotator preferences can guide us towards more helpful model responses, this approach is both intellectually unsatisfying and inefficient in terms of compute and the amount of data required @xcite . the use of a black-box approach also leads to the possibility for models to associate longer responses with helpfulness, thereby prioritizing length over substance @xcite @xcite . this poses the risk of valuing verbose but irrelevant responses, highlighting the need for a more refined understanding of 'helpfulness' in model training. training more effective domain-general language models requires elucidating what humans find helpful in responses. @xcite use the aspects of creativity and humor as a measure of the helpfulness of instruction-following models. while these attributes can enhance the utility of responses in specific settings (e.g. story writing), they do not contribute to helpfulness in other scenarios (e.g. text classification) and could even be counterpro-ductive in formal business settings. insights from domain-specific language models trained for particular tasks offer valuable guidance on what constitutes helpfulness. in summarization tasks, @xcite identify accuracy, coverage, and coherence as important components of the overall quality of summaries. similarly, @xcite emphasize the importance of relevance, factuality, and information completeness in the long-form question-answering task. these findings suggest that the factors contributing to helpfulness in language models may vary significantly across different applications and contexts. we propose that the helpfulness of domaingeneral model responses can be assessed by their correctness, coherence, complexity, and verbosity. correctness refers to the inclusion of all pertinent facts without errors. coherence refers to the consistency and clarity of expression. we believe both aspects are critical for users to trust model responses across all tasks. complexity represents the response's intellectual depth, reflecting whether the content is basic or requires profound expertise (i.e. whether the response can be written by anyone with basic language competency or requires deep domain expertise to author), and is important because people tend to trust experts more. verbosity refers to the amount of detail included in the response, which has been found to correlate positively with human preferences @xcite , potentially because lengthy responses likely contain additional relevant information. while not exhaustive, we believe this set of factors provides a foundational framework for curating data to enhance the helpfulness of language models. to demonstrate the contributions of these attributes, we: ## dataset in this section, we present details on the data collection methodology of helpsteer outlining the underlying motivation, the prompt selection strategy, the procedure for generating responses, and the measures implemented to ensure the quality of annotations. ## experiments in this section, we detail our methodology in assessing whether helpsteer can effectively guide models towards improved helpfulness, factual accuracy, coherence, and appropriate levels of complexity and verbosity. we employ a blend of automated and human evaluations to gauge these aspects. automatic evaluation is conducted once on every model, because they effectively have tiny or no variance across runs. additionally, we describe the process of training a steerlm model @xcite utilizing the helpsteer dataset. the section also covers the baseline models against which we compare the performance of the steerlm model, providing a comprehensive view of its effectiveness in model alignment. ## ablation studies to better understand the contributions of each helpsteer attribute, we conduct ablation studies in which we exclude one or more attribute(s) when training the attribute-condition sft model and inference. as seen in the ablation of correctness shows the importance of the model's factuality to its overall helpfulness. when a model is not explicitly trained to be truthful, it can substantially hurt its overall helpfulness, as shown in the large drop in mt bench (7.54 → 6.92). on the other hand, such a model can generate more complex-sounding responses, resembling materials for a higher reading age (fkgl). the possibility of language models being optimized to generate complex-sounding responses that are poorly grounded in real world facts re-iterates the importance of explicitly training models to be factual in addition to being helpful. finally, comparing the ablation of helpsteer attributes to open assistant attributes suggests that helpsteer attributes contribute more towards the model's helpfulness since its drop in mt bench is more substantial (7.54 → 6.9 vs. 7.54 → 7.36). using both sets of attributes together complements each other, resulting in the most helpful model (mt bench 7.54) while training with either set of attributes alone can still far outperform a vanilla sft model (mt bench 6.29). ## steerability demo a further advantage steerlm offers is the ability to customize responses at inference time, providing end-users the ability to adapt to different scenarios. we present an analysis of steering the attributes of complexity and verbosity, which can be important for different user needs. in educational settings, presenting information with lower complexity aids comprehension for beginning learners. conversely, for experts, higher complexity can enhance their advanced understanding. when we increase the complexity attribute, the reading grade-level (fkgl) of the response gradually increases from 8.489 to 9.104. for instance, when prompted with a question, what is gravity? a low complexity response looks like gravity is the force that pulls everything towards each other. it is what keeps the planets in orbit around the sun, and what causes the tides in the ocean to rise and fall ... while a high complexity one looks like gravity is a force that exists between any two objects with mass or energy, causing them to attract each other. gravity is a result of the curvature of spacetime caused by the presence of mass or energy ... in business settings, different stakeholders (e.g. executives, managers, and engineers) might need answers to similar questions at varying levels of detail. as we dial up verbosity, mean response length increases from 422.3 to 1125.2 characters. ## conclusion we create helpsteer, a multiple attribute helpfulness preference dataset for steerlm with 37k samples, covering attributes of correctness, coherence, complexity and verbosity in addition to overall helpfulness. we openly release this dataset with cc-by-4.0 license. training a llama 2 70b model with the steerlm technique leveraging help-steer produces a model that scores 7.54 on mt bench, highest among models trained with truly open-source datasets. beyond helpfulness, this model is also more truthful and coherent compared to llama 2 70b chat and other baselines while being user-steerable in verbosity and complexity.