Value Alignment from Unstructured Text (2024)

Abstract: Aligning large language models (LLMs) to value systems has emerged as a significant area of research within the fields of AI and NLP. Currently, this alignment process relies on the availability of high-quality supervised and preference data, which can be both time-consuming and expensive to curate or annotate. In this paper, we introduce a systematic end-to-end methodology for aligning LLMs to the implicit and explicit values represented in unstructured text data. Our proposed approach leverages the use of scalable synthetic data generation techniques to effectively align the model to the values present in the unstructured data. Through two distinct use-cases, we demonstrate the efficiency of our methodology on the Mistral-7B-Instruct model. Our approach credibly aligns LLMs to the values embedded within documents, and shows improved performance against other approaches, as quantified through the use of automatic metrics and win rates.

## introduction large language models (llms) have become increasingly powerful and widely used, leading to growing interest in value alignment @xcite . this is also requisite for the systems to behave in accordance to particular value systems @xcite , which may originate from individuals, communities, companies, or countries. traditional approaches to value alignment often rely on highquality human-curated supervised data and preference data @xcite , which can be costly and time-consuming to produce. moreover, these methods align models to values that are explicitly prescribed by human curators, potentially overlooking nuanced information and context during training @xcite . particularly, popular alignment approaches including reinforcement learn-ing from human feedback (rlhf) @xcite @xcite and non-rl approaches such as direct preference optimization (dpo) @xcite , kahneman-tversky optimization (kto) @xcite , etc. rely on the paired and unpaired preference-data, with or without needing reference reward @xcite . such datasets comprise of an accepted and a rejected response by human curators to a given query. curating such datasets can be expensive and furthermore, the aligned models using such datasets can often overfit to the preferences of the majority group @xcite . additionally, there are also alignment approaches that rely on a carefully curated set of rules or principles @xcite . however, in most real-world use-cases, value systems are more likely to be embedded within unstructured text, such as documents, rather than as human-curated supervised, preference data, or a carefully curated set of rules. furthermore, this also calls for methods to optimize llms to these set of value systems quickly rather than rely on a single model with "universal" values. the majority of widely used datasets for general alignment are built using hand-crafted instructions @xcite , preference data @xcite , or principles diligently designed to elicit human feedback. these datasets often rely on expensive, proprietary llms for response generation or label annotation. the curation process involves creation of samples, which encompasses flat lists of values, or red-team prompts. for example, in @xcite , constitutional ai (cai) aligns llms through a constitution with normative principles written into it. one of the sources, for these principles, is united nations universal declaration of human rights (udhr). in cai, the values of the udhr are funneled through expensive process of curation for principles and then subjected to rigorous redteaming process to capture human feedback. there is a clear and pressing requirement for developing methodologies that can align models to value systems that are encoded in unstructured text. although such a text may not encompass all specific contexts, necessitating additional finetuning of the models, what we aim to establish is a robust baseline which can be iteratively improved with other approaches such as human-preference based alignment. we also aim for models that do not adhere to a single "universal" value system, but can be easily adapted to any value system. in this paper, we present a novel, systematic technique for aligning llms with both the implicit and explicit values embedded within unstructured documents. our approach automates the process of prodding values from these documents through synthetic data, thereby eliminating the need for manual curation and human feedback. we empirically show that our method surpasses other techniques in aligning llms with the values present in unstructured data. our proposed end-to-end approach is capable of handling entire documents, such as a corporate policy, and is not limited to documents with flat lists of principles or rules. it is worth pointing out that our proposed end-to-end approach can handle entire documents, such as a corporate policy, and is not limited to documents with flat lists of principles or rules. our method's ability to automatically extract, create specialized synthetic data and align to valuesfrom an unstructured text document has significant implications for the development of ethical and responsible llms and for variety of applications. the main contributions of this work are: instruct data sdg scenario data sdg document dpo, orpo, simop etc. sft figure 1: end-to-end view: our alignment method involves instruct and scenario sdgs steps, which are then leveraged for sft and preference optimization. ## alignment from unsupervised data in this section, we detail the stages required in aligning a llm according to the values embodied in unstructured data. figure 1 provides an overview of the system, which involves two primary components: a) synthetic data generation (sdg) of instructions ("instruct" for short) and preference/scenario data for different chunks of the document, b) supervised fine-tuning (sft) and preference optimization methods to align the model's responses to the values. for the reminder of this work, we will use the terms "preference data" and "scenario data" interchangeably. overall, the combination of these two components is crucial for two reasons -the first being that we ensure the implicit and explicit values of the document are reflected in synthetic data and second that the values are effectively baked-in into the llm through alignment algorithms. this method ensures that the llm responses are constrained under the values in the data. ## experimental setup we exhibit the efficiency and effectiveness of our method through two distinct use-cases. we compare various competitive methods using several metrics as detailed in the following sections. ## experimental results and discussion we conduct all our empirical experiments using an instruct version, 'mistral-7b-instruct-v0.2', from the mistral family as a "base" model, for both the use-cases. in order to create d sft and d pref , the inference in run on the sparse mixture of experts model, 'mixtral-8x7b-instruct-v0.1'. starting with the seed model π ϕ , we train a sft model, π sft , using d sft and then utlitize the final sft model as the reference model to further perform dpo. we also perform continual fine-tuning of the seed model on the raw extract text with simple a causal lm loss. we refer to this model as 'c-fine-tuned'. additionally, for rag, we retrieve from indexed chunks to augment the context to create the final prompt. in all our rag setup, we restrict number of retrieved chunk(s) to be one. in it is note worthy, contrary to the expectations, that integrating rag to an aligned model resulted in a surprise decline in performance. this observation was consistent even under situation where we had perfect retrieval. we hypothesize that this behavior may be due to the conflict between parametric and non-parametric memory, which is an active and recent line of research studied by the community @xcite and is beyond the scope of this article. we leave further exploration of this as a future work. while acknowledging the previous observation, it is crucial to emphasize our method's ability to achieve alignment to values and efficient operation without relying on expensive non-parametric or auxiliary memory resources. ## conclusion in this study, we introduce a novel approach for aligning large language models with values that are implicitly and/or explicitly embedded within unstructured data. by leveraging a large pretrained, teacher model we first create high-quality and diverse synthetic instruct and scenario data to prod the values. these sets of synthetic data are then utlitized to supervise finetune and preference optimize in order to instill the values within a llm. the efficacy of our proposed methodology is demonstrated through empirical study across two distinct use-cases, which underscores the potential of our approach in alignment without the necessity of auxiliary memory and expensive human curated data.