Value FULCRA : Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Value (2024)

Abstract: Value alignment is crucial for the responsible development of Large Language Models (LLMs). However, how to define values in this context remains largely unexplored. Existing work mainly specifies values as risk criteria formulated in the AI community, e.g., fairness and privacy protection, suffering from poor clarity, adaptability and transparency. Leveraging basic values established in humanity and social science that are compatible with values across cultures, this paper introduces a novel value space spanned by multiple basic value dimensions and proposes BaseAlign, a corresponding value alignment paradigm. Applying the representative Schwartz’s Theory of Basic Values as an instantiation, we construct FULCRA, a dataset consisting of 20k ( LLM output, value vector ) pairs. LLMs’ outputs are mapped into the K -dim value space beyond simple binary labels, by identifying their underlying priorities for these value dimensions. Extensive analysis and experiments on FULCRA: (1) reveal the essential relation between basic values and LLMs’ behaviors, (2) demonstrate that our paradigm with basic values not only covers existing risks but also anticipates the unidentified ones, and (3) manifest BaseAlign’s superiority in alignment performance with less data, paving the way for addressing the above three challenges.

## introduction as large language models (llms) exhibit incredible abilities across tasks @xcite @xcite , unexpected social risks also emerge @xcite @xcite . given the broad impacts of llms, value alignment becomes increasingly imperative to foster their sustainable development @xcite @xcite . the most widely adopted value principle requires llms to be helpful, honest and harmless, a.k.a., 'hhh' @xcite . in practice, these values are specified as concrete safety criteria formulated by the ai community, e.g., fairness and privacy protection @xcite @xcite , as shown in fig. 1 (a ). however, given the complexity, variability, and subjectivity of values, existing methods fail to handle three challenges. (1) clarity: the alignment goal should be an unambiguous, precise and comprehensive proxy of human values @xcite , guaranteeing 'the purpose we really desire' @xcite , while current value principles are tailored to a limited number of specific scenarios and may be in conflict with each other. (2) adaptability: the values ought to be compatible with varying contexts, evolving model capabilities, shifting social norms and diverse cultures beyond observed safety issues @xcite @xcite , while existing ones are usually tied to issues observed at this stage and a particular culture without considering the pluralism. (3) transparency: the framework must allow interpreting llms' risky actions via discerning their underlying values, helping human validation and calibration @xcite , which is critical but ignored by existing work. in moral philosophy and social science, the basic values, which encompass value foundations to explain the essential motivation behind individual behaviors and characterize cultural groups, has been established, such as schwartz's theory of basic values @xcite and moral foundations theory @xcite , holding the potential to address these challenges. inspired by them, we introduce a novel value space spanned by k basic value dimensions for the formalization, evaluin this value space, since the basic value dimensions are derived from universal requirements of human existence, serve as essential motivation and can be combined together to clearly characterize any type of human desires, providing clarity. a particular value type of a scenario or a country (e.g., the us value) is represented as a k-d vector with weights reflecting the relative importance of each dimension. thus, diverse values across situations (either observed or unidentified) and cultures can be flexibly covered in our value space by setting different weights, achieving adaptability. llms' actions are discerned to specific priorities of basic values, and the alignment process can be completed by adjusting the priorities, allowing some transparency. in this work, we adopt the most representative schwartz's theory of basic values @xcite as an instantiation. it is fundamental and comprehensive to model complex human values by identifying ten motivationally distinct basic values and 58 fine-grained value items @xcite , which has been widely applied in economics and political science research @xcite . noting that schwartz's theory is just one representative instantiation and our proposed framework is general to be extended to other basic value theories. based on our value space, we construct ful-cra 1 , a dataset composed of 20k (llm output, value vector) pairs. largely diverging from existing value datasets containing merely human utterances for human value analysis @xcite , all responses in fulcra are generated by mainstream llms from value- to conclude, we have three main contributions: ## related work value alignment of llms multiple approaches are investigated to align llms with humans. in-context learning (icl) regulates llms' behaviors by instructions without changing parameters @xcite . to better align llms with human preference, supervised fine-tuning (sft) @xcite and reinforcement learning from human feedback (rlhf) algorithms @xcite @xcite are developed. besides, more variants are introduced for stable optimization @xcite @xcite @xcite and relieving annotation cost @xcite @xcite . in these approaches, various alignment goals are explored @xcite , ranging from instruction-following ability @xcite to typical responsible ai issues @xcite @xcite . as llms advance, values extend to ethical and social risks like privacy and illegal problems @xcite @xcite ), yet they fall short in encompassing potential and unforeseen ones. the higher-level value principle hhh offers a comprehensive framework for alignment goals @xcite . human-annotated preferences @xcite and red-teaming @xcite have also been utilized. despite their prominence, the three challenges stated in sec 1 are still unresolved. ## the fulcra dataset we instantiate our basic value paradigm with schwartz's theory and present a dataset of (llm output, value vector) pairs, namely fulcra. in this section, we elaborate on the dataset composition, annotation method and data quality validation. ## dataset analysis fig. 2 presents the distribution of ten basic values in the fulcra dataset. more social-focus value dimensions like security, conformity and benevolence are detected in the llm's response to adversarial situations. besides, the llm aligns well with the value of achievement to complete tasks successfully, even surpassing those socialfocus values. this section delves into detailed case studies and a visualization analysis to justify the essential advantages of our basic value paradigm. ## basic value evaluation in our paradigm, a core component is the basic value evaluator, which can 1) measure llms' underlying values to reveal potential risks and 2) perform as a reward model for subsequent alignment. using the subset of alpaca's responses to beavertails prompts in our fulcra, we fine-tune a pretrained language model (plm) as the evaluator. ## basic value alignment leveraging basic values, we further propose a novel value alignment approach, basealign. this is elaborated and empirically validated in the next. ## conclusion concentrating on a largely unexplored question, i.e., how to define values for llms alignment, we leverage basic values from social sciences to build a novel value space and propose a corresponding value alignment paradigm, basealign. instantiated on the representative schwartz's theory of basic values, we construct fulcra, a dataset consisting of 20k (llm output, value vector) pairs. the analysis of this dataset affirms the advantages of our proposed basic value paradigm on clarity, adaptability, and transparency. furthermore, experimental results show that the basealign is effective, data-efficient and adaptable to various values. as an initial study on the basic value paradigm, we will extend this paradigm to other value theories and explore more robust alignment approaches.