Axiomatic Preference Modeling for Longform Question Answering (2023)

Abstract: The remarkable abilities of large language models (LLMs) like ChatGPT and GPT-4 partially stem from the post-training processes involving human preferences encoded within a reward model as part of a Reinforcement Learning from Human Feedback (RLHF) regimen. These reward models (RMs) often lack direct knowledge of why, or under what principles, the preferences annotations were made. In this study, we identify principles that guide RMs to better align with human preferences, and then develop an axiomatic framework to generate a rich variety of preference signals to uphold them. We use these axiomatic signals to train a model for the scoring answers to longform questions. Our approach yields a Preference Model with only about 220M parameters that agrees with gold human-annotated preference labels more often than GPT-4. The contributions of this work include: training a standalone preference model that can score human- and LLM-generated answers on the same scale; developing an axiomatic framework for generating training data pairs tailored to certain principles; and showing that a small amount of axiomatic signals can help small models outperform GPT-4 in preference scoring. We intend to release our axiomatic data and model.

## introduction recent advances in large language models (llms) has seen the introduction of diverse post-training strategies, including reinforcement learning from human feedback (rlhf) and reinforcement learning from ai feedback (rlaif). these techniques have helped bridge the "alignment gap" between the responses of raw pretrained language models and responses that resonate more closely with human preferences @xcite . these techniques steer llms to prefer one response over another based on feedback signals from either human annotators, or from another llm instructed to follow certain principles @xcite @xcite 2023). rlhf in particular involves the construction of a "reward model" (rm) which is trained to encode these human preferences and output a scalar score for any given response @xcite @xcite . primarily, a rm-based approach to training llms separates what to learn from how to learn it @xcite . the problem with most rms used in rlhf posttraining is that they are taught to regress a single scalar preference score annotated by humans without clear knowledge of why they made that decision or what principles they operated under. we term models trained in this fashion as naive preferencemodels. furthermore, the underlying preference pairs used to train the rm do not come from diverse sources, often being sampled from the same @xcite we propose five principled axioms to construct rich contrastive signals for training preference models llm they are trained on @xcite @xcite . it is also not clear that rms can reliably score human-written and llm-generated responses on the same scale, which is more challenging than previously anticipated due to vast differences such as style, as shown in figure 1 . without clear signals of which principle informs the preference decision, and diverse sources of training examples upholding it, a rm may not be aligned with the expectations of human stakeholders. for instance, studies have shown that rlhffinetuned llms may fall short of key expectations -e.g. by failing to support claims with evidence, or making claims that sound convincing but are untrue -showing that there are still prevalent gaps in alignment for these scenarios @xcite @xcite . in this work, we define principles (axioms) that humans desire in longform answers around the concepts of usefulness, relevance, grounded-ness, truthfulness, and thoroughness similar to @xcite . a distinguishing feature of our study is that we then use these principles to construct candidate answer pairs "axiomatically" such that one answer is clearly preferred along a certain principle. some of these axiomatic pairs are constructed from abundant sources of weak human preferences in the form of "upvotes" from community-based question answering (cqa) sites like stackexchange foot_0 . in figure 2 we illustrate how axiomatic pairs are generated for a single question. we define the principles in appendix a, and describe how the axioms uphold those principles in section 2. prior work used axioms to diagnose and correct failure modes in information retrieval systems @xcite @xcite . similarly, our axioms target known failure modes of modern llms, such as hallucinating incorrect statements that appear factual @xcite or being distracted by irrelevant context @xcite . the axioms also enforce new capabilities, such as incorporating evidence, or addressing multiple perspectives. we believe our axiomatic framework provides richer, more targeted underlying preference pairs than, say, sampling from the same llm with different temperatures. moreover, the rms in existing studies are often not released nor the subject of close study compared to the llms they post-train. they can be quite costly, sometimes holding as many parameters as the llms they train. while there are many studies on rms to address safety and toxicity issues @xcite @xcite @xcite , there are fewer on longform question answering @xcite . our approach is driven by the intuition that the act of identifying failure modes -or verifying an answer is free of them -is cognitively simpler (requiring fewer parameters) than the act of generating an answer, especially in the presence of authoritative evidence from a search engine. a separate, smaller rm also has many advantages: it is a controllable whitebox whose behavior is steerable, quantifiable, and decoupled from the llms it supervises; it allows for generalization to unseen examples without having to annotate them; and it is cheaper to run at scale. ## axiomatic preference modeling learning a preference model for longform question answering can be formulated as a learningto-rank problem @xcite . given a question q and a set of candidate answers a 1 , a 2 , ..., a n , the goal of the preference model is to find a partial ordering of the answers by training on pairs that best align with real human preferences @xcite . existing neural architectures such as transformer @xcite are adept at solving learning-to-rank problems (nogueira and cho, 2020), and do even better under contrastive learning regimes @xcite . a preference model pm takes as input a question q, answer a, and outputs a scalar pm (q, a) ∈ r a.k.a "preference score"; it has an optional input reserved for evidence passages e denoted pm (q, e, a). we instantiate pm as a transformerbased cross-encoder @xcite , f , whose input is a linearized sequence of tokens x constructed from the concatenation of q and a, denoted @xmath0. the output scalar is obtained from a linear regressor layer on the final transformer layer's cls token. we further construct contrastive pairs of sequences such that the answer in one sequence x + = q ⊙ a + is preferred over a negative answer to the same question x -= q ⊙ a -. at training time, the sequences are fed into f separately with the objective to score the positive example higher: f (x + ) > f (x -). we choose the margin loss to accomplish this goal: ## experimental methods implementation details for all our experiments, the preference model is initialized from a t5flan (chung et al., 2022) base model. we train each model on a different combination of axiomatic pairs with a learning rate of 5e-6 warmed up linearly over 1k steps. we control for differences in training data size by mixing the data and training for exactly 16k steps -just under one epoch -to avoid any overfitting. we sample training examples uniformly at random according to the question (aka "posts") so that posts with many answers do not dominate. for each question, we group all pairs of its answers into the batch. the maximum sequence length of the concatenation of question, evidence, and answer tokens is 2048, with the question capped at 256. ## results throughout these results, we compare preference models trained on various combinations of the axiomatic data, e.g. "pm 0-2 " denotes training with data pairs from axioms 1 and 2 added to the original pairs from axiom 0. ## related work early works on scoring llm outputs like lamda and blenderbot3 collect scores of single inputoutput pairs rather than preference scores between pairs of candidate outputs @xcite . more recent reward models (rms) fall into two camps. the first is training separate regressor models like those used in rlhf, which are often a single reward model to encode a one dimension of human preferences @xcite @xcite @xcite or many dimensions @xcite . the second camp uses llms instructed to give feedback based on principles or a "constitution" @xcite , with the drawback of being costly to query. other approaches seek more fine-grained reward signals or multiple reward models, such as collecting relevance, correctness, and completeness signals on both sentence-and response-levels using separate reward models @xcite . sparrow collects "targeted judgements" from human annotators to better characterize which of 23 rules a llm violated (mostly around toxicity and safety), and then train multiple targeted classifiers ( glaese 2022). the coexistence of rule-based and trained reward functions is also explored in @xcite . process supervision has emerged as a promising direction to provide feedback at each step of a complex multi-step task @xcite . retrieval augmentation has been shown in several studies to mitigate hallucination of incorrect statements in llms, by either finetuning llms with grounding documents @xcite , or inserting them to the context windows without fine-tuning llms @xcite . other methods infuse retrieved knowledge in the decoding stage for knowledge-intense question-answering tasks @xcite . ## conclusions we show that augmenting human preference data with axiomatically generated responses leads to effective pm that can score both human-written and llm-generated answers on the same scale under a variety of scenarios, including open-book search scenarios. while the bulk of the work in this paper went into generating training data rather than modeling, we stress that high quality training signals which illuminate nuanced differences between responses to the same question is what drives our pm 's quality, allowing it to exceed other public reward models with more than 10x parameters. notably, our resulting preference models is better aligned with gold human preferences than gpt-4, despite having only 220m parameters. future work can expand the pm to multi-turn conversations, and then leverage it to post-train llms.