The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values (2023)

Abstract: Human feedback is increasingly used to steer the behaviours of Large Language Models (LLMs). However, it is unclear how to collect and incorporate feedback in a way that is efficient, effective and unbiased, especially for highly subjective human preferences and values. In this paper, we survey existing approaches for learning from human feedback, drawing on 95 papers primarily from the ACL and arXiv repositories. First, we summarise the past, pre-LLM trends for integrating human feedback into language models. Second, we give an overview of present techniques and practices, as well as the motivations for using feedback; conceptual frameworks for defining values and preferences; and how feedback is collected and from whom. Finally, we encourage a better future of feedback learning in LLMs by raising five unresolved conceptual and practical challenges.

## introduction incorporating human feedback into large language models (llms) is a welcome development to create models that are better aligned with human preferences or values, and exhibit traits such as helpfulness, honesty and harmlessness @xcite or safety, quality and groundedness @xcite . however, learning from human feedback introduces new biases and challenges, and there are many unresolved questions in this fast-moving field of research. it is important to take stock of current practices, possible blindspots, and new frontiers of research, so that tangible progress can continue to be made. in this paper, we adopt the dual aim to both survey existing literature on human feedback learning, then draw on the regularities, commonalities and critiques of this survey to also provide recommendations for future work. we review 95 articles that use human feedback to steer, guide or tailor the behaviours of language models. this includes making models' responses more coherent and engaging @xcite ; assisting models to infer user intent @xcite ; rejecting and rebutting unsafe requests @xcite ; or minimising the risk of hallucination @xcite . we source articles primarily from the acl and arxiv repositories, coding each according to a detailed conceptual and methodological schema. our review makes three contributions: ## the past in this section, we review 22 articles released between 2014-2019 that use human feedback but with older generation model architectures. highlighting these works ensures that foundational research is adequately attributed for advancements in today's models, and demonstrates the evolution from indirect or proxied human feedback. ## the present turning our attention to the heart of our review, this section discusses the 50 papers that incorporate human feedback to steer llm behaviours. ## challenges and recommendations for the future drawing on our analysis of the reviewed papers, we identify five key challenges for future researchers. these challenges are divided into conceptual and practical issues. the conceptual challenges (c1-c3) revolve around the difficulty of specifying a clear set (or sets) of preferences and values. even assuming the resolution of the conceptual challenges, practical challenges remain in converting conceptual ideals into empirical signals, which in turn steer language model behaviours. however, these values are only unobjectionable because they are abstract and not precisely defined @xcite . these terms can be considered what levi-strauss and laclau call 'empty signifiers' @xcite ; terms that are viewed positively but are inscribed with different meanings by different people. for example, when @xcite design a constitution to produce outputs as "ethical and harmless as possible", this can have varying interpretations based on an individual's own ethical frameworks and sociocultural background. establishing priorities over sets of preferences or values to embed in llms, and ensuring consistent interpretation of conceptual meaning across people, is a persistent challenge which cannot alone be resolved via purely technical solutions. one possible approach is to draw on legal theory, and values protected in human rights law @xcite . translating abstract shared values into decisions is a core function of legal systems and legal theory offers a long history of scholarship which combines the philosophical and practical. one approach along these lines was proposed by @xcite which applies a principle of subsidiarity to govern the personalisation of generative ai systems for different use cases. we also advocate for anchoring closely to existing legal systems as a matter of democratic principle: it is dangerous for moral and value judgements with broad societal impacts to be made by small independent groups. colloquially, values are understood to be stronger than preferences, and potentially carry greater normative weight as guiding principles or life goals @xcite . as such, users may have greater concerns about an llm misaligned with their values than with their preferences; so, it is important to be clear about which is being discussed. within the broad terms, there are many meanings: 'preferences' have been defined as 'instrumental utility' @xcite @xcite , 'stylistic taste' @xcite @xcite , and 'behavioural principles' @xcite . 'values' definitions are based on 'instrumental and intrinsic value' @xcite , 'acceptable social behaviours' @xcite , or 'making decisions which are unlikely to be harmful' @xcite . the differences between individual (subjective) and global (objective) preferences is often blurred-for example, which properties of a "better" summary are universal, and which depend on subjective appreciation, like writing style and tone. clearer definitions of preferences and values in the context of alignment would serve to motivate and clarify what we are aligning llms to. to aid transparent communication, it should be clearly documented whether researchers aspire to follow the prescriptive or subjective paradigm of data annotation, rather than leaving it unspecified @xcite . increased interdisciplinary communication with practitioners in other fields would impart wisdom on measuring the perspectives and behaviours of human subjects. for example, human-computer interaction literature shows how interfaces and incentives can be optimally designed to avoid participant response bias @xcite @xcite ; experimental psychology and behavioural economics research show how the presentation of scales and order effects influence ratings @xcite @xcite and that human preferences are unstable, intransitive and vulnerable to experimental artefacts @xcite @xcite @xcite . researchers should consider techniques to model the noise and distribution of human feedback @xcite or establish post-hoc consensus @xcite , rather than ignoring disagreement by aggregating responses. however, there are trade-offs: the specific nuances and minutiae captured in fine-grained feedback might heighten biases and reduce generalisability when drawn from unrepresentative samples-which we now discuss. when soliciting human feedback, attempts should be made to diversify who is given a voice, such as by applying democratic or jury-based principles in how these voices are weighted @xcite and by employing bottom-up participatory approaches @xcite @xcite ; or to seek top-down sampling that better represents the population being studied @xcite . mirroring the move in other areas of nlp to document and explore annotator disagreement @xcite @xcite @xcite , each item of feedback should be associated with a pseudo-anonymised annotator id. so far as privacy allows, documentation of annotator background should be provided in a data statement @xcite . ## conclusion this review provided an overview of incorporating human feedback into llms, with a focus on subjective preferences and values that lack 'ground truth alignment'. we have witnessed two notable shifts in the field from past to present-first, a move away from specialist systems towards general purpose language agents capable of handling many nlp subtasks via instruction or open-ended dialogue; second, more use of direct human feedback which surpasses the limitations of user simulations or automated metrics. while the shift to incorporate human voices directly into llm development is welcome, it introduces new challenges that require careful navigation. some challenges are more tractable than others-for example, practitioners will always have to deal with the complexities and intricacies of unstable and idiosyncratic preferences across end users of their model, but can take practical steps to better approximate this distribution by diversifying the recruitment of feedback providers. external scrutiny is crucial to ensure the integrity and reliability of research efforts. our review shows that many influential papers lack open and externally-validated peer review, especially those published by big industry labs like google deepmind, anthropic, google, and openai. furthermore, the majority of reviewed papers do not release model artefacts, or only do so behind a paywalled api. to foster progress, we advocate for a greater degree of open, interdisciplinary and democratically-grounded discussion on how humans can meaningfully shape future llm behaviours in a way which is well-bounded, operationalisable, and equitable.