Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation (2023)

Abstract: Natural language generation has witnessed significant advancements due to the training of large language models on vast internet-scale datasets. Despite these advancements, there exists a critical challenge: These models can inadvertently generate content that is toxic, inaccurate, and unhelpful, and existing automatic evaluation metrics often fall short of identifying these shortcomings. As models become more capable, human feedback is an invaluable signal for evaluating and improving models. This survey aims to provide an overview of recent research that has leveraged human feedback to improve natural language generation. First, we introduce a taxonomy distilled from existing research to categorize and organize the varied forms of feedback. Next, we discuss how feedback can be described by its format and objective, and cover the two approaches proposed to use feedback (either for training or decoding): directly using feedback or training feedback models. We also discuss existing datasets for human-feedback data collection, and concerns surrounding feedback collection. Finally, we provide an overview of the nascent field of AI feedback, which uses large language models to make judgments based on a set of principles and minimize the need for human intervention. We also release a website of this survey at feedback-gap-survey.info.

## introduction for generation systems to be widely useful, they must generate text that is not only fluent and high-quality, but also well-aligned with human desires and specifications @xcite @xcite @xcite . achieving such ambitious goals requires large language models (llms) to evolve beyond traditional training methods. recent improvements in this space have centered on incorporating human feedback @xcite @xcite , intended to serve as a guiding force toward the desired outcomes, much like feedback mechanisms in physical machines @xcite . typically, state-of-the-art language generation systems are obtained by training probabilistic, autoregressive llms on massive amounts of data using maximum likelihood estimation (mle). however, the data used to train these models is generally scraped from the internet, often containing noise, social biases, and errors @xcite . this combination may result in a misspecification of target behavior @xcite , and may lead to models that generate toxic, inaccurate, and unhelpful content @xcite . the evaluation challenge is compounded as these models are often assessed by automatic metrics that rely on superficial features such as word overlap with reference text. however, these metrics often fail to correlate with human-perceived text quality, particularly when models are overly optimized for these metrics @xcite @xcite @xcite . 1 considering human-perceived quality can help bridge the gap between machine and human generated text and better align the system with desired outcomes @xcite . feedback, as a concept, encompasses a wide range of interpretations @xcite ; however, some universal characteristics can be identified, such as its format, its intended results, and the ways it is utilized as a part of the model development process. in this survey, we focus on the role of human feedback for improving language generation. we start by formalizing human feedback, creating a taxonomy of feedback types and uses ( §2). we characterize feedback by its format and objective, relating to desired model behavior ( §3). we explore direct feedback optimization strategies, such as reinforcement learning with human reward functions ( §4) and indirect approaches utilizing trained feedback models as proxies ( §5). we look at human-feedback data datasets and their collection, discussing their influence on models ( §6). lastly, we cover recent work leveraging ai feedback from llms for feedback reduction ( §7). ## directly leveraging human feedback in an ideal scenario, we would directly leverage human feedback to improve generation for both training and decoding. ## improving generation using human feedback models directly using human feedback to improve model behavior is not feasible in the general case: asking humans to provide feedback for every model output is both expensive and time-consuming. ## collecting and using human feedback collecting human feedback can be expensive and may present issues for the inexperienced. we highlight existing datasets, their collection methods, and considerations for those creating their own preference datasets. annotator variability remains largely unexplored @xcite , though evidence suggests well-constructed annotation guidelines are necessary @xcite to avoid systemic bias away from the intended task. in general, there is not much discussion in the literature as to how choices made in the feedback collection process impact the final generalization ability of the model, as this has not been studied in a controlled setting. however, there is an appreciation for the importance of data quality in feedback collection, as researchers make efforts to filter out annotators based on their agreement with gold labels, as well as based on interannotator agreement @xcite @xcite note that judging data quality is difficult for more open-ended forms of feedback such as dialogues. despite this, they were able to achieve good results without detailed data quality controls. the impact of collection methods on final results may be a direction for future research, but for now, we present considerations for data collection along different axes below that experimenters should keep in mind, based on previous studies: task dataset & their descriptions collection method platform feedback type language assistant hh-rlhf (bai et al., 2022a; perez et al., 2022a) explicit upwork, mturk ranking language assistant shp (ethayarajh et al., 2023) implicit scraped from reddit ranking/score summarization summarize-from-feedback (stiennon et al., 2020) explicit upwork ranking question answering feedbackqa (li et al., 2022) explicit mturk score, nl question answering stackexchange (lambert et al., 2023) implicit stackoverflow ranking translation wmt metrics shared task (freitag et al., 2022b) explicit pro translation workflow mqm, da ## ai feedback feedback models have been crucial in advancing generation techniques by effectively leveraging feedback. however, they are heavily reliant on human input: for example, @xcite found that across various preference model sizes, utilizing fewer than 1,000 comparisons resulted in only chance improvements. moreover, employing static feedback can make consistency challenging, causing changes in the model's output distribution. ai-generated feedback, an emerging research area, focuses on harnessing the llm's own abilities to enhance the model without human intervention. two primary approaches have emerged in this domain: self ai feedback the first approach involves using the same model to provide feedback and improve its output. in this scenario, the model engages in a continuous self-improvement process, learning from its evaluations and refining its capabilities accordingly. examples of this approach include prompting models to generate harmful responses and revising them for harmlessness @xcite , or employing rule-based reward models for rlhf fine-tuning (openai, 2023a). techniques such as iterative output revision through few-shot prompting @xcite @xcite @xcite have been explored using llms like gpt-3.5 @xcite and @xcite . notably, these techniques demonstrate potential when applied to llms trained to adhere to human instructions and align outputs with human preferences. this suggests that incorporating human feedback during training equips ai models to comprehend task requirements better, align outputs with directives, and function as dependable feedback mechanisms, thereby minimizing human intervention. the capacity to offer valuable ai feedback may depend on the model being trained with human feedback. external ai feedback this approach utilizes a separate feedback model to critique the task model's outputs @xcite @xcite @xcite . a key advantage is that the feedback model need not be large or general-purpose, making smaller feedback models an appealing option when abundant feedback is available. ## conclusion recent developments in large language models have emphasized the need for human feedback to ensure models have desirable behaviour and generate helpful and harmless text. we provide an overview of a recent line of research on leveraging (human) feedback to improve natural language generation. despite the relative infancy of this field, several important observations emerge when considering existing works: