System-Level Natural Language Feedback (2024)

Abstract: Natural language (NL) feedback offers rich insights into user experience. While existing studies focus on an instance-level approach, where feedback is used to refine specific examples, we introduce a framework for system-level use of NL feedback. We show how to use feedback to formalize system-level design decisions in a human-in-the-loop-process – in order to produce better models. In particular this is done through: (i) metric design for tasks; and (ii) language model prompt design for refining model responses. We conduct two case studies of this approach for improving search query and dialog response generation, demonstrating the effectiveness of system-level feedback. We show the combination of system-level and instance-level feedback brings further gains, and that human written instance-level feedback results in more grounded refinements than GPT-3.5 written ones, underlying the importance of human feedback for building systems.

## introduction users interacting with a machine learning system offer feedback, either actively or passively. the feedback can be binary ratings @xcite , preference feedback @xcite ) and natural language (nl) feedback @xcite . among them, nl feedback is the most general due to its free-form nature, as opposed to the limited choices in other feedback forms. hence, it is crucial to harness the potential of nl feedback to improve a system. existing research on nl feedback typically adopts one of two strategies. the first uses feedback as an auxiliary target in addition to the original task, just like in multitask learning @xcite . the second modifies the original output based on per-instance feedback. the system can either be fine-tuned with the new output @xcite or iteratively self-critique and self-refine at inference time @xcite . one common limitation of these studies is that they only focus on instance-level learning, where each feedback only serves the instance for which it was received. furthermore, they often assume the availability of feedback for each and every example, which is not practical in real-world scenarios, where feedback is often sparse. this paper asks the following question: can we aggregate instance-level nl feedback to make system-level design decisions that improve language generation systems? we answer this question by proposing a general framework for aggregating instance-level nl feedback. a set of criteria (i.e., system-level feedback) are first derived from instance-level feedback through a human-inthe-loop process involving clustering and summarization. those criteria then guide the design of instruction-following language model prompts to refine (i.e., correct) examples, and the development of metrics that align with users' needs. we conduct two case studies of the proposed framework on information-seeking dialog tasks where we improve both the query generator and the response generator of an internet-augmented dialog system. the experimental results point to the effectiveness of system-level feedback. our contributions are: learning from human feedback as language models increasingly integrate into people's daily life, aligning them with human needs becomes essential @xcite . as a result, researchers have been working on utilizing various human feedback, including preference feedback @xcite , binary feedback @xcite @xcite , nl feedback @xcite @xcite @xcite , and so on. so far, the use of nl feedback is relatively less explored, with most studies focusing on instance-level feedback where each instance receives its own feedback @xcite . in this work, we propose a general framework for deriving system-level feedback from instance-level feedback, and show the effectiveness of system-level feedback alone and its complementarity with instance-level feedback. ## experimental setup: dialog systems we study our framework within dialogue system deployment, a context where users naturally offer nl feedback, such as "that's not correct" for incorrect responses @xcite . our case studies focus on information-seeking dialogues, where users interact with dialog agents to obtain answers or relevant information @xcite . ## combining system-level feedback and instance-level feedback and instance-level feedback, i.e. both the desired criteria and the specific example-based feedback (see appendix a.2). we introduce three systems for comparison. the system does not use instance-level feedback. ## conclusion in this paper, we present a framework that harnesses system-level nl feedback. by using a set of instance-level feedback, we derive systemlevel feedback for refinement prompt engineering and metric design. we show the effectiveness of system-level feedback through two case studies: generating queries and formulating dialogue responses. we further combine system-level and instance-level feedback in the refinement data construction process, and observe that the resulting trained response generator makes considerable improvements versus either alone. finally, we explore the possibility of substituting instance-level human feedback with gpt-3.5 feedback. we find that human feedback stands out in capturing main issues, while gpt-3.5 feedback is lengthy and less focused.