Let Me Teach You: Pedagogical Foundations of Feedback for Language Models (2024)

Abstract: Natural Language Feedback (NLF) is an increasingly popular mechanism for aligning Large Language Models (LLMs) to human preferences. Despite the diversity of the information it can convey, NLF methods are often hand-designed and arbitrary, with little systematic grounding. At the same time, research in learning sciences has long established several effective feedback models. In this opinion piece, we compile ideas from pedagogy to introduce FELT, a feedback framework for LLMs that outlines various characteristics of the feedback space, and a feedback content taxonomy based on these variables, providing a general mapping of the feedback space. In addition to streamlining NLF designs, FELT also brings out new, unexplored directions for research in NLF. We make our taxonomy available to the community, providing guides and examples for mapping our categorizations to future research.

## introduction the last few years introduced a new paradigm for finetuning large language models (llms): learning from human feedback @xcite @xcite @xcite @xcite to augment their capabilities beyond their pretraining @xcite @xcite . this alignment has yielded less toxic and harmful models @xcite that are preferred by users @xcite . whether learning from feedback is done by directly learning from human preferences @xcite @xcite @xcite , or with reinforcement learning from feedback (rlf) using human-curated (rlhf; @xcite @xcite or ai-generated (rlaif; figure 1 : connecting feedback research in nlp to foundations of feedback in the learning sciences. @xcite @xcite feedback, all variants have been shown to be successful in several metrics -from encouraging honest behaviors, to reducing toxicity, to being preferred by evaluators @xcite . other approaches, such as imitation learning @xcite @xcite , and feedback modeling @xcite @xcite @xcite have had similar success. feedback has thus emerged as an important source of information for model improvement and evaluation, directing them toward desired objectives and behaviors @xcite . despite the observed benefits of learning from feedback, a systematic study of what constitutes helpful feedback has so far remained absent. for example, rlf requires a reward model (rm) to be trained on numerical or ranking-based feedback data @xcite ) -a format limited in the amount of information it conveys @xcite . to counteract this limitation, works have begun exploring natural language feedback (nlf; @xcite @xcite . however, these works rely on "intuitive guesses" about what constitutes useful feedback -leading different works to explore distinct conceptualizations of nlf, preventing a systematic comparison. to establish a more concrete foundation for learning from feedback, we survey comprehensive studies of feedback from the field of learning sciences, which investigate feedback as an essential component of instruction and learning. we unify feedback approaches for nlp into a common framework, grounded on our surveyed pedagogical foundations (as summarized in figure 1 ). to this end, we first present the most relevant feedbackrelated models from the learning sciences ( §3). taking inspiration from these pedagogical models, we create a novel framework, felt, that expansively maps the various features of the llm feedback space to pedagogical foundations ( §4), and identify both dimensions studied by previous works as well as several important aspects of feedback that remain underexplored. we then focus on the least explored one, feedback content, and introduce a novel comprehensive taxonomy systematizing both the content and the delivery of natural language feedback ( §4.2), shining a light on the underspecification of current approaches to feedback, and suggesting promising areas of future research. contributions we (i) present a survey of pedagogical feedback formulations and models; (ii) organize the variables that influence feedback (and its processing) into a schematic framework, specifically adapted for llms; (iii) propose a general and extensive taxonomy of feedback content; and (iv) propose areas of future research based on gaps between our taxonomy and the current landscape of llm research on feedback. ## feedback in nlp in this section, we survey current conceptualizations and applications of feedback in nlp, before exploring the feedback models and perspectives developed in the domain of learning science ( §3). ## feedback in learning sciences to construct a comprehensive, theory-grounded model of feedback that addresses the limited exploration of nlf in nlp, we build off the work of @xcite , who conducted a systematic review of 15 relevant and influential works on feedback models research in education. we provide a brief overview of the key points of each of these works and use them to subsequently propose a framework for the feedback ecosystem ( §4) and a taxonomy for feedback content ( §4.2). ## felt: unifying the two worlds in the pedagogical research surveyed in the previous section ( §3), feedback emerges as both a complex ecosystem, and a rich, but systematized, information source. in this section, we consolidate different pedagogical feedback models, and conceptualize a novel feedback framework adapted for llms, felt (feedback, errors, learner, task), displayed in figure 2 . we briefly analyze each of its components, and dedicate a specific section ( §4.2) to the formalization of feedback content. ## conclusion we survey the most influential feedback models from the learning sciences, creating a novel framework, felt, as well as a taxonomy of natural language feedback (nlf). both enable the systematic exploration of feedback, allowing for objective conclusions on the impact of a piece of feedback and the optimization of nlf. beyond a survey of the space of natural language feedback, we explore how both felt and the fine-grained feedback content dimensions underlie many existing techniques in natural language feedback in ai, making specific recommendations for enhancing feedback formulations for a wide range of proposed topics.