Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback (2022)

Abstract: Large language models (LMs), while powerful, are not immune to mistakes, but can be difficult to retrain. Our goal is for an LM to continue to improve after deployment, without retraining, using feedback from the user. Our approach pairs an LM with (i) a growing memory of cases where the user identified an output error and provided general feedback on how to correct it (ii) a corrector model , trained to translate this general feedback into specific edits to repair the model output. Given a new, unseen input, our model can then use feedback from similar, past cases to repair output errors that may occur. We instantiate our approach using an existing, fixed model for script generation , that takes a goal (e.g., “bake a cake”) and generates a partially ordered sequence of actions to achieve that goal, sometimes containing errors. Our memory-enhanced system, , learns to apply user feedback to repair such errors (up to 30 points improvement), while making a start at avoiding similar past mistakes on new, unseen examples (up to 7 points improvement in a controlled setting). This is a first step towards strengthening deployed models, potentially broadening their utility. Our code and data is available at https://github.com/allenai/interscript

## introduction language models (lms) have achieved remarkable success on many tasks @xcite , but they are still prone to mistakes @xcite . correcting mistakes by retraining is not always easy due to the cost and/or unpredictability of how additional training data will change the model. instead, our goal is to allow users to correct such errors directly through interaction, without retraining -by giving corrective feedback on the model's output. our approach is to maintain a growing, dynamic memory of such figure 1 : given a frozen model b, we train a corrector model g to apply feedback from a user about errors made by the original model. in the example, b has generated a script with an error in, stating that "driving" and "getting in a car" can occur in any order (red box). the user provides general feedback ("get in a car before driving"), and g operationalizes this to generate a corrected graph (by predicting and applying a graph edit operation) in which "get in car" happens first (green box). the feedback is stored in a memory m so it can also be retrieved to repair similar, future errors. we consider the class of problems where the model's output is repairable, namely a structured output that is (typically) nearly correct, and fixable through a small number of edit operations. our system is general and admits a general graph based input, so in principle it applies to a large number of tasks. in this paper, we apply our approach to the task of script generation that provides a natural setting for users to critique, and has applications in smart assistants @xcite . we use an existing, fixed model: proscript @xcite that satisfies the constraint of the model's output to be repairable. proscript takes as input a goal to achieve (expressed in natural language), and outputs a partially ordered sequence of stepsa script -required to achieve that goal. our interest here is not in proscript itself, but in what to do when proscript's output contains an error. this instantiation of our approach is illustrated in figure 1 . here, proscript has generated a script x to achieve the goal "see an alligator", but the script contains an error: it states that the steps of "driving to the zoo" and "get in car" can be applied in any order. to repair this, the user provides the general feedback "get in a car before driving". the corrector model g then takes that feedback and the erroneous script, translates it into appropriate edit operations on the script, and applies those edits to generate a corrected script (y in figure 1 ). the feedback is stored in memory m so it can also be retrieved in the future. our system, fbnet, comprises the corrector module g, the memory m, and searching and writing operations. to train our system, we collect examples of bad outputs, general feedback, and specific edits that the feedback should translate to (section 4.2). this allows g to learn how to translate general feedback into specific edits to apply. pairing g with the memory m allows fbnet to repair new, unseen scripts containing similar errors to the one the user corrected. our approach loosely follows some early ai systems that maintained a memory of the output problems and how to fix them @xcite @xcite , but here, we use neural methods and interact with a user to provide corrective feedback. it also builds on the idea of allowing users to specify edits in natural language, e.g., nledit @xcite , except we use general user feedback (then translated to examplespecific edits by g) and add a memory so that feedback can also be automatically reused. we evaluate fbnet along two dimensions: (a) how well does fbnet interpret nl feedback? (b) how well can fbnet learn from prior mistakes? we find that (a) it uses nl feedback effectively to repair script errors, with +30% (absolute) improvement over a baseline that does not use feedback, and that (b) it makes a start at avoiding past mistakes (+7% (absolute) improvement in a controlled setting). although these results are only for a single deployment of our general approach, they suggest that memory-based architectures can help deployed models continue to improve with time, without retraining, potentially broadening their utility. ## related work there have been numerous approaches to using user feedback to improve model performance, including: in shrdlurn, the user provides feedback by identifying which of the system's alternative interpretations of a user command is correct @xcite . these methods all augment/replace the standard use of automated answer feedback (if available), e.g., testing whether a semantic parse correctly executes to the correct answer, e.g., @xcite , sometimes using unsupervised techniques to generate additional training data, e.g., bifi @xcite . our work expands on the above approaches in two important ways. first, users provide general feedback in nl, that can potentially be applied to multiple cases (rather than just correcting a specific instance). the corrector model g is trained to operationalize that advice in different ways for different examples appropriately, in contrast to (say) nledit where the user-provided specific corrective edits for a single example only. second, we use a feedback memory, allowing feedback to be reused. while adding external mem- (right) fbnet maintains a memory m of corrective feedback, and searches for feedback from prior queries with similar error intent as x using a retrieval function ω. x is then concatenated to the retrieved feedback to form the input to the corrector model g. users can also give new feedback which is added to m. in this work, we focus on script generation models that might generate an erroneous script which are correctable using an edit (feedback). ## experiments we empirically evaluate two questions: rq1. how well does fbnet interpret nl feedback? specifically, we measure how well fbnet can translate general feedback fb from a user into the correct repair edit on an imperfect script x. the main focus of rq1 is to test the performance of g in the pipeline (fig. 2 ) rq2. how well can fbnet learn from prior mistakes? we make the same measurement, but using feedback fb recalled from similar, previous examples. the main focus of rq2 is to test the performance of m and ω. metrics to compare the gold edit y e * and the generated edit y ê, we use standard metrics used to evaluate generated text. we report the following metrics: we report these metrics over the entire edit: em, bleu, rouge. the components of y e broadly follow a template: <edit @xmath0. this allows comparison of the location or edit type in y e * and y ê: em loc , bleu loc , rouge loc and em type , bleu type , rouge type baseline as baseline, we train a model that does not use any feedback (we call this, @xcite and is trained only with @xmath1. the language model used in this baseline and fbnet is the same (t5-xxl), allowing a meaningful comparison. ## scope in principle, we could apply fbnet to any task that satisfies the assumptions ( §3.2). however, our approach has some limitations in practice, several of which merit further detailed follow-up work. advancements in these directions would further increase the applicability of fbnet. still, there are several applications @xcite where our approach would currently apply in principle, or is easy to set up. ## summary our goal is to create a system that can continuously improve the structured output of a model. our approach is to train an error correction model that uses natural language (nl) feedback to correct errors in that output. we have presented the first step towards this goal, showing that an error correction module can learn to interpret nl feedback successfully, resulting in 40% fewer errors in script generation. we have also described ongoing work on the next step, namely adding a memory layer where human feedback is stored and later retrieved efficiently. together, these offer a possible path to systems that can continuously improve their output over time, with progressively less feedback and without retraining.