Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement (2022)

Abstract: Our goal is a teachable reasoning system for question-answering (QA), where a user can interact with faithful answer explanations, and correct its errors so that the system improves over time. Our approach is to augment a QA model with a dynamic memory of user feedback, containing user-supplied corrections toerroneous model beliefs that users identify during interaction. Retrievals from memory are used as additional context for QA, to help avoid previous mistakes in similar new situations - a novel application of memory-based continuous learning. With simulated feedback, we find that our system (called TeachMe) continually improves with time, and without model retraining, requiring feedback on only 25% of training examples to reach within 1% of the upper-bound (feedback on all examples). Similarly, in experiments with real users, we observe a similar trend, with performance improving by over 15% on a hidden test set after teaching. This suggests new opportunities for using frozen language models in an interactive setting where users can inspect, debug, and correct the model’s beliefs, leading to improved system’s performance over time.

## introduction our goal is a teachable question-answering (qa) system -one that a user can interact with to see faithful explanations for its answers, debug errors, and correct them so that the system gradually improves over time (sometimes referred to as explanatory interactive machine learning (xil) @xcite ). while the benefits of such a system are evident @xcite , the challenges are evident also: despite recent progress in explainability @xcite , it is often hard to understand how a model arrived at an answer, and even harder to correct it if it made given a new question, facts retrieved from memory are used as additional context for the model, influencing its answers and proofs. (b) if the user disagrees with an answer, they localize the error in the explanation and offer corrective feedback, which is added to memory. (c) these new facts can then be retrieved if the query is re-asked, helping the system avoid repeating mistakes. note that these also help improve answers on new, similar questions that are asked later, helping the system improve over time. a mistake. in contrast, people are typically able to provide a chain of reasoning for their decisions, and may change their mind if a flaw in their knowledge or reasoning is exposed. our goal is to similarly have machines provide reasoned answers to questions, showing how the answer follows from its internal knowledge (and possibly externally available information), and where it is capable of changing its answer if errors in that knowledge are identified. our approach has three components. first, the system produces answers supported by an entailment-based chain of reasoning, showing how the answer follows from the system's own internal beliefs 2 . second, if an answer is wrong, a user can inspect the reasoning to diagnose and correct the failure. for example in figure 1 , the system incorrectly concludes that "a magnet can pick up a penny" from its over-general (false) belief that "metals are magnetic". the user can thus correct the mistake by asserting that "not all metals are magnetic", in particular copper. finally, to store and apply the user's feedback, we augment the model with a dynamic memory. given a new question (or re-asking an old question), teachme retrieves usersupplied facts from this memory. these are then used as context while generating an entailmentsupported answer to the question, e.g., step (c) in figure 1 . this helps override prior, erroneous model beliefs, thus biasing teachme to avoid similar mistakes in future -a novel application of memory-based continual learning to belief maintenance, in which the model itself remains fixed (frozen) and retraining is not required. we evaluate teachme using both simulated and real user feedback. with simulated feedback, using two existing datasets obqa @xcite and quartz @xcite , we find that teachme is able to continuously improve with time, without retraining, requiring only a quarter of the feedback annotations available in the original dataset to reach within 1% of the upper-bound (using all gold annotations). similarly with real users, we find that after they interact with teachme on a small set of questions, the system's performance on a hidden test set similarly improves (by over 15%) without retraining. our contributions are thus: ## related work guiding frozen language models and memory: our use of context to modify a (run-time) frozen model's behavior is similar to retrieval-based qa @xcite , where retrieved context can improve qa performance. in our case, however, retrieval is from a dynamic memory of user-supplied facts, rather than a static corpus, the memory serving to expand and override model beliefs. it also can be seen as a form of prompt engineering @xcite , except using relevant facts rather than few-shot qa examples, and with novelty on the interactive collection and management of those facts. teachme's memory-based feedback is inspired by the feedback mechanism of beliefbank @xcite , in which retrieved memories were similarly used as context to guide future qa. in be-liefbank, however, memories were previous system answers, without any mechanism for explaining its reasonong nor being corrected by a user. in contrast, teachme's memories are provided by a user, identified through interaction with system explanations. teachme's memory is also related to work by @xcite where user feedback memories were used but in different ways, namely to repair erroneous model outputs via post-processing @xcite , or to clarify user intent in gpt3 prompts @xcite . in contrast, teachme's feedback contains corrections and elaborations to the model's internal beliefs themselves. more generally, while the idea of memory for improved performance is not new, our way of using memory is novel: to the best of our knowledge, teachme is the first system that allows a user to find, extend, and correct its reasoning errors, and the memory allows the resulting system to improve over time (continual learning). feedback and interaction: interaction has been successfully used to learn in interactive recommender systems, e.g., @xcite , conversational systems, e.g., blenderbot @xcite , knowledge graphs @xcite , and procedural tasks @xcite . interaction has also been used for data augmentation, by having users identify model biases and provide additional corrective training examples to reduce those biases @xcite . in contrast, our work focuses on learning corrective feedback in the context of reasoning. early ai attempts at having users debug rule-based representations had limited success, e.g., teiresias @xcite , convince @xcite . our work can be viewed as a modern formulation of this goal, using linguistic expressions of the knowledge stored latently in a model. continual learning: finally, our system performs a kind of continual learning @xcite , aiming to correct specific errors that appear. recent work has explored "model editing" -editing model parameters to fix incorrect answers or add new knowledge @xcite @xcite . however, to date these approaches have only been demonstrated in a limited context (e.g., correcting a single error), and even then can lead to uncontrollable out-of-scope changes @xcite . in contrast, our goal is not just to correct a specific error, but to have that correction generalize to new problems, and without damaging the model's basic problem-solving acumen. thus, our work leaves the model fixed, and seeks improvement in the broader system in which the model is embedded, exploring an alternative and potentially more interpretable architecture towards this goal. ## approach we adopt a question-centric approach to teaching and interaction, in which the user (teacher) asks the system (student) a question that they know the answer to, to probe the system's knowledge. the system then answers it along with a faithful entailmentbased explanation. if the system's answer is wrong, the user can interact with the explanation to identify the erroneous system beliefs that lead to the incorrect answer, and correct them. corrections are stored in a dynamic memory used to influence, and ideally improve, future system behavior. we instantiate this approach in a system called teachme, which has three key components: 1. answering questions: given a user's question, teachme searches for an entailmentbased line of reasoning for different candidate answers, and selects the best. 2. interaction: the user can inspect, locate, and correct errors in the system beliefs that led to incorrect answers. 3. dynamic memory: teachme maintains a dynamic memory of user-corrected beliefs, used to help answer future questions. we now describe each in turn. given a question, teachme generates multiple answers and proofs, discards those not consistent with its own beliefs (verification), and presents the best to the user @xcite . if the answer is wrong, the user interacts to identify erroneous model beliefs, and add corrections to memory, which in turn modifies future qa behavior without model retraining. ## experiments and results our goal is that teachme's memory-augmented architecture will allow users to teach the system in a general way, adding to and correcting model beliefs so that its performance improves on new, unseen questions. to evaluate this, we use both both simulated and real users. in both cases, users first provide feedback on a set of training questions, populating the memory. then, with no further interaction, we measure whether teachme's performance has improved on a set of hidden test questions. in all cases, teachme's model is frozenany improvements are purely via memory updates. algorithm 1 teachme's overall control algorithm 1: procedure answer(q: question, a: answer choices, m : memory of useful facts ) 2: @xmath0: @xmath1, @xmath2,a) 4: for @xmath3: // generate a hypothesis hi for each choice ai 6: @xmath4,ai) 7: for @xmath5: // 1. generate a proof 9: generate a proof (pi ⊢ hi) of hi with first 10: @xmath6.1.2) † 11: // 2. add the proof to the list of proofs so far: 12: @xmath7, s(hi), @xmath8: <e best , score best , a @xmath9: