Memory-assisted prompt editing to improve GPT -3 after deployment (2022)

Abstract: Large LMs such as GPT-3 are powerful, but can commit mistakes that are obvious to humans. For example, GPT-3 would mistakenly interpret “What word is similar to good?” to mean a homophone, while the user intended a synonym. Our goal is to effectively correct such errors via user interactions with the system but without retraining, which will be prohibitively costly. We pair GPT-3 with a growing memory of recorded cases where the model misunderstood the user’s intents, along with user feedback for clarification. Such a memory allows our system to produce enhanced prompts for any new query based on the user feedback for error correction on similar cases in the past. On four tasks (two lexical tasks, two advanced ethical reasoning tasks), we show how a (simulated) user can interactively teach a deployed GPT-3, substantially increasing its accuracy over the queries with different kinds of misunderstandings by the GPT-3. Our approach is a step towards the low-cost utility enhancement for very large pre-trained LMs.

## introduction language models are now better than ever before at generating realistic content, but still lack commonsense @xcite . one failure mode due to a lack of commonsense is in misunderstanding a user's intent. the typical remedy of retraining with more data is prohibitive due to the cost and infrastructure requirements. in such cases, even if users repeatedly observe the model making a mistake, there are no avenues to provide feedback to the model to make it more accurate and personalized over time. our goal is to allow users to correct such errors directly through interaction, and without retraining by injecting the knowledge required to correct the our memory enhanced gpt-3 implementation. user: what word is similar to good? gpt-3: the homophone of good is: wood. user: "similar to" means "with similar meaning". ## related work in @xcite , we show that using a memory of user feedback can be used to repair erroneous model in a supervised setting. in this work, we build upon the recent advances in few-shot prompting to modify gpt-3's behavior by adding user feedback to the query (prompt). like others, we use gpt-3 with few-shot prompting, where the prompt consists of a prefix pref ix containing a few inputoutput "training" examples of the task, followed by the input x, e.g., a question, to operate on. however, while prior work has focused on constructing better prefixes, e.g., dynamically selecting good "training" examples based on the question @xcite , or even representing the prefix latently @xcite , our work elaborates the input x itself to clarify the intended task, by adding user feedback f b from previous misunderstandings. similarly, our work can be seen as a form of retrieval-augmented qa. extensive prior work has used retrievals from a text corpus to aid qa, e.g., @xcite ; @xcite , or retrievals of prior qa pairs for nearest-neighbor qa @xcite . in contrast, we retrieve from a dynamic memory of user feedback. the idea of failure-driven reminding and dynamic memory date back several decades, e.g., @xcite . our work resurrects these ideas in a modern context. learning from instruction has become important for large lms that can perform a task based on direct instruction rather than examples @xcite . our work extends this by adding an adaptive component when those instructions are misinterpreted. while it may not be possible for a user to provide meaningful feedback on the output itself, giving feedback on the understanding of the instruction is more feasible. our approach aims to modify the model's behavior through prompting, given a wrong answer. an alternative, recently explored approach is "model editing" -updating the model itself by modifying its parameters to fix incorrect answers @xcite @xcite . model editing approaches have to date been limited due to uncontrollable out-of-scope changes @xcite . in contrast, our goal is not just to correct a prediction, but to generalize that correction for new problems by collecting feedback to clarify the misunderstanding without damaging the model's basic problem-solving acumen. finally, our work is a simple example of debugging and learning via dialog. while system debugging through dialogue has been explored in many contexts @xcite @xcite , our contribution is a dialogue about the model's understanding of the user's intent. ## experiments baselines we compare memprompt (memoryassisted prompt editing) with two baselines: • no-mem this is the standard gpt-3 4 in few-shot prompting mode (hyper-parameters listed in appendix §c). input is p # x i (i.e., question x i appended to prompt p). it generates answer y i and its understanding of the user's intent u i . the most recent subset of m of memory inserted is inserted in the prompt. the ethical reasoning tasks (ert) involve long examples, and the initial prompt itself takes close to the max allowed tokens. thus, the grow-prompt setup is only provided for the lexical relations and word scrambling tasks. ## conclusion we present memprompt, a novel, memoryenhanced gpt-3 that allows users to interact and improve the model without retraining. a key insight is to have the model articulate not just its answer but also its understanding of the user's intent, providing an avenue for feedback. we show that deployed systems with fixed large-language models can still be improved by interacting with end-users, potentially improving their performance and broadening their utility.