R efute B ench: Evaluating Refuting Instruction-Following for Large Language Models (2024)

Abstract: The application scope of large language models (LLMs) is increasingly expanding. In practical use, users might provide feedback based on the model’s output, hoping for a responsive model that can complete responses according to their feedback. Whether the model can appropriately respond to users’ refuting feedback and consistently follow through with execution has not been thoroughly analyzed. In light of this, this paper proposes a comprehensive benchmark, RefuteBench , covering tasks such as question answering, machine translation, and email writing. The evaluation aims to assess whether models can positively accept feedback in form of refuting instructions and whether they can consistently adhere to user demands throughout the conversation. We conduct evaluations on numerous LLMs and find that LLMs are stubborn, i.e. exhibit inclination to their internal knowledge, often failing to comply with user feedback. Additionally, as the length of the conversation increases, models gradually forget the user’s stated feedback and roll back to their own responses. We further propose a recall-and-repeat prompts as a simple and effective way to enhance the model’s responsiveness to feedback.

## introduction the advent of large language models (llms) has ushered in transformative advances in natural language processing, enabling a wide array of applications that leverage their generative capabilities. these models are designed to interact with users through multiple rounds of instruction and responses @xcite @xcite . one significant advantage of such multi-round interaction is the query-responsefeedback pipeline, where the user first poses a query, checks llms' responses, and provides feedback for llms to improve. such a paradigm has facilitated various techniques such as self-correct, self-refine, and multi-agent debate @xcite @xcite @xcite . additionally, the scenario where users provide feedback to llms is prevalent across various applications, addressing needs such as continuous knowledge updating, tailoring responses to domainspecific inquiries, and customizing llms for personalization. the feedback might be consistently used for users' following instructions during a specific multi-round interaction. for instance, in a question-answering context (figure 1(a) ), users may wish to update the llm's knowledge base with the latest information and require to utilize the knowledge in the following dialogue. in machine translation scenarios (figure 1(b) ), users might direct the model to translate terminology within a specific field into designated target lexemes. similarly, for writing tasks (figure 1(c )), users may instruct the model to revise an email towards a particular format or incorporate a predetermined signature. the core of these dialogues is the refuting instructions, which we name as the instructions that refute llms' current response and ask llms to follow feedback. even though efforts @xcite @xcite have been devoted to evaluating how llms can respond to instructions, the extent to which llms are amenable to these refuting instructions remains an open question. to address this issue, we introduce a novel benchmark, designated as refutebench. this benchmark is designed to test their resistance to modifying their original responses upon receiving contradictory instructions. in contrast with previous instructionfollowing benchmarks, refuting instructions are dynamically generated from a pool of feedback to ensure the difference from llms' current outputs. refutebench reflects both controlled and real-world settings by considering single-feedback and multi-feedback interactions, respectively, to capture the complexity of human-agent dialogues. in the single-feedback setting, the llm is refuted only once immediately after its response to the initial query (see figure 1 ). after the llm responds to the refuting instruction, we further provide a verification query to test whether the llm can apply the feedback. we consider two types of verification queries. one is the memorization query, which is the same as the initial query. another one is the generalization query, a query that is different from the initial query, but also viable for the same feedback. in the multi-feedback setting, we simulate real-world applications where the user provides multiple queries and multiple refuting instructions through interaction with llms. the queries are selected to be in the same domain and each one could be a generalization query for previous queries. our evaluation considers three representative tasks, as shown in figure 1 . by carefully benchmarking seven instruction-finetuned models from both close-source and open-source over tasks, we have the following findings: (1) generally, all evaluated llms demonstrate a tendency to adhere to their pre-existing knowledge to a certain degree. notably, gpt-4 and claude-2 exhibit the highest flexibility, whereas other robust models, such as chatgpt and mistral-7b-instruct-v0.2, display significant resistance to change; (2) it is challenging for llms to apply the feedback to generalization queries. compared to the memorization query, the generalization query leads to 10% to 20% per-formance degradation; (3) all evaluated models gradually forget the feedback and fall back to their internal knowledge as the dialogue proceeds. (4) whether the models accept the user's feedback at immediate response is crucial for the following application of the feedback. based on the observations, we further propose a simple and effective strategy, recall-and-repeat, to address the above-mentioned issue. by finetuning the multilingual bert (kenton and toutanova, 2019) to classify which user instruction contains feedback and is viable for the current query, we retrieve the most relevant feedback and design a prompt that asks the model to first confirm and then repeat the feedback. the experimental results show that our method strongly improves llms' response rate. to our knowledge, we are the first to propose a benchmark for refuting instructions, analyze different llms, and introduce a principled solution. we release our dataset and codes to facilitate future work foot_0 . with regard to stubbornness, xie et al. (2024) evaluates the stubbornness of llms, from the perspective of conflicts between models' parametric memory and external evidence. different from their work, we focus on a continuous interaction scenario and the stubbornness regarding models' acceptance of user feedback. ## problem definition here, we give a formal definition of refuting instructions and our evaluation settings. single-feedback first, we consider a clean setting. given a query q and a llm m, the initial response is defined as r 1 = m(q). as the initial response r 1 may not fulfill users' needs, the user can now provide a refuting instruction f to illustrate his feedback. thus far, the immediate llm response after feedback is modeled by r 2 = m(q; r 1 ; f ). note that the immediate model response after feedback could be responses like "i understand.". hence, with r 2 , we evaluate the feedback acceptance (fa) that measures whether the model positively accepts or adopts the feedback. then, we apply a verification query, q, to test whether the llm can apply the user's feedback. there are two types of verification queries as discussed before, the memorization query qfor and the generalization query qgen . the memorization query is the same as the initial query q, aiming to evaluate whether the model memorizes the user's requirement, while the generalization query is different from the original query but also fits the requirement of the refuting instruction f . a concrete example is in machine translation when the user asks to translate the word "apple" to 苹果公司 (apple company in chinese)", the generalization query could be another source sentence that also contains the word "apple" in it. we detail how we choose the generalization query in section 4 for each task. then, based on q, the llms response is given by r 3 = m(q; r 1 ; f ; r 2 ; q). a matching metric d is used to evaluate whether the refuting instruction f is followed by response r 3 , @xmath0, r 3 ). we refer to the setting as the single-feedback setting, as we provide the feedback only once. to isolate the effect of different llm's capabilities on downstream tasks, we ensure that the provided feedback is not fulfilled by the model's initial response. a set of candidate feedback is prepared, and feedback already accomplished by the llm is filtered. in this way, each llm is asked to edit their response exactly once, and our evaluation is dynamic. multi-feedback additionally, we introduce the multi-feedback setting that is more challenging and closely aligned with the real-world interaction with llms. taking machine translation as an example, a real-world example of multi-feedback is when a user is repeatedly querying the llm to translate sentences, possibly from a document or a similar domain. during the process, the user gives feedback when the response is not unsatisfactory. different from the single-feedback setting, there might be multiple instructions among rounds of interactions, and each instruction might contain several feedback requirements, e.g., several words to be translated to certain target language words in machine translation. formally, a turn of interactions with llm contains the following four steps: (1) the user makes a query q; ## data collection our evaluation considers three tasks, question answering (qa), machine translation (mt), and email writing. each one of them represents one important capability of llms, knowledge retention, multilingual comprehension, and writing proficiency. the statistics of all three tasks are in ## analysis 6.1 correlation between fa and rr in the previous section, a strange observation is that 'strong' models like chatgpt and mitral-7b-instruct-v0.2 achieve low scores in our benchmark. through fa scores, we find that these models tend to reject our requests immediately after feedback and then stick to their internal knowledge in the following response, while models like gpt-4 and claude-2 are more flexible and listen to our requests. thus, we propose a hypothesis that if a model rejects the requests in the first place, it cannot respond accordingly. we plot both the rr and fa of qa in figure 2 , where the setting is memory and con-@xmath1.2). we observe that the metrics of rr and fa are positively correlated, where the pearson correlation coefficient is 0.92, 0.58, and 0.68 in the tasks of qa, mt, and email writing, respectively. the performance indicates that when the model can positively accept the feedback, it can generate the response in further queries with a higher probability. it also implies that if the model cannot accept the responses, it does not tend to memorize the information and is stubbornly relies on it internal information. for example, mistral-7b-instruct-v0.2 is strongly stubborn in the qa task and only accepts the feedback in 8.0%, and it achieves rr in low value (15.0%). interestingly, when chatgpt carries out the mt task, although it can accept the feedback in the response, it does not insist on the feedback information, and answer the same question in its knowledge. the correlation between rr and fa indicates that if we can instruct the llms to positively accept the feedback information, it can further perform more satisfactorily in further related queries and instructions. ## conclusion in this paper, we proposed a benchmark, re-futebench, that focuses on evaluating how stubborn llms are through multi-round interactions. we designed refuting instructions from knowledge, multilinguality, and writing to comprehensively test models' performance. we observed that llms demonstrate the tendency to adhere to their internal knowledge. a simple and effective method was further proposed to improve the response rate to the refuting instruction.