Paper: Ask Again, Then Fail: Large Language Models’ Vacillations in Judgement (ICLR-2024)

Reviewer comment (Writing):
I didn't understand the second sentence in footnote 1.

Author response:
We are sorry for the ambiguity. When generating a response, the model usually provides a lot of thought process, ending with an answer. However, there is currently no particularly effective way to automatically evaluate these intermediate thought processes, so we can only assess the model based on the final answer it provides. To enable automated evaluation, we instruct the model to output the final result in a specified format (i.e., Answer:). We hope our response is helpful to you.

Reviewer comment (Writing):
Modification Rate (M. Rate) was not clear to me.

Author response:
Sorry for the confusion. We hope to explain the concept of Modification Rate (M. Rate) through an example. Suppose there is an evaluation test set with 1000 samples, and the model answered 10 correctly in the initial question and answer. We continue to ask follow-up questions for these 10 samples, and after the follow-up question, the model only answered 5 samples correctly. So M. = 10/1000 - 5/1000 = 5%, and M. Rate = (10 - 5) / 10 = 50%.

The rationale for employing both M. and M. Rate to assess the judgement consistency of LLMs primarily stems from the fact that in scenarios where initial performance is poor, the potential for further decrease in model performance is constrained. Consequently, relying solely on M. might not provide an accurate reflection of the model's judgement consistency.
For example, in the above example, although the model's performance only decreased by 5% after follow-up question, 50% of the samples answered correctly in the first round were answered incorrectly in the second round, indicating that the model's judgement consistency is low. Therefore, considering these two indicators together can provide a more accurate and comprehensive reflection of the model's judgement consistency.

Thank you for your valuable feedback. We hope our response has resolved your confusion.

Reviewer comment (Experiments):
**Limited Scope of LLMs**: The study evaluates a specific set of LLMs (ChatGPT, PaLM2-Bison, and Vicuna-13B), potentially limiting the generalizability of the findings to other models in the rapidly evolving landscape of conversational AI.

Author response:
Thank you for the insightful comment! We will address your concerns as follows:

> Q1: Considering the rapid advancements in AI technologies, how might the results differ when applied to newer or upcoming LLMs? Is there room for future research to address this limitation?

A1 (also response to weakness 1): 
Your concern is indeed very necessary. Considering the rapid development of large language models, the latest LLMs may have improvements in various aspects, and we also believe it is necessary to explore whether this issue remains universal on the latest LLMs. With limited computing resources, we have evaluated the judgement consistency of several of the latest and most capable closed-source and open-source models, such as `GPT-4-1106-preview`[1], `UltraLM-13B-v2.0`[2], `XwinLM-13B-v0.2`[3], and `Zephyr-7B-Beta`[4], on the benchmarks MultiArith, StrategyQA, and CoinFlip, as per the experimental setup in the paper. We report the experimental results below.

The experimental results show that even the most advanced LLMs generally exhibit noticeable fluctuations in judgement consistency when faced with user questioning, negation, or misleading inputs. Consequently, we posit that this challenge will persist in the realm of LLMs, even with the advent of newer, more advanced models in the future. This issue is universal across all LLMs and is currently underemphasized, which underscores the importance of our research. Given this context, it is unlikely that newly developed models will be able to fully address these challenges in the near term.


*Note 1: We chose models based on [AplacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) rankings and our computational resources we could afford.*

*Note 2: Due to the costs associated with calling the GPT-4 API, we only sampled 100 samples from the test sets of each of the three datasets for evaluating the judgement consistency of GPT-4. For all other models, the number of samples used for evaluation strictly adhered to the evaluation settings outlined in our paper.*

[1] https://openai.com/blog/new-models-and-developer-products-announced-at-devday

[2] https://huggingface.co/openbmb/UltraLM-13b-v2.0

[3] https://huggingface.co/Xwin-LM/Xwin-LM-13B-V0.2

[4] https://huggingface.co/HuggingFaceH4/zephyr-7b-beta

Reviewer comment (Evaluation):
**Lack of Real-World Application**: The research focuses on theoretical evaluation and proposed mechanisms; it would strengthen its impact by discussing practical implications and real-world applications of the proposed solutions.

Author response:
> Q2: Can you provide insights into how the proposed mechanisms and solutions could be practically applied in real-world scenarios, especially in fields where LLMs are extensively used, such as customer support or healthcare?

A2 (also response to weakness 3): Thank you for your constructive feedback. We agree that discussing how this mechanism can be integrated with practical applications can indeed help strengthen the impact of our research.

Currently, LLMs mainly appear as virtual assistants in real life. Considering that they may be questioned by users or have disagreements with users during the interaction process, we believe it is necessary to use the mechanism we proposed to evaluate the model's judgement consistency in the face of interference before they are officially put into use. If the judgement consistency is low, the mitigation methods in the paper can be considered to improve their judgement consistency in the face of interference to some extent. This can not only enhance the user experience and satisfaction but also improve the reliability of the model-generated content in some fields where virtual assistants participate in actual decision-making. 

Here are some potential impacts and applications of our proposed mechanism and mitigation methods in real-life scenarios:

- **Customer Support**: LLMs are widely used as virtual bots in the customer support field, primarily for answering user questions, solving problems, and providing advice. In this process, users may question the bot's responses or disagree with the bot-generated answers. For this application scenario, the quality assurance and monitoring team of virtual bots can use our proposed mechanism to evaluate the judgement consistency of customer support virtual bots when facing user interference. After comprehensive and reliable analysis of the results, the development team can implicitly concatenate the mitigation methods from the paper as model input after the user's question to improve the judgement consistency of virtual bots when facing interference, thereby enhancing the quality and reliability of customer support services and increasing user satisfaction and trust.

- **Healthcare**: LLMs can serve as virtual assistants in healthcare, assisting in areas such as diagnosis, medical image review, and drug development. For example, when an LLM serves as a virtual medical assistant in reviewing medical images and submitting the results to doctors for diagnosis, our proposed mechanism can be used to repeatedly evaluate the consistency of its judgements with different interference questions. If the consistency reaches a preset threshold, the judgement can be submitted as auxiliary material to the doctor; otherwise, we may reasonably suspect that the judgement's reliability is low, and the mitigation methods from the paper can be used to improve judgement consistency. If the consistency still fails to meet the preset threshold after applying the mitigation methods, the patient case can be marked to remind the doctor to exercise caution. It is important to note that the final judgement should be made by the doctor, and the judgements and recommendations provided by the virtual medical assistant serve as reference information to support the doctor.

It is important to note that although our proposed mechanism and mitigation methods can assess and improve the model's judgement consistency to some extent, considering the complexity of real-world situations and the high requirements for consistency in some application scenarios, more efforts are needed in model training strategies and more comprehensive evaluations in the future to completely solve this problem. 

Thank you for your insightful suggestions, and we hope our response has been helpful to you.

Reviewer comment (Novelty):
The overall novelty of this work is a bit limited given that prior work (many of which are also cited by the authors) has investigated the "sycophantic" behavior of LLMs, and the proposed methods in the paper are quite similar to the ones in prior work. For example, the paper by [Turpin et al.] which the authors seem to miss studies LLM's behavior when there exists bias in the context, where one of the settings is exactly about putting human user's belief (in a wrong answer) in the context, which is close to the type L (leading questions) prompt explored in this paper. Similar findings are also present in [Perez et al., 2022] as cited. [Wang et al., 2023a] as cited explores using another conversational LLM conditioned on a wrong solution to engage in a debate with the original LLM; the "follow-up" responses by the simulated user there also share many similarities with the ones proposed (expressing disagreement, doubt, different opinions, etc.).

Author response:
Thank you for the valuable feedback!

>Q1 (from weakness 1): The overall novelty of this work is a bit limited given that prior work (many of which are also cited by the authors) has investigated the "sycophantic" behavior of LLMs, and the proposed methods in the paper are quite similar to the ones in prior work. For example, the paper by [Turpin et al.] which the authors seem to miss studies LLM's behavior when there exists bias in the context, where one of the settings is exactly about putting human user's belief (in a wrong answer) in the context, which is close to the type L (leading questions) prompt explored in this paper. Similar findings are also present in [Perez et al., 2022] as cited. [Wang et al., 2023a] as cited explores using another conversational LLM conditioned on a wrong solution to engage in a debate with the original LLM; the "follow-up" responses by the simulated user there also share many similarities with the ones proposed (expressing disagreement, doubt, different opinions, etc.).

A1: Sorry for the oversight. We will add [Turpin et al.] to the related work, and we appreciate your detailed and friendly reminder.

Thank you for your insightful comments. Although our work indeed intersects with several studies you mentioned regarding the reliability of large language models, it distinguishes itself in several key aspects:

- **Novel Research Perspective.** Unlike [Wang et al., 2023a], which designs debate-like dialogues with invalid solutions for each sample, and [Turpin et al., 2023], which introduces bias features into model inputs for multiple-choice questions (like modifying the order of options), our Follow-up Questioning Mechanism is closer to scenarios that ordinary users might encounter in real-life use of LLMs. Furthermore, the simpler and more conversational follow-up questions in our mechanism are more general and in line with the habits of everyday users than the templates or methods designed for each sample in other approaches.

- **Comprehensive Scenario Design.** As you mentioned, the research methods of the related work [Wang et al., 2023a][Turpin et al., 2023] only resemble one type of question in our proposed Follow-up Questioning Mechanism (leading questions), neglecting questioning and negation, common dialogue scenes in interactions between users and LLMs. Moreover, our experimental results show that questioning and negation also cause significant fluctuations in the judgement consistency of LLMs when facing interference.

- **Beyond Sycophancy, Further Discoveries.** Our study not only corroborates the sycophantic behavior mentioned by [Perez et al., 2022] but also reveals a new finding: **the model may become cautious and neutral in the face of interference**, a behavior not extensively covered in previous studies. As analyzed in the Error Analysis (refer to pages 6 to 7), we categorized errors into four types through human observation, where error categories 2, 3, and 4 can be attributed to sycophancy. However, it's worth noting the existence of category 1 (Unable to answer), refer to Figure 5 in the paper. In cases of error category 1, the model opts for a cautious and neutral stance, avoiding direct answers. This behavior is crucial for understanding the practical usability of LLMs, as it reflects an attitude distinctly different from sycophancy when faced with challenges, negations, or misleading information, rather than just flattery.

Our research aims to demonstrate through comprehensive analysis that conversational large language models show unreliable judgement when faced with disturbances like questioning, negation, and misleading. Although our study shares some thematic overlaps with the cited works, it contributes new perspectives and insights into the reliability and practical application of LLMs. Together with these insightful related studies, our work is vital for the future development and real-world deployment of LLMs.

We hope our clarification can address your concerns.