Paper: Adaptive Chameleon  or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (ICLR-2024)

Reviewer comment (Novelty):
The authors may discuss the literature on LLM distraction with irrelevant contexts, for example, "Large Language Models Can Be Easily Distracted by Irrelevant Context, ICML 2023", when presenting results with irrelevant evidence. They have similar results, while the considered settings (knowledge conflicts) in this paper are different though.

Author response:
**Answer 2**:  Thank you for bringing this paper to our attention. We have read this work and agree that it provides insightful conclusions relevant to our study. In response to your suggestion, we have updated our submission and cited this paper in the irrelevant evidence part of Section 4.2. We appreciate your valuable recommendation.

Reviewer comment (Evaluation):
The last paragraph of Section 3.4 is unclear. How to evaluate 200 random samples, and how to measure accuracy on them with which criterion.

Author response:
**Answer 3**:  Thanks for the question. The rationale is the following: we hope to use a state-of-the-art NLI model to further improve the quality of the synthesized data. Before doing that, we need to evaluate the quality of the NLI model itself to see its fitness to this specific purpose. The 200 random examples are a manually labeled dataset for this evaluation:  We sample 200 of the generated examples (including both parametric memory and counter-memory) and manually annotate whether the example entails  the corresponding claim (memory answers and counter-answers).  
The labels are *supportive* (entailment in the NLI task) or *not supportive* (either neutral or contradiction in the NLI task). Then we evaluate the state-of-the-art NLI model over this dataset and calculate its accuracy. We have updated the evaluation details in the Appendix B.5.

Reviewer comment (Reproducibility):
The parametric knowledge LLMs output would have randomness. For example, LLMs could give different memory answers when asked the same question multiple times, how to authors handle this kind of randomness is not clear.

Author response:
A1: Yes, randomness is an worthnoting issue. To alleviate it, we conducted preliminary experiments to evaluate the stability of 10 different instructions and adopted the most stable one for follow-up experiments. Additionally, to ensure the reliability of the generated parametric knowledge, we further adopt the 'answer consistency' step to filter out the evidence that may be generated out of/influenced by the randomness, as illustrated in Figure 2.

Reviewer comment (Evaluation):
Beyond just investigating the textual output of LLMs, it would be interesting to see the LLM’s uncertainty when encountering knowledge conflicts.

Author response:
Thanks for the great suggestion. With Llama2-7B as a case study, we report the log probabilities for the token it generates, after normalizing over all three tokens representing memory answer, counter-answer, and uncertain.
Specifically, we mainly explore two scenarios:

Firstly, in the single-source setting where only counter-memory is presented, we sampled 1,000 examples that Llama2-7B gives a counter-answer. In the Figure shown in [Anonymous Link](https://anonymous.4open.science/r/iclr2024_rebuttal_1967-5ED8/1_hist_llama2-7b-1c.pdf), Llama2-7B shows high confidence when generating the counter-answer and 91.3% of examples have a memory answer probability of 95% or greater. **This demonstrates the high receptiveness to the external evidence, even when it conflicts with LLM's parametric memory.**

Secondly, in the multi-source scenario where two supportive and two contradictory pieces of evidence are presented, we sample 1,000 instances that Llama2-7B favors the counter-answer.
Figure in this [Anonymous Link](https://anonymous.4open.science/r/iclr2024_rebuttal_1967-5ED8/1_hist_llama2-7b-2p2c.pdf) shows that Llama2-7B is confident in its memory answer response, based on the token log probability. Specifically, 96.3\% of the examples show a log probability of 95\% or greater for the counter-answer.
**Both *high confidence* shown here and *high frequency* (65\%) shown in Table 6 of using memory-aligned evidence indicate the confirmation bias of LLMs.**

Thanks again for the suggestion, we have also updated this discussion in the Appendix A.2.

Reviewer comment (Novelty):
For LLMs would be distracted by irrelevant context part, I recommend citing this work: Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., ... & Zhou, D. (2023, July). Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning (pp. 31210-31227). PMLR.

Author response:
We have read this paper and find its insights quite compelling. Following your suggestion, we have updated our submission to include a citation to this work in the irrelevant evidence part of Section 4.2. Thank you for pointing out this relevant research.

Reviewer comment (Experiments):
One of the two main results of the paper “LLMs are highly receptive to external evidence if that is the only evidence, even when it conflicts with their parametric memory” is not well-supported in the paper. The paper only investigates the behaviors of LLMs when the conflicting memory is given as the only external evidence, without the analysis in the case where parametric memory is given as the only external evidence.

Author response:
**Answer 1:**  Thanks for the suggestion. In Step 4 of Figure 1 (Answer Consistency), to ensure the “firmness” of the parametric memory, we have filtered out the cases where an LLM doesn’t choose the memory answer when given the parametric memory as evidence. Therefore, the remaining examples are guaranteed to produce the memory answer when the “parametric memory is given as the only external evidence”, so we didn’t include that. If we add those filtered cases back to the final dataset and present the parametric memory as the only external evidence, the results are as follows:

  |            | PopQA | StrategyQA |
  | ---------- | ----- | ---------- |
  | ChatGPT    | 95.3% | 96.3%      |
  | GPT-4      | 96.1% | 97.4%      |
  | PaLM2      | 91.6% | 97.3%      |
  | Qwen-7B    | 94.6% | 94.4%      |
  | Lllama2-7B | 95.3% | 92.7%      |
  | llama2-70B | 97.7% | 99.3%      |
  | Vicuna-7B  | 87.6% | 93.1%      |
  | Vicuna-33B | 83.4% | 94.5%      |

Not too surprisingly, LLMs are highly receptive to the parametric memory (unfiltered) when it is presented as the only external evidence.

Reviewer comment (Theory):
In section 3.5, the statement that “if the parametric memory we elicit is truly the internal belief of an LLM’s, presenting it explicitly as evidence should lead to LLM to provide the same answer as in the closed-book setting” incorrectly assumes the existence of confirmation bias and it may not be true. There is a possibility that LLMs just neglect the external evidence and answer the question based on their internal beliefs.

Author response:
**Answer 3**: It seems that two different (though related) concepts are being conflated here: an LLM’s firm parametric memory and confirmation bias. Confirmation bias only applies when there are *multiple pieces of external evidence*, and it refers to the behavior of an entity that favors information conforming to its prior belief while ignoring contrary information. Eliciting the firm parametric memory is a prerequisite step for testing the existence of confirmation bias—if we don’t confidently know an LLM’s “prior belief” (parametric memory), the test of confirmation bias will be built on a shaky foundation. We do not assume the existence of confirmation bias. Our goal is to test its existence.

With the clarification, to directly respond to the question, if an LLM just answers the question based on its internal belief without considering the external evidence, then it becomes exactly the same situation as the close-book setting in Step 1. The LLM should simply give the same answer. Also, considering that we have already shown, under the single-evidence setting, that LLMs are highly receptive to external evidence even when it’s counter-memory, it’s highly unlikely that the LLM just ignores the external evidence (in this case its parametric memory). 

We would also like to echo the results in Q2, which show that even if we add these filtered examples back to the final dataset, the same level of confirmation bias still largely holds.

Reviewer comment (Evaluation):
The paper concludes that “the effectiveness of our generated counter-memory also shows that LLMs can generate convincing dis- or misinformation, sufficient to mislead even themselves”, while giving counter-answers does not necessarily mean LLMs are mislead. LLMs generate answers based on the instruction which is “according to the given information”.

Author response:
**Answer 5**: That’s an insightful thought. As detailed in Appendix Table C.9, to prevent LLMs from just doing reading comprehension (i.e., only using the given information) when presenting only parametric memory/counter-memory, we intentionally designed the instruction: “According to the given information *and your knowledge*, answer the question.” Thus, LLMs are encouraged to have a balanced consideration of both the provided external evidence and their internal beliefs.

Further, we include (1) the reading-comprehension-like instruction “According to the given information, answer the question” to explore if LLMs can faithfully use the given information only and (2) no instruction to explore how LLMs would perform without any explicit instruction. The additional experiments are based on 1000 randomly sampled examples.

The receptiveness of four LLMs given different instructions is shown in the table below. From this, we can observe:
1) Without any instruction, LLMs still utilize external evidence in most cases, indicating their inherent tendency to rely on external information.
2) When 'and your knowledge' is incorporated into the instruction, most LLMs actually accept less external evidence compared to the reading-comprehension instruction.
But **even with such an instruction that prevents LLMs from blindly using external evidence, LLMs still exhibit high receptiveness** — which is why we claimed they may be misled by dis- or misinformation.

  |           | w/ “and your knowledge” (result in paper) | w/o “and your knowledge” | w/o instruction |
  | --------- | ----------------------------------------- | ------------------------ | --------------- |
  | ChatGPT   | 90.1%                                     | 96.4%                    | 92.2%           |
  | GPT-4     | 87.1%                                     | 96.1%                    | 86.1%           |
  | PaLM2     | 82.8%                                     | 90.9%                    | 95.5%           |
  | Llama2-7B | 95.7%                                     | 89.1%                    | 87.0%           |

Reviewer comment (Presentation):
In Figure 2, it might be better to exclude the percentage of counter-answer, as showing both may draw attention to the comparison between the percentage of memory-answer and counter-answer instead of the existence of memory-answer.

Author response:
**Answer 7**:  Thanks for the suggestion. We would like to highlight that LLMs may give uncertain answers (i.e., abstention), as described in Section 3.2 and 3.6 in the paper.
Therefore, the sum of 'memory answer' and 'counter answer' may not be 100%.
So we present the ratios of both memory answer and counter-answer.

Reviewer comment (Experiments):
The counter-memory construction method, or the framework in general, is limited to question answering settings only, while knowledge conflicts may happen in other scenarios.

Author response:
**Answer 8**: Thanks for the suggestion. Our scenario is primarily focused on question answering and knowledge conflict is indeed prevalent across various domains. Given that QA is the most common way of interaction between human users and the widely-used tool-augmented LLMs (generative search engines) and the limited scope of one single research paper, we chose to investigate the QA scenario in a more comprehensive and thorough way. The main goal of this work is the comprehensive and controlled experiments of knowledge conflict in the most common scenario, as well as the discussions and important findings. Our construction method provides qualified counter-memory to serve  the main goal and we leave a better unified construction method as our future work.

Reviewer comment (Reproducibility):
Where do the same-type entities used for substitution come from?

Author response:
**Answer 9**: The same-type entities used for substitution in our study are derived from PopQA, which provides 16 relationship questions. The objects within the same relationship category are of consistent entity types. We have updated the details in the Appendix B.1.

Reviewer comment (Presentation):
Which dataset does Figure 2 employ? PopQA or StrategyQA or both?

Author response:
**Answer 10**: Figure 2 is based on the PopQA dataset. Given PopQA's entity-centric nature and the challenges associated with performing entity substitutions in StrategyQA, which lacks this focus, we opted to use PopQA for this analysis.

Reviewer comment (Writing):
This terminology of “counter-memory” conflicts with the parametric and non-parametric memory. Better to use a direct and more specific terminology.  
> We refer to external evidence that conflicts with parametric memory as counter-memory.

Author response:
**Answer 1:**  Thank you for your suggestion regarding terminology. We acknowledge the challenge in accurately and concisely describing the concept of "evidence conflicting with parametric memory." After considerable literature survey and discussion among the authors, we decided to choose the term "counter-memory", partly because its resemblances to "counter-factual." Meanwhile, we are very open for suggestions and would be happy to change to better alternatives.