Paper: Resolving Knowledge Conflicts in Large Language Models (ICLR-2024)

Reviewer comment (Presentation):
The most shorting is the orginize of the whole article.

Author response:
Thanks for your suggestion. Though we believe that grouping by tasks and grouping by content sections are both valid writing strategies, and we employ grouping by sections (settings, results, and proposed approaches) with subtitles of "Task1", "Task2", and "Task3" in each of them to help with the comparison on methodologies and results across different tasks, similar to other works on knowledge conflicts [1-2], we will clarify better the organization of the paper and emphasize on the separation of the conceptual design from the method in the final version.

Reviewer comment (Reproducibility):
In Task 1, it is unclear which kind of prompt was used in other baselines, like Few-shot prompting and Chain-of-Thought prompting (CoT). The method in Table.1 indicates using the prompt "Does the given context conflict with what you know? Yes/No." It is questioned whether the method includes some examples as demonstrations or just the mentioned prompt.

Author response:
We present all prompts in Appendix G. To reiterate: 

For the few-shot prompting, we used "*Does the given context conflict with what you know? Yes/No Examples: Context: Acetone is an organic compound and the simplest amide derived from acetic acid. It is a white crystalline solid that is soluble in water and commonly used as a solvent in laboratories. It is also utilized in the production of pharmaceuticals, dyes, and plastics. Answer: Yes Context: {context} Answer:*".

For chain-of-thought prompting, we used "*Does the given context conflict with what you know? Yes/No Examples: Context: Acetone is an organic compound and the simplest amide derived from acetic acid. It is a white crystalline solid that is soluble in water and commonly used as a solvent in laboratories. It is also utilized in the production of pharmaceuticals, dyes, and plastics. Answer: Based on my knowledge, Acetone is an organic compound; Acetone is a type of ketone instead of amide; Acetone is a colorless liquid instead of white solid; Acetone is soluble in water; Acetone is commonly used as a solvent in laboratories; Acetone is utilized in the production of acrylic plastics, polycarbonates and epoxy resins instead of dyes. The number of inconsistency is three, which is greater than zero, so the answer is Yes. Context: {context} Answer:*``.

Four in-context exemplars are included for few-shot prompting approaches (few-shot, CoT, GKP, self-ask, break-down, and ours) as documented in Section 3.2.

Reviewer comment (Experiments):
The method is simple but potentially useful; however, it is unclear how the authors plan to combine their method with existing prompting methods, like CoT, because in practice there are many multi-hop questions where it is not known in which step the knowledge conflict happens, similar to solving a math problem without knowing which conflicted knowledge leads to a wrong intermediate result.

Author response:
Thank you for acknowledging the usefulness of our method and for your relevant question! Indeed, our proposed approaches (as fully documented in Appendix G) do incorporate existing prompting methods, including the CoT. 

Regarding your concern on multi-hop questions, we conduct **an additional experiment** utilizing the two-hop dataset Compositional Celebrities [1]. We randomly choose 400 examples and replace the LLM responses to one of the sub-questions with a different answer from the in-context corpus, creating 400 positive examples and 400 negative examples. Employing the CoT prompting method results in a precision of 0.795, a recall of 0.995, and an F1 score of 0.884 in Task 1. This aligns with our conclusion that LLMs can perform well above random in identifying the existence of knowledge conflicts within contextual information, even under multi-hop settings.

|           |  n  |  TP |  TN |  FP |  FN |  Acc  | Precision | Recall |   F1  |
|-----------|:---:|:---:|:---:|:---:|:---:|:-----:|:---------:|:------:|:-----:|
| Zero-shot | 743 | 147 | 298 |  73 | 225 | 0.599 |   0.668   |  0.395 | 0.497 |
|  Few-shot | 796 | 164 | 224 | 174 | 234 | 0.487 |   0.485   |  0.412 | 0.446 |
|    CoT    | 756 | 373 | 285 |  96 |  2  | 0.870 |   0.795   |  0.995 | 0.884 |

[1] Press Ofir et al. Measuring and narrowing the compositionality gap in language models. EMNLP 2023, Findings.

Reviewer comment (Evaluation):
The hallucination is possible in LLM's answer. It seems that this is not well addressed in the paper.

Author response:
We appreciate your observation regarding the framework being limited to word-level knowledge edits and the potential for hallucinations in the LLM's answer. They are good future directions and we also consider and discuss them in Appendix B.

We agree that real-world knowledge conflicts might be more complex, but we found that **LLMs even fail to address these *easy* knowledge conflicts in our work generated by word-level edits**. We also admit that there is a chance of hallucination, but since our entity list is generated by LLMs (Section 2.1), it is biased towards entities already well-known to LLMs, reducing the likelihood of hallucination. Moreover, it is still debatable whether hallucinations should be considered as parametric knowledge, and our primary focus is on conflicts/knowledge differences instead of whether the parametric knowledge is factually accurate or not. The abilities of identifying the existence of knowledge conflicts, pinpointing specific conflicting segments, and generating distinct responses based on all conflicting information are the desiderata that we emphasize and evaluate in this work.

Reviewer comment (Theory):
Could the assumption of a single parametric knowledge answer be relaxed? How would results change?

Author response:
We conduct **an additional experiment** to explore the possibility of multiple parametric knowledge answers. We sample 600 questions (with conflicts) across different knowledge domains, and employ gpt-3.5-turbo to generate 3 answers in a zero-shot manner with a temperature of 1.5 to try to induce multiple parametric knowledge answers, following self-consistency [1]. We present the results in the following table:



|                       | ExactMatch | Synonyms | Invalid/Incorrect |
|-----------------------|------------|----------|-------------------|
| Mathematics           | 8          | 17       | 5                 |
| Biology               | 9          | 18       | 3                 |
| Chemistry             | 7          | 20       | 3                 |
| Physics               | 4          | 23       | 3                 |
| Psychology            | 6          | 20       | 4                 |
| Computer Science      | 7          | 21       | 2                 |
| History               | 11         | 18       | 1                 |
| Literature            | 15         | 14       | 1                 |
| Sociology             | 3          | 27       | 0                 |
| Economics             | 9          | 19       | 2                 |
| Political Science     | 10         | 16       | 4                 |
| Philosophy            | 5          | 21       | 4                 |
| Anthropology          | 6          | 24       | 0                 |
| Geology               | 8          | 17       | 5                 |
| Linguistics           | 1          | 25       | 4                 |
| Art History           | 12         | 10       | 8                 |
| Environmental Science | 8          | 19       | 3                 |
| Health Sciences       | 7          | 22       | 1                 |
| Communications        | 9          | 20       | 1                 |
| Music                 | 3          | 20       | 7                 |
| **Total**             | **148**    | **391**  | **61**            |

The results show that 25% of the additional parametric answers matched the top answer exactly with 65% of synonyms, while others tend to be invalid or incorrect answers. Therefore, we conclude that the vast majority of questions have only one parametric answer, thus the results and findings won’t change significantly if we relax this assumption.

[1] Wang, Xuezhi, et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2022.

Reviewer comment (Novelty):
The paper seems to lack a connection to previous works in the field of knowledge conflict. The organization of the entire article is isolated and does not introduce other benchmarks/analysis, such as DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering (ACL 2023), or other method comparisons that specifically address knowledge conflict issues.

Author response:
> Besides, this paper seems to lack a connection to previous works in the field of knowledge conflict. The organization of the entire article is isolated and does not introduce other benchmarks/analysis[1] or other method comparisons that specifically address knowledge conflict issues.
>
> [1] DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering. ACL 2023

Thank you for the pointer to DisentQA, which explores and analyzes the ability to disentangle parametric and contextual knowledge, and it is relevant to Task 3 in our study. DisentQA focuses on training data augmentation methods and relies on Natural Questions, while we focus on instruction-based methods and employ synthetic datasets generated by our proposed framework, as NQ has been reasonably addressed in existing works [1-2] and we want to refrain from assuming existing datasets as the parametric knowledge. We further introduce the entity shuffling approach which addresses the limitation of their counterfactual data augmentation process which can only be employed for questions whose answers are named entities, and break down experimental results by different knowledge domains and conflict generation methods. While there are differences between our work and DisentQA, it is a valuable reference and we will add it to the related work in the final version. 

While there are several works on simulating knowledge conflicts and analyzing the behavior of LLMs in the face of knowledge conflicts (as we cited in the paper), including DisentQA, we are the first to define the desiderata and propose concrete objectives towards resolving knowledge conflicts.

Reviewer comment (Experiments):
The limited size of the knowledge conflict dataset proposed in this paper makes the analysis unconvincing. For example, in Figure 4, the authors argue that the ability to tackle knowledge conflict varies across knowledge domains. However, according to Table 7 in the appendix, on average there are only about 100 test cases per domain, which is far from enough to claim that knowledge conflict varies across knowledge domains.

Author response:
> The limited size of the knowledge conflict dataset proposed in this paper makes the analysis unconvincing. Take Figure 4 as an example, the authors argue that the ability to tackle knowledge conflict varies across knowledge domains. However, according to Table 7 in the appendix, on average there are only about 100 test cases per domain, which I think is far from enough to claim that knowledge conflict varies across knowledge domains.

As shown in Table 7, we employ approximately 450 test cases per knowledge domain: the "Total" column is the amount of test cases in each knowledge domain. With 20 distinct knowledge domains, there are a total of 9,000 test cases in the experiments. This quantity should be sufficient for drawing meaningful conclusions and is higher than in similar works that study knowledge conflicts [3-4].

[1] Zhang, Hang et al. Poolingformer: Long Document Modeling with Pooling Attention. ICML, 2021.

[2] Xuguang Wang et al. No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehension. EMNLP 2020, Findings.

[3] Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge clashes. ArXiv, abs/2305.13300, 2023.

[4] Neeman Ella et al. DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering. ACL 2023.