Paper: Towards Understanding Factual Knowledge of Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Some points in the main results could be verified through better experiment settings, such as the point that "LLMs without instruction tuning underperform those with instruction tuning by 16.0%". It's better to conduct a peer-to-peer comparison since the instruction tuning is not the only factor of the two groups. For example, we can compare LLaMA with Alpaca and Vicuna, OPT vs. OPT-IML, T5 vs. Flan-T5, BLOOM vs. BLOOMz. I am not saying that we need to compare all of them, but the comparison could be made between the instruction tuning model and its backbone.

Author response:
Thank you for your constructive feedback. As you recommended, we have conducted additional analyses to compare models with their instruction-tuned counterparts in Appendix A.9.

The comparisons between LLaMA and its instruction-tuned versions, Alpaca and Vicuna, can be found in Table 2. Furthermore, we have conducted extra tests under the few-shots with CoT setting for T5-11B vs. Flan-T5-11B and BLOOM-6.7B vs. BLOOMz-6.7B. For T5, the accuracy was 18.6%, and the Macro F1 was 25.2%. In contrast, as shown in Table 2, Flan-T5 achieved an accuracy of 38.4% and a Macro F1 of 38.4%. Similarly, BLOOM’s performance was at an accuracy of 6.6% and Macro F1 of 12.2%, whereas BLOOMz showed a marked improvement with an accuracy of 27.5% and a Macro F1 of 27.7%.
|             | Multifaceted |      | Structural |      | Adversarial |      | Temporal |      | Real-World |      | Domain-Specific |      | Multi-lingual |      | Overall |      |
| :---------: | ------------ | ---- | ---------- | ---- | ----------- | ---- | -------- | ---- | ---------- | ---- | --------------- | ---- | ------------- | ---- | ------- | ---- |
|             | ACC.         | F1   | ACC.       | F1   | ACC.        | F1   | ACC.     | F1   | ACC.       | F1   | ACC.            | F1   | ACC.          | F1   | ACC.    | F1   |
|  LLaMA-7B   | 38.3         | 33.9 | 44.1       | 32.1 | 43.2        | 46.1 | 41.6     | 30.0 | 26.4       | 26.3 | 23.6            | 25.0 | 27.8          | 27.7 | 35.3    | 31.4 |
|  Alpaca-7B  | 38.6         | 28.8 | 48.0       | 23.6 | 46.4        | 35.1 | 49.6     | 26.1 | 24.5       | 19.9 | 42.9            | 26.8 | 24.2          | 17.7 | 39.4    | 26.2 |
|  Vicuna-7B  | 44.2         | 36.0 | 49.7       | 36.3 | 59.0        | 59.2 | 50.1     | 37.6 | 49.0       | 41.8 | 44.3            | 38.6 | 46.7          | 43.1 | 48.5    | 40.6 |
|   T5-11B    | 26.3         | 31.3 | 29.2       | 27.6 | 16.5        | 25.1 | 24.9     | 28.5 | 2.2        | 2.9  | 3.9             | 7.3  | 27.7          | 18.1 | 18.6    | 25.2 |
| Flan-T5-11B | 49.2         | 49.4 | 43.5       | 33.7 | 54.7        | 56.6 | 31.6     | 30.6 | 31.1       | 29.4 | 35.6            | 34.6 | 23.3          | 14.4 | 38.4    | 38.4 |
| BLOOM-6.7B  | 10.7         | 13.5 | 0.8        | 3.5  | 2.0         | 3.7  | 3.7      | 7.7  | 5.4        | 8.5  | 11.8            | 15.6 | 9.8           | 15.9 | 6.6     | 12.2 |
| BLOOMz-6.7B | 34.2         | 27.3 | 32.6       | 27.4 | 38.2        | 42.4 | 27.2     | 21.4 | 17.8       | 24.0 | 24.8            | 24.0 | 17.4          | 21.6 | 27.5    | 27.7 |

These peer-to-peer comparisons consistently demonstrate that models undergoing instruction tuning generally outperform their non-tuned backbones. With few exceptions (e.g., LLaMA vs. Alpaca in terms of Macro F1), we observed an average performance improvement of 11.3% across all compared models. This indicates that instruction tuning has a significant positive impact on the performance of LLMs.

Reviewer comment (Experiments):
For the "Using the CoT method, we observed a relative boost in performance in LLMs subjected to instruction tuning and RLHF", the tested models may not have strong CoT ability. It's good to see LLaMA-2 or GPT-4 and other recent models that claim to have strong reasoning abilities. Since CoT is not a generic ability that every LLM has.

Author response:
Thank you for your valuable comments. To address your query, we specifically tested the performance of GPT-4-0613 on the Pinocchio dataset under the "Few shots with CoT" setting. **In this context, GPT-4 achieved an accuracy of 49.9% and a Macro F1 score of 47.2%. These scores represent the best performance across all tested models, showing an average improvement of 2.1% over GPT-3.5-Turbo.**

|                    |Multifaceted |         | Structural |          | Adversarial |          | Temporal |          | Real-World |          | Domain-Specific |          | Multi-lingual |          |
| :--------------: | ----------- | -------- | ---------- | -------- | ----------- | -------- | -------- | -------- | ---------- | -------- | --------------- | -------- | ------------- | -------- |
|                  | ACC.        | F1       | ACC.       | F1       | ACC.        | F1       | ACC.     | F1       | ACC.       | F1       | ACC.            | F1       | ACC.          | F1       |
| Text-Davinci-002 | 47.7        | 47.7     | **50.8**   | 38.4     | 64.2        | 64.3     | 33.9     | 31.1     | **51.7**   | 41.4     | 36.4            | 36.1     | 43.1          | 39.5     |
| Text-Davinci-003 | 51.1        | 47.8     | 44.3       | 33.7     | 64.1        | 63.7     | 41.4     | 35.1     | 48.0       | 42.8     | 40.4            | 41.4     | **43.7**      | **43.6** |
| GPT-3.5-Turbo    | 53.6        | **53.1** | 44.8       | 37.8     | 67.4        | 67.4     | 37.4     | 33.9     | 50.4       | 43.1     | 38.7            | 40.3     | 41.3          | 41.1     |
| GPT-4            | **56.4**    | 51.3     | 43.2       | **38.6** | **73.7**    | **73.3** | **45.2** | **39.6** | 49.9       | **45.3** | **50.4**        | **49.1** | 41.8          | 41.3     |

Reviewer comment (Evaluation):
You can include an additional metric to report the cases that LLMs failed to generate meaningful answers. Since LLM may refuse to answer according to security or other customized policies, treating all unanswered questions as wrong predictions may be unfair.

Author response:
During our experiments, we observed that only a negligible fraction of questions were left unanswered by the LLM due to security or other customized policies. Specifically, when using GPT-3.5-turbo as an example, out of 20,713 queries, fewer than 20 (< 1‰) responses were cases of refusal to answer. The main reason is that we have already excluded claims that could cause safety issues during the annotation progress.

Reviewer comment (Evaluation):
Sec-4.2 could be improved, especially the discussion around temporal analysis, "GPT-3.5-Turbo exhibits superior performance when dealing with outdated data as compared to updated data." I think the numbers are close, and it may not be sufficient to support the claim.

Author response:
We have revised this section to reflect a more cautious and accurate statement that aligns better with the observed data. We appreciate your guidance in improving the clarity and precision of our discussion.

Reviewer comment (Novelty):
There are some missing references for testing LLM's factuality:  
- SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models  
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation  
- FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios

Author response:
Thank you for your constructive suggestion. We have included all these references in the related work section (Page 9).

Reviewer comment (Evaluation):
Data leaky may happen. Note that the questions and answers are annotated based on the text of previous public datasets. The pretraining data of evaluated LLMs is likely to contain the text. This may cause data leaky, and LLMs might have memorized/learned shortcuts to answer the question. It makes the evaluation results on the benchmark less convincing.

Author response:
Thank you for raising the critical issue of data leakage. We have conducted several measures to alleviate data leakage.
- We transformed the original dataset's statements into a question-and-answer (QA) format to mitigate this concern. This adaptation reduced the likelihood of data leakage. 
- It is also worth noting that the labels from the original fact-checking dataset, typically categorized as "Refutes" or "Supports," do not directly apply to our new dataset. In the original context, these labels indicate whether the provided evidence supports or contradicts a given claim. The model determines whether a factual question is "Factual" or "Non-Factual" based on its inherent knowledge.
- Furthermore, we manually revised their labels for questions that underwent factual changes, and around 7% of the statements experienced such factual alterations.

Lastly, we conducted experiments to evaluate the data contamination. Following the methodology proposed in [1], we evaluated the claim in the original dataset and found a data leakage rate of 21.3%. After reannotating the original claims into question-answer pairs manually (by annotators), this rate decreased to 7.4%. We compare this rate with other datasets below. When compared with other benchmarks, our benchmark has a significantly lower contamination rate. 

| Dataset | Contamination Rate |
| :-----| ----: |
| MMLU | 29.8% |
| HellaSwag | 25.6% |
| ARC | 16.3% |
| Ours | **7.4%** |

*Reference:*

[1] An Open Source Data Contamination Report for Llama Series Models. ArXiv 2023.

Reviewer comment (Experiments):
Overclaimed contribution (as well as title) to the research question. I think the main contribution of this paper is the newly proposed benchmark. In the experiment, the authors research the extent and scope of factual knowledge within LLMs. However, the authors only test LLMs on the benchmark. There are no baselines (LLMs, datasets) as a comparison. Only showing accuracy numbers is not sufficient to give the answer to this research question.

Author response:
To address your concern,  we have now provided performance results for GPT-3.5-Turbo in the few shots with CoT settings on various datasets, as shown in the table below. 
Due to time and resource constraints, we will gradually include more LLMs for comparison in future revisions of the paper.

| Datasets    | Accuracy | F1   |
| ----------- | -------- | ---- |
| FEVER       | 56.3     | 55.7 |
| Feverous    | 36.4     | 31.0 |
| FoolMeTwice | 60.2     | 61.1 |
| VitaminC    | 42.8     | 42.7 |
| Politifact  | 36.2     | 35.5 |
| PubHealth   | 56.3     | 49.5 |
| X-Fact      | 35.8     | 33.1 |

Reviewer comment (Evaluation):
May I ask how you design the seven aspects for evaluation? Any high-level intuition?

Author response:
Thank you for your question regarding the design of the seven evaluation aspects. **To develop these aspects, we first undertook a comprehensive survey of numerous existing fact-checking datasets, the summary of which is presented in Appendix A.6, Table 9. ** Our analysis focused on identifying the various challenges these datasets aim to address regarding factuality.

We found that the challenges tackled by these datasets generally align with the seven aspects we have listed in our dataset. These aspects appear individually or in combination across the surveyed datasets, indicating their relevance and importance in fact-checking. This realization led us to intentionally design our evaluation framework around these seven specific challenges, ensuring that our benchmark is not only comprehensive but also directly addresses the core difficulties encountered in current fact-checking tasks.

Reviewer comment (Evaluation):
Can 92.4% annotation accuracy (as well as 85.6% inter-annotator agreement rate) promise the quality of the dataset? Does that mean when the test accuracy of LLMs is larger than ~= 85%, it's meaningless to use this benchmark for evaluation?

Author response:
The 92.4% annotation accuracy and 85.6% inter-annotator agreement rate indeed signify a high-quality dataset. Notably, the primary source of disagreement among annotators was the "Not Enough Information" (NEI) label. According to research by Guo et al. [1], defining the NEI category is inherently challenging, which can account for some of the disagreements.
Moreover, compared to commonly accepted benchmarks in the field, our dataset's quality is notably superior. For instance, the FEVER [2] dataset has an inter-annotator agreement rate of 0.68, LIAR [3] is at 0.82, and FEVEROUS [4] stands at 0.65. Our dataset surpasses these widely used benchmarks, underscoring its reliability.

*References:* 

[1] A Survey on Automated Fact-Checking. TACL 2021

[2] FEVER: a Large-scale Dataset for Fact Extraction and VERification. NAACL 2018

[3] “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection. ACL 2017

[4] FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information. NeurIPS 2021

Reviewer comment (Experiments):
Data is derived from existing datasets but the motivation for transformation is not clearly specified

Author response:
>(2) There is no clear motivation in the paper on why these datasets were chosen — are these the only specific 7 categories of questions? Or was it the distribution of the known datasets and the data points? 

**As detailed in Appendix A.6, Table 9, our selection was based on an extensive survey of existing fact-checking datasets.** This survey aimed to identify and summarize the different challenges these datasets address associated with factuality.

We discovered that these datasets commonly target the seven question categories outlined in our study. These categories, appearing individually or in various combinations in the datasets we reviewed, represent the most significant and prevalent challenges in current fact-checking tasks. Based on this comprehensive analysis, we deliberately designed our benchmark to encompass these seven specific challenges, ensuring that our evaluation framework is both relevant and comprehensive.

Reviewer comment (Novelty):
Most conclusions of the experiments are already made in isolation prior to this work. The novelty in experiments and conclusion should be well specified.

Author response:
>(4) It’s important for the authors to clarify the important of new conclusions in comparison on what already exists.

Thank you for your thoughtful feedback. Please allow us to clarify and highlight the important findings we had based on the proposed benchmark:

- **Divergent Responses to Prompting Techniques:** We've observed that the efficacy of prompting techniques varies significantly between complex reasoning and factual knowledge tasks. Advanced prompting strategies like increased shot counts, complex Chains of Thought (CoT), and self-consistency show promise in reasoning tasks including mathematics reasoning, logical reasoning, symbolic reasoning, etc. However, these methods fall short in factual queries (refer to Table 5). Our conjecture is that this divergence might stem from fundamental differences in how LLMs handle factual-based tasks versus those requiring complex reasoning. In factual-based tasks, the LLMs should first remember the facts correctly and then perform reasoning on the relevant facts. If the facts are wrong in the first place, simply enhancing the reasoning is not helpful. Prior prompting techniques for reasoning can not help LLMs to better elicit factual knowledge.

- **LLMs' Overconfidence in Facts:** LLMs display a higher level of confidence in their answers to factual questions, which can be misleading. Despite multiple iterations of the same question (as self-consistency prompting suggests), LLMs often replicate their errors (Table 5). After human inspections, we found that LLMs tend to make the same factual mistakes no matter whether we ask them how many times or adopt a decoding strategy to increase the diversity of answers.

- **Self-Reflection in Correcting Factual Errors:** Self-refinement prompts LLMs to reassess their answers, showing significant improvements for factual questions (Table 5). After manual inspection, we found that when LLMs re-evaluate their earlier responses with CoTs, they can sometimes detect and rectify errors within their own reasoning pathways. This finding is illustrates a potential method for tempering overconfidence and enhancing fact-based accuracy in LLM responses.

- **The Interplay of Structured and Unstructured Knowledge:** Contrary to initial assumptions that mixing structured (tables) and unstructured (text) knowledge sources complicates answering factual questions, our study reveals that LLMs are adept at integrating the two. This blend can even facilitate the recall of structured knowledge (Figure 3(b)). This equivalency between structured and combined knowledge sources is significant because it suggests that LLMs may use textual contexts to reinforce and access tabular facts, a valuable consideration for knowledge representation and retrieval strategies.

- **Challenges in Factual Precision:** LLMs exhibit proficiency in distinguishing between entities but struggle with precise numerical information (Figure 3(c)). Our manual analyses suggest that although LLMs effectively memorize and differentiate entity details, they have difficulties with exact figures like numbers and dates. This distinction is crucial for understanding LLM limitations in tasks requiring precision and lays the groundwork for focused improvements in numeral cognition for LLMs.

Reviewer comment (Evaluation):
Dataset Design: _“...multi-choice questions are a simple but good proxy to evaluate the potential of advanced abilities of LLMs…”_: I think this claim should be supported by prior work. I tend to disagree with this statement as Chen and Durrett (2019) concludes that, for some datasets, multiple-choice questions could be easily gamed (i.e., spurious correlations between questions and labels). Link: https://arxiv.org/pdf/1904.12106.pdf

Author response:
>(1)The claim "...multi-choice questions are a simple but good proxy to evaluate the potential of advanced abilities of LLMs…" should be supported by prior work.

Multiple-choice questions offer a practical approach to assessing current LLMs' complex capabilities, of which GPT-4 is a prime example [1]. Key benchmarks such as the MMLU [2], HellaSwag [3], ARC [4], BIG-Bench Hard [5], and TruthfulQA [6], all of which utilize multi-choice formats, serve distinct purposes in evaluating various aspects of GPT -4's proficiency. Precisely, the MMLU gauges an LLM's academic and professional knowledge breadth and depth. HellaSwag is designed to test commonsense reasoning, and ARC focuses on one's ability to tackle challenging questions. BIG-Bench Hard presents complex reasoning tasks that challenge even the most advanced LLMs, while TruthfulQA measures how LLMs mimic human falsehoods.
Furthermore, the evaluation of language generation brings its own challenges, as a universal metric for measurement is currently lacking [7], which multiple-choice questions help mitigate by offering straightforward classification accuracy for assessment [1]. Also, prior studies [8] underscore that LLMs demonstrate reliable calibration when faced with multiple-choice scenarios, suggesting that this format is adept at extracting a model's latent knowledge. 
In response to your insight, we have included this discussion in our paper, and rewritten section 2.2 **ANNOTATION AND QUALITY CONTROL** in the rebuttal version.

*References:*

[1] GPT4 Research Report. https://openai.com/research/gpt-4

[2] Measuring Massive Multitask Language Understanding. ICLR 2021

[3] HellaSwag: Can a Machine Really Finish Your Sentence? ACL 2019

[4] Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. ArXiv 2018

[5] Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. ACL 2023

[6] TruthfulQA: Measuring How Models Mimic Human Falsehoods. ACL 2022

[7] A Survey of Evaluation Metrics Used for NLG Systems. ACM CSUR 2022

[8] Language Models (Mostly) Know What They Know. Anthropic Technical Report 2022

>(2) According to Chen and Durrett (2019), multiple-choice questions could be easily gamed (i.e., spurious correlations between questions and labels).

Chen and Durrett (2019) focused on multi-hop question-answering datasets, where a model is required to reason over multiple supporting documents to answer the question. In order to verify that it is possible to pick the correct answer without consulting the related documents, they construct a "no context" baseline. This "no context" baseline achieves higher performance on the multiple-choice WikiHop dataset, comparable to some state-of-the-art QA systems. This result shows that WikiHop can solve reasonably well without using multiple documents, indicating that the dataset is potentially gamed.

Following Chen and Durrett (2019), we developed the same "no context" baseline to investigate the spurious correlations between questions and labels in our dataset. The results are shown below:

| Method | Performance |
| :-----| ----: |
| No Context | **28.3** |
| LLaMA-7B | 31.6 |
| Alpaca-7B | 37.8 |
| Vicuna-13B | 45.2 |
| GPT-3.5 | 47.0 |

Our experimental findings show that our dataset does not exhibit the same level of vulnerability to the exploitation of question-label correlations as observed by Chen and Durrett (2019) in the WikiHop dataset. With performance improvements of 16.9 points by Vicuna-13B and 18.7 by GPT-3.5 over the "no context" baseline, our results offer compelling evidence that our dataset is more resilient to such biases, contrary to the reported susceptibilities within WikiHop.

We extended our analysis to include a direct comparison with several established multiple-choice question-answering benchmarks, such as the WikiHop mentioned above, as well as with other prevalent benchmarks like TruthfulQA and ARC utilized in evaluating LLMs. The performance of the "no context" baseline across these benchmarks is displayed in the table below:

| Dataset | Performance |
| :-----| ----: |
| WikiHop | 59.7 |
| TruthfulQA | 34.5 |
| ARC | 33.2 |
| Ours | **28.3** |

Evidently, our proposed dataset presented the most challenge to the "no context" baseline, marking the lowest performance compared to other datasets. The notable performance on WikiHop, with a "no context" baseline score of 59.7%, underscores the presence of spurious correlations that facilitate gaming that dataset. On the contrary, the lower baseline performances on TruthfulQA and ARC suggest that such issues are less prevalent. Our dataset, therefore, not only stands out as the least prone "to be gamed" but also underscores its robustness and the high level of rigor needed to tackle it effectively.

Reviewer comment (Novelty):
A significant number of related papers have not been cited. There are numerous strands of prior work that investigate the factual knowledge and reasoning abilities of large language models (LLMs), including multi-hop reasoning, temporal and geographical questions, new entities, and complex inference. (I’m listing a few of them here)  
- Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (https://arxiv.org/abs/2101.02235)  
- SituatedQA: Incorporating Extra-Linguistic Contexts into QA (https://arxiv.org/abs/2109.06157)  
- A Dataset for Answering Time-Sensitive Questions (https://arxiv.org/abs/2108.06314)  
- Entity Cloze By Date: What LMs Know About Unseen Entities (https://arxiv.org/abs/2205.02832)  
- RealTime QA: What's the Answer Right Now? (https://arxiv.org/abs/2207.13332)  
- StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models (https://arxiv.org/abs/2205.11388)  
- Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge (https://arxiv.org/abs/2305.01651)  
- FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (https://arxiv.org/abs/2310.03214)

Author response:
>(3) A significant number of related papers have not been cited.

Thank you for providing the list of the related work missing in the paper. We have included the paper you listed in the rebuttal version. 

Furthermore, we extensively survey other related efforts in these topics. **As shown in Table 13 in the Appendix (page 36), we comprehensively compared 37 research efforts. We also included a new section called RELATED WORK: QUESTION ANSWERING DATASETS in Appendix A.10 (pages 32-35), where we discuss a significant number of related work in detail (more than 50 references are added)**.

Reviewer comment (Experiments):
_“...but in Pinocchio, we only need to judge the factuality of the question.”_ Does this mean all supporting facts are not included in the questions?

Author response:
>(4) “...but in Pinocchio, we only need to judge the factuality of the question.” Does this mean all supporting facts are not included in the questions?

Apologies for any confusion earlier. Please allow us to clarify: in our proposed dataset, the focus is exclusively on assessing the factual accuracy of the questions themselves, without the inclusion of external supporting facts.
In typical fact-checking datasets, labels like "Supports" or "Refutes" are used to signal whether external evidence confirms or denies a particular claim. These datasets are aimed at fostering the development of systems able to emulate the comprehensive process journalists go through to fact-check a claim, which involves sifting through various sources for relevant evidence and consequently reaching a conclusion regarding the claim's accuracy. This process relies heavily on evidence gathering, a critical aspect of traditional fact-checking operations.

However, our Pinocchio benchmark is designed with the objective of gauging the factual knowledge stored within LLMs. The benchmark evaluates LLMs' capacity to reason using the factual knowledge they have previously assimilated, thus, intentionally excluding the provision of external supporting facts. In simple terms, we are testing the LLMs' ability to verify facts based solely on the information they have internalized without referencing additional evidence. We are less concerned with an LLM's ability to process and infer from 'external evidence' to formulate conclusions.
To create this specific dataset, we have re-evaluated each statement from the original fact-checking datasets by employing annotators to research various sources and verify facts, which led us to re-annotate the claims. This method ensures that our dataset appropriately tests the LLMs' factuality without relying on the inclusion of supporting facts within the questions.