Paper: Chain-of-Verification Reduces Hallucination in Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The utilization of few-shot learning for enabling LLMs to perform planning, verification, and generation (3-shot as detailed in the Appendix) raises a potential concern that the results might be influenced by variations in the few-shot examples used.

Author response:
We have now added the few-shot prompts used in our supplementary material for reproducibility.  We agree that changing the few-shot prompts can lead to a variance in the final output. In our comparative analysis, we use the same prompts to generate the baseline response for all our baseline approaches.

Reviewer comment (Experiments):
What criteria were employed for the selection of few-shot examples, and what is the potential influence of employing different examples on the results?

Author response:
1. Few-shot prompts:
      1. Our few-shot prompts for the list-based QA benchmarks were randomly chosen from the train set. We however keep these 
                  prompts constant across our variants.
      2. For the biographies dataset, we choose some of the most viewed people's pages on Wikipedia and craft our prompts 
                  based on their biographies.  We find few-shot examples that allow us to show the model our desired verification responses. 
                  We have added the few-shot prompts used to the supplementary material.

Reviewer comment (Evaluation):
Could you provide an estimate of the additional computational overhead that the proposed method would introduce?

Author response:
We perform an analysis of the inference overhead. Please see the common response and Section 8 in our appendix.

Reviewer comment (Experiments):
There is an absence of comparative analysis with other methods aimed at mitigating the hallucination issue. It remains to be clarified whether CoVe offers an enhancement in performance relative to other methods.

Author response:
We have now added three new state-of-the-art baselines to the paper. Our work deals with hallucinations specifically for longform generations. We have thus updated Table 3 with baselines using SelfCheckGPT (NLI and LLM) and ChatProtect - three state-of-the-art papers that deal with hallucinations in longform generations. We find our approach, CoVe, outperforms both SelfCheckGPT (with NLI and LLM)  and ChatProtect. Please see our common response for more details.

Reviewer comment (Writing):
More instances of prompts are required. For example, in Section 3.3, some examples of prompts need to be provided to distinguish between the several variants of verification variants.

Author response:
We add our few-shot prompts in the supplementary material.

Reviewer comment (Experiments):
How do we verify that the model can do plan verification, execution verification, and verified response well after a few shots? What happens if you use zero-shot CoVe?

Author response:
Like several reasoning tasks, it is hard to measure the performance of the intermediate parts of the chain owing to the absence of labels, although clearly you can measure the end-to-end performance, or hand annotate some data. For the Wikidata task we manually measure the accuracy of the verification questions and we find that the Llama 65B is around 70% accurate at answering factoid questions related to the professions and birth-places of people. We describe this in the paper under Shortform verification questions are more accurately answered than longform queries.

Reviewer comment (Novelty):
Is it possible to skip the step of generating a baseline response by directly generating questions similar to plan verification based on the query and generating the final response based on the response? Does CoVe have any advantages over this method?

Author response:
That’s an interesting approach to try. For some tasks, it is not likely to work as you will not know which questions to ask, e.g. consider list-based questions (e.g. “who are some politicians born in NYC“), the only question you can really ask at the start is the list-based question itself.

Reviewer comment (Reproducibility):
More details of prompts are needed in the supplemental materials, or it would be difficult for readers to follow the ideas.

Author response:
We had added our few-shot prompts in the supplementary material

Reviewer comment (Theory):
What is the time complexity of the proposed method and other competitors?

Author response:
We perform a comparative analysis of the inference overhead encountered. Please check the common response and Section 8 in our appendix.

Reviewer comment (Novelty):
The technical contribution is thin. I had a hard time justifying the paper’s technical contribution since CoVE looks very similar to [1].

Author response:
The paper “Measuring and Narrowing the Compositionality Gap in Language Models” (which is cited) as the title suggests concentrates on compositional questions with multiple hops such as “Who won the Master’s Tournament the year Justin Bieber was born?” – and breaking into subquestions. It does not focus on hallucinations (does not mention that word at all). It uses a method called self-ask to build this subquestion decomposition. In our work questions are used for a different purpose: to verify hallucinations, rather than decompose a multi-hop question, so this is different. Further, we develop joint, 2-step, factored and factor+revise approaches and compare these and show how they are important – none of which is related to this paper.

Reviewer comment (Experiments):
The two Wikipedia QA datasets (Wikidata and Wiki-category list) are created from simple templates and are rather toyish. I am not sure how much they add to the paper.

Author response:
We would like to clarify that we do not create the Wiki-Category benchmark ourselves but use a subset of an existing benchmark: Quest (https://aclanthology.org/2023.acl-long.784.pdf).
Wikidata uses real data from Wikipedia, but the queries are synthetically constructed. 
We also report results on the MultiSpanQA task which does use real queries from humans (from Google). Overall, list-form answers are very natural in assistant-based applications. List-generation questions can be excellent ways to understand LLM hallucinations and rapidly test approaches. List generations mimic longform generations with the advantage that they can be automatically, efficiently and perfectly evaluated, while in other tasks evaluation is much more tricky.
We agree that questions of the form “Name some politicians born in Boston” appear deceptively trivial **but large-scale commercially deployed LLM systems fail in producing correct answers**. We have provided several qualitative examples of this in the supplementary material.

Reviewer comment (Experiments):
Hallucinations are especially tricky to address in generation; the only generation task considered is biography generation, which is way less challenging than most real-world applications.

Author response:
Hallucinations mitigation in longform generations has very recently started receiving significant attention and a lot of benchmarks don’t yet exist for this problem. We chose these four tasks because (i) they are known to produce hallucinations for state-of-the-art models (as shown in experiments), both open source and commercial products; and (ii) ways of evaluating these tasks exist. In particular, the FactScore metric was recently built for the biography task, and hence several other papers, other than ours, are biographies as well (see new baselines). 
We agree that it would be interesting to see how CoVe would perform on further benchmark tasks as they are developed. Scientific, legal and medical domains are great applications and will warrant deep analysis beyond a methods paper such as this one. Hence, this is exciting future work beyond this paper.

Reviewer comment (Experiments):
The paper aims to address hallucination problems. However, I do not find this reflected in the designs of the experiments: 3 out of 4 experiments are question answering, and the remaining one is a rather confined biography generation task. Therefore I find the experiments of this paper weak. Including a more diverse and challenging set of tasks can strengthen the results.

Author response:
Hallucinations mitigation in longform generations has very recently started receiving significant attention and a lot of benchmarks don’t yet exist for this problem. We chose these four tasks because (i) they are known to produce hallucinations for state-of-the-art models (as shown in experiments), both open source and commercial products; and (ii) ways of evaluating these tasks exist. In particular, the FactScore metric was recently built for the biography task, and hence several other papers, other than ours, are biographies as well (see new baselines). 
We agree that it would be interesting to see how CoVe would perform on further benchmark tasks as they are developed. Scientific, legal and medical domains are great applications and will warrant deep analysis beyond a methods paper such as this one. Hence, this is exciting future work beyond this paper.

Reviewer comment (Experiments):
Wikipedia is a domain that most models have a lot of exposure to; it would be more interesting to see how CoVE performs in other domains such as scientific, legal, and medical domains.

Author response:
Hallucinations mitigation in longform generations has very recently started receiving significant attention and a lot of benchmarks don’t yet exist for this problem. We chose these four tasks because (i) they are known to produce hallucinations for state-of-the-art models (as shown in experiments), both open source and commercial products; and (ii) ways of evaluating these tasks exist. In particular, the FactScore metric was recently built for the biography task, and hence several other papers, other than ours, are biographies as well (see new baselines). 
We agree that it would be interesting to see how CoVe would perform on further benchmark tasks as they are developed. Scientific, legal and medical domains are great applications and will warrant deep analysis beyond a methods paper such as this one. Hence, this is exciting future work beyond this paper.

Reviewer comment (Evaluation):
It seems that CoVE can potentially negatively impact the model’s helpfulness: in Table 1, the number of factual outputs reduces; similarly, in Table 3, CoVE produces fewer facts. Some discussion on this would be interesting.

Author response:
Any method that reduces hallucinations will reduce the number of facts. As we note in our research questions, we are interested in whether CoVe can selectively remove fictitious facts with a minimal impact on the number of truthful facts. 
The FactScore metric is normalized by number of facts. So a shorter generation does not automatically mean fewer hallucinations. Note that we also built a baseline whereby the number of facts is reduced by chopping off the ends of passages to match the same number of facts as our best model and we see CoVe significantly outperforms this baseline. We describe this in our passage: CoVe outperforms existing hallucination mitigation baselines.

Reviewer comment (Evaluation):
I am concerned about making strong conclusions solely based on the FactScore metric: will a model receive higher FactScore by producing shorter outputs with less information? Complementing it with models’ helpfulness, generation quality, and controlling for the number of facts might be necessary.

Author response:
Any method that reduces hallucinations will reduce the number of facts. As we note in our research questions, we are interested in whether CoVe can selectively remove fictitious facts with a minimal impact on the number of truthful facts. 
The FactScore metric is normalized by number of facts. So a shorter generation does not automatically mean fewer hallucinations. Note that we also built a baseline whereby the number of facts is reduced by chopping off the ends of passages to match the same number of facts as our best model and we see CoVe significantly outperforms this baseline. We describe this in our passage: CoVe outperforms existing hallucination mitigation baselines.

Reviewer comment (Theory):
A key assumption behind CoVE is that even when the model is not able to generate factual outputs, it might still be able to identify nonfactual statements. A similar observation is mentioned by [2], but they argue that retrieval is necessary. Can the authors elaborate their views on whether or not the model needs external information to identify factual errors in its own outputs?

Author response:
Our work along with several prior works, eg SelfCheckGPT, ChatProtect have shown that it is possible to mitigate hallucinations without retrieval. We agree that it is impossible to guarantee veracity without access to a verified source of information. We would argue that even retrieval as shown by several  Retrieval Augmented Generation (RAG) style approaches still cannot guarantee this without assuming that everything on the internet is true. We also would like to note that errors can still creep into the RAG pipeline for example the PerplexityAI result from our paper which uses a retrieval system.

Reviewer comment (Experiments):
**Absence of necessary baselines**. As the authors pointed out in the Related work sections, there are many relevant works based on prompting, to reduce the hallucination of LLMs [1,2,3] or improve LLMs’ reasoning [4,5]. However, these baselines are never compared through the experiments now. Therefore, it’s hard to verify the effectiveness of the proposed CoVe compared to them.

Author response:
We have added three new state-of-the-art baselines to the paper. Our work deals with hallucinations specifically for longform generations. We have thus updated Table 3 with baselines using SelfCheckGPT (NLI and LLM) and ChatProtect - three state-of-the-art papers that deal with hallucinations in longform generations. We find our approach, CoVe, outperforms both SelfCheckGPT (with NLI and LLM)  and ChatProtect.

Reviewer comment (Experiments):
**Difficulty of direct comparison**. Currently, only zero-shot (or CoT) results are presented for LLaMA2 and few-shot results for LLaMA65B, respectively. It makes be hard to compare both models as there is no overlap. To ease their comparison, including the results of LLaMA2 few-shot and LLaMA zero-shot is strongly encouraged.

Author response:
Llama65B is not an instruction-tuned model and hence produces gibberish (without stopping) when prompted zero-shot. This is expected behavior for a base language model, hence we do not report it (we will clarify this in the paper). When prompted zero-shot with “Let’s think step-by-step”, the model generated erroneous steps, for example, to check information on the web instead of generating the required answer, and did not produce answers.  
We have updated our appendix on the unsuccessful experiments of getting Llama 65B to generate the desired response when prompted zero-shot. In any case, these serve as baselines to CoVe rather than being interesting in themselves to compare. The CoVe experiments are also few-shot using Llama65B, hence the baseline few-shot Llama65B comparison. Llama 2 (which should be superior) is there as a baseline just to show that zero-shot instruction-tuned model baselines do not actually fare better.

Reviewer comment (Presentation):
**Inconsistency across Tables**. While Factor+revise is presented in Table 3 and it achieves the best score, there are no such results in Tables 1 and 2. Does this method not perform well on the setups in Tables 1 and 2? To facilitate the understanding of the working mechanism of the proposed framework.

Author response:
The Factor+revise method adds an additional step where the LLM can check for inconsistencies between the generated content and the answered verification question. 

Consider the example

          “Trump received a BS in Business from the University of Pennsylvania” 
           And the verification question and output:

          What degree did Donald Trump graduate with from the University of Pennsylvania in 1968? 
          Answer: A BS in Economics. 

The Factor+revise method adds a separate step that has the LLM check if these two outputs are consistent, inconsistent or partially consistent and then recomposes the output based on the consistent parts. 

For the set QA tasks checking for inconsistencies and removing an answer from the list is more trivial, as it is just an entity match.

Example:

         Name some politicians born in New York, NY
        A: Hillary Clinton

        Where was Hillary Clinton born?
        A: Chicago (Chicago != New York)

CoVe - 2 step is a variant of CoVe-Factored where the Verification questions are evaluated in one step (thus reducing the need to prompt the LLM several times for each verification question). This is feasible to use on tasks such as set QA but as the number of verification questions grows answering them all can be difficult in the fixed context size. 
We hence don’t experiment with this approach on the harder benchmarks.

Reviewer comment (Presentation):
**More qualitative examples**. While the authors present some examples in Figures 1 and 3, it would be better to present more examples to help the understand of readers.

Author response:
Figure 1 and 3 show full examples of CoVe. 
There is also analysis in section 9 of how and why ChatGPT fails on a number of tasks, and why verification questions can work.
Finally, we have now added the prompts used as part of our supplementary material