Paper: Evaluating and Improving Generation Consistency of Large Language Models via A Divide-Conquer-Reasoning Approach (ICLR-2024)

Reviewer comment (Experiments):
The proposed method has a limited scope and is not easily generalizable to broader instruction-following tasks. In the experiment, the method required hand-crafted prompts for each specific task, which can be overwhelming and even infeasible when evaluating a wide spectrum of tasks. Other LLM-based evaluators can handle this common scenario more effectively.

Author response:
Our work focuses on the evaluation and improvement of generated text consistencies. We do not claim DCR as a generic evaluation metric and our experiments demonstrated that DCR achieves the state-of-the-art performance on consistency evaluation compared with the existing baseline methods. Our DCR is easily generalizable to assess the consistency between any two text sequences without constraints on their format, that is, they can be either generated texts by LLM or written by human experts. 

We agree that our DCR framework requires hand-craft prompts for specific tasks, and acknowledge that this is a general hurdle shared by all works relying on LLMs, which include G-Eval, GPTScore, and self-refine.  Specifically, in G-Eval, different prompts will need to be composed for different aspects: consistency, coherence, etc.  Self-refine defines multiple customized prompts to perform their INIT - FEEDBACK – REFINE components. Our current solution is to structure our prompts in a modularized manner so task-specific content can be updated easily. An automated prompt-tuning procedure is beyond the focus of our study. If you believe there exists coinciding work on an LLM-based evaluator that offers a more effective solution, we kindly ask you to point us to it.

Reviewer comment (Evaluation):
The per-sentence assessment protocol is used to identify and fix inconsistencies, but it may be prone to LLM's overconfidence -- it prefers its predictions even if it is wrong.

Author response:
Although the experiments lack human evaluations on sentence-level judgments, our DCR method provides an overall numerical score by calculating and integrating sentence-level scores through an automatic metric converter (AMC) component. As shown in Fig.3 and the following table result, we verify the F1-score, precision, and recall performance of our method on sentence-level and paragraph-level evaluations compared with human evaluations. Such high accuracy provides solid support for our reason-assisted improver (RAI) which aims at reducing inconsistencies via multi-round iterations. We agree with the reviewer that LLMs have the potential tendency to be overconfidence. However, as demonstrated by our experiments, such overconfidence can be mitigated through the careful design of our framework, resulting in high accuracy that is well-aligned with human intuition.

| Metric  Name  | F1 | Precision  | Recall | 
| --------  | --------  |--------  |--------  |
| SUMMEVAL | 0.933 | 0.929 | 0.936|
| QAGS-XSUM | 0.811 | 0.718 | 0.930|
| QAGS-CNN| 0.871 | 0.842 | 0.901|

Reviewer comment (Evaluation):
The evaluation focuses solely on consistency, but other criteria like factuality and coherence are also important for text generation. It's unclear why only consistency was considered.

Author response:
Assessing the consistency of LLM-generated content is crucial for ensuring AI safety and has become a critical step in improving the reliability of many real-world application domains that rely on LLM by preventing the generation of misinformation and harmful content. For example, consistency checking can significantly enhance the chain of thought reasoning in LLMs [1]. Recent studies employ consistency checking to detect hallucinations [2] based on pre-trained LLMs and instruction tuned LLMs [3]. Although these methods exhibit promising results on several specific tasks, including mathematical reasoning and factual assessment, the potential failures of self-consistency are often overlooked [4]. 

This is essentially due to a lack of a generic, automatic, and reliable strategy that assesses the consistency of two responses, let alone remediating such inconsistency after identifying them. We highlight that our objective is not to propose a generic or comprehensive evaluation pipeline to handle broader metrics as G-Eval or GPTScore claimed. Instead, we mainly focus on the consistency scope and have achieved our state-of-the-art performance across multiple benchmarks in semantic, factual, and summarization consistency tasks, compared to existing baseline methods. Furthermore, our approach also substantially reduces nearly 90\% output inconsistencies, showing promise for effective hallucination mitigation and reduction.

[1] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.

[2] Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896, 2023.

[3] Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. arXiv preprint arXiv:2305.15852, 2023.

[4] Angelica Chen, Jason Phang, Alicia Parrish, Vishakh Padmakumar, Chen Zhao, Samuel R Bowman, and Kyunghyun Cho. Two failures of self-consistency in the multi-step reasoning of llms. arXiv preprint arXiv:2305.14279, 2023.

Reviewer comment (Writing):
Why it is stated in the Intro that DCE “does not rely on LLM“, considering Chatgpt/GPT4 is required for most components of the method?

Author response:
Our complete statement on Page 2 is that “Our approach does not rely on LLMs, which are prone to hallucination,  **to output numeric scores without justification**”. Our intention was to emphasize that the DCR framework does not rely on LLMs to **directly** generate scores, which is in contrast to previous methods such as G-Eval, which may be possibly hallucinated.  As we discussed in Section 2, existing LLM-based evaluators have a major drawback as the generated verbal scores by LLMs are prone to hallucinations, resulting in abnormally higher ratings for LLM-generated content that diverge from human judgment. Such methods also generate no actionable insight to justify the score or mitigate inconsistencies after identifying them. 

To overcome these issues, we intentionally avoid using LLM in the critical steps that involve generating numerical evaluation scores but instead use LLMs to perform reasoning which they have been proven to excel in. To operationalize this, we introduce an automatic metric converter (AMC) that aims to quantitatively measure the consistency evaluation by converting the reasons a numeric score system. AMC functions as a binary sentiment classifier that classifies the reasons to be either positive (marked by “+1" if the sentence is consistent), or negative (marked by “-1” otherwise) for each sentence, then utilizes this score array to calculate a comprehensive score to evaluate consistency, as explained in Section 3.2.  We believe such a numerical score calculated by AMC is more stable than the verbal score directly output by LLMs, as is verified by experiments. 

We clarify this sentence in the revised version as “**Our approach does not rely on LLMs to directly output numeric scores without justification, which are prone to hallucination**”.

Reviewer comment (Experiments):
How is the reason-assisted improver (RAI) compared to self-refine?

Author response:
Thanks for the suggestion.  As self-refine contains the process of INIT - FEEDBACK - REFINE, we summarize the difference between DCR with self-refine as follows.

- DCR aims at achieving a similar goal as self-refine in providing supervision-free refinements to LLM-generated content. The focus of the two frameworks is different as self-refine aims at more general purposes with the ability to customize prompts for individual tasks whereas DCR aims at detecting and mitigating inconsistency.

- There is no specific prompt dedicated to consistency checking within the context of self-refine, where it is simply defined as a criterion of "Consistent" in one specific task, i.e., dialogue-response. Also, self-refine uses 6 in-context examples for feedback generation, but the DCR is a zero-shot approach that does not require few-shot examples.

- Comparing RAI to the REFINE step in self-refine, both approaches use LLM as the improver. Thus, it boils down to the feedback input and the prompt for performance differences. As the FEEDBACK prompt is more general regarding dialogue in self-refine, the feedback given from DCE is more specific.

Reviewer comment (Evaluation):
G-Eval does not use references during evaluation, so comparing the proposed method to G-Eval may not be fair since they use different evaluation approaches.

Author response:
We apologize for the confusion caused by the definition of “reference” in our DCR framework. We would like to make the following clarifications: 

1. Our DCR method employs the same setting as G-Eval, which does not rely on a golden reference written by the human expert or ground truth labels.  

2. Our DCR method aims to offer a generic way to assess the consistency between two text sequences. For instance, in the summarization task, one text sequence may be the original document, while the other might be the generated text summarization by LLMs. To distinguish between the two text sequences, in our paper we referred to one of the text sequences as “reference” and the other as “candidate”. Accordingly, we formulate the problem to check whether the candidate is consistent with the reference or not, as presented in Section 2.  We realize that such wording might have caused confusion and will change them to more neutral terms such as “text sequence 1” and “text sequence 2”.

Below we provide a detailed explanation of the “reference” used in our experiments:
- **Paraphrase detection tasks**. For example, in the Quora Question Pairs (QQP) dataset, each question pair is annotated with a binary value indicating whether the two questions are paraphrases of each other.   We consider “question1” as the “reference” and “question2” as the “candidate”, and our task is to evaluate if the candidate is consistent with the reference in semantic meaning.  
- **Summarization tasks**. For example, SummEval datasets include original source articles, machine summaries, and human summaries. Our “reference” in this task is the original source article, and our “candidate” is the machine summaries. Our task is to check the factual consistency between them without relying on any additional golden reference or ground truth.

Reviewer comment (Writing):
The authors use a "reference" document to help with consistency evaluation. However, from the prompts shown in Appendix A.2, it's unclear what "reference" means. In some cases, it seems to mean the "true answer" (like a gold standard in common situations), while in others, it refers to the "article" (or source document in common contexts). Could the authors clarify what they mean by "reference"?

Author response:
1. Our DCR method employs the same setting as G-Eval, which does not rely on a golden reference written by the human expert or ground truth labels.  

2. Our DCR method aims to offer a generic way to assess the consistency between two text sequences. For instance, in the summarization task, one text sequence may be the original document, while the other might be the generated text summarization by LLMs. To distinguish between the two text sequences, in our paper we referred to one of the text sequences as “reference” and the other as “candidate”. Accordingly, we formulate the problem to check whether the candidate is consistent with the reference or not, as presented in Section 2.  We realize that such wording might have caused confusion and will change them to more neutral terms such as “text sequence 1” and “text sequence 2”.

Below we provide a detailed explanation of the “reference” used in our experiments:
- **Paraphrase detection tasks**. For example, in the Quora Question Pairs (QQP) dataset, each question pair is annotated with a binary value indicating whether the two questions are paraphrases of each other.   We consider “question1” as the “reference” and “question2” as the “candidate”, and our task is to evaluate if the candidate is consistent with the reference in semantic meaning.  
- **Summarization tasks**. For example, SummEval datasets include original source articles, machine summaries, and human summaries. Our “reference” in this task is the original source article, and our “candidate” is the machine summaries. Our task is to check the factual consistency between them without relying on any additional golden reference or ground truth.

Reviewer comment (Reproducibility):
G-EVAL is described as a method that doesn't rely on a reference (reference-free method), where "reference" seems to mean a gold reference written by the human expert. If this study's prompts use the "gold reference" or the "source", it would be helpful if the authors could specify whether the proposed divide-and-conquer technique is based on a reference or not, and if it uses a source or not.

Author response:
1. Our DCR method employs the same setting as G-Eval, which does not rely on a golden reference written by the human expert or ground truth labels.  

2. Our DCR method aims to offer a generic way to assess the consistency between two text sequences. For instance, in the summarization task, one text sequence may be the original document, while the other might be the generated text summarization by LLMs. To distinguish between the two text sequences, in our paper we referred to one of the text sequences as “reference” and the other as “candidate”. Accordingly, we formulate the problem to check whether the candidate is consistent with the reference or not, as presented in Section 2.  We realize that such wording might have caused confusion and will change them to more neutral terms such as “text sequence 1” and “text sequence 2”.

Below we provide a detailed explanation of the “reference” used in our experiments:
- **Paraphrase detection tasks**. For example, in the Quora Question Pairs (QQP) dataset, each question pair is annotated with a binary value indicating whether the two questions are paraphrases of each other.   We consider “question1” as the “reference” and “question2” as the “candidate”, and our task is to evaluate if the candidate is consistent with the reference in semantic meaning.  
- **Summarization tasks**. For example, SummEval datasets include original source articles, machine summaries, and human summaries. Our “reference” in this task is the original source article, and our “candidate” is the machine summaries. Our task is to check the factual consistency between them without relying on any additional golden reference or ground truth.

Reviewer comment (Experiments):
The lack of comparisons with other consistency methods like SMART, NLI, and Bleurt makes me question the thoroughness of the paper.

Author response:
Thanks for your suggestion. We have added some additional experiments for comparison. Since SMART has not released open-source code yet, we cannot evaluate its performance in our experiment setting, i.e., the summary-level correlation on SummEval datasets. Instead, we implement the system-level correlation used in SMART, for a fair comparison of these baseline methods. As shown in the table below, our method outperforms SMART in terms of all metrics, including the SMART series provided in their paper. 

| Metric   | S1-CHRF| S2- CHRF | SL- CHRF | S1-BLEURT| S2-BLEURT | SL-BLEURT | DCE-AMC (ours) |
| -----------------  | ------- |------- |------- |------- |------- |------- |------- |
| Kendall tau | 0.733 | 0.700 | 0.733 | 0.667 | 0.750 | 0.567 | **0.799** |