Paper: Improving Language Models via Plug-and-Play Retrieval Feedback (ICLR-2024)

Reviewer comment (Novelty):
While REFEED looks promising approach to improve large language model in a plug-and-play framework, its goals are not specifically clarified. Because of the wide range of goals, it is necessary to compare and discuss with other approaches, such Prompt tuning, Chain-of-Thought (CoT), Tree of Thoughts (ToT), and, Retrieval Augmented Generation (RAG) have not been described. Figure 1, 2, and 3 can be interpreted as a kind of RAG.

Author response:
Thank you for incorporating the related work discussion into your paper. Here's a polished summary of the differences between your work and existing approaches:

First, in comparison to Retrieval-Augmented Generation (RAG), our \textsc{ReFeed} method also leverages retrieval as a crucial component of our pipeline. However, unlike conventional methods that directly use retrieval to enhance model performance in complex reasoning and factual accuracy, our research introduces a novel application. We demonstrate that retrieved documents can be effectively employed as feedback to refine language model outputs, significantly improving factual accuracy.

Second, our approach diverges from recent advancements such as Chain-of-Thought, Tree-of-Thought reasoning, and Self-Refinement. These methods do not utilize external knowledge to augment language model reasoning and factual accuracy. Instead, they concentrate on improving the language model's reasoning abilities through diverse prompt designs.

Reviewer comment (Evaluation):
Backbone language models are limited, and it is difficult to determine whether the effect is due to the approches or the emergent nature of the language model.

Author response:
Thank you for highlighting the importance of backbone model selection. Our initial decision to use Davinci-003 and Codex as backbone models was made to ensure a fair comparison with baseline methods (e.g., GenRead, RePLUG) that we have discussed in the paper. 

Besides, the ChatGPT and GPT-4 system might involve many user data for instruction tuning, which could have contamination with many academic benchmark dataset, such as NQ and TriviaQA. 

Lastly, in OpenAI’s announcement that both ChatGPT and GPT-4 will be subject to ongoing updates in their model parameters. These continual modifications would lead to non-reproducible experiments, potentially compromising the reliability of our research outcomes.

In response to your concern, we have conducted additional experiments using ChatGPT (gpt-3.5 turbo) and GPT-4, which reinforce the adaptability and effectiveness of our approach across different model architectures. These results are now detailed in the revised version of our paper, providing a comprehensive view of our method's applicability.


| Experiment on ChatGPT (gpt-3.5-turbo)                  | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 34.3 / 41.5 | 58.7 / 63.7 | 31.7 / 33.6 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.5 / 48.1 | 66.3 / 71.1 | 34.1 / 36.0 |


| Experiment on GPT4 (gpt-4)                             | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 34.8 / 49.6 | 64.6 / 72.8 | 30.8 / 33.7 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 32.5 / 46.5 | 59.9 / 67.1 | 31.6 / 37.5 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 36.8 / 54.4 | 66.3 / 74.0 | 36.9 / 42.6 |

From the above table, we can obserbe:

(1) The retrieval-feedback consistently improves the system, compared to both standard closebook QA method, and retrieve-then-read pipeline.

(2) The traditional retrieve-then-read system even sometimes hurt by noisy retrieval. On the contrary, our ReFeed could avoid misled issue if the first-round generation is good.

Reviewer comment (Reproducibility):
Reproducibility, limitations, and lack of qualitative evaluation do not confirm its validity. Table 1 is inadequate because the settings are not clear.

Author response:
Thank you for your suggestions. We acknowledge these aspects are crucial for confirming the validity of our research.

Regarding reproducibility, while this was not included in the main paper, comprehensive details have been provided in Appendices A.1 and A.2. Additionally, to further ensure reproducibility, we will make all codes and data publicly available for sure.

Regarding the limitations of our study, we have addressed this by adding a dedicated section on page 9 of our paper. This section aims to provide a transparent overview of the potential constraints and boundaries of our research.

Regarding qualitative evaluation, all methods were evaluated in a few-shot QA/generation setting. Although the domains vary, with focusing on commonsense QA [1] and on open-domain QA [2], methods share the objective of using feedback to improve the performance of large language models like GPT-3. Table 1 mainly reveals the main differences in method design and application domain, without mentioning their implementation details.

[1] Rethinking with retrieval: Faithful large language model inference. 

[2] Check your facts and try again: Improving large language models with external knowledge and automated feedback.

Reviewer comment (Novelty):
The paper's novelty is in question, as the process of using an initial output for retrieval and then refining it does not appear to be a novel approach, and the absence of citation to related work, such as [1], raises concerns. Additionally, the paper lacks in-depth insights into the understanding of retrieve-then-read pipelines. The paper altogether seems more like a system report that shows the effectiveness of each trick, not an academic research.

Author response:
We acknowledge the concerns regarding the recent work by Jiang et al. on Active Retrieval Augmented Generation (2023). **It's important to note that our paper was published within 10 days on a third-party open-access repository. Due to the anonymity policy, we cannot specify the exact repository and date, but we emphasize that our work was indeed novel at the time of its publication and is contemporary with Jiang et al.'s research.**

Despite the similarities in the pipeline (generate-retrieve-regenerate) between Active RAG and our ReFeed approach, we offer distinct contributions. Firstly, one-pass decoding in Active RAG may limit potential matches to diverse relevant information. Our research demonstrates that multi-pass decoding, or diverse generation, can facilitate more comprehensive retrieval feedback based on varying outputs.

Secondly, we address a significant issue in the pipeline where retrieved documents might mislead the language model, leading to a correct output being inaccurately revised. We propose an ensemble strategy that allows for the re-evaluation of answer trustworthiness, thereby enhancing the reliability of the output. This aspect of our research presents a novel approach to managing the complexities inherent in retrieve-then-read pipelines.

Reviewer comment (Reproducibility):
How are the two enhanced modules integrated together?

Author response:
The two enhanced modules are built in two different steps. First, the diverse generation is built in the first step when generating initial answer, instead of generating only one answer, the system generates multiple diverse answer via sampling. Then, we use each generated initial answer to retrieve a set of documents and remove the duplicated documents. Lastly, the ensemble method is built in the second step when producing the final answer. The final output will be based on all initial answer likelihood and refined answer likelihood.

Reviewer comment (Experiments):
It is unclear whether the proposed approach would work with another backbone than davinci.

Author response:
Thank you for highlighting the importance of backbone model selection. Our initial decision to use Davinci-003 and Codex as backbone models was made to ensure a fair comparison with baseline methods (e.g., GenRead, RePLUG) that we have discussed in the paper. 

Besides, the ChatGPT and GPT-4 system might involve many user data for instruction tuning, which could have contamination with many academic benchmark dataset, such as NQ and TriviaQA. 

Lastly, in OpenAI’s announcement that both ChatGPT and GPT-4 will be subject to ongoing updates in their model parameters. These continual modifications would lead to non-reproducible experiments, potentially compromising the reliability of our research outcomes.

In response to your concern, we have conducted additional experiments using ChatGPT (gpt-3.5 turbo) and GPT-4, which reinforce the adaptability and effectiveness of our approach across different model architectures. These results are now detailed in the revised version of our paper, providing a comprehensive view of our method's applicability.


| Experiment on ChatGPT (gpt-3.5-turbo)                  | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 34.3 / 41.5 | 58.7 / 63.7 | 31.7 / 33.6 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.5 / 48.1 | 66.3 / 71.1 | 34.1 / 36.0 |


| Experiment on GPT4 (gpt-4)                             | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 34.8 / 49.6 | 64.6 / 72.8 | 30.8 / 33.7 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 32.5 / 46.5 | 59.9 / 67.1 | 31.6 / 37.5 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 36.8 / 54.4 | 66.3 / 74.0 | 36.9 / 42.6 |

From the above table, we can obserbe:

(1) The retrieval-feedback consistently improves the system, compared to both standard closebook QA method, and retrieve-then-read pipeline.

(2) The traditional retrieve-then-read system even sometimes hurt by noisy retrieval. On the contrary, our ReFeed could avoid misled issue if the first-round generation is good.

Reviewer comment (Experiments):
I'm skeptical that taking average probabilities of the output before and after the refinement would work in all models.

Author response:
Thanks for your question! Indeed, there is a fundamental limitation in ensuring that the language model is appropriately calibrated. And we admit our method should be based on a relatively well-calibrated language model. 

If the model is not well-calibrated, one way is to better measure uncertainty using semantic-based metrics [1], instead of using probability only. 

We add a limitation discussion in the paper to point out the issue and remind our ensemble strategy could only be applied on well-calibrated model. Besides that, our main retrieval feedback framework is not relying on model calibration.

Reviewer comment (Evaluation):
How would verify that the LLM is well-calibrated in order to decide whether a refinement is more plausible or not? If the model is not well-calibrated, how would you proceed?

Author response:
Thanks for your question! Indeed, there is a fundamental limitation in ensuring that the language model is appropriately calibrated. And we admit our method should be based on a relatively well-calibrated language model. 

If the model is not well-calibrated, one way is to better measure uncertainty using semantic-based metrics [1], instead of using probability only. 

We add a limitation discussion in the paper to point out the issue and remind our ensemble strategy could only be applied on well-calibrated model. Besides that, our main retrieval feedback framework is not relying on model calibration.

Reviewer comment (Experiments):
How would the approach generalize for other models than davinci?

Author response:
Thank you for highlighting the importance of backbone model selection. Our initial decision to use Davinci-003 and Codex as backbone models was made to ensure a fair comparison with baseline methods (e.g., GenRead, RePLUG) that we have discussed in the paper. 

Besides, the ChatGPT and GPT-4 system might involve many user data for instruction tuning, which could have contamination with many academic benchmark dataset, such as NQ and TriviaQA. 

Lastly, in OpenAI’s announcement that both ChatGPT and GPT-4 will be subject to ongoing updates in their model parameters. These continual modifications would lead to non-reproducible experiments, potentially compromising the reliability of our research outcomes.

In response to your concern, we have conducted additional experiments using ChatGPT (gpt-3.5 turbo) and GPT-4, which reinforce the adaptability and effectiveness of our approach across different model architectures. These results are now detailed in the revised version of our paper, providing a comprehensive view of our method's applicability.


| Experiment on ChatGPT (gpt-3.5-turbo)                  | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 34.3 / 41.5 | 58.7 / 63.7 | 31.7 / 33.6 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.5 / 48.1 | 66.3 / 71.1 | 34.1 / 36.0 |


| Experiment on GPT4 (gpt-4)                             | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 34.8 / 49.6 | 64.6 / 72.8 | 30.8 / 33.7 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 32.5 / 46.5 | 59.9 / 67.1 | 31.6 / 37.5 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 36.8 / 54.4 | 66.3 / 74.0 | 36.9 / 42.6 |

From the above table, we can obserbe:

(1) The retrieval-feedback consistently improves the system, compared to both standard closebook QA method, and retrieve-then-read pipeline.

(2) The traditional retrieve-then-read system even sometimes hurt by noisy retrieval. On the contrary, our ReFeed could avoid misled issue if the first-round generation is good.

Reviewer comment (Reproducibility):
W1: Some important details are missing, e.g., how to conduct de-duplication.

Author response:
We appreciate your emphasis on the need for detailed methodological descriptions. Regarding the de-duplication process, we first use each generated initial answer to retrieve a set of documents. For each document retrieved, we calculate a corresponding similarity score. Subsequently, we merge all retrieved documents and rank them based on their similarity scores, keeping only the top-k documents. This approach ensures that we maintain diversity in the information we gather and use.

There are two main reasons for choosing BM25 as our retrieval method. First, BM25 demonstrates excellent adaptability and generalizability across a variety of contexts. Second, dense retrieval methods, such as DPR, are often trained with QA datasets or synthetic query-document pairs. Therefore, evaluating such dense retrieval methods might disproportionately reflect their ability to provide retrieval feedback, as the retrieval more accurate than BM25. However, demonstrating that BM25 performs well within our retrieval feedback framework suggests that there is potential for significant improvement with more advanced retrieval methods.


Lastly, we want to summarize out motivation of each component:

(1)	General Retrieval Feedback Pipeline: Language models often generate information that is either incorrect or hallucinated. The retrieved information allows the language model to reassess the initial outputs and, if necessary, refine them to produce new answers.

(2)	The diverse retrieval strategy enhances the diversity of the outputs and enables more comprehensive retrieval feedback based on these varied outputs.

(3)	Ensemble method is motivated that there are times when the retrieved documents may mislead the language model, leading to a correct output being revised into an incorrect one. So, the ensemble technique takes into account both the initial and refined outputs, ultimately enhancing the overall performance.

Reviewer comment (Experiments):
W2: Not applied to the state-of-the-art LLMs (e.g., GPT-4).

Author response:
Thank you for highlighting the importance of backbone model selection. Our initial decision to use Davinci-003 and Codex as backbone models was made to ensure a fair comparison with baseline methods (e.g., GenRead, RePLUG) that we have discussed in the paper. 

Besides, the ChatGPT and GPT-4 system might involve many user data for instruction tuning, which could have contamination with many academic benchmark dataset, such as NQ and TriviaQA. 

Lastly, in OpenAI’s announcement that both ChatGPT and GPT-4 will be subject to ongoing updates in their model parameters. These continual modifications would lead to non-reproducible experiments, potentially compromising the reliability of our research outcomes.

In response to your concern, we have conducted additional experiments using ChatGPT (gpt-3.5 turbo) and GPT-4, which reinforce the adaptability and effectiveness of our approach across different model architectures. These results are now detailed in the revised version of our paper, providing a comprehensive view of our method's applicability.


| Experiment on ChatGPT (gpt-3.5-turbo)                  | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 34.3 / 41.5 | 58.7 / 63.7 | 31.7 / 33.6 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.5 / 48.1 | 66.3 / 71.1 | 34.1 / 36.0 |


| Experiment on GPT4 (gpt-4)                             | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 34.8 / 49.6 | 64.6 / 72.8 | 30.8 / 33.7 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 32.5 / 46.5 | 59.9 / 67.1 | 31.6 / 37.5 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 36.8 / 54.4 | 66.3 / 74.0 | 36.9 / 42.6 |

Reviewer comment (Experiments):
Q2: Following W2, I would really like to know how the proposed method can be applied to state-of-the-art models like GPT-4. Although it might not be reproducible for a certain metric number, it can still give some signs about the performance upper bound. Similarly, I wonder if the method can be applied to a conventional smaller neural language model (e.g., BART and T5) since the proposed is a plug-and-play method.

Author response:
Thank you for highlighting the importance of backbone model selection. Our initial decision to use Davinci-003 and Codex as backbone models was made to ensure a fair comparison with baseline methods (e.g., GenRead, RePLUG) that we have discussed in the paper. 

Besides, the ChatGPT and GPT-4 system might involve many user data for instruction tuning, which could have contamination with many academic benchmark dataset, such as NQ and TriviaQA. 

Lastly, in OpenAI’s announcement that both ChatGPT and GPT-4 will be subject to ongoing updates in their model parameters. These continual modifications would lead to non-reproducible experiments, potentially compromising the reliability of our research outcomes.

In response to your concern, we have conducted additional experiments using ChatGPT (gpt-3.5 turbo) and GPT-4, which reinforce the adaptability and effectiveness of our approach across different model architectures. These results are now detailed in the revised version of our paper, providing a comprehensive view of our method's applicability.


| Experiment on ChatGPT (gpt-3.5-turbo)                  | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 34.3 / 41.5 | 58.7 / 63.7 | 31.7 / 33.6 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.5 / 48.1 | 66.3 / 71.1 | 34.1 / 36.0 |


| Experiment on GPT4 (gpt-4)                             | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 34.8 / 49.6 | 64.6 / 72.8 | 30.8 / 33.7 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 32.5 / 46.5 | 59.9 / 67.1 | 31.6 / 37.5 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 36.8 / 54.4 | 66.3 / 74.0 | 36.9 / 42.6 |

Reviewer comment (Experiments):
Q3: Following W3, there are still several hyper-parameters (e.g., k in the module-1) in the proposed method. I wonder how they affect the performance (instead of "because other papers use this number").

Author response:
Previous research, as mentioned in papers [1] and [2], indicates that increasing the value of 'k' beyond 10 leads to only marginal improvements while simultaneously contributing to higher complexity. Additionally, as noted in [3], expanding the context length excessively for LLMs can result in “lost in the middle issue”, i.e., LLM ignores the middle part of a long text input.

To further investigate the effect of varying 'k', we conducted experiments using ChatGPT (gpt-3.5-turbo) on datasets such as NQ, TriviaQA, and HotpotQA. The results are shown in the following tables (metric: left EM, right accuracy). The experimental results are consistent with findings in [1] [2]. As shown in the Table when ‘k’ is set as 10 performs significantly better than when ‘k’ is set as 1 or 5, however, when ‘k’ is set as 20, the improvement is marginal over when ‘k’ is set as 10. 

We also observe that when 'k' is set as a small number, such as 1, the language model could be misled because it tends to trust the only one given documents. 

| K=1 with ChatGPT (gpt-3.5-turbo)                       | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 23.6 / 30.0 | 40.5 / 44.8 | 23.2 / 24.0 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 34.0 / 41.9 | 65.9 / 68.7 | 27.1 / 28.3 |

| K=5 with ChatGPT (gpt-3.5-turbo)                       | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 30.5 / 37.6 | 50.4 / 55.4 | 28.5 / 29.0 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 34.3 / 42.1 | 67.5 / 70.8 | 30.0 / 30.4 |

| K=10 with ChatGPT (gpt-3.5-turbo)                      | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 34.3 / 41.5 | 58.7 / 63.7 | 31.7 / 33.6 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.5 / 48.1 | 66.3 / 71.1 | 34.1 / 36.0 |

| K=20 with ChatGPT (gpt-3.5-turbo)                      | NQ          | TriviaQA    | HotpotQA    |
|--------------------------------------------------------|-------------|-------------|-------------|
| Closebook question answering (Question -> Answer)      | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 |
| Retrieve-then-Read (Question -> Retrieval -> Answer)   | 34.6 / 42.0 | 59.5 / 64.5 | 31.8 / 33.8 |
| ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.8 / 48.2 | 66.5 / 71.2 | 34.4 / 36.5 |


[1] Prompting GPT-3 To Be Reliable. ICLR 2023

[2] REPLUG: Retrieval-Augmented Black-Box Language Models. EMNLP 2023.

[3] Lost in the Middle: How Language Models Use Long Contexts. TACL 2023.