Paper: Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editing (ICLR-2024)

Reviewer comment (Writing):
The choice of using a reviewer to estimate scores instead of directly relying on end-task scores is an interesting one? Is it for computational reasons/costs to query GPT4? An elaboration would help (I may have misunderstood this aspect). I had to re-read the sections describing the author, reviewer, selector multiple times along with the algo block to be sure of what's going on. This section could benefit from some more details in an image and/or writing.

Author response:
***Regarding the Use of a Reviewer for Scoring:*** `The choice of using a reviewer to estimate scores instead of directly relying on end-task scores is an interesting one? Is it for computational reasons/costs to query GPT4? An elaboration would help (I may have misunderstood this as aspect). I had to re-read the sections describing the author, reviewer, selector multiple times along with the algo block to be sure of whats going on. This section could benefit from some more details in an image and/or writing.`

***Response***: We appreciate your observation about our use of the LLM-Reviewer. Our decision was driven by both efficiency and effectiveness:

- **Correlation with End-Task Accuracy:** 

One significant challenge in evaluating auto-prompt tasks, such as those in Instruction Induction, Big Bench, and the OpenLLM Leaderboard, lies in the limited size of the datasets. Typically, these datasets contain fewer than 1,000 entries, a stark contrast to the tens or hundreds of thousands of data points commonly used in traditional machine learning. This small sample size inevitably leads to high variance and instability in performance metrics when using a standard train-test split. As a result, the derived accuracy from such a small dataset may not reliably reflect the true task performance.

In contrast, LLMs like the one we employ in LLM-Reviewer are trained on vast and diverse data sources, encompassing a wide range of tasks. This extensive training equips LLMs with a robust understanding and capability to assess task performance more reliably. Our approach leverages this strength of LLMs. As shown in Figure 9, our analysis reveals a significant positive correlation between the evaluations made by LLM-Reviewer and the actual end-task accuracy. This correlation not only validates our methodology but also underscores the reliability of LLM-Reviewer as a proxy for task performance assessment.

Moreover, as evidenced in Table 1 below, the LLM-Reviewer outperforms the Author-Only approach in accuracy and stability. This further emphasizes the advantage of integrating LLM-Reviewer in the evaluation process, particularly when dealing with tasks characterized by limited data availability. By utilizing LLM-Reviewer, we are able to mitigate the challenges posed by small datasets and provide a more accurate and stable measure of task performance.

- **Computational Efficiency:**  Opting for the LLM-Reviewer's score rather than direct task accuracy for prompt selection notably enhances computational efficiency. This method reduces the frequency of querying the inference LLM (e.g., GPT-4), thus saving computational resources.

In the camera-ready version of the paper, we will use the additional one page to expand on these motivations and provide additional details for clearer understanding. We also plan to include a comparative analysis showing the performance enhancement when incorporating the reviewer, as outlined in the table below:

Table 1: Test Accuracy (%) Change on **Author-Reviewer vs. Author-only**

| Task | Author-Reviewer vs. Author-only |
| -------- | ------- |
| orthography starts with | +5.8%|
| common concept | +1.9% |
| rhymes | +3.1% |
| movie recommendation | +4.3% |
| logical fallacy detection | +3.9% |
| presuppositions as nli | +1.5% |
| winowhy | +3.6% |
| epistemic reasoning | +4.2% |
| SST-2 | +4.7% |
| QQP | +2.4% |
| adversarial SST-2 | +4.9% |
| adversarial QQP | +5.1% |
| NER (English) | +3.4% |

Reviewer comment (Experiments):
While the paper only uses GPT4, I was wondering if the authors could experiment with open-access models such as Falcon 180B and other smaller-scale models. I am sure that the authors would agree "LLMs" cannot do whatever the paper describes in general -- if someone does not use the GPT4 OpenAI model, is there a takeaway from this paper? A study of how this behavior changes with scale (of model sizes) could be a useful addition.

Author response:
***On Experimenting with Other LLMs:*** `While the paper only uses GPT4, I was wondering if the authors could experiment with open-access models such as Falcon 180B and other smaller-scale models. I am sure that the authors would agree "LLMs" cannot do whatever the paper describes in general -- if someone does not use the GPT4 OpenAI model, is there a takeaway from this paper? A study of how this behavior changes with scale (of model sizes) could be a useful addition.`

***Response***: Your question about the generalizability of our approach with different LLMs is very pertinent. We agree that not all LLMs exhibit the same level of proficiency in instruction-following. It's also crucial to note that model performance is influenced not only by size but also by factors like training methods and data curation. These aspects significantly impact a model's ability to follow instructions. To address this:

- **Additional Experiments with Diverse Models:** To further validate the versatility of our approach, we have extended our experimentation to include models such as Mistral-7B and Llama2-70B. Mistral-7B stands out as a top-performing model in the category of those under 30 billion parameters. Meanwhile, Llama2-70B represents larger-scale models, offering a comprehensive view across a broad spectrum of model sizes. These additional tests are designed to demonstrate the adaptability and robustness of our method, across model scale and inherent capabilities, thereby underscoring its broad applicability. We report the results in Table 2 and 3 below.


Table 2: Test Accuracy (%) Comparison on APE and Evoke for **Mistral-7B**


| Task | APE | Evoke | 
| -------- | ------- | ------- |
| orthography starts with | 29.3 |70.4|
| common concept | 3.3|22.7|
| rhymes | 66.9 | 74.7|
| movie recommendation | 26.4 | 72.7|
| logical fallacy detection | 48.4| 77.3 |
| presuppositions as nli | 21.6| 29.1|
| winowhy |18.2| 60.9 |
| epistemic reasoning | 52.6 | 69.3 |
| SST-2 |62.2 |70.0 |
| QQP |54.1| 73.1 |
| adversarial SST-2 | 51.3 | 64.2 |
| adversarial QQP | 47.5 |66.4 |
| NER (English) | 58.1 |83.5 |

Due to limited time and computational resources, we have not finished all experiments on Llama2-70B, we'd like to firstly present available result below.

Table 3: Test Accuracy Comparison (%) on APE and Evoke for **Llama2-70B**


| Task | APE | Evoke |
| -------- | ------- | ------- |
| orthography starts with | 39.8 |74.2|
| common concept | 3.6 | 21.9 |
| adversarial SST-2 | 67.3 |91.7 |
| adversarial QQP | 68.2 |89.6 |
| NER (English) |62.5|86.3 |

In summary, the applicability of Evoke extends beyond GPT-4, demonstrating compatibility with a wide range of commonly used open-source LLMs. 

We are grateful for the reviewer's insightful comments, which have provided us with an opportunity to enhance the clarity, detail, and overall effectiveness of our method. We welcome any further questions and are eager to provide additional information as needed.

Reviewer comment (Presentation):
Some of the detailed tables and figures could be relocated to the appendix to allow ample space for a more comprehensive description of the experimental settings and the Evoke method itself. A more detailed explanation of the Evoke process would enhance the paper's clarity.

Author response:
We agree with the suggestion to relocate some detailed tables and figures to the appendix. This will create space for a more thorough description of the experimental settings and the Evoke method. In the camera-ready version, we will include additional explanatory text to distinguish our main algorithm from [1], ensuring greater clarity for readers.

Reviewer comment (Experiments):
The rationale behind selecting the Instruction Induction and Big Bench Instruction Induction datasets is not clearly articulated. Some of these tasks may be practical, while others could be considered more akin to toy tasks. Additionally, the use of the LLM as merely GPT-4, a black-box large language model, raises the question of whether the Evoke method is equally effective with other large language models, such as Llama 2 or ChatGPT.

Author response:
- **Dataset Choice:** The selection of the Instruction Induction and Big Bench Instruction Induction datasets was strategic, aligning with current practices in auto-prompt work [1,2,3]. These datasets offer a broad spectrum of task difficulties, reflecting real-world LLM applications. We chose them for their relevance and to facilitate comparisons with existing and future studies in auto-prompting.
- **LLM Choice:** We selected GPT-4 for its state-of-the-art capabilities, as noted in [4]. However, it can also be extended to smaller scale, open source LLMs. To demonstrate this, we conducted additional experiments with Mistral-7B and Llama2-70B for inference, showcasing significant improvements across various tasks. The detailed results are presented in Table 1 of our response.


Table 1: Test Accuracy (%) Comparison on APE and Evoke for **Mistral-7B**


| Task | APE | Evoke | 
| -------- | ------- | ------- |
| orthography starts with | 29.3 |70.4|
| common concept | 3.3|22.7|
| rhymes | 66.9 | 74.7|
| movie recommendation | 26.4 | 72.7|
| logical fallacy detection | 48.4| 77.3 |
| presuppositions as nli | 21.6| 29.1|
| winowhy |18.2| 60.9 |
| epistemic reasoning | 52.6 | 69.3 |
| SST-2 |62.2 |70.0 |
| QQP |54.1| 73.1 |
| adversarial SST-2 | 51.3 | 64.2 |
| adversarial QQP | 47.5 |66.4 |
| NER (English) | 58.1 |83.5 |

Due to limited time and computational resources, we have not finished all experiments on Llama2-70B, we'd like to firstly present available result below.

Table 2: Test Accuracy Comparison (%) on APE and Evoke for **Llama2-70B**


| Task | APE | Evoke |
| -------- | ------- | ------- |
| orthography starts with | 39.8 |74.2|
| common concept | 3.6 | 21.9 |
| adversarial SST-2 | 67.3 |91.7 |
| adversarial QQP | 68.2 |89.6 |
| NER (English) |62.5|86.3 |

Reviewer comment (Evaluation):
Comparing Evoke to other baseline methods may be perceived as somewhat unfair, given that Evoke leverages supervised information from the training dataset. This distinction should be acknowledged when discussing the comparative results.

Author response:
We appreciate the concern regarding the fairness of comparisons. It's important to note that other baseline methods also leverage supervised information from training datasets. For human prompt engineers, viewing sample data is necessary to understand the given task, and during prompt iterations, failed cases of a certain prompt are also taken into account. For APE, it uses instruction induction to initialize the prompt, along with sample data. All approaches assume the availability of a given dataset with input-output pairs; where Evoke differs is in its more effective usage of this set.

Reviewer comment (Novelty):
A key concern revolves around the distinctions between Evoke and "Large Language Models as Optimizers" by DeepMind. Clarifying these differences is pivotal for a comprehensive understanding of both approaches.

Author response:
As previously mentioned, our method's efficiency and dual role of the LLM in Evoke are key differentiators from [1]. These aspects are crucial for understanding the unique contributions of our work.

Reviewer comment (Reproducibility):
Could you provide more information regarding the number of steps involved in generating prompts for each task? Additionally, is the prompt length substantial for each task?

Author response:
- **Number of Steps:** In our prompt generation process, each 'step' represents a stage in refining or expanding the prompt to better suit the task at hand. For all tasks, we have standardized the process to involve T=5 steps. This decision was grounded in a thorough empirical analysis. Initially, we experimented with a higher count of T=10 steps. However, we observed that beyond the fifth step, there was no significant increase in accuracy - the benefits started to plateau. This finding indicated that five steps struck an optimal balance, effectively enhancing prompt quality without unnecessary complexity. By maintaining this uniformity across different tasks, we not only ensure the comparability of our results but also optimize the efficiency of the process. This approach ensures that each prompt is effective yet concise, tailored to each task without excess verbosity.

- **Prompt Length Variation:** The length of the generated prompts indeed varies depending on the specific requirements of each task. For instance, the logical fallacy detection task necessitates detailed definitions within the prompt, while simpler tasks like movie recommendations do not.

Reviewer comment (Experiments):
Efficiency may be a concern when dealing with very large training datasets, as it necessitates running the large language model multiple times on the entire training dataset to obtain evaluation scores. A discussion on potential optimizations or trade-offs for such scenarios would be beneficial.

Author response:
We acknowledge that efficiency is a concern, especially with large datasets. However, using the generated prompt to obtain its performance on a training set is common for all auto-prompt works (including our baseline [2] and the work recommended by the reviewer [1]), as this process is used to evaluate the prompt during prompt editing.

Reviewer comment (Evaluation):
Section 4.4 highlights the effectiveness of the LLM-Selector. However, it remains unclear why, if the generated prompt is superior, there isn't consistently better or at least equivalent performance on easier examples. This phenomenon is briefly mentioned without a deeper exploration or analysis. Further elaboration would enhance the paper's clarity.

Author response:
Sorry for the confusion; The difference in performance between 'Hard' and 'Easy' samples is a deliberate part of our methodology to showcase the effectiveness of our selection process.

In our approach, we categorize samples into 'Hard' and 'Easy' based on their difficulty level:

In our study, 'Hard' samples are chosen for their high level of difficulty, while 'Easy' samples represent those with the lowest difficulty. The LLM-Author utilizes these samples to refine and edit prompts. The difficulty scores for these samples are determined using the LLM-Selector, with the specific prompts for this process detailed in Appendix B.

We find that this differentiation between 'Hard' and 'Easy' is crucial in demonstrating the versatility of our model. The significant drop in performance when the selection is reversed (i.e., when 'Hard' methods are applied to 'Easy' samples and vice versa) validates our selection method's effectiveness. This finding underscores the importance of appropriate sample selection in optimizing model performance, a point we discuss in greater detail in Section 4.4.

We hope this explanation clarifies the rationale behind our methodological choices and their impact on the model's performance. We remain open to further discussions and insights that can refine our approach.

Reviewer comment (Experiments):
Iteration-Dependent: The efficacy of the approach may heavily depend on the number of iterations and the quality of feedback loops, which could vary significantly based on the complexity of the task and the initial prompt quality.

Author response:
We appreciate your concern about the iteration-dependent nature of our approach. To clarify, we have standardized the number of iterations (T) to 5 across all tasks in our experiments. We have initially tried T up to 10, and setting T=5 was informed by preliminary experiments showing converging improvements in task accuracy at this iteration count, irrespective of task complexity. Furthermore, we experimented with varying initial prompts, including task descriptions and empty strings, to assess the sensitivity of our Evoke algorithm to initial prompt quality. Our findings, detailed in Table 1 of our response, demonstrate that Evoke's performance is robust to variations in initial prompt quality, consistently outperforming baseline methods.

Reviewer comment (Evaluation):
Complexity and Overhead: The Evoke model might introduce additional complexity and computational overhead since we need to do the iterative selection process T times.

Author response:
We acknowledge the reviewer's point about potential complexity and computational overhead. To contextualize this, we compared Evoke's process with the conventional human prompt engineering workflow, which typically involves several iterative steps of testing, analyzing, and refining prompts. Interestingly, Evoke operates more efficiently in terms of computation time while achieving comparable or superior accuracy. This efficiency gain is particularly notable given that Evoke's iterative process, even set at its most computationally intensive mode, matches the computational demands of a single iteration in the human-driven process. Furthermore, by limiting T to 5 in our experiments, we achieved a balanced trade-off between performance optimization and computational resource usage.

Additionally, in all experiments, we set T to 5, resulting a strong optimized prompt.

Reviewer comment (Reproducibility):
Are all the prompts generated by Evoke concatenated together during the evaluation phase?

Author response:
- **During Training (Prompt Editing) Evaluation Phase:** In the training phase of our method, each prompt generated by Evoke undergoes an independent evaluation process. We have adopted this strategy deliberately to mitigate any potential interactions or biases that could arise from prompt concatenation. This independent assessment of each prompt allows us to accurately gauge the effectiveness of every individual prompt generated, ensuring that our method's learning process is informed by clear, unbiased feedback.

- **During Inference Evaluation Phase:** For the evaluation phase during inference, our approach is different. Here, we utilize only the final optimized prompt generated by Evoke for calculating accuracy on the test set. This decision is based on our objective to evaluate the practical applicability of the final prompt in real-world scenarios. By focusing on the final prompt, we are able to assess the ultimate effectiveness of the Evoke algorithm in producing a single, highly effective prompt for practical use.

Reviewer comment (Experiments):
How do the quality of initial prompts affect the effectiveness of the Evoke method throughout iterations?

Author response:
The influence of initial prompt quality on the efficacy of Evoke was an important aspect of our investigation. To thoroughly examine this, we conducted additional experiments using two alternative initial prompts: a task description and an empty string. As illustrated in Table 1, Evoke's performance showed relative insensitivity to the quality of the initial prompt. These results, further underscore the robustness of our approach.

Table 1: Change in Task Accuracy Using Different Initial Prompts

| task | empty string vs. instruction induction | task description vs. instruction induction |
| -------- | ------- | ------- |
| orthography starts with | 1.52%| 0.59%|
| common concept | 0.13% | 1.28% |
| rhymes | -0.20% | 0.40% |
| movie recommendation | 1.27% | 1.35% |
| logical fallacy detection | -0.84% | 0.28% |
| presuppositions as nli | -0.48% | 0.51% |
| winowhy | -0.06% | 0.09% |
| epistemic reasoning | -1.23% | 0.52% |
| SST-2 | 0.08% | 1.28% |
| QQP | -0.69% | 0.40% |
| adversarial SST-2 | -0.25% | 0.71% |
| adversarial QQP | -0.53% | 1.50% |
| NER (English) | 0.61% | 0.97% |