Paper: Prompt Optimization via Adversarial In-Context Learning (ICLR-2024)

Reviewer comment (Novelty):
The novelty is limited: The resampling method (as the prompt modifier) is already proposed in APE (Zhou et al., 2023). The prompt used to modify instructions in this work is very similar to the prompt used in APE but without proper citation.

Author response:
To the best of our knowledge, this is the first work exploring optimizing discrete prompts for in-context learning (ICL) in the adversarial learning style. We propose adv-ICL which can effectively optimize the prompts. Despite simplicity, our method shows strong improvements over the baselines and achieves significant improvements over previous approaches while requiring very few training data.

That being said, novelty itself is a subjective matter. The whole deep learning field is built on empirical results with old ideas. The whole GPT series had very limited “novelty” in the traditional sense, except the prompting formulation. They just made it work. Our method also works. Papers like ours are the reason for why we have great AI technologies today.

Reviewer comment (Presentation):
The presentation needs to be improved; it is suggested to include a running example in the method section to explain the generator, discriminator, and prompt modifier, and how they interact with each other.

Author response:
We have updated an example in Figure 1 of the updated version. Please check it.

Reviewer comment (Presentation):
The limitations are missing.

Author response:
The limitation section is not required for ICLR. Please check the author guide (https://iclr.cc/Conferences/2024/AuthorGuide).

Reviewer comment (Evaluation):
Question: In Table 3, too many iterations \(T\) or samples \(m\) seem to harm the performance; could you give some explanation or hypothesis about why?

Author response:
This is an interesting point! We observed this phenomenon in the experiments and were also curious about it. Our current hypothesis is that, while in-context learning holds great promise, it does not behave similar to SGD based learning. For example, having more demonstrations or data samples in the prompt does not help. Also, as demonstrated by [1], a critical threshold exists for the number of training examples, and surpassing this threshold leads to a decline in performance. As a result, more iterations or more samples do not necessarily help. 

Given the complexity of in-context learning, we simply determine the best hyperparameters based on empirical results. We perform hyperparameter search based on the validation data of a small set of tasks. These hyperparameters do generalise to the other tasks. For details about the hyperparameter search process, you can refer to Section 2.4 in the original paper and Section 2.5 in the revised version.

Reviewer comment (Novelty):
Misc: There is a concurrent work you may discuss in related work: LLM as optimizers [1].

Author response:
We have included it in our related work discussion. The paper presents a novel LLMs optimizer that leverages feedback from LLMs to refine prompts effectively. This paper lies in the field of self-refinement and RL optimization m ethods, as explored in previous works [3] and [4]. However, our distinctive contribution lies in the study of adversarial in-context learning. We have included a discussion with the LLM optimizer in our related work.

Reviewer comment (Theory):
The convergence of the proposed approach. It seem that the prompts are modified by chance rather than gradient signals. This poses a serious concerns on the convergence and efficiency of the proposed approach. Yet the paper does not discuss this aspect in details.

Author response:
> W1. The convergence of the proposed approach.

In the updated draft, we included a new theoretical analysis section that shows the proposed algorithm converges to the desired equilibrium under mild assumptions. 

Here is the sketch of the analysis. 

### **Theoretical Results**
In this section, we theoretically analyze whether such a minimax objective in the form of in-context learning can achieve the desired equilibrium as in the original GAN scenario. We assume access to models with infinite capacities powering the discriminator $D$, generator $G$, and prompt modifier $M$ and that in each iteration, we sample a sufficient number of prompts from $M$ to update both $G$ and $D$. Let $p_{data}$ be the distribution of the training data, and $p_g$ be of the generated data from $G$. 

Considering a language model $\mathcal{M}$ which can be $D$ or $G$ performing a corresponding task $T_1$ and evaluated by a metric $E_1$,  we further assume that:
1. $M$ is powerful enough to modify the initial prompt of $\mathcal{M}$ for $T_1$, covering all possible prompt variants performing the task $T_1$. 
2. $\mathcal{M}$ is a powerful enough language model that there exists a prompt $\mathcal{P}$ of $T$ for $\mathcal{M}$ that given $\mathcal{P}, \mathcal{M}$ can achieve the globally optimal result on $E_1$ for the task $T$.
3. There exists a prompt sampled by $M$ that maximizes $\mathcal{M}$ on $E_1$ globally on $T$.

The assumption 3 is a result of the assumptions 1, and 2, and the assumption about our access to infinite capacities language models. Indeed, given $\mathcal{M}$, from assumption 2, there exists a globally optimized prompt $\mathcal{P}$ of $T_1$ for it such that it can achieve the globally optimal state on $E_1$ for the task $T_1$. Furthermore, since $M$ is powerful enough in modifying the initial prompt (ass. 1), plus $M$ samples a sufficiently large number of prompts for each iteration (ass. 2), $M$ can generate $\mathcal{P}$ with a non-zero probability, which conclude the assumption 3.

From the assumptions, we have the following propositions.

**Proposition 1 (Goodfellow et al., 2014).** For G fixed, the optimal discriminator D can be described in a closed form, denoted as D*.

**Proposition 2.** For each training iteration, for a fixed $G$, the optimal discriminator $D^*$ can be achieved.

**Proposition 3 (Motivated by Goodfellow et al., 2014).** If $G$ and $D$ have enough capacity, and at each training step, the discriminator is allowed to reach its optimum $D^*$ given $G$, and $p_g$ is updated so as to improve the criterion 

The proofs for the propositions are presented in *Appendix A.1* of our revised manual script. Our conclusion is that *with strong enough $D,G,M$, the framework adv-ICL converges*.

Reviewer comment (Experiments):
Disregarded Prompts: During the iterative training process, when a new prompt is introduced to optimize the loss function, the model (either the generator or the discriminator) engages in in-context learning, regardless of whether the new prompt is accepted or discarded. It's worth noting that discarded prompts can also impact the model's training, but the paper does not explore this aspect in detail.

Author response:
> W2a. It's worth noting that discarded prompts can also impact the model's training 

We respectfully think that the reviewer misunderstood our method. First, it is worth noting that our method does not require any update in the models’ parameters, it only updates the prompts of the discriminator and generator. Second, we only forward the best-performing prompt from the previous training iteration to the next training iteration for modification, and it is reasonable for us to further optimize this best-performing prompt without considering the discarded ones.

Reviewer comment (Evaluation):
Unclear Origin of Benefits: The observed performance improvements in the experiments may not be solely attributed to the accepted examples but could also be influenced by the discarded ones. Furthermore, given that the optimization process relies on the stochastic behavior of large language models, it's possible that the learning effect results from the repeated trial-and-error of prompt modifications occurring during the training iterations rather than from improved prompts. Therefore, the assumption that improvements in experimental results are solely due to adversarial training may not be valid.

Author response:
> W2b. Effect results may come from the repeated trial-and-error of prompt modifications

In addition to our arguments in W2a, we further conduct experiments to verify whether the prompt modifier module worked as expected. Specifically, we deactivate the discriminator and only employ a prompt modifier to repeatedly optimize the prompt. Our experimental results with Vicuna and ChatGPT are provided below:

| Model | WebNLG | Translation (RO->EN) | YELP | GSM8K |
| -------- | ------- | ------- | ------ | ------ |
| Vicuna | 52.5 | 72.1 | 71.0 | 40.7 |
| Vicuna with Adv-ICL w/o Discriminator | 50.1 | 71.4 | 72.1 | 40.2 |
| Vicuna with Adv-ICL | 59.3 | 73.4 | 73.6 | 43.9 |
|  |  |  |  |  |
| ChatGPT | 60.9 | 78.8 | 69.8 | 79.4 |
| ChatGPT with Adv-ICL w/o Discriminator | 61.2 | 77.4 | 64.5 | 71.6 |
| ChatGPT with Adv-ICL | 63.6 | 80.4 | 71.9 | 82.3 |

In the majority of cases, when the discriminator is deactivated and only the prompt modifier is relied upon under Vicuna and ChatGPT, there is a noticeable decline in performance. This decline can be attributed to the inherent randomness of the prompt modifier, which lacks the ability to discern a more effective prompt. This observation underscores the substantial significance of the discriminator and adversarial loss in the optimization process.

Reviewer comment (Experiments):
Is there a method to modify the prompts that doesn't involve trial-and-error? How does it compared to gradient descent?

Author response:
> Q1. Avoid trial-and-error and adopt gradient descent.

We have conducted experiments by incorporating the best-performing prompts from multiple previous iterations as feedback to refine the prompt in subsequent iterations (refer to our response to Reviewer iBKM Q1 for more information). However, when it comes to optimizing a discrete prompt, directly applying gradient descent is not feasible. In [5], the concept of 'gradient' in the context of a discrete prompt is defined, and our approach consistently outperforms it. For further details, please check Table 1 and Table 2 in the paper.

In the case of a soft prompt such as [6] [7], gradient descent can be employed for optimization. Nevertheless, there is no universally applicable loss function. For instance, in tasks like summarization, reference-based metrics are ineffective for measuring performance accurately [8].

Apart from all the pros and cons of gradient based approaches, the biggest problem is that it cannot be applied on state-of-the-art closed source models. We recognize the importance of doing experiments with open sourced models like Vicuna but the closed source models such as ChatGPT currently have better performance. So to know the effectiveness of our method, we need to perform experiments on closed source models.

Reviewer comment (Evaluation):
What are the training time and memory usage of your approach compared to the baselines?

Author response:
> Q2. Training time and Memory usage

Our method does not involve updating any parameters of the models and only relies on the inference state of LLMs. The training time for any models and settings is under 10 minutes, and the memory usage for vicuna-13b does not exceed 32GB.

Reviewer comment (Evaluation):
Does your GANs framework converge effectively? What's the learning curves for the generator and discriminator, respectively?

Author response:
> Q3. Does your GANs framework converge effectively?

Please refer to the response to W1 for more details.

Reviewer comment (Reproducibility):
How reliably can the experimental results be reproduced considering the stochastic nature of the optimization process and an LLM itself?

Author response:
> Q4. Randomness

We have rerun our experiments three times on WebNLG, RO->EN, YELP, and GSM8K. Below are our results with format run1/run2/run3:

| Model | WebNLG | Translation (RO->EN) | YELP | GSM8K |
| -------- | ------- | ------- | ------ | ------ |
| Vicuna | 59.3/59.2/59.5 | 73.4/74.1/73.2 | 73.6/73.6/73.5 | 43.9/44.3/44.1 |
| ChatGPT | 63.6/63.5/63.8 | 80.4/80.6/80.6 | 71.9/71.8/71.9 | 82.3/82.5/82.2 |

The results clearly demonstrate that Adv-ICL consistently delivers stable results, thereby highlighting its reliability in faithfully reproducing our experimental findings.

Reviewer comment (Experiments):
In terms of fairness, it seems that not all methods in the experiments use the prompts generated (or modified) by the same prompt modifier. Why?

Author response:
> Q5. Not all methods in the experiments adopt the same prompt modifier.

We conducted experiments with various baselines, each employing distinct prompt modification strategies. In order to make sure that we stay faithful to the originally proposed formulations, we used prompt modification strategies proposed by these techniques.

Reviewer comment (Experiments):
Some more RL-based prompt optimization baselines (e.g., Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, & Zhiting Hu. (2022). RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning.) could be used in the evaluation section to provide more insight.

Author response:
These RL-based baselines could not be applied to the closed source models such as ChatGPT and text-davinci. To test the effectiveness of our method, we need to perform experiments on these models because they have better performance. This is why we made the decision to not compare with RL-based methods. Specifically, [1] and [2] require access to LLMs for inserting MLPs. 

We compared adv-ICL with state-of-the-art prompt learning methods including GPS, APO and APE and outperformed them consistently, which indicates the effectiveness of our method.

Reviewer comment (Experiments):
It seems that in order to make edits to the prompts, the prompt modifier is prompted with a template text, and the last best-performing generator and discriminator prompts. What if you provide more feedback to the prompt modifier, such as the last best-performing prompt, alongside its predecessor, and how much better this last prompt performed than the other?

Author response:
We conducted an experiment that involved integrating the most successful prompts from previous iterations as feedback for the next iteration. In this process, we utilized previous best-performing prompts, namely $P_1, P_2, ..., P_k$, as inputs to the prompt modifier module when generating the $(k+1)$-th prompt. 

We applied the method to four representative tasks, including data-to-text generation task WebNLG, machine translation, sentiment classification Yelp, and reasoning GSM8k, using Vicuna and ChatGPT. The experimental results are shown below:

| Model | WebNLG | Translation (RO->EN) | YELP | GSM8K |
| -------- | ------- | ------- | ------ | ------ |
| Vicuna | 52.5 | 72.1 | 71.0 | 40.7 |
| Vicuna with Adv-ICL (prompt modifier with history) | 56.9 | 74.0 | 74.2 | 42.2 |
| Vicuna with Adv-ICL | 59.3 | 73.4 | 73.6 | 43.9 |
|  |  |  |  |  |
| ChatGPT | 60.9 | 78.8 | 69.8 | 79.4 |
| ChatGPT with Adv-ICL (prompt modifier with history) | 62.1 | 79.8 | 72.1 | 80.9 |
| ChatGPT with Adv-ICL | 63.6 | 80.4 | 71.9 | 82.3 |

In the case of Vicuna, incorporating additional feedback into the prompt modifier proves effective for tasks such as translation and classification. However, this approach falls short when applied to data-to-text and reasoning tasks. On the other hand, for ChatGPT, augmenting the prompt modifier with more feedback does not yield improved performance.

Reviewer comment (Evaluation):
For human evaluation, it is mentioned that annotators are tasked with verifying whether the sampled instruction/demonstration is semantically similar to the original one or not. What if you also add a module that can automatically verify this, check the content preservation score, and also consider this metric when choosing a prompt?

Author response:
As shown in the “Human evaluation of prompt modifier performance” part of Section 3.3, most of the prompts outputted by the prompt modifier are semantically correct. Specifically, the correct percentage of  text-davinci-002, ChatGPT, and Vicuna are 88%, 91%, and 83% respectively. So we do not think that it is necessary to automatically verify modifications.