Paper: Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL (ICLR-2024)

Reviewer comment (Experiments):
My main concern is that it lacks a very important comparison with “on-line” methods. If the proposed method can achieve similar (or just a little bit lower) performance compared with “on-line” methods and it is significantly cheaper, it would be a good evidence that the proposed method can potentially be useful practically.

Author response:
We deeply appreciate your time and effort devoted to reviewing our paper. And we are happy to have your acknowledgment of the significance and soundness of our work. Your insightful comments on the relationship between Prompt-OIRL and the online methods remind us to better present and contrast those approaches in our revision, and we have added a corresponding discussion section to our revised manuscript in Appendix D (New).**

---

 In the following, we would start by noting that the online methods are the foundations for our query-dependent offline approach, and **additional prompts discovered by online algorithms can improve the performance of Prompt-OIRL.**

1. Additional prompt generated by the online method improves Prompt-OIRL’s performance upper bound.

As the number of available prompts increases, the best-of-n performance among those prompts will be monotonically better. In addition to the evidence we have shown in the paper (Figure 6), the same conclusion can be drawn by comparing the performance upper bound of Prompt-OIRL on different tasks. 

In the table below, we use UB-Train-6 to denote the best performance (performance Upper-Bound) that can be achieved using the training prompts on the test queries, and use UB-Test-10 to denote the best performance that can be achieved using the 10 test prompts. Finally, we use UB-16 to denote the best performance that can be achieved using all those 16 prompts.

| Dataset | LLM        | UB-Train-6 | UB-Test-10 | UB-16 |
|---------|------------|--------------------------|-------------------------|--------------------------|
| MAWPS   | GPT3.5-turbo | 0.942                | 0.941                  | 0.957                   |
| SVAMP   | GPT3.5-turbo | 0.882                | 0.869                  | 0.903                   |
| GSM8K   | GPT3.5-turbo | 0.872                 | 0.873                   | 0.914                    |
| MAWPS   | LLaMA2-7B   | 0.891                 | 0.855                  | 0.926                   |
| SVAMP   | LLaMA2-7B   | 0.876                 | 0.837                  | 0.907                   |
| GSM8K   | LLaMA2-7B   | 0.582                  | 0.501                   | 0.648                    |
| MAWPS   | TigerBot-13B | 0.929                | 0.642                  | 0.935                   |
| SVAMP   | TigerBot-13B | 0.928                 | 0.913                 | 0.954                   |
| GSM8K   | TigerBot-13B | 0.754                 | 0.7                     | 0.822                    |

From the results, we can observe that the best performance can be achieved by the test prompts are usually worse than the best of the training prompts — this is not surprising given the fact that those training prompts are generated and optimized by experts or online algorithms. That said, we can also observe that selecting from the combination of all those 16 prompts can always lead to a significant improvement. Such an observation verifies the idea that adding additional prompts that are generated by the online prompts, either to the training set or to the test set, can be beneficial to Prompt-OIRL.

2. Monotonic Performance Improvement with the Number of Prompts.

To add additional empirical evidence to our discussion, we can observe from Figure 6 in the main text that all methods, including all baseline methods and our proposed method, can have improved performance when more available prompts are available.

The table below contextualizes the averaged performance improvement over different LLMs and tasks. 
We use “Improv. w/ +train” to highlight the improvement achieved using 6 training prompts as compared to only using 1 prompt. We use “Improv. w/ +test” to highlight the improvement achieved using additional test prompts as compared to only selecting from the training prompts (i.e., the improvement of Prompt-OIRL over BoTr Eqn(2)). 

| Method      | K = 1    | K = 2    | K = 3    | K = 4    | K = 5    | K = 6    | Improv. w/ +train |
|-------------|----------|----------|----------|----------|----------|----------|--------------------|
| LLM-Conf.   | 0.4751   | 0.5184   | 0.5427   | 0.5543   | 0.5594   | 0.5601   | + 17.89%           |
| BoTr Eqn(1) | 0.4751   | 0.5379   | 0.5654   | 0.5800   | 0.5893   | 0.5944   | + 25.11%           |
| BoTr Eqn(2) | 0.4751   | 0.5553   | 0.5910   | 0.6130   | 0.6295   | 0.6393   | + 34.56%           |
| Prompt-OIRL | 0.5904   | 0.6095   | 0.6225   | 0.6331   | 0.6410   | 0.6459   | + 9.4%             |
| Improv. w/ +test | +24.27% | +9.76% | +5.33% | +3.23% | +1.83% | +1.03% |          -          |

**The monotonic improvement in all those comparisons demonstrates the benefit of integrating additional prompts, which can either be discovered through an existing online algorithm or crafted by human experts.**

---

Once again, we thank the reviewer for their effort in improving our work. If there should be any remaining concerns or questions, we are keen to do our utmost to address them.

Reviewer comment (Evaluation):
While the method looks promising, I still expect to see potential discussion about the limitations, e.g., stability of inverse RL?

Author response:
Thank you for raising the concern about stability and robustness. In general, our offline-IRL approach enjoys high stability since the reward modeling step is based on gradient boosting methods and optimized through supervised learning. Similar to other offline-RL literature, learning with a fixed dataset also avoids the dilemma of exploration-exploitation trade-off which is usually challenging in the online approach. (e.g., in OPRO, APE, and APO [1-3]). In our work, we applied xgboost with a universal hyper-parameter setting to demonstrate the robustness of our approach.

A temporary challenge for Prompt-OIRL’s wide application lies in the availability of open-accessible data. We will elaborate on this point with more details in A3 below.

Reviewer comment (Writing):
The scope is limited by arithmetic reasoning but the title seems a more generic framework that can be used to solve more broader tasks across different NLP tasks.

Author response:
Thank you for the suggestion. To avoid potential misunderstanding, we have updated our title with a subtitle in our revision
to better highlight the use case studied in our work. The updated title is 
> *Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL
— A Case Study on Arithmetic Reasoning*

Reviewer comment (Experiments):
What's the current bottleneck if the proposed framework is applied to other instruction prompt optimization tasks, listed in ORPO[1], APE[2], and APO[3] baseline methods?

Author response:
Thank you for this question. We would answer this question by first providing our rationale in focusing on the arithmetic reasoning tasks and then highlighting a challenge and opportunity for the community — inspired by the success of Prompt-OIRL.

1. Motivation of centering on the arithmetic reasoning tasks

The current scope of our work is centered on arithmetic reasoning tasks as the arithmetic reasoning ability is considered to be one of the emergent abilities of LLMs [4] that has “been essential yet challenging for language models and NLP models broadly” [5,6]. On the other hand, the rich demonstration data and expert knowledge in zero-shot prompting developed for arithmetic reasoning tasks enable accurate and rigorous study of the potential and effectiveness of the proposed method.

2. Challenge and opportunity: Accessibility of Offline Data - we would call on the community’s attention by demonstrating the value of those assets through the success of Prompt-OIRL

To apply the key insight of Prompt-OIRL to more general natural language generation tasks, the challenge is on the **open-accessibility of the offline datasets.**

Intuitively, one may think the challenge lies in the scarcity of prompting knowledge (i.e., the number of expert-crafted prompts). Yet as a matter of fact, an important observation we have made in our work is that in arithmetic reasoning datasets is that learning a reward model for query-dependent prompt evaluation and optimization **is helpful even in the scarce-expert knowledge setting**. Therefore, the scarcity of expert-crafted prompts is not a vital challenge. 

On the other hand, the accessibility of open-sourced offline datasets — under the form of (query, prompt, prompted response) — is more important. In the era of LLMs, only a few researchers release their prompted responses when releasing their discovered prompting strategies. Even having access to their code, to fully reproduce their results still require intensive interactions with the LLMs, leading to a huge amount of cost. 

Motivated by the discovery made by Prompt-OIRL, those offline datasets — not only those interactive communication logs with LLMs for the discovered well-performing prompts but also those logs for the failed prompting attempts — can be useful for learning a better reward model. Therefore, with the success of Prompt-OIRL, we would like to call on the community to release not only code but also the interaction logs (i.e., offline demonstration dataset) as an asset. Learning with those demonstrations saves energy and computation by avoiding repetition, and enhances the development of prompt engineering.

**Actions taken** We have updated our manuscript and added this discussion to the appendix.

Reviewer comment (Experiments):
Without query dependence, what's the performance drop? Can you prove the necessity of that?

Author response:
We thank the reviewer for raising the question. 

Yes, the query-dependent objective is necessary and can be proved through our ablation studies. In our work, the improvement of query dependency can be empirically verified by comparing BoTr using Eqn(1), which is query-independent, and Eqn(2), which is query-dependent. Taking query-dependency into consideration, the performance on arithmetic reasoning tasks can be improved up to 7\%. The improvement over different demonstration availability is presented in Figure 6.

Reviewer comment (Experiments):
Is that possible to compare with the GPT-4 or PaLM 2 model as well?

Author response:
Thank you for the question.

We additionally experiment with GPT-4 on the MAWPS dataset. With a stronger language model, the reward modeling step of Prompt-OIRL will be more challenging, this is because of the increased imbalance of the data used for training the reward model (i.e., most prompts lead to a correct answer). That said, under such a more challenging setting, the efficacy of Prompt-OIRL and the query-dependent objective based on the learned reward model have been demonstrated. 

| Method      | K = 1       | K = 2       | K = 3       | K = 4       | K = 5       | K = 6       |
|-------------|-------------|-------------|-------------|-------------|-------------|-------------|
| BoTr Eqn(1) | 0.8781      | 0.9068      | 0.9131      | 0.9190      | 0.9245      | 0.9300      |
| BoTr Eqn(2) | 0.8781      | 0.9352      | 0.9513      | 0.9609      | 0.9685      | 0.9757      |
| Prompt-OIRL | 0.8944      | 0.9401      | 0.9526      | 0.9607      | 0.9672      | 0.9745      |

As we have discussed in A3, the challenge of applying Prompt-OIRL to different settings is mainly on their data availability, which can be generated as by-product when researchers evaluating different prompts. Given the computational efficiency advantage of Prompt-OIRL, deploying our method on new tasks/LLMs will be smooth as long as the offline data is available.

**Actions taken** We have added the new results to our updated manuscript. (Appendix E.3)

Reviewer comment (Experiments):
Dataset generation requires a large number of model interactions for every new task and model, as no cross-task or -model generalization has been demonstrated.

Author response:
### Q6. Generalization over tasks, and requirements on data.

### A6.
Through the empirical results in our main text and A4, we demonstrated that Prompt-OIRL improves over baselines in both limited training data size settings and scarce expert-crafted prompting strategy settings. In this section, we follow the reviewer’s suggestion to further investigate the generalization ability of Prompt-OIRL over different tasks. 


1. Prompt-OIRL achieves promising results in when generalized over tasks

We agree with the reviewer that the generalization of Prompt-OIRL over tasks can have great value in practical deployments. To verify the generalization ability of the learned reward model, we experiment by using reward models that are trained on different tasks in Prompt-OIRL when evaluating and optimizing prompts for the MAWPS task with GPT3.5-turbo. The following table shows the results we get using the three different reward models trained with MAWPS, SVAMP, and GSM8K, individually. 

| Model  | Method      | K = 1  | K = 2  | K = 3  | K = 4  | K = 5  | K = 6  |
|--------|-------------|-------|-------|-------|-------|-------|-------|
| MAWPS  | BoTr Eqn(1) | 0.6598 | 0.6988 | 0.7118 | 0.7155 | 0.7173 | 0.7175 |
|        | BoTr Eqn(2) | 0.6598 | 0.7742 | 0.8150 | 0.8350 | 0.8473 | 0.8546 |
|        | Prompt-OIRL | 0.7637 | 0.8032 | 0.8379 | 0.8750 | 0.8916 | 0.8944 |
| SVAMP  | BoTr Eqn(1) | 0.6598 | 0.6959 | 0.7078 | 0.7101 | 0.7124 | 0.7163 |
|        | BoTr Eqn(2) | 0.6598 | 0.7367 | 0.7529 | 0.7587 | 0.7593 | 0.7626 |
|        | Prompt-OIRL | 0.7222 | 0.7497 | 0.7757 | 0.7804 | 0.7883 | 0.7903 |
| GSM8K  | BoTr Eqn(1) | 0.6598 | 0.6753 | 0.6922 | 0.6968 | 0.6967 | 0.6926 |
|        | BoTr Eqn(2) | 0.6598 | 0.6880 | 0.6982 | 0.7016 | 0.7007 | 0.7092 |
|        | Prompt-OIRL | 0.6617 | 0.6884 | 0.6985 | 0.7015 | 0.7012 | 0.7145 |

It is worth noting that, under such a cross-task setting, the BoTr Eqn(1) baseline selects the best prompt according to the training set performance on the training dataset of the reward model. For instance, when generalizing the reward model learned from the GSM8K prompting demonstration dataset to the MAWPS task, its Best-of-Training prompt is chosen to be the best-performing prompt on the GSM8K dataset.

According to the results, we find the reward model learned with the SVAMP dataset generalizes better to the MAWPS task than the reward model learned with the GSM8K dataset, this is not surprising as those two tasks are more similar as compared to the GSM8K dataset which always requires multiple step reasoning. Importantly, the performance of Prompt-OIRL consistently surpasses using the best-of-training baselines, demonstrating its potential in practical deployments where generalization over tasks is considered to be an essential advantage.



2. Generalization over LLM requires a strong assumption

Following the common practice in RLHF, where reward models are also used, Prompt-OIRL uses LLM-specific reward models in prompt evaluation. This is due to the fact that different language models are often trained with different model architectures, training settings, and data. On the other hand, generalizing the learned reward model across language models assumes the prompting mechanism is agnostic to all of those differences, which is unlikely to hold. Empirically, the recent paper of OPRO [2] has shown evidence that the optimal prompting strategies for different language models are different.

We thank the reviewer again for their insightful comments, as because of the promising results in cross-task generalization, the potential impact and applicability of our proposed method is greatly improved. 

**Actions taken** We have added additional experiments studying the generalization of Prompt-OIRL over tasks (Appendix E.2). The promising results demonstrate the potential of Prompt-OIRL in real-world deployments.



**_Reference_**

[2. OPRO] Yang, Chengrun, et al. "Large language models as optimizers." arXiv preprint arXiv:2309.03409 (2023).

---

Once again, we thank the reviewer for their effort in improving our work. If there should be any remaining concerns or questions, we are keen to do our utmost to address them.

Reviewer comment (Reproducibility):
Many experimental details remain unclear or are only discussed in the appendix. E.g. the modelling of the proxy reward is not discussed at all in the main text, with the appendix suggesting a combination of an LLM embedding of prompt and query followed by an XGBT.

Author response:
### Q1. Can you describe in detail the modeling of the proxy reward and conduct an ablation over the approaches mentioned but rejected in Appendix C.2

### A1. 

Thank you for raising your concern and sharing your interest in the details of the reward modeling. In our initial draft, we deferred the detailed information about the offline dataset and the discussion on the reward model to the appendix due to space constraints in the main text. We have moved the key contents of reward modeling implementation into the main text to make it further self-contained. 

Moreover, we agree with the reviewer that providing a more detailed empirical comparison and analysis will be invaluable not only for enhancing the understanding of our method but may also contribute to the general community in understanding the learning with embeddings. 
We have extended our discussion on reward modeling (in Appendix C) to include the following empirical evidence we observed in experiments.


1. MLPs easily converge to the trivial classifiers

For the MLP model, we have tried different choices of its hyper-parameters, including unit numbers, layers, various drop-out choices, and dual-channel architecture with each channel process query and prompt embeddings individually. However, we find all of those choices tend to converge to the trivial solution that predicts either all 0 or all 1 for the binary classification task. Such a reward model does not actually have the prediction ability in inference time. In practice, with such a reward mode, we can do nothing better than select the best-performing prompt on the training dataset. Therefore, the performance of BoTr Eqn.(1) represents the best-achievable performance of using a dummy MLP reward model.


2. XGBoost is robust on hyper-parameter for reward modeling

For the xgboost method, we set a universal hyper-parameter for all tasks and LLMs. When deploying Prompt-OIRL in practice, a case-by-case engineering on the hyper-parameters on reward modeling should be able to further boost the performance of the algorithm, yet this is out of our research scope. In our paper, we experiment with a single universal hyper-parameter setting for all LLMs and tasks, demonstrating the robustness of the proposed method.


3. Instance-wise prediction of outcome is better than pair-wise prediction (preference-based learning)

Another alternative for reward modeling is based on preference-based learning, which is effective in some conventional inverse RL tasks [1. T-REX]. To be specific, for every training query $x^{(i)}$, there may exist multiple prompts (denoted as $p_+^{(i)}$) that lead to a correct answer, and some other prompts ($p_-^{(i)}$) that lead to a wrong answer. 


We can also organize the offline prompt demonstration in the form of preference-based dataset $(x^{(i)}, p_-^{(i)}, p_+^{(i)} )$, and learn from those pair-wise preference data. In this pair-wise preference approach, the learned reward model takes both prompts and the query as input, and outputs the preferred prompt. Given a new query in the inference time, such a reward model can be applied to all K candidate prompts with K-1 comparisons to find the best prompt. (so it is more computationally expensive than the direct reward modeling method used in our work). We empirically studied whether such an approach can lead to better performance on the MAWPS dataset with the GPT-3.5-turbo model. The results are shown in the table below:


| # Training Prompts | 1          | 2          | 3          | 4          | 5          | 6          |
|--------------------|------------|------------|------------|------------|------------|------------|
| BoTr Eqn(1)        | 0.6598     | 0.6988     | 0.7118     | 0.7155     | 0.7173     | 0.7175     |
| BoTr Eqn(2)        | 0.6598     | 0.7742     | 0.8150     | 0.8350     | 0.8473     | 0.8546     |
| BoTr Eqn(2) Paired | 0.6598     | 0.7195     | 0.7203     | 0.7194     | 0.7236     | 0.7247     |
| Prompt-OIRL        | 0.7637     | 0.8032     | 0.8379     | 0.8750     | 0.8916     | 0.8944     |


It can be concluded that the pair-wise reward model can not achieve better performance than the direct reward modeling used in Prompt-OIRL. 


All our offline datasets and code for processing those datasets, as long as the MLP implementation and pair-wise reward modeling implementation will be made publicly available and contributed as an asset for future research.




**Actions taken** We have updated our manuscript accordingly to include extended details for the reward model training. Revisions have been highlighted in red in our updated manuscript.



---

**_Reference_**

[1. T-REX] Brown, Daniel, et al. "Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations." International conference on machine learning. PMLR, 2019.

Reviewer comment (Evaluation):
While the use of LLMs for embedding computation of prompts is unproblematic when choosing the best-of-n (with n=6) fixed prompts, it might be prohibitively expensive for different policy optimization approaches requiring a substantially larger number of prompts to be embedded. This should be highlighted more prominently and not presented as a major advantage of this method.

Author response:
### Q5. While the use of LLMs for embedding computation of prompts is unproblematic when choosing the best-of-n (with n=6) fixed prompts, it might be prohibitively expensive for different policy optimization approaches requiring a substantially larger number of prompts to be embedded. This should be highlighted more prominently and not presented as a major advantage of this method.

### A5. 

Using the embedding models to get prompt or query embeddings is in general much cheaper than calling the LLMs in getting responses. 
For instance, the gpt-3.5-turbo API charges 0.001 USD per 1k tokens for input, and 0.002 USD for output; as a comparison, the embedding model of ada v2 charges only 0.0001 USD per 1k token.

Moreover, we would like to note those prompt embeddings are re-usable. For instance, getting the embedding for 100 prompts like “let’s try to solve it step by step, and make sure to get the correct answer” will cost about 100 prompts *  20 tokens/per prompt * 0.0001 USD / 1000 tokens = 0.0002 USD. 

In comparison, **interaction with the LLMs to get the prompted response is much more expensive.** A single query-answer pair will lead to a cost of approximately 0.0006 USD. Evaluating the performance of a single prompting strategy by interacting with gpt-3.5-turbo on a medium-sized dataset containing 5000 samples will lead to a 3 USD expense.

Therefore, the cost of Prompt-OIRL in generating embeddings ($\approx$0.0002 USD) is significantly cheaper than the cost of evaluating any prompt with LLMs ($\approx$3 USD). The gap in cost will be even more significant when multiple prompts need to be evaluated (i.e., evaluating K different prompts will cost $\approx$3K USD).

**Actions taken** We have added an additional section of discussion in our manuscript to discuss the expenses of embedding. (New: Appendix C.7)

Reviewer comment (Experiments):
Can you describe in detail the modeling of the proxy reward and conduct an ablation over the approaches mentioned but rejected in Appendix C.2.

Author response:
### Q1. Can you describe in detail the modeling of the proxy reward and conduct an ablation over the approaches mentioned but rejected in Appendix C.2

### A1. 

Thank you for raising your concern and sharing your interest in the details of the reward modeling. In our initial draft, we deferred the detailed information about the offline dataset and the discussion on the reward model to the appendix due to space constraints in the main text. We have moved the key contents of reward modeling implementation into the main text to make it further self-contained. 

Moreover, we agree with the reviewer that providing a more detailed empirical comparison and analysis will be invaluable not only for enhancing the understanding of our method but may also contribute to the general community in understanding the learning with embeddings. 
We have extended our discussion on reward modeling (in Appendix C) to include the following empirical evidence we observed in experiments.


1. MLPs easily converge to the trivial classifiers

For the MLP model, we have tried different choices of its hyper-parameters, including unit numbers, layers, various drop-out choices, and dual-channel architecture with each channel process query and prompt embeddings individually. However, we find all of those choices tend to converge to the trivial solution that predicts either all 0 or all 1 for the binary classification task. Such a reward model does not actually have the prediction ability in inference time. In practice, with such a reward mode, we can do nothing better than select the best-performing prompt on the training dataset. Therefore, the performance of BoTr Eqn.(1) represents the best-achievable performance of using a dummy MLP reward model.


2. XGBoost is robust on hyper-parameter for reward modeling

For the xgboost method, we set a universal hyper-parameter for all tasks and LLMs. When deploying Prompt-OIRL in practice, a case-by-case engineering on the hyper-parameters on reward modeling should be able to further boost the performance of the algorithm, yet this is out of our research scope. In our paper, we experiment with a single universal hyper-parameter setting for all LLMs and tasks, demonstrating the robustness of the proposed method.


3. Instance-wise prediction of outcome is better than pair-wise prediction (preference-based learning)

Another alternative for reward modeling is based on preference-based learning, which is effective in some conventional inverse RL tasks [1. T-REX]. To be specific, for every training query $x^{(i)}$, there may exist multiple prompts (denoted as $p_+^{(i)}$) that lead to a correct answer, and some other prompts ($p_-^{(i)}$) that lead to a wrong answer. 


We can also organize the offline prompt demonstration in the form of preference-based dataset $(x^{(i)}, p_-^{(i)}, p_+^{(i)} )$, and learn from those pair-wise preference data. In this pair-wise preference approach, the learned reward model takes both prompts and the query as input, and outputs the preferred prompt. Given a new query in the inference time, such a reward model can be applied to all K candidate prompts with K-1 comparisons to find the best prompt. (so it is more computationally expensive than the direct reward modeling method used in our work). We empirically studied whether such an approach can lead to better performance on the MAWPS dataset with the GPT-3.5-turbo model. The results are shown in the table below:


| # Training Prompts | 1          | 2          | 3          | 4          | 5          | 6          |
|--------------------|------------|------------|------------|------------|------------|------------|
| BoTr Eqn(1)        | 0.6598     | 0.6988     | 0.7118     | 0.7155     | 0.7173     | 0.7175     |
| BoTr Eqn(2)        | 0.6598     | 0.7742     | 0.8150     | 0.8350     | 0.8473     | 0.8546     |
| BoTr Eqn(2) Paired | 0.6598     | 0.7195     | 0.7203     | 0.7194     | 0.7236     | 0.7247     |
| Prompt-OIRL        | 0.7637     | 0.8032     | 0.8379     | 0.8750     | 0.8916     | 0.8944     |


It can be concluded that the pair-wise reward model can not achieve better performance than the direct reward modeling used in Prompt-OIRL. 


All our offline datasets and code for processing those datasets, as long as the MLP implementation and pair-wise reward modeling implementation will be made publicly available and contributed as an asset for future research.




**Actions taken** We have updated our manuscript accordingly to include extended details for the reward model training. Revisions have been highlighted in red in our updated manuscript.



---

**_Reference_**

[1. T-REX] Brown, Daniel, et al. "Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations." International conference on machine learning. PMLR, 2019.

Reviewer comment (Experiments):
Can you report the performance of the (on average) best (on the test set) prompt and what this prompt is for the different models and datasets? Can you similarly report the performance of always choosing the best prompt for every query (corresponding to the setting where the proxy reward models the success flawlessly)?

Author response:
### Q2. Can you report the performance of the (on average) best (on the test set) prompt and what this prompt is for the different models and datasets? Can you similarly report the performance of always choosing the best prompt for every query (corresponding to the setting where the proxy reward models the success flawlessly)?

### A2.
We thank the reviewer for pointing out those valuable performance references: those results can have reference value in understanding the improvement and future potential of Prompt-OIRL. We would also note that in practice, we will not be able to identify those best prompts according to their performance as the test label is not accessible.

1. Best test-set prompting strategy on different datasets:
We use Best of Train to denote the best performance of training prompts (i.e., out of the 6 training prompts listed in Table 2 of our paper) on the test queries, and use Best of Test to denote the best performance of the first 10 held-out test prompts. Details of those test prompts are provided in Appendix C.3.  

| Dataset | LLM        | Best of Train | Train Prompt Index | Best of Test | Test Prompt Index |
|---------|------------|---------------|--------------------|--------------|-------------------|
| MAWPS   | GPT3.5-turbo | 0.855       | 0                  | 0.841       | 5                 |
| SVAMP   | GPT3.5-turbo | 0.718       | 2                  | 0.736       | 2                 |
| GSM8K   | GPT3.5-turbo | 0.672       | 1                  | 0.670       | 2                 |
| MAWPS   | LLaMA2-7B   | 0.646        | 0                  | 0.587       | 8                 |
| SVAMP   | LLaMA2-7B   | 0.637        | 0                  | 0.588       | 8                 |
| GSM8K   | LLaMA2-7B   | 0.248        | 0                  | 0.241       | 5                 |
| MAWPS   | TigerBot-13B | 0.536       | 0                  | 0.669       | 0                 |
| SVAMP   | TigerBot-13B | 0.645       | 1                  | 0.673       | 8                 |
| GSM8K   | TigerBot-13B | 0.3940       | 3                  | 0.393       | 7                 |

2. Assuming we have an oracle reward model (i.e., on test time, we assume we can always pick the prompt that achieves a correct answer), this should correspond to an empirical performance upper bound of Prompt-OIRL

| Dataset | LLM        | Train Oracle | Test Oracle | Train + Test Oracle |
|---------|------------|--------------------------|-------------------------|--------------------------|
| MAWPS   | GPT3.5-turbo | 0.942                | 0.941                  | 0.957                   |
| SVAMP   | GPT3.5-turbo | 0.882                | 0.869                  | 0.903                   |
| GSM8K   | GPT3.5-turbo | 0.872                 | 0.873                   | 0.914                    |
| MAWPS   | LLaMA2-7B   | 0.891                 | 0.855                  | 0.926                   |
| SVAMP   | LLaMA2-7B   | 0.876                 | 0.837                  | 0.907                   |
| GSM8K   | LLaMA2-7B   | 0.582                  | 0.501                   | 0.648                    |
| MAWPS   | TigerBot-13B | 0.929                | 0.642                  | 0.935                   |
| SVAMP   | TigerBot-13B | 0.928                 | 0.913                 | 0.954                   |
| GSM8K   | TigerBot-13B | 0.754                 | 0.7                     | 0.822                    |


Take-Away Messages:

1. The optimal prompt is highly relevant to the task and LLM used. In 3 out of the 9 settings, the optimal test-time prompting strategy exists in the test prompts, rather than the training prompts generated by previous research.
2. The oracle performance can be limited by the ability of LLMs.
3. Increasing the diversity of prompts improves the oracle performance. 

**Actions taken** We have updated our manuscript and added the results with the oracle reward model in Table 6 and Table 7.

Reviewer comment (Evaluation):
How does Prompt-OIRL differ from BoTr Eqn.2 when trained on all considered prompts? Its description suggests that also selects the best of n prompts but it achieves worse performance in Figure 6.

Author response:
### Q3. How does Prompt-OIRL differ from BoTr Eqn.2 when trained on all considered prompts? Its description suggests that also selects the best of n prompts but it achieves worse performance in Figure 6.

### A3.
As the reward model of Prompt-OIRL can evaluate any given prompt, in test time, it can choose a prompt not only from the training set (i.e., out of the 6 training prompts), but also from the test set (in our experiments, we use 110 held-out test prompts, as detailed in Appendix C.3). The performance gain of Prompt-OIRL over BoTr Eqn.2 is based on the generalization ability of Prompt-OIRL on test prompts.

The table below contextualizes the performance improvement averaged over different LLMs and tasks.  We use “OIRL/Eqn(2)” to highlight the improvement achieved using additional test prompts as compared to only selecting from the training prompts (i.e., the improvement of Prompt-OIRL over BoTr Eqn(2)). 


| Method      | K = 1    | K = 2    | K = 3    | K = 4    | K = 5    | K = 6    | 
|-------------|----------|----------|----------|----------|----------|----------|
| LLM-Conf.   | 0.4751   | 0.5184   | 0.5427   | 0.5543   | 0.5594   | 0.5601   |
| BoTr Eqn(1) | 0.4751   | 0.5379   | 0.5654   | 0.5800   | 0.5893   | 0.5944   |  
| BoTr Eqn(2) | 0.4751   | 0.5553   | 0.5910   | 0.6130   | 0.6295   | 0.6393   | 
| Prompt-OIRL | 0.5904   | 0.6095   | 0.6225   | 0.6331   | 0.6410   | 0.6459   |  
| OIRL/Eqn(2) | +24.27% | +9.76% | +5.33% | +3.23% | +1.83% | +1.03% | 

The monotonic improvement demonstrate the benefit of integrating additional prompts in the test set.

**Actions taken** We have updated our presentation to further clarify the usage of the test prompts, and further added additional discussion and Table 8 in our appendix to highlight this point.