Paper: InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The paper could benefit from a broader evaluation scope. Considering additional tasks such as reasoning QA GSM8K, machine translation, or summarization could provide a more comprehensive view of the method's effectiveness.

Author response:
**Q1:** The paper could benefit from a broader evaluation scope, e.g., GSM8k, machine translation

**A1:** Thanks for your suggestion! We add new experiments of GSM8K, AQUA, and SVAMP by evaluating the zero-shot performance. Following [1], the reasoning template is designed as `I have some instruction examples for solving school math problems. Instruction: Let’s figure it out! Instruction: Let’s solve the problem. Instruction: Let’s think step by step. Write your new instruction that is different from the examples to solve the school math problems. Instruction:` The results are reported in the table below:
|Dataset| Method | Instruction | Results |
|---| ---    | ---        | ---|
|GSM8k| CoT    | Let's think step by step | 0.718 |
|   | Ours   | Let's use the instruction to solve the problem | 0.743|
|AQUA| CoT    | Let's think step by step | 0.511 |
|   | Ours   | Let's break down the problem | 0.543|
|SVAMP| CoT    | Let's think step by step | 0.763 |
|   | Ours   | Let's use the brain | 0.795|

[1] USE YOUR INSTINCT: INSTRUCTION OPTIMIZATION USING NEURAL BANDITS COUPLED WITH TRANSFORMERS. https://arxiv.org/abs/2310.02905

Reviewer comment (Experiments):
The inclusion of only two comparative baselines, APE and Uniform, limits the robustness of the evaluation. Expanding the set of comparative baselines could provide a more holistic understanding of the method's performance relative to existing work. For example, RLprompt and Autoprompt are also two good prompt search methods.

Author response:
**Q2:** Comparison with other prompt search methods, e.g., RLprompt.

**A2:** We discussed the key difference with the RLprompt in our related work section: [2] (RLprompt) relies on RL to optimize hard prompts while InstructZero optimizes an instruction in the output space of an open-source model $g(\cdot)$ **without RL** by BO of a soft prompt to g(·). Moreover, the instructions found by our method are all fluent while the instructions found by RLprompt are not, e.g., in Yelp Review (a text classification task), RLprompt found `StaffAreaFocusHardware Advisory` as their best prompt. Hence, InstructZero is a better pipeline to optimize human-readable instructions, which is more challenging than the task of RLprompt.

[2] RLprompt: Optimizing Discrete Text Prompts with Reinforcement Learning. https://arxiv.org/pdf/2205.12548.pdf

Reviewer comment (Evaluation):
There is ambiguity in the claim about the method being applicable for zero-shot evaluation. While it's true that the proposed method employs a black-box LLM API for zero-shot generation, the Bayesian Optimization (BO) process requires labeled data. This seems to contradict the zero-shot claim and may constitute an overstatement.

Author response:
**Q3:** There is ambiguity in the claim about the method being applicable for zero-shot evaluation. 

**A3:** Thank you for your feedback! We did include a few labeled data as input demos for the open-source models during instruction optimization. However, the inference and evaluation of the optimized instructions on the black-box LLMs are zero-shot. Hence, the application of the optimized instructions is indeed zero-shot and it does not overclaim. We will make this clear in the polished manuscript.

Reviewer comment (Evaluation):
I wonder why the results of APE are so weak in Figure 1.

Author response:
Thank you for letting us know that most of your concerns have been addressed! We appreciate you for raising your score! Your constructive input remains invaluable to us, and we appreciate your dedication to enhancing the quality of our manuscript. Thank you for your time and consideration.

Every concern of you matters a lot to us and we carefully addressed every one of them. **Our response to W3 regarding the weak score of APE can be found at the very beginning of our general response (at the very top of this page)**. This is due to the number of demos (i.e., 5 demos, compared to 100 in the original APE paper) used in both APE and our method (for a fair comparison in a challenging setting). We also provided an experimental comparison between APE and our method when using more demos (100 demos for APE, 10 demos for ours). Our method still finds better instructions than APE even using only 10% of demos used by APE. 

Would you mind confirming if this addressed your concern in W3? If you have any further questions, please let us know and we are happy to discuss and elaborate on more details. Thank you for reading our responses and providing valuable feedback! If this reply has addressed your concerns in W3, we would really appreciate it if you would consider raising your score.

Reviewer comment (Theory):
I have some questions and concerns about the instruction-coupled kernel. First of all, it seems that to calculate this kernel between a pair of input soft prompts, you need to have the evaluated scores for both soft prompts (correct me if I'm wrong)? If this is the case, then when you calculate the vector $\boldsymbol{k}$ in equations 4 and 5, this instruction-coupled kernel cannot be used to calculate these kernel values and therefore these kernel values will simply use the normal squared exponential or matern kernel? In this case, I wonder how much this instruction-coupled kernel actually helps the performance of the Bayesian optimization, because the vector $\boldsymbol{k}$, which directly measures the distance between a new soft prompt and other previously evaluated soft prompts and therefore has a huge influence on the uncertainty measure, cannot make use of it.

Author response:
$\mathbf K$ in Eq. (4)-(5) are computed using Eq. (9). It depends on both the soft prompts' similarity $l(\cdot,\cdot)$ and their generated instructions' similarity $s(\cdot,\cdot)$. Since its computation depends on the similarity between explored soft prompts $\{p_1, ..., p_m\}$ from previous iterations and their evaluation scores $\{H(p_1), ..., H(p_m)\}$, $K$ can be predetermined before computing Eq. (4)-(5). $k(p, p_i)$ and $k(p,p)$ of Eq. (4)-(5) only measure the kernel similarity in the soft prompt space using $l(\cdot,\cdot)$ so we do not need to evaluate every $p$'s performance during optimization. 

Instruction-coupled kernel $\mathbf K$ aims to align the exploration in the soft prompt space (what the BO directly does in InstructZero) and the optimization of the textual instruction (which is the final goal). To this end, the kernel in BO is expected to reflect the similarity of the generated instructions for the target task. So we apply the instruction-coupled kernel in computing $\mathbf K$. However, to compute $\mathbf k$, we cannot afford to evaluate every soft prompt $p$'s generated instruction during the maximization of the acquisition function and its evaluation is not differentiable. So in practice we only apply the soft prompt space kernel $l(\cdot,\cdot)$ in computing $k(p, p_i)$ and $k(p,p)$ of Eq. (4)-(5).  

In all our experiments, the above approach performs promisingly and we haven't observed any degradation of using the kernel.

Reviewer comment (Experiments):
About the ablation study (Section 4.3), it looks like the scores "w/o Manual" is in general better than "Manual"? This is also puzzling because it implies that the meta-prompt used by APE may not be useful...

Author response:
We agree that the meta-prompt used by APE may not be useful. The effectiveness of the meta-prompt may vary on models or tasks. For example, an effective meta-prompt for GPT-3, as in the experiments of the APE paper, may not be the best on other models.

Reviewer comment (Experiments):
The proposed method InstructZero seems to only optimize the zero-shot performance of the instructions instead of few-shot performance. However, since you already have access to these input-output exemplars which are used as input to the open-source LLM, why don't you also use them as input to the black-box LLM to improve the performance? So this may bring into question how practical the experiments are.

Author response:
For a fair comparison, we follow the APE pipeline, which only uses these input-output examplars for instruction inductions and conducts zero-shot evaluations of the induced instructions.