Paper: BatchPrompt: Accomplish more with less (ICLR-2024)

Reviewer comment (Experiments):
How to determine good hyperparameters like batch size and voting rounds for BatchPrompt? More analysis could be provided on computational efficiency tradeoffs.

Author response:
Thank you. First, we set the batch size 32 and voting rounds 5 as default. We provide the detailed token usage against voting rounds/batch size in Fig. 4, as well as in our “general response to reviewer PJJX, pc53, Nbfh”, with detailed numbers. Detailed explanation for choosing batch size and voting round in our response to R3-Q2. 

In general, hyperparameter choices can be important. The ablation on batch size and voting rounds are intended to illustrate this. While there is no absolute strategy guaranteed to work on all tasks or in all domains, “batch size 32 and voting rounds 5” is a good default setting, and a short calibration is possible for any task. This process is similar to fine-tuning the parameter of neural networks. We cannot assume the best batch size/epoches when training a neural network at the very begining, but need a validation process to achieve this. To find the best parameters for a specific task, we can use a small validation dataset with ~100 randomly chosen data to validate. For example, a user can start from batch size 16 and 1 voting rounds, and iteratively increase batch size or voting rounds or both, until performance stabilizes, and according to priority of cost-savings and accuracy. The optimal parameters still depend on users’ needs and priorities.

Reviewer comment (Experiments):
It seems that gpt-3.5-turbo suffers from performance degradation when using BatchPrompt, while gpt-4 does not. Is this caused by the model scale? I believe it is helpful to add an analysis of BatchPrompt on the LLaMA series with different model sizes.

Author response:
Thanks to the reviewer for the kind suggestion. GPT-3.5-turbo indeed suffers more from performance degradation compared with gpt-4, and it is caused by the model scale. BatchPrompt performance improves as LLM performance improves.. Also, it has been the trend that more advanced LLMs allow more tokens (32k for Gpt-4, 8k for GPT-3.5-Turbo), which also makes larger batches possible. In this way, every update that increases the available context window of LLMs adds to the leverage now possible with BatchPrompt. 

Benchmarking with Llama is a good idea, and will make a good addition to our revision. We agree with your hunch, that a larger more generally-performant model will improve BatchPrompt performance.

Reviewer comment (Experiments):
It would be helpful evaluate the method on more diversed tasks, e.g. reasoning, knowledge-intensive, creative writing tasks. Does the type or difficulty of the instruction affect the performance of BatchPrompt?

Author response:
We also evaluate the method in GSM8K, which is an arithmetic reasoning task. Also, the task presented in  our “general response to reviewer PJJX, pc53, Nbfh”. 
BatchPrompt should generally be applicable to all close-ended tasks, e.g., reasoning, translation, summarization, etc., while might be difficult to be applied to tasks with open-ended solutions/answers, e.g., text generation, creative writing, etc.
On difficulty of instruction, we agree that a difficult-to-answer, or ambiguously-worded instruction could hamper the performance. Finding the best instruction should be important, while might not be the major task for BatchPrompt. We will leave this to future work.

Reviewer comment (Evaluation):
It seems that each item in the new batch (with only one prompt) could not be computed parallelly as original. Whether it will increase the time cost? It might be better to add time and flops metrics in the experiments.

Author response:
Thank you for the question. As our motivation is frugality for LLMs, we initially did notconsider the time cost. We use OpenAI API, and inference time for GPT can depend on number of concurrent users, which is not stable. A statistic can be found below: https://gptforwork.com/tools/openai-api-and-other-llm-apis-response-time-tracker . A detailed comparison of response time against tokens:

https://community.openai.com/t/gpt-3-5-and-gpt-4-api-response-time-measurements-fyi/237394 . Also, as we are directly using pre-trained LLMs (gpt-3.5-turbo, gpt-4), we might not be able to calculate an accurate number of flops. 

In the link above, we see that response time is in direct proportion to number of completion tokens. Since we use many fewer tokens compared with SinglePrompt (see token comparison in Table 2-4, Fig. 4), we will use proportionally less time.

Reviewer comment (Reproducibility):
I think the “batchprompt” could be used in both training and test phases, right?

Author response:
Very interesting idea. We did not consider BatchPrompt for training efficiency, only for robust inference. This could be a nice area of future research.

Reviewer comment (Experiments):
In BPE, the weight for confidence is directly 1. What about to generate the weights scores directly by the LLM without whether confident?

Author response:
Right. Weights can be generated by the LLM. We simplify to 1 and \alpha, for sake of exposition. We can also generate a score directly. Please refer to Appendix D, where we have a description:

```
[Conf-Description]: ’You not only need to generate The label/answer, but also your confidence. If you are confident in your output class, append a ”(confident)” at the end of the label; else, append a ”(not confident)”.’ 
```

Here, we could  change this description to:

```
[Conf-Description]: ’You not only need to generate The label/answer, but also your confidence. Please append a confidence score between 0 and 1 at the end of the label to represent your confidence in the result you generate.
```

Then we can multiply the score with the corresponding result, and select the result with maximum score as final output in the majority voting phase. 

We experimented with allowing a range of confidence values, but the results did notchange. Therefore we chose the binary representation. We will explain this in the next paper version.

Reviewer comment (Experiments):
All tasks are very short answer type tasks, using tasks that generate longer answers might be very hard to experiment using the batchprompt approach. Couldn't see a discussion on this topic in the paper. Thoughts?

Author response:
Sorry for the lack of clarity. BatchPrompt can be used for all close-ended tasks. Task answers need not be short, but must have the same format in each voting round, in order to determine agreement. This is required for our ensemble (majority voting) step. We achieve this goal through task specification. 

Please note the prompt in Appendix D. For all three tasks in the paper, we write the following description at the end our task specification:

```
Below are the outputs you need to generate. ”X” can be ’0’ or ’1’. 
Label for Input 0: [class X] [Place-Holder-Conf] 
Label for Input 1: [class X] [Place-Holder-Conf] ......

Please make sure each generated label is in format of [class X].
Please make sure to generate [BATCH-SIZE] labels. You may include other additional sections here.
```

The motivation for writing this is to encourage the model to map answers of varied formats  to a binary label, and to unify the format of the answer to make majority voting possible.

However, what if the answer cannot be binary label?
 
Please note the work in Appendix C. We provide a general experimental result on GSM8K, the arithmetic reasoning task, whose answers cannot be a binary label. The answer to the arithmetic reasoning task can be long chain-of-thought answers. That said, we can still use the task specification to map the answer to a specific number/format, enabling a majority vote. In Appendix D, we provide prompts for GSM8K: 

```
Please make sure to write your a series of intermediate reasoning steps. Please make sure the final sentence is ”The answer is xxx.”, and the answer should be a number. Please make sure to generate [BATCH-SIZE] labels each time. 
```

We add this at the end of each task specification to make answer formats consistent in each voting round, and make ensembling possible.

We recognize that for open-ended tasks like text generation, it is not obvious how BatchPrompt could be applied. BatchPrompt requires an “equality” function to enable majority voting. Suppose two open-ended responses are considered to “say the same thing”. By providing this kind of semantic equality function, BatchPrompt can be extended to this task. We leave this as future work.

Reviewer comment (Experiments):
It is not clear how this system would be used with all its advantages in a deployment scenario - batching real world prompts with very different instructions might have unpredictable behavior, any thoughts?

Author response:
If we understand correctly, the question is about batching different tasks into each model call, e.g. 

```
“Instruction #1, Question #1, Instruction #2, Question #2, …”
```

versus

```
“Instruction #1, Question A, Question B, Question C, …”
```

In the former case, we lose the token-saving benefit from sharing instructions, but still get other robustness and cost-saving benefits from permutation, ensembling, and early stopping. BatchPrompt is designed for processing large volumes of data where the LLM performs a specific function, e.g. “label sentiment of all emails in an organization”, or “map each request to one of k functions/tools/teams”. We expect LLMs will take many such roles in software systems.