Paper: IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Ideally, rather than (arbitrarily?) choosing k = 18 & k = 100, you should provide an automated approach to automatically assess the smallest amount of annotations that leads to the best possible performance; you only introduce Auto-IDEAL in "4.4" but you do not seem to claim it as a major contribution. Why?

Author response:
**1. [Re Weakness 1.1 - Part 1 & Question 3 :  Why choose 18 and 100 as the annotation budget?]**

Thanks for your suggestion! We would like to clarify that the annotation budget is not arbitrarily selected but based on the following non-trivial reasons. 
- Firstly, we strictly followed the same setting as Vote-k, which also used 18 and 100 as the annotation budget, to enable easy comparison with the most important baseline and perform deep analysis. 
- Secondly, we included 18 as the annotation budget because all annotated examples can fit the prompt of the large language models within context limits. Therefore, the prompt retrieve stage can be ignored, and the evaluation results can naturally represent the quality of the selected examples. 

We appreciate your valuable suggestion and have revised the paper to make it clearer **in section 4.2**.


**3. [Re Weakness 1.1 - Part 2: You should provide an automated approach to automatically assess the smallest amount of annotations.]**

Thank you for your valuable feedback! We argue that the concept of “the smallest amount of annotations” should not exist in selective annotation problems. Given that we do not know the distribution of the test examples, we believe that the larger the annotated examples, the better the expected performance, as it will be easier to retrieve the most relevant examples from a larger data pool. In an ideal case, all available examples should be annotated, and the algorithm could achieve the best-expected performance. However, in a more realistic setting, users have a limited budget, and the algorithm could select the best subset to annotate under the user’s budget. Regarding your suggestion of finding the smallest subset, we think a more meaningful setting would be if users have an expected performance threshold, and they expect there exists an algorithm that could help them find the smallest subset to annotate to reach the threshold

Reviewer comment (Evaluation):
In Table 2, please add an additional row that shows the best-known performance on each dataset; without it, it is impossible to quantify the practical impact of IDEAL (i.e., is it "just" improving on Vote-k, or does it come close to the best-known performance?).

Author response:
**4. [Re Weakness 2.1: Please also add an additional row to show the best known performance.]**

Thanks for your suggestions. The main purpose of Table 2 is to compare the performance of IDEAL with other methods that could select a “coreset” from the large scale unlabeled data to annotate. Regarding the “best-known” performance, we can observe from Table 2 that IDEAL achieves the best performance compared with all baselines. If we misunderstand your suggestion, please let us know. Thanks!

Reviewer comment (Experiments):
In Table 2, please also add to the Table results for k = 1,000 and even k = 10,000. How big are these improvements, if any?

Author response:
**5. [Re Weakness 2.2: Please also add the results when k=1000 even k=10000.]**

Thank you for your suggestions. We would like to clarify that the value of K = 10000 exceeds the size of our dataset. To simulate the process of obtaining unlabeled samples, we followed the same setting as Vote-k, which is described on page 4 of Vote-k. In this setting, the sample size was set to 3000. Due to limited rebuttal time and computational resources, we will try our best to conduct additional experiments when k=1000 during the rebuttal period and update the results accordingly once complete as shown in the following reply. Thank you again for your suggestion!

Reviewer comment (Evaluation):
In Table 2, please explain why we see IDEAL(100) < IDEAL(18) on RTE. Is there a general lesson worth sharing?

Author response:
**6. [Re Weakness 2.3: please explain why we see IDEAL(100) < IDEAL(18) on RTE.]**

Thank you for your suggestion. It is indeed an interesting scenario that was also observed in previous work Vote-k, where large annotation budgets do not necessarily lead to better performance (Vote-K (100) < Vote-K (18) in MWoZ, GeoQ, and DBpedia). However, in most cases (8 out of 9), IDEAL(100) performs better. We believe this particular scenario exists because a large annotated data pool may include some “bad” examples that have a similar embedding with the users’ query but are not relevant actually. Overall, a large annotation budget typically leads to better performance from a statistical perspective.

Reviewer comment (Experiments):
In Table 2, explain why Vote-k outperforms IDEAL(100) on GeoQ (Tables 2 & 8). In particular, given that Table 8 suggests there is a fair amount of variability among the various trials, ideally you should increase the number of trials to 10 or even larger. In the current setting, according to Table 8, for "100" Vote-k and IDEAL obtain the same best result (60.5) on GeoQ; similarly, for "18", there is a similar "tie" on HellaSwag, and Vote-k gets a better max on GeoQ.

Author response:
**7. [Re Weakness 2.4: Why Vote-k outperform IDEAL(100) on GeoQ?]**

Thank you for your constructive suggestion. Vote-k is insightful and needs a careful balance between diversity and representativeness. In some cases, e.g., on GeoQ (m=100), a good balance may be achieved. However, since the balance is tricky and complex, in most cases, Vote-k cannot work well, which motivates us to develop more advanced algorithms. 

We also kindly argue that the comparison of the two methods needs to be considered at the statistical level. IDEAL outperforms Vote-K in 17 out of 18 cases, which could indeed demonstrate the superiority of IDEAL over Vote-K. We totally agree that in an ideal case, as researchers, we should develop methods that are better than baselines in all cases. This is what we aim to achieve in our future research career and the follow-up work of our method. We appreciate your suggestions!


**8. [Re Weakness 2.5 & Question 4: Why not include more trials in the experiments?]**

Thank you for your suggestion. We included three different runs for each dataset for two reasons:

- To ensure a fair comparison and ease of analysis, we used exactly the same setting as the most important baseline Vote-k, which also used three different runs.

- Some experiments are especially costly. For example, each run of Mowz and GeoQ costs over 130+ USD dollars using GPT-3.5 Turbo. Considering our limited budget, we cannot afford large-scale experiments in some specific datasets.
Due to limited rebuttal time and computational resources, we will try our best to conduct additional experiments to run more trials during the rebuttal period and update the results accordingly once complete, as shown in the following reply. Thank you again for your suggestion!

Reviewer comment (Experiments):
Section 4.3.1 should cover all datasets, not only two; worst case scenario, if space is an issue, you can summarize the results on all datasets here, and refer the reader to the appropriate appendix.

Author response:
**9. [Re Question 2 & Weakness  4, 5 and Weakness 3 - part 1: Why don't you use all datasets in all experiments?]**

Thank you for your insightful comments! We have the following three reasons for not performing experiments on all datasets for all sections in our paper, including ablations.

(1) We intend to keep the same setting as previous work Vote-k for convenient comparisons and analysis, which also does not perform experiments on all datasets in the ablation sections. Moreover, to demonstrate the robustness of our method in different scenarios, we cover diverse datasets in different sections.

(2) Performing selective annotation experiments is extremely costly in terms of both time and API fees. For example, it costs over 120 USD for a single run of GPT-3.5-turbo on Mowz and GeoQ datasets, which is unaffordable considering our limited budget.

(3) We believe that using partial datasets in the main experiments to perform ablation is acceptable and does not result in a loss of generality. There are a large number of research works in both selective annotation [1] and more general In-Context learning examples selection [2][3][4] that only uses a partial dataset to perform the ablation experiments. 

We sincerely appreciate your suggestion, and we will try our best to perform experiments on all datasets in the ablation study during the rebuttal period, as shown in the following reply. Thank you again for your suggestion!

[1] Su, H., Kasai, J., Wu, C. H., Shi, W., Wang, T., Xin, J., ... & Yu, T. (2022). Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975.

[2] Chang, T. Y., & Jia, R. (2023, July). Data curation alone can stabilize in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8123-8144).

[3] Nguyen, T., & Wong, E. (2023). In-context Example Selection with Influences. arXiv preprint arXiv:2302.11042.

[4]Li, X., & Qiu, X. (2023). Finding supporting examples for in-context learning. arXiv preprint arXiv:2302.13539.

Reviewer comment (Experiments):
Section 4.3.1 should cover all datasets, not only three of them. Also: (i) why have Table 2 for "18" rather than "100," when IDEAL(100) outperforms IDEAL(10) on all datasets, and (ii) similarly to the above comment on stability, you should use at least 10 randomized trials.

Author response:
**9. [Re Question 2 & Weakness  4, 5 and Weakness 3 - part 1: Why don't you use all datasets in all experiments?]**

Thank you for your insightful comments! We have the following three reasons for not performing experiments on all datasets for all sections in our paper, including ablations.

(1) We intend to keep the same setting as previous work Vote-k for convenient comparisons and analysis, which also does not perform experiments on all datasets in the ablation sections. Moreover, to demonstrate the robustness of our method in different scenarios, we cover diverse datasets in different sections.

(2) Performing selective annotation experiments is extremely costly in terms of both time and API fees. For example, it costs over 120 USD for a single run of GPT-3.5-turbo on Mowz and GeoQ datasets, which is unaffordable considering our limited budget.

(3) We believe that using partial datasets in the main experiments to perform ablation is acceptable and does not result in a loss of generality. There are a large number of research works in both selective annotation [1] and more general In-Context learning examples selection [2][3][4] that only uses a partial dataset to perform the ablation experiments. 

We sincerely appreciate your suggestion, and we will try our best to perform experiments on all datasets in the ablation study during the rebuttal period, as shown in the following reply. Thank you again for your suggestion!

[1] Su, H., Kasai, J., Wu, C. H., Shi, W., Wang, T., Xin, J., ... & Yu, T. (2022). Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975.

[2] Chang, T. Y., & Jia, R. (2023, July). Data curation alone can stabilize in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8123-8144).

[3] Nguyen, T., & Wong, E. (2023). In-context Example Selection with Influences. arXiv preprint arXiv:2302.11042.

[4]Li, X., & Qiu, X. (2023). Finding supporting examples for in-context learning. arXiv preprint arXiv:2302.13539.


**10. [Re Weakness 3 - part 2: Why have Table 2 for "18" rather than "100”?]**

Thanks for your suggestions. In most cases, we choose to use 18 as the annotation budget. This is because all annotated examples can fit the prompt of the large language models within context limits. Therefore, the prompt retrieval stage is ignored and the evaluation results can naturally represent the quality of the selected examples. In section 4.3.2, the main purpose of the experiment is to demonstrate that the selected annotated examples could work well in multiple LLMs. To exclude the influence of other factors, we chose 18 as the annotation budget.

Reviewer comment (Experiments):
Section 4.4 should cover all datasets, not only five of them.

Author response:
**9. [Re Question 2 & Weakness  4, 5 and Weakness 3 - part 1: Why don't you use all datasets in all experiments?]**

Thank you for your insightful comments! We have the following three reasons for not performing experiments on all datasets for all sections in our paper, including ablations.

(1) We intend to keep the same setting as previous work Vote-k for convenient comparisons and analysis, which also does not perform experiments on all datasets in the ablation sections. Moreover, to demonstrate the robustness of our method in different scenarios, we cover diverse datasets in different sections.

(2) Performing selective annotation experiments is extremely costly in terms of both time and API fees. For example, it costs over 120 USD for a single run of GPT-3.5-turbo on Mowz and GeoQ datasets, which is unaffordable considering our limited budget.

(3) We believe that using partial datasets in the main experiments to perform ablation is acceptable and does not result in a loss of generality. There are a large number of research works in both selective annotation [1] and more general In-Context learning examples selection [2][3][4] that only uses a partial dataset to perform the ablation experiments. 

We sincerely appreciate your suggestion, and we will try our best to perform experiments on all datasets in the ablation study during the rebuttal period, as shown in the following reply. Thank you again for your suggestion!

[1] Su, H., Kasai, J., Wu, C. H., Shi, W., Wang, T., Xin, J., ... & Yu, T. (2022). Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975.

[2] Chang, T. Y., & Jia, R. (2023, July). Data curation alone can stabilize in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8123-8144).

[3] Nguyen, T., & Wong, E. (2023). In-context Example Selection with Influences. arXiv preprint arXiv:2302.11042.

[4]Li, X., & Qiu, X. (2023). Finding supporting examples for in-context learning. arXiv preprint arXiv:2302.13539.

Reviewer comment (Experiments):
Please add to the paper a paragraph or two about the actual/expected cost of annotation for the various domains in the empirical evaluation.

Author response:
**12. [Re Comments 2: please add to the paper a paragraph or two about the actual/expected cost of annotation for the various domains in the empirical evaluation?]**

In our revised version, we have added one section in the Appendix to further illustrate the cost of annotation (Appendix F) . Thanks for your insightful suggestion.

Reviewer comment (Writing):
Please intuitively explain the "end-to-end" term that you are using in the Abstract, Introduction, and later on in the Appendices, but nowhere else in the rest of the paper. If it is an important concept, it should also appear in the main body of the paper; if it is not, you should remove all references to it as it is distracting to see it mentioned early on, never clarified, and then simply ignored.

Author response:
**13. [Re Comments 3: Please intuitively explain the "end-to-end" term that you are using in the Abstract, Introduction, and later on in the Appendices.]**

Thank you for your feedback. We refer to IDEAL as an end-to-end algorithm because it directly selects a subset of unlabeled data for annotation from a large-scale pool in one step, without any intermediate procedure. In contrast, Vote-k selects a subset of data in two stages. First, it selects a small portion of data for diversity and annotates them manually. Second, these annotated data act as prompts for predictions on all other unlabeled data, and the remaining ones that need to be annotated are chosen based on diverse confidence scores. This approach adds inference costs due to the predictions on unlabeled data. We mentioned the difference between IDEAL and Vote-k in our original submission.

Reviewer comment (Presentation):
Both in Fig 3's caption and in the abstract, please summarize the running time improvements of IDEAL vs Vote-k. The current "largely reduces the time cost" is too vague. Ideally, the last sentence of the abstract should end something like this: "... improving the performance by X% while reducing the running time by Y%".

Author response:
**14. [Re Comments 4:  please summarize the running time improvements of IDEAL vs  Vote-k?]**

Thank you for your feedback. In our original submission, we summarized the numerical improvement with respect to time consumption on page 7. “Specifically, under the same hardware conditions, IDEAL achieves a 7.8× lead on average over Vote-k.”. In response to your constructive suggestion, we also have added this information to the caption to further emphasize the advantage of our method in the revised version. Thank you again for your suggestions!

Reviewer comment (Writing):
p 1: "pretty performance" → "strong performance"

Author response:
**15. [Re Nitpicks: P1 - P5]**

Thanks for your suggestions. We have revised the paper according to your suggestions from P1, P2, P4, P5 which is shown in blue color. For P3, we would like to clarify that the iteration doesn’t start from (d).  
We deeply appreciate the time and effort you spent to further improve our submission. Thanks!

Reviewer comment (Writing):
p 1: Please add a reference and/or brief summary for "requires substantial manpower and financial resources"

Author response:
**15. [Re Nitpicks: P1 - P5]**

Thanks for your suggestions. We have revised the paper according to your suggestions from P1, P2, P4, P5 which is shown in blue color. For P3, we would like to clarify that the iteration doesn’t start from (d).  
We deeply appreciate the time and effort you spent to further improve our submission. Thanks!

Reviewer comment (Theory):
p 2: "Vote-k devoids theoretical guarantees" → "Vote-k lacks theoretical guarantees"

Author response:
**15. [Re Nitpicks: P1 - P5]**

Thanks for your suggestions. We have revised the paper according to your suggestions from P1, P2, P4, P5 which is shown in blue color. For P3, we would like to clarify that the iteration doesn’t start from (d).  
We deeply appreciate the time and effort you spent to further improve our submission. Thanks!

Reviewer comment (Presentation):
p 3: For Fig 2, please explain if/why the iterations start at (d).

Author response:
**15. [Re Nitpicks: P1 - P5]**

Thanks for your suggestions. We have revised the paper according to your suggestions from P1, P2, P4, P5 which is shown in blue color. For P3, we would like to clarify that the iteration doesn’t start from (d).  
We deeply appreciate the time and effort you spent to further improve our submission. Thanks!

Reviewer comment (Novelty):
Why don't you claim Auto-IDEAL as a major contribution?

Author response:
**2. [Re Weaknesses 1.2 & Question 1: Why don’t include Auto-IDEAL as a major contribution.]**

Although Auto-IDEAL has demonstrated better performance, it has two main drawbacks that may hinder its usability in practice. We list the reasons below.

- Auto-IDEAL suffers from a model inference cost, especially in the era of large language models as a service. Specifically, when performing automatic annotation, Auto-IDEAL has to make predictions for all unlabeled examples to achieve automatic annotation. In our experiments on GeoQuery datasets, it costs more than 120 USD for a single run. Considering this, we think it is not as practical as the native IDEAL.

- Auto-IDEAL may fail when the initial examples to be labeled are not very similar to the initial examples labeled, even though they have a similar embedding. Specifically, Auto-IDEAL performs automatic annotation by following the information diffusion process from the initial annotated subset to the examples to be labeled with similar embedding. When the examples to be labeled are not relevant to the initial examples labeled, it may lead to incorrect automatic annotation and poor performance.

Considering these drawbacks, we decided to use native IDEAL as our core contribution but regarded Auto-IDEAL as a potential extension to IDEAL in a case study in our paper. We also illustrate it more clearly **in Appendix G** in the revised version.

We may leave the potential improvement of Auto-IDEAL as our future work. Thank you for your suggestions!

Reviewer comment (Experiments):
Why don't you use all datasets in all experiments, but rather different subsets in 4.3.1, 4.3.2, and 4.4?

Author response:
**9. [Re Question 2 & Weakness  4, 5 and Weakness 3 - part 1: Why don't you use all datasets in all experiments?]**

Thank you for your insightful comments! We have the following three reasons for not performing experiments on all datasets for all sections in our paper, including ablations.

(1) We intend to keep the same setting as previous work Vote-k for convenient comparisons and analysis, which also does not perform experiments on all datasets in the ablation sections. Moreover, to demonstrate the robustness of our method in different scenarios, we cover diverse datasets in different sections.

(2) Performing selective annotation experiments is extremely costly in terms of both time and API fees. For example, it costs over 120 USD for a single run of GPT-3.5-turbo on Mowz and GeoQ datasets, which is unaffordable considering our limited budget.

(3) We believe that using partial datasets in the main experiments to perform ablation is acceptable and does not result in a loss of generality. There are a large number of research works in both selective annotation [1] and more general In-Context learning examples selection [2][3][4] that only uses a partial dataset to perform the ablation experiments. 

We sincerely appreciate your suggestion, and we will try our best to perform experiments on all datasets in the ablation study during the rebuttal period, as shown in the following reply. Thank you again for your suggestion!

[1] Su, H., Kasai, J., Wu, C. H., Shi, W., Wang, T., Xin, J., ... & Yu, T. (2022). Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975.

[2] Chang, T. Y., & Jia, R. (2023, July). Data curation alone can stabilize in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8123-8144).

[3] Nguyen, T., & Wong, E. (2023). In-context Example Selection with Influences. arXiv preprint arXiv:2302.11042.

[4]Li, X., & Qiu, X. (2023). Finding supporting examples for in-context learning. arXiv preprint arXiv:2302.13539.

Reviewer comment (Experiments):
How did you end up with the two values of 18 and 10 (especially the "18" one)?

Author response:
**1. [Re Weakness 1.1 - Part 1 & Question 3 :  Why choose 18 and 100 as the annotation budget?]**

Thanks for your suggestion! We would like to clarify that the annotation budget is not arbitrarily selected but based on the following non-trivial reasons. 
- Firstly, we strictly followed the same setting as Vote-k, which also used 18 and 100 as the annotation budget, to enable easy comparison with the most important baseline and perform deep analysis. 
- Secondly, we included 18 as the annotation budget because all annotated examples can fit the prompt of the large language models within context limits. Therefore, the prompt retrieve stage can be ignored, and the evaluation results can naturally represent the quality of the selected examples. 

We appreciate your valuable suggestion and have revised the paper to make it clearer **in section 4.2**.

Reviewer comment (Experiments):
Why do we see such large variability in performance among the three different runs? Why didn't you try more runs, so that you can get a better understanding of the scale of the variability issue?

Author response:
**8. [Re Weakness 2.5 & Question 4: Why not include more trials in the experiments?]**

Thank you for your suggestion. We included three different runs for each dataset for two reasons:

- To ensure a fair comparison and ease of analysis, we used exactly the same setting as the most important baseline Vote-k, which also used three different runs.

- Some experiments are especially costly. For example, each run of Mowz and GeoQ costs over 130+ USD dollars using GPT-3.5 Turbo. Considering our limited budget, we cannot afford large-scale experiments in some specific datasets.
Due to limited rebuttal time and computational resources, we will try our best to conduct additional experiments to run more trials during the rebuttal period and update the results accordingly once complete, as shown in the following reply. Thank you again for your suggestion!

Reviewer comment (Novelty):
This paper is related to "Li et al., Finding Support Examples for In-Context Learning", which also adopts the idea of coreset selection in in-context example selection, and "Diao et al., Active prompting with chain-of-thought for large language models". It is suggested to include discussion with these papers in the related work.

Author response:
We appreciate the suggestions. We have revised the paper according to the suggestion and include the discussion of these two papers **in the related work section in the revised version**.  Thanks for your valuable feedback!

Reviewer comment (Theory):
The theoretical analysis is about the influence lower bound. How the influence score relates to better diversity and representativeness needs further illustration, since it is the major consideration of the proposed method.

Author response:
Thank you for your feedback! We would like to clarify that our method is not intended to select examples to achieve better diversity and representativeness. Instead, we aim to investigate a new paradigm to quantify potential selective annotation subsets through an information diffusion process that avoids the intractable explicit balance between data diversity and representativeness, as mentioned in our paper’s abstract. Furthermore, we provide a visualization for the selected examples in Figure 6 in our paper that compares Vote-$k$ and IDEAL. Vote-$k$ takes diversity into consideration, which may lead to the inclusion of outliers.

Reviewer comment (Experiments):
A minor suggestion is that the experiment datasets can also include some novel LLM benchmarks that are probably not trained on the experiment LLMs.

Author response:
Thank you for your constructive suggestions! We agree that benchmark leakage is a significant research problem that has garnered a lot of attention in the research community [1]. However, the research problem in our paper is selective annotation in In-Context learning, which does not involve the processing of training data selection. Due to the limited rebuttal time, we will take a deep dive into the benchmark leakage problem in In-Context Learning in our future research and investigate its impact on selective annotation problems. We have also added an additional paragraph in **Appendix G** to discuss the benchmark leakage problems. Thank you for your suggestions!

[1] Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., ... & Han, J. (2023). Don't Make Your LLM an Evaluation Benchmark Cheater. arXiv preprint arXiv:2311.01964.

Reviewer comment (Experiments):
This paper uses small LMs less than 7B, and ChatGPT. Will the method still be effective on medium-sized LMs such as LLaMA2-chat 7B, 13B? And do you use any instruction for testing on ChatGPT?

Author response:
Thank you for your valuable suggestions. We conducted experiments and evaluated the results of our method and baselines on LLaMA2-7b-chat and LLAma2-13b-chat using the same setting as Section 4.3.4. Our observations indicate that IDEAL consistently outperforms the baselines. This demonstrates that our method is a general approach that works well in medium-sized LLMs.

**---------------------------- LLaMA2-7b-chat ----------------------------**


|        | MRPC | MNLI | RTE  |
|--------|------|------|------|
| Random | 64.6 | 37.9 | 56.3 |
| Vote-K | 66.0 | 39.1 | 58.6 |
| IDEAL  | **67.2** | **39.4** | **60.5** |

**---------------------------- LLaMA2-13b-chat ----------------------------**

|        | MRPC | MNLI | RTE  |
|--------|------|------|------|
| Random | 65.8 | 38.8 | 58.5 |
| Vote-K | 66.5 | 40.1 | 59.2 |
| IDEAL  | **66.8** | **40.9** | **60.7** |

Thank you for your feedback. To ensure a fair and easy comparison, we used the same instructions and prompts as the other LLMs used in our experiments. This approach allows us to more strictly show the differences between multiple LLMs and verify the generalization ability of our method on different LLMs. We have elaborated on this procedure in more detail in our revised version (**Section 4.3.4**).

Reviewer comment (Theory):
There is no time complexity in the proposed method, which is very crucial if it needs to run for every inference.

Author response:
**1. [Re  weakness 1 & question 1: Whether it is required to run the influence function for n times to retrieve n shots?]**

Thank you for your constructive suggestion. To retrieve n shots, the influence function indeed needs to be run n times. We also included a time-complexity analysis **in the Appendix E** in the revised version. Here is a high-level overview of the time-complexity analysis for our IDEAL. 

- Information diffusion process: The time complexity of quantifying the influence of a specific subset is O(a + b), where a is the number of edges and b is the number of nodes. This is because it involves running an independent cascade diffusion process (BFS-like traversals) through the graph.

- Greedy algorithm: The greedy algorithm iteratively selects the example that provides the maximum marginal gain in influence. When the annotation budget is n, the time complexity is O(n*(a+b)). This is because, in the worst case, the algorithm needs to evaluate the influence for each of the n nodes in the network, and for each node, the algorithm needs to perform a simulation with a time complexity of O(m + n).

Regarding time consumption, we compared IDEAL with Vote-$k$ in Figure 3 in the log scale. We observed that IDEAL achieves a **7.8× lead** on average over Vote-k. This time consumption improvement benefits from the fact that IDEAL doesn’t involve any model prediction time cost, which is totally unsupervised compared to Vote-$k$. Therefore, the proposed method is more efficient.

Reviewer comment (Theory):
It is not clear how the proposed method considers the correlations among the retrieved data points.

Author response:
**2. [Re weakness 2 & question 2: How to consider the correlations among the selected examples when performing greedy search?]**

We appreciate your suggestion and address your concerns as follows. 
- Our method does not require consideration of correlations among retrieved data points. In fact, this is precisely the advantage of our method. IDEAL aims to maximize the influence of the entire selected subset, rather than considering each example separately.
- Taking the example you provided, if the influence of A+B is greater than C+B, our algorithm would choose A+B over C+B, even though C has a greater influence than A. As mentioned in section 2.2, IDEAL aims to maximize the influence of the entire selected subset of examples, which avoids an intractable explicit balance between data diversity and representativeness. 

Additionally, we also have supplemented more details in the revised paper **(section 2.2)** based on your advice. Thanks for this suggestion!

Reviewer comment (Experiments):
It is better to compare with more demonstration selection methods such as similarity-based and diversity-based methods which are widely used in practice.

Author response:
**3. [Re weakness 3 & question 4 : Compare with more demonstration selection methods in NLP fields including similarity-based or diversity-based.]**

Thanks for the suggestion. We address your concern below. 
- We kindly argue that the research problem in our paper is the selective annotation problem, which involves curating a core subset from a large-scale data pool to annotate. The “similarity-based or diversity-based” methods you mentioned are demonstrated retrieval methods that are not what we investigated in this research. Our method is agnostic to example retrieval methods and can be combined with all potential retrieval methods. 
- In particular, we also investigated different retrieval methods **in Section 4.3.3** when combined with IDEAL. We observed that similarity-based methods achieve better performance.

Reviewer comment (Theory):
For each inference of LLM, it is unclear whether it is required to run the influence function for n times to retrieve n shots, and if so, a complexity analysis of running time is very important.

Author response:
**1. [Re  weakness 1 & question 1: Whether it is required to run the influence function for n times to retrieve n shots?]**

Thank you for your constructive suggestion. To retrieve n shots, the influence function indeed needs to be run n times. We also included a time-complexity analysis **in the Appendix E** in the revised version. Here is a high-level overview of the time-complexity analysis for our IDEAL. 

- Information diffusion process: The time complexity of quantifying the influence of a specific subset is O(a + b), where a is the number of edges and b is the number of nodes. This is because it involves running an independent cascade diffusion process (BFS-like traversals) through the graph.

- Greedy algorithm: The greedy algorithm iteratively selects the example that provides the maximum marginal gain in influence. When the annotation budget is n, the time complexity is O(n*(a+b)). This is because, in the worst case, the algorithm needs to evaluate the influence for each of the n nodes in the network, and for each node, the algorithm needs to perform a simulation with a time complexity of O(m + n).

Regarding time consumption, we compared IDEAL with Vote-$k$ in Figure 3 in the log scale. We observed that IDEAL achieves a **7.8× lead** on average over Vote-k. This time consumption improvement benefits from the fact that IDEAL doesn’t involve any model prediction time cost, which is totally unsupervised compared to Vote-$k$. Therefore, the proposed method is more efficient.

Reviewer comment (Theory):
The method does not clearly address how to consider the correlations among the retrieved samples; for example, in greedy search, there may be cases where A+B is better than C+B, even though C has more influence than A.

Author response:
**2. [Re weakness 2 & question 2: How to consider the correlations among the selected examples when performing greedy search?]**

We appreciate your suggestion and address your concerns as follows. 
- Our method does not require consideration of correlations among retrieved data points. In fact, this is precisely the advantage of our method. IDEAL aims to maximize the influence of the entire selected subset, rather than considering each example separately.
- Taking the example you provided, if the influence of A+B is greater than C+B, our algorithm would choose A+B over C+B, even though C has a greater influence than A. As mentioned in section 2.2, IDEAL aims to maximize the influence of the entire selected subset of examples, which avoids an intractable explicit balance between data diversity and representativeness. 

Additionally, we also have supplemented more details in the revised paper **(section 2.2)** based on your advice. Thanks for this suggestion!

Reviewer comment (Reproducibility):
It is unclear how the proposed method leverages the features of each data sample (nodes in the graph), such as whether these features are embedded using sentence transformers or GNN-based methods, and what the cost of applying these methods is.

Author response:
**4. [Re question 3: How to get the embedding of the examples when constructing the graph? And what is the cost?]**

As we illustrated in our paper **(page 3 in our submission)**, we obtain the embedding of the examples through Sentence-BERT which follows the same setting as Vote-$k$. The cost of this process is negligible and can be ignored, as it takes less than 5 minutes for all datasets in our experiments.

Reviewer comment (Experiments):
The experiments lack comparison against many existing demonstration selection methods in the NLP field, for example, similarity-based or diversity-based methods widely employed in LLMs.

Author response:
**3. [Re weakness 3 & question 4 : Compare with more demonstration selection methods in NLP fields including similarity-based or diversity-based.]**

Thanks for the suggestion. We address your concern below. 
- We kindly argue that the research problem in our paper is the selective annotation problem, which involves curating a core subset from a large-scale data pool to annotate. The “similarity-based or diversity-based” methods you mentioned are demonstrated retrieval methods that are not what we investigated in this research. Our method is agnostic to example retrieval methods and can be combined with all potential retrieval methods. 
- In particular, we also investigated different retrieval methods **in Section 4.3.3** when combined with IDEAL. We observed that similarity-based methods achieve better performance.

Reviewer comment (Experiments):
The experiments would benefit from including results for more datasets, especially tasks beyond text classification, and more LLMs.

Author response:
**5. [Re question 5:  It would be great to see some results for more datasets (especially more tasks beyond the text classification) and more LLMs.]**

Thank you for your constructive feedback. We address your question below. 

- Following the same setting with previous selective annotation work Vote-K in the same area, we have demonstrated the results of IDEAL for datasets beyond text classification, including Multi-Choice, Dialogue, and Generation in **Section 4.2**. We have also conducted experiments for different LLMs **in section 4.3.4**. 
- Additionally, we have performed additional experiments on medium-sized LLMs LLaMA2-chat 7B and 13B using the same setting as Section 4.3.4, and presented the results below. Our observations indicate that IDEAL consistently outperforms the baselines. This demonstrates that our method is a general approach that works well in medium-sized LLMs 

**------------------------------ LLaMA2-7b-chat ------------------------------**

|        | MRPC | MNLI | RTE  |
|--------|------|------|------|
| Random | 64.6 | 37.9 | 56.3 |
| Vote-K | 66.0 | 39.1 | 58.6 |
| IDEAL  | **67.2** | **39.4** | **60.5** |

**------------------------------ LLaMA2-13b-chat ------------------------------**

|        | MRPC | MNLI | RTE  |
|--------|------|------|------|
| Random | 65.8 | 38.8 | 58.5 |
| Vote-K | 66.5 | 40.1 | 59.2 |
| IDEAL  | **66.8** | **40.9** | **60.7** |

Reviewer comment (Experiments):
W1: Some state-of-the-art studies like MDL [a] TopK (Liu et al., 2022) are not cited and/or compared in the experiments.

Author response:
**1. [Re Weakness 1 & Question 1:  Comparisons with Self-adaptive in context learning and TopK.]**

Thank you for your constructive feedback! In our revised version, we have included a detailed discussion **in the related work section (Appendix A)** for Wu et al., 2022 and TopK (Liu et al. 2022 and Gao et al). 

We kindly clarify that the research problem in our paper is different from these papers and the methods are also agnostic. Specifically, 
Wu et al., 2023, is the first work to formally define the problem of self-adaptive in-context learning, which aims to search for the best in-context example and corresponding order with a set of annotated candidate examples.
Liu et al. 2022 and Gao et al. 2021  focus on the example retrieval problem which proposes to use the nearest neighbors of a given test query as the corresponding In-Context examples. 

On the contrary, the research problem of our paper is the selective annotation problem, which aims to curate a core set from large-scale unlabeled data to annotate. This problem is agnostic to the both self-adaptive In-Context Learning problem and example retrieval. 

Additionally, we also provide ablation experiments in Section 4.3.3 to quantify the effect of different example retrieval methods. We believe that all three research problems are meaningful and worth exploring in the research community. Thanks again for your suggestion!

Reviewer comment (Evaluation):
W2: The rationales behind some experimental results are not explained, such as the reverse performance of MNLI in Table 5.

Author response:
**2. [Re Weakness 2 & Question 2: The rationales behind the reverse performance of MNLI in Table 5.]**

We sincerely appreciate your constructive suggestions. We address your concerns as follows. 

- The performance drop of MNLI in Table 5 is predictable because Auto-IDEAL is an automatic annotation method that relies on half of the selected examples being automatically annotated rather than manually labeled ground truth. 

- Auto-IDEAL performs automatic annotation by following the information diffusion process from the initial annotated subset to the examples to be labeled. In some datasets, when the examples to be labeled are not very relevant to the initial examples labeled (but with a similar embedding), it may lead to a performance drop. We acknowledge that this is a potential drawback of Auto-IDEAL. Therefore, we did not regard Auto-IDEAL as the main contribution of our paper but chose a more trivial method IDEAL. 

We appreciate your suggestions and have elaborated more specifically on this issue in the revised versions in **Appendix G.**

Reviewer comment (Evaluation):
W3: Lack of analysis on how the proposed method

Author response:
**3. [Re Question 3 & Weakness 3: It would be great to apply IDEAL to different LLMs.]**

Thank you for your constructive suggestions! 
- We totally agree that IDEAL is a general selective annotation method. We would like to clarify that we have shown the evaluation results of different methods using multiple LLMs **in Figure 5** in our submission. Our evaluations reveal that IDEAL consistently outperforms the baselines across all models tested. 

- Additionally, we further performed more experiments on medium-sized LLMs LLaMA2-7B-chat, LLaMA2-13B-chat using the same setting as Section 4.3.4, and showed the results below. We observed that IDEAL still achieves the best performance compared to the baselines.  

**------------------------------ LLaMA2-7b-chat ------------------------------**

|        | MRPC | MNLI | RTE  |
|--------|------|------|------|
| Random | 64.6 | 37.9 | 56.3 |
| Vote-K | 66.0 | 39.1 | 58.6 |
| IDEAL  | **67.2** | **39.4** | **60.5** |

**------------------------------ LLaMA2-13b-chat ------------------------------**

|        | MRPC | MNLI | RTE  |
|--------|------|------|------|
| Random | 65.8 | 38.8 | 58.5 |
| Vote-K | 66.5 | 40.1 | 59.2 |
| IDEAL  | **66.8** | **40.9** | **60.7** |

Reviewer comment (Experiments):
Q1: Following W1, it would be great if the authors could patch more comparisons to more state-of-the-art works during the author feedback period.

Author response:
**1. [Re Weakness 1 & Question 1:  Comparisons with Self-adaptive in context learning and TopK.]**

Thank you for your constructive feedback! In our revised version, we have included a detailed discussion **in the related work section (Appendix A)** for Wu et al., 2022 and TopK (Liu et al. 2022 and Gao et al). 

We kindly clarify that the research problem in our paper is different from these papers and the methods are also agnostic. Specifically, 
Wu et al., 2023, is the first work to formally define the problem of self-adaptive in-context learning, which aims to search for the best in-context example and corresponding order with a set of annotated candidate examples.
Liu et al. 2022 and Gao et al. 2021  focus on the example retrieval problem which proposes to use the nearest neighbors of a given test query as the corresponding In-Context examples. 

On the contrary, the research problem of our paper is the selective annotation problem, which aims to curate a core set from large-scale unlabeled data to annotate. This problem is agnostic to the both self-adaptive In-Context Learning problem and example retrieval. 

Additionally, we also provide ablation experiments in Section 4.3.3 to quantify the effect of different example retrieval methods. We believe that all three research problems are meaningful and worth exploring in the research community. Thanks again for your suggestion!

Reviewer comment (Evaluation):
Q2: Following W2, I wonder why there are some inconsistent behaviors of the proposed method, instead of only getting the message "the proposed method performs still not bad". This is important because it would decide the choice between IDEAL and Auto-IDEAL.

Author response:
**2. [Re Weakness 2 & Question 2: The rationales behind the reverse performance of MNLI in Table 5.]**

We sincerely appreciate your constructive suggestions. We address your concerns as follows. 

- The performance drop of MNLI in Table 5 is predictable because Auto-IDEAL is an automatic annotation method that relies on half of the selected examples being automatically annotated rather than manually labeled ground truth. 

- Auto-IDEAL performs automatic annotation by following the information diffusion process from the initial annotated subset to the examples to be labeled. In some datasets, when the examples to be labeled are not very relevant to the initial examples labeled (but with a similar embedding), it may lead to a performance drop. We acknowledge that this is a potential drawback of Auto-IDEAL. Therefore, we did not regard Auto-IDEAL as the main contribution of our paper but chose a more trivial method IDEAL. 

We appreciate your suggestions and have elaborated more specifically on this issue in the revised versions in **Appendix G.**

Reviewer comment (Experiments):
Q3: Following W3, because the proposed method should be general, it is reasonable to apply the method to different LLMs. It would be great if the authors could also conduct some related studies.

Author response:
**3. [Re Question 3 & Weakness 3: It would be great to apply IDEAL to different LLMs.]**

Thank you for your constructive suggestions! 
- We totally agree that IDEAL is a general selective annotation method. We would like to clarify that we have shown the evaluation results of different methods using multiple LLMs **in Figure 5** in our submission. Our evaluations reveal that IDEAL consistently outperforms the baselines across all models tested. 

- Additionally, we further performed more experiments on medium-sized LLMs LLaMA2-7B-chat, LLaMA2-13B-chat using the same setting as Section 4.3.4, and showed the results below. We observed that IDEAL still achieves the best performance compared to the baselines.  

**------------------------------ LLaMA2-7b-chat ------------------------------**

|        | MRPC | MNLI | RTE  |
|--------|------|------|------|
| Random | 64.6 | 37.9 | 56.3 |
| Vote-K | 66.0 | 39.1 | 58.6 |
| IDEAL  | **67.2** | **39.4** | **60.5** |

**------------------------------ LLaMA2-13b-chat ------------------------------**

|        | MRPC | MNLI | RTE  |
|--------|------|------|------|
| Random | 65.8 | 38.8 | 58.5 |
| Vote-K | 66.5 | 40.1 | 59.2 |
| IDEAL  | **66.8** | **40.9** | **60.7** |

Reviewer comment (Experiments):
Some experimental settings only include the baselines Random and Vote-k. More methods as mentioned in 4.3.2 can be also included.

Author response:
Thank you for your valuable feedback. We would like to clarify that the research problem in this paper is a selective annotation problem and we have compared our method with alternative selective annotation methods, excluding vote-k and random search in **Section 4.3.2.** The evaluation results show that IDEAL outperforms all alternative methods, which could demonstrate the superiority of our method.

Reviewer comment (Experiments):
The experiments can be supplemented by including other methods not just Random and Vote-k, as mentioned above.

Author response:
Thank you for your valuable feedback. We would like to clarify that the research problem in this paper is a selective annotation problem and we have compared our method with alternative selective annotation methods, excluding vote-k and random search in **Section 4.3.2.** The evaluation results show that IDEAL outperforms all alternative methods, which could demonstrate the superiority of our method.

Reviewer comment (Reproducibility):
The paper claims that it conducts experiments three times and reports the average score. What/where is the randomness? It mainly comes from examples selected or model predictions? Or both of them?

Author response:
The proposed method’s randomness arises from two sources: example selection and model prediction. 
- The randomness in the example selection process is due to the independent-cascade diffusion process used to quantify the influence of a specific subset. To ensure stability, we repeat the information diffusion process ten times when quantifying the influence of each potential subset (as illustrated in **Appendix D.2 and Section 2.2 of the revised version draft**). This procedure is a part of our algorithm.

- The randomness in the model prediction process occurs in specific models like GPT-3.5-turbo. When calling GPT-3.5-turbo through OpenAI API and the temperature is not 0, it suffers from randomness.

Reviewer comment (Experiments):
For the process of information diffusion, in the main paper, it seems that the process is only performed once. In the Appendix, the paper discusses that multiple processes are performed. I suggest that this description can be moved to the main paper. Also, what is the influence of the times of information diffusion?

Author response:
Thank you for your feedback. In our paper, we repeat the information diffusion process ten times when quantifying the influence of each potential subset (**as illustrated in Appendix D.2**). We have moved this content from the Appendix to the main paper (**Section 2.2**).

Reviewer comment (Evaluation):
Additional elaboration is required to explain why the combination of IDEAL/vote-k and similarity retrieval proves effective, while IDEAL/Vote-k alone, coupled with the random selection of supporting examples, does not work well.

Author response:
Thanks for the suggestion. The experimental results suggest that examples with more similar embeddings are more likely to be relevant. Therefore, the combination of IDEAL/vote-k and similarity retrieval is more likely to perform better than its corresponding combination with a random retrieval method.

Reviewer comment (Experiments):
Recently, Chain-of-Thought (CoT) prompting has gained widespread adoption in in-context learning research. Several studies employing this method have reported enhanced model performance. However, an unexplored aspect of this approach is the model's sensitivity and variance concerning the retrieved prompts when used in conjunction with CoT. This work does not address whether such sensitivity exists and whether CoT can be applied for better performance.

Author response:
Thank you for your constructive feedback. We kindly argue that our method is agnostic to the chain-of-thought technology. Our approach involves selecting a subset of “core” examples to annotate. Once annotated, we can use any prompting technology, including chain-of-thought, to further boost the final performance.

Reviewer comment (Evaluation):
Is there any consideration given by the authors to the cost associated with encoding every unlabeled training instance using Sentence-BERT when comparing the methods? As far as I can discern, this aspect was not addressed in the paper.

Author response:
Thank you for your suggestions. For ease of analysis, we calculated the embedding using the same settings as previous works Vote-$k$. Specifically, as illustrated in our paper (**page 3 in our submission**), we obtained the embedding of the examples through Sentence-BERT. The cost of this process is negligible and can be ignored, as it takes less than 5 minutes for all datasets in our experiments.

Reviewer comment (Experiments):
It seems that the instance embedding has a large influence on the proposed method since it needs to build a directed graph using the embeddings. I am interested in changing the previously used pre-trained models and seeing the robustness of the proposed method.

Author response:
Thank you for your suggestions. For ease of analysis, we calculated the embedding using the same settings as previous works Vote-$k$. Specifically, as illustrated in our paper (**page 3 in our submission**), we obtained the embedding of the examples through Sentence-BERT. The cost of this process is negligible and can be ignored, as it takes less than 5 minutes for all datasets in our experiments.

Reviewer comment (Experiments):
The paper shows the label distribution brought by the proposed method. What about the distribution of original data (before selecting)? Are their label distributions similar? Besides, a balanced distribution is expected to bring better performance in follow-up tasks? Or otherwise. I am interested in this.

Author response:
We present the label distribution of the origin data **in the first row of Table 9**. From the statistical results, we can observe that our method achieves the smallest ratio between the numbers of the most frequent class and the least frequent class in 2 out of 3 cases. This scenario implies that our method could mitigate the label skew problem in the origin data.