Paper: Dynamic Demonstrations Controller for In-Context Learning (ICLR-2024)

Reviewer comment (Evaluation):
It would be interesting to discuss what causes the observations in the pilot experiments.

Author response:
**A1**: Thanks for your constructive feedback. We speculate that adding a demonstration to a prompt will have two effects: (1) Providing more information to the prompt, resulting in improvement in performance. (2) Causing the distribution of the prompt to become more different from the pre-training corpus of LLMs, leading to difficulty in understanding the prompt and reducing performance. When more demonstrations are added, the direction of the change in performance depends on which effect is more influential.  For different datasets and LLMs, when adding more demonstrations, the strengths of Effect (1) and Effect (2) are different, leading to the variation observed in pilot experiments and also causing the difference in the optimal k.

Reviewer comment (Experiments):
It is important to conduct experiments comparing demonstration number selection and demonstration selection. The original Table 2 shows that the proposed method can further improve based on the demonstration selection. However, it is still unclear which one is more effective among demonstration selection and dynamic demonstration number selection.

Author response:
**A2**: Thanks for your precious suggestion. As we demonstrated in our paper, KATE (a demonstration selection method) obtains more competitive performance when adapted to these ten classification datasets than our demonstration number selection method. However, KATE uses a default number to select demonstrations and is not able to decide how many demonstrations to select is good. Our method is orthogonal to KATE while also being complementary. Thus, combining two methods can achieve better performance.

Reviewer comment (Evaluation):
In terms of the limitations, when will the method fail, and when will the method have a good performance?

Author response:
**A3**: Thanks for your valuable comment. Regarding the $D^2Controller$ method, some LLMs exhibit a minor decline in performance on the MPQA, SST-2, and MR datasets compared to the default setting. One possible reason is that these datasets have relatively shorter average demonstration lengths (shown in Table 6), leading to encoded semantic representations that contain less information. Thus, the similarities measured by IICScore based on these representations are inaccurate. In this case, selecting an appropriate demonstration number for these datasets may be more challenging. In contrast, most of the datasets with relatively longer average demonstration lengths perform well with $D^2Controller$.

Reviewer comment (Evaluation):
What may cause the observations in Pilot experiments? For instance, in Figure 2, what aspects of the datasets cause the different optimal k for different datasets?

Author response:
**A1**: Thanks for your constructive feedback. We speculate that adding a demonstration to a prompt will have two effects: (1) Providing more information to the prompt, resulting in improvement in performance. (2) Causing the distribution of the prompt to become more different from the pre-training corpus of LLMs, leading to difficulty in understanding the prompt and reducing performance. When more demonstrations are added, the direction of the change in performance depends on which effect is more influential.  For different datasets and LLMs, when adding more demonstrations, the strengths of Effect (1) and Effect (2) are different, leading to the variation observed in pilot experiments and also causing the difference in the optimal k.

Reviewer comment (Experiments):
How does $D^2$ Controller compare to a simple baseline where $k$ is optimized as a hyperparameter using a validation set?

Author response:
**A1**: Thanks for your valuable comment. Following your suggestion, we randomly sample some examples (i.e., 100, 200, 300) to construct validation sets as a baseline for choosing k. The results are as follows (note that the results we report are the average performance of ten datasets):

|                     | GPT-2 1.5B | Cerebras-GPT 2.7B | Cerebras-GPT 6.7B | OPT 13B |
| ------------------- | ---------- | ----------------- | ----------------- | ------- |
| **Default setting** | 60.0         | 63.0                | 71.7              | 74.5    |
| **Validation-100**  | 64.9       | 68.3              | 72.6              | 75.8    |
| **Validation-200**  | 65.4       | 68.5              | 71.8              | 76.1    |
| **Validation-300**  | 64.9       | 68.3              | 72.6              | 76.4    |
| $D^2Controller$     | **67.0**         | **69.3**              | **74.0**                | **76.6**    |

Based on these results, we can observe that using these examples does not lead to the optimal choice of k, and almost all of the results are inferior to $D^2Controller$. This further underscores the effectiveness of using IICScore to select a small number of representative examples.

Reviewer comment (Evaluation):
How does $D^2$ Controller work with `classification` tasks where options have no consistent meanings? For instance, below is a datapoint taken from the BigBenchHard dataset:
  ```
  Jane quited her job on Mar 20, 2020. 176 days have passed since then. What is the date today in MM/DD/YYYY?
  (A) 09/12/2020
  (B) 11/12/2020
  (C) 12/12/2020
  (D) 09/12/1961
  (E) 09/17/2020
  ```
In this case, computing IICScore per class makes less sense. Could the author provide more insights on this?

Author response:
**A2**: Thanks for your inspiring comment.  The BigBenchHard dataset mentioned above is essentially not a classification task but rather a QA (Question-Answering) task. A possible way to work with it is to convert the QA form into classification form. Specifically, we pair the question sentence with each answer and label the pair with correct answer as "yes" and others as "no". Thus, the task is transformed to a 2-class classification task and we can apply our $D^2Controller$ to the dataset.

Reviewer comment (Theory):
How does $D^2$ Controller work when $k < |c|$?

Author response:
**A3**: We are sorry that we did not understand exactly the meaning of the above comment. We try answering it:

According to our understanding, |c| denotes the number of examples for a class c. When k < |c|, we can certainly sample k-shot examples from each class. When k > |c|, we can sample all examples from each class. In this paper, all ten of our datasets meet the condition where k < |c|.

Reviewer comment (Experiments):
In my opinion, adding GPT-3 in Section 5.4 could make the analysis stronger.

Author response:
**A5**: Thanks for your precious suggestion. We have reported the results of GPT-3 (175B) in Section 5.4. Based on these results, we can observe that our previous analysis is also applicable to GPT-3. This change has indeed strengthened our analysis, providing further support for our research findings. Thanks for your suggestion again.

|                                             | GPT-2 0.3B | GPT-2 0.8B | GPT-2 1.5B | Cerebras-GPT 2.7B | Cerebras-GPT 6.7B | GPT-3 175B |
| ------------------------------------------- | :--------- | :--------- | :--------- | :---------------- | :---------------- | :--------- |
| **KATE**                                    | 54.1       | 58.7       | 66         | 65.4              | 73.0                | 82.2       |
| **KATE + $D^2Controller$**                  | 54.4       | 59.2       | 64.0         | 67.1              | 72.6              | 82.6       |
| **Contextual Calibration**                  | 54.9       | 59.3       | 62.2       | 68.3              | 72.4              | 78.9       |
| **Contextual Calibration+ $D^2Controller$** | 54.5       | 62.4       | 67.0         | 68.7              | 74.0                | 80.1       |

|                         | GPT-2 0.3B | GPT-2 0.8B | GPT-2 1.5B | Cerebras-GPT 2.7B | Cerebras-GPT 6.7B | GPT-3 175B |
| ----------------------- | ---------- | ---------- | ---------- | ----------------- | ----------------- | ---------- |
| **Random**              | 54.1       | 58.7       | 66.0         | 65.4              | 73                | 81.3       |
| **$D^2Controller$-ED**  | 54.4       | 59.2       | 64.0         | 67.1              | 72.6              | 79.1       |
| **$D^2Controller$-Cos** | 54.9       | 59.3       | 62.2       | 68.3              | 72.4              | 80.4       |
| $D^2Controller$         | 54.5       | 62.4       | 67.0         | 68.7              | 74.0                | 82.0         |

|                             | GPT-2 0.3B | GPT-2 0.8B | GPT-2 1.5B | Cerebras-GPT 2.7B | Cerebras-GPT 6.7B | GPT-3 175B |
| --------------------------- | ---------- | ---------- | ---------- | ----------------- | ----------------- | ---------- |
| **$k_{\max}$-shot setting** | 54.1       | 58.7       | 66.0         | 65.4              | 73.0                | 81.4       |
| **$D^2Controller$**         | 54.5       | 62.4       | 67.0         | 68.7              | 74.0                | 82.0         |

Reviewer comment (Experiments):
The authors report a setting where they combine KATE and $D^2$ Controller, this is interesting. Now, KATE is selecting $k$ different IC examples per test data point, where $k$ is determined by $D^2$ Controller at a dataset level. While I understand the setting, could the authors provide some insights on, is it necessary, or is there a way to dynamically determine the $k$ for every test data point?

Author response:
**A6**: Thanks for your constructive review. It is meaningful but impractical to dynamically determine the k for every test data point. Here we design a possible way to achieve this goal. First, we select m groups of different k-shot samples for each test data point based on similarity, and then use the IICScore method to construct a validation set for the m groups of k-shot samples. Subsequently, we calculate the accuracy of each group of k-shot samples on the validation set, and finally choose the group with the highest accuracy among the m groups to determine the k value. 

Compared to our method, dynamically selecting k values for each test data would incur a significant computational cost, roughly tens of times more time-consuming. Moreover, as the number of test data points increases, the time cost will also multiply, which is impractical in real-world applications.

Reviewer comment (Evaluation):
Could the authors provide some insights on why sometimes LLMs fail to benefit from more IC examples? Do stronger LLMs (e.g., gpt-4) suffer less from this?

Author response:
**A7**: Thanks for your constructive feedback. We speculate that adding an in-context example to a prompt will have two effects: (1) Providing more information to the prompt, resulting in improvement in performance. (2) Causing the distribution of the prompt to become more different from the pre-training corpus of LLMs, leading to difficulty in understanding the prompt and reducing performance. When more IC examples are added, the direction of the change in performance depends on which effect is more influential. For some datasets on some LLMs, the Effect (2) is stronger, leading to performance degradation when more IC examples are included.

Besides, similar to Section 3, we have conducted pilot experiments with the GPT-4 model on five text classification datasets. Due to budgetary constraints, for each dataset, we use five different seeds to test the model's performance in the 1-shot setting, the default setting (4-shot), and $k_{max}$-shot setting. Note that the maximum input length of the GPT-4 model we use is 8192 tokens, so the maximum shot number for SST-5, CR, MPQA, RTE, and CB is 32, 128, 256, 32, and 16. The results are as follows:

| GPT-4                       | SST-5    | CR       | MPQA     | RTE      | CB       |
| --------------------------- | -------- | -------- | -------- | -------- | -------- |
| **1-shot setting**          | 45.3±4.4 | 83.7±1.3 | 67.4±1.0 | 82.7±3.0 | 89.3±1.8 |
| **Default setting(4-shot)** | 45.7±5.0 | 92.2±2.2 | 83.8±0.3 | 89.1±1.4 | 83.9±2.5 |
| **$k_{\max}$-shot setting** | 43.6±0.8 | 95.9±0.3 | 90.2±1.1 | 88.7±0.6 | 82.7±1.0 |

From the perspective of a general trend, when the input increases from a 1-shot setting to $k_{max}$- shot setting, the accuracy improves on the CR, MPQA, and RTE datasets while declines on the SST-5 and CB datasets. Moreover, in the above table, the RTE dataset achieves the best performance in the default setting, rather than $k_{max}$- shot setting. Thus, increasing the number of demonstrations in stronger LLM like GPT-4 does not necessarily improve performance.

Reviewer comment (Evaluation):
In Section 5.4.4, the authors mention that they get better performance with fewer demonstrations. Maybe a more straightforward way to present this is to report (on average) how many tokens their method queries an LLM, and how does that compare to prior work (default $k$).

Author response:
**A8**: Thanks for your valuable suggestion. We reported the average number of tokens used by three methods (default k, maximum k, and ours) to query LLM:

|                 | GPT-2 1.5B | Cerebras-GPT 2.7B | Cerebras-GPT 6.7B | OPT 13B |
| --------------- | ---------- | ----------------- | ----------------- | ------- |
| **Default k**   | 455.49     | 516.87            | 516.87            | 516.87  |
| **Maximum k**   | 678.29     | 1345.72           | 1345.72           | 1345.72 |
| $D^2Controller$ | 603.98     | 885.51            | 1187.37           | 725.89  |

Based on these results, we can observe that our method uses fewer tokens to achieve better performance compared to maximum k. Especially on some LLMs, such as Cerebras-GPT 2.7B and OPT-13B, $D^2Controller$ saves almost 30% and 50% tokens.  Meanwhile, although our method uses more tokens compared to the default k, it achieves an average relative improvement of 5.4% on ten datasets.

Reviewer comment (Presentation):
I have read the authors response including those answering other reviewers' questions. I appreciate the authors' effort on clarifying things so I'm happy to raise my score a bit. **However, please note, I give 6 -> 8 only because there is no option of 7. I don't think the current version is as mature as 8.** (E.g., the authors have included quite a bit of new experiments during the rebutal period, mainly in the appendices. It may require some efforts to merge some into the main content, with some non-trivial rewriting.)

Author response:
We are very happy to hear that the reviewer has raised the score! We thank you again for the invaluable feedback and insights. We really appreciate it!

Reviewer comment (Experiments):
This is basically a hyperparameter selection paper, and as such it is missing a key baseline: what if one uses as many examples as possible for selecting k? That would correspond to the classic setting of having your dataset split into training, validation and test sets. While it would be more computationally expensive at the hyperparameter selection time, the key concern in practical applications of LLMs is the inference speed at test time, which would not be affected by using more validation examples to select k.

Author response:
**A1**: Thanks for your valuable comment. Following your suggestion, we randomly sample more examples as a baseline to select k. Specifically, we construct three different sizes of validation sets (100, 200, and 300) to select k. The results are as follows (note that the results we report are the average performance of ten datasets):

|                     | GPT-2 1.5B | Cerebras-GPT 2.7B | Cerebras-GPT 6.7B | OPT 13B |
| ------------------- | :--------: | :---------------: | :---------------: | :-----: |
| **Default setting** |     60.0     |        63.0         |       71.7        |  74.5   |
| **Validation-100**  |    64.9    |       68.3        |       72.6        |  75.8   |
| **Validation-200**  |    65.4    |       68.5        |       71.8        |  76.1   |
| **Validation-300**  |    64.9    |       68.3        |       72.6        |  76.4   |
| **$D^2Controller$** |     **67.0**     |       **69.3**        |        **74.0**         |  **76.6**   |

Based on these results, we can observe that using more examples does not lead to the optimal choice of k, and almost all of the results are inferior to $D^2Controller$. This further underscores the effectiveness of using IICScore to select a small number of representative examples.

Reviewer comment (Presentation):
Some aspects of the paper were difficult to understand, see the next section of the review. I found Figure 5 very dense and difficult to understand. I did not find the motivation for IICScore clearly explained and compelling.

Author response:
**A6**: The leftmost part of Figure 5 shows the iteration process over different k-shot settings (From 1-shot setting to 16-shot setting). The rest of the figure shows an example of how to evaluate the 4-shot setting, which is divided into 3 steps: 

1. Sampling $N_s$ groups of 4-shot in-context examples from the training dataset.

2. Sampling evaluation examples from the rest of the training dataset according to IICScore. 

3. Computing average prediction accuracy of the selected groups of in-context examples on the evaluation examples.

**A7**: Thanks for your valuable feedback on the motivation of IICScore. To save costs, we hope to evaluate the performance of each k-shot setting with a small number of evaluation examples. However, when the number of evaluation examples is small, the evaluation results can be biased. To alleviate the problem, we hope to select a set of evaluation examples that are comprehensive to reflect the performance. Specifically, we co-opt the idea of the fit ability and the generalization ability in ML to achieve comprehensiveness:

* The fit ability reflects how well a model can capture seen patterns, and here we adapt the idea by evaluating the performance of a group of in-context examples using similar examples. 

* The generalization ability is the ability to deal with unseen data, and we analogously use dissimilar examples to evaluate. 

To measure similarities, considering the type of task is text classification, we propose IICScore which takes both intra and inter-class distances into account to measure similarities. 

In sum, the motivation behind evaluation examples selection is to construct a set that is small in scale but comprehensive, and IICScore is the metric to help us find these examples.

Reviewer comment (Experiments):
Is your Oracle baseline using the test set examples? If it is, your explanation as to why it is not practical on Page 7 is a bit confusing. Because Oracle would not be a possible practical method, it’s just a hypothetical baseline from above.

Author response:
**A4**: Yes, our Oracle baseline uses test set examples. In this paper, the Oracle baseline is actually an approximation to the real upper bound, obtained by iterating through all k-shot settings on the test set. As we mentioned in Section 5.1, to control the GPT-3 inference costs, we randomly sample 256 examples from each dataset to evaluate accuracy. Thus, the iteration is possible on such small-scale of test examples. However, in real-world scenarios, the number of test examples is much larger than 256. Consequently, it becomes impractical to iterate through all k-shot settings to obtain the real Oracle, due to the substantial resource and time requirements.

Reviewer comment (Experiments):
What set of in-context examples is used at the test time? Is it one the N sets you used to select k, or is it another one?

Author response:
**A5**:  The set of in-context examples at test time is one of the N sets we used to select k.