Paper: Demonstration Distillation for Efficient In-Context Learning (ICLR-2024)

Reviewer comment (Experiments):
One concern with the paper is the lack of a zero-shot baseline. For example, for BoolQ, the authors report a 3-shot accuracy of about 60% in Table 3. This seems to be about the same as the _zero_-shot accuracy for GPT-3 (which should match or be outperformed by ChatGPT used by the authors) published here: https://paperswithcode.com/sota/question-answering-on-boolq?tag_filter=188. **If there are no benefits to including in-context examples in the first place, then it is no surprise that there is no performance hit from compressing input examples with any method, including DGS.** The zero-shot baseline can be seen as an 'infinite compression baseline': it compresses inputs of any size into zero tokens. I would like the authors to demonstrate that DGS outperforms this zero-shot baseline.

Author response:
We fully understand the reviewer's question on why the performance on GSM8K degrades when changing from 8-shot demonstrations to 16-shot demonstrations. We have conducted experiments on GSM8K using 0-shot, 4-shot, 8-shot, and 16-shot randomly sampled demonstrations without any distillation (each of the results is averaged over three repeated runs, and all the experiments use ChatGPT-0925):
  1. 0-shot: 72.5
  2. 4-shot: 75.5
  3. 8-shot: 77.2
  4. 16-shot: 77.1

The results above show that GSM8K is not a kind of task whose 0-shot evaluation outperforms few-shot, and it actually fits the general expectation of ICL. The interesting phenomenon is that the average performance of 16-shot demonstrations is still slightly outperformed by that of 8-shot. We reason that it is because 16-shot demonstrations are too long in context length (typically over 2000 tokens) and thus lead to higher difficulty for LLM to fully understand the context and relate to the test question, eventually failing to further improve the ICL performance. What our DGS does is just to alleviate such a long-context problem. As is demonstrated in Table 1, for 16-shot random demonstrations containing 2014 tokens and 8-shot longest demonstrations containing 3020 tokens, by distilling the overlong demonstrations to a reasonable length of around 1000 tokens so that the LLM can more easily understand, DGS both improves the original performance by 0.3% and 1% respectively.

Reviewer comment (Evaluation):
Table 1 shows that there are _no_ improvements as the number of in-context examples is increased for GSM8K. In fact, an increase in the number of in-context learning (ICL) examples leads to decreased performance. On the one hand, this supports concerns that a zero-shot baseline would outperform DGS. On the other hand, this makes me sceptical that DGS is actually a method that 'ensure generalizability across diverse query types'. For standard ICL tasks, I would expect the performance to improve (and eventually stagnate) as more examples are provided in-context.

Author response:
We fully understand the reviewer's question on why the performance on GSM8K degrades when changing from 8-shot demonstrations to 16-shot demonstrations. We have conducted experiments on GSM8K using 0-shot, 4-shot, 8-shot, and 16-shot randomly sampled demonstrations without any distillation (each of the results is averaged over three repeated runs, and all the experiments use ChatGPT-0925):
  1. 0-shot: 72.5
  2. 4-shot: 75.5
  3. 8-shot: 77.2
  4. 16-shot: 77.1

The results above show that GSM8K is not a kind of task whose 0-shot evaluation outperforms few-shot, and it actually fits the general expectation of ICL. The interesting phenomenon is that the average performance of 16-shot demonstrations is still slightly outperformed by that of 8-shot. We reason that it is because 16-shot demonstrations are too long in context length (typically over 2000 tokens) and thus lead to higher difficulty for LLM to fully understand the context and relate to the test question, eventually failing to further improve the ICL performance. What our DGS does is just to alleviate such a long-context problem. As is demonstrated in Table 1, for 16-shot random demonstrations containing 2014 tokens and 8-shot longest demonstrations containing 3020 tokens, by distilling the overlong demonstrations to a reasonable length of around 1000 tokens so that the LLM can more easily understand, DGS both improves the original performance by 0.3% and 1% respectively.

Reviewer comment (Experiments):
Have you tried going through QA pairs individually instead of asking the models to distill them in batches? It seems like this approach could be less error-prone.

Author response:
We agree with the reviewer that distilling QA pairs individually instead of using a batch mode will possibly be less error-prone. But there are two noteworthy points. First, going through QA individually scales the latency cost by multiple API requests, and also increases the token cost since each pair of QA shares a large proportion of similar tokens in their input prompts(e.g. Distillation criteria provided for the LLM) and generated content from the LLM. Second, it is shown empirically that the batch mode we employ now already has a low error frequency, and is generally robust enough for the experimented demonstration distillation tasks.

Reviewer comment (Presentation):
In the prompt for the specialist (figure 8), why is the ground truth answer or the LLM's prediction for the original, long input not included? This information could make it much easier to judge the usefulness of the distillation.

Author response:
In our experiments, a target question is randomly sampled from the training set and does have a ground truth answer. We agree with the reviewer that including the ground truth answer to the target question will make it easier to judge the usefulness of the distillation, especially for the specialist. However, we choose not to include it out of the consideration of simulating real-world use cases. Ideally, a user may provide a target question along with the demonstrations for DGS, hoping that the target question can positively guide and check the distillation output. Since it is not always the case that the user knows the ground truth answer to the target question he provides, we think it is reasonable to assume that the ground truth answer is not given to DGS, and DGS should do all the distillation and checking only based on the target question itself.

Reviewer comment (Writing):
In the description, "If the computed score exceeds 90, it indicates that the distilled demonstrations are well-suited for the specific question. In this case, the demonstrations, having been evaluated by both the Generalist and Specialist, proceed to the Distillist for further refinement" — why, if the demonstrations are 'well-suited', are they passed to the distillist again? Is DGS applied iteratively, or does the distillist perform another distinct function at this stage? When does this process end?

Author response:
For the termination criterion of our iterative DGS loop, we would like to state it more clearly as follows. Generally speaking, we hope the distilled demonstrations are as concise as possible under the condition of effectiveness (correctness). The distillist and generalist are mostly dedicated to the correctness of distillation and do not pay much attention to conciseness. Thus we hope the specialist can somehow not only check the effectiveness of the distilled demonstrations with regard to the user-provided question but also promote the degree of conciseness. We implement this idea by designing the special termination criteria in the specialist: if the computed score given by the specialist exceeds 90, it means the current distilled demonstrations can still serve the purpose of helping the LLM to answer the final question well, which indicates room for further distillation that continues to condense the currently eligible demonstrations to make it more concise. However, if the computed score falls below 90, then we think the demonstrations are now overly distilled, and cannot guide the solution of the final answer well enough. Therefore, the distillation process is terminated at this point, breaking out the DGS loop with the distilled demonstrations of the last round (i.e. The demonstrations that are most concise before being overly distilled).

Reviewer comment (Reproducibility):
"Demonstrations exceeding 2000 tokens are partitioned into two segments, each of which is distilled independently before concatenation. Additionally, since the input question participates in the distillation process, a question is randomly selected from the training set before distillation starts." Why not use the question that actually belongs to the demonstration? The distillist prompt mentions "several User-Response pairs" but is apparently called with a single input instead.

Author response:
We would also like to clarify the components of the input to distillist. The distillist mainly receives 2 things. The first is the "User-Response" pairs, which are namely the demonstrations to be distilled. The second is a randomly sampled question from the training set that serves the purpose of checking if the distilled demonstrations are still eligible by the specialist. It is noteworthy that the randomly sampled question cannot in the meanwhile be part of the demonstrations to be distilled. Otherwise, it won't be able to be used for the specialist's checking procedure where the distilled demonstrations are prepended to the randomly sampled question for inference and evaluation.

Reviewer comment (Experiments):
The claim "DGS proves consistently effective, regardless of how the demonstration and question are selected, as will be demonstrated in subsequent experiments" requires clarification. Where is this shown?

Author response:
We fully understand the reviewer's question on how we demonstrate the robustness of DGS against the randomness posed by the varieties of demonstrations to be distilled. To address this, we have conducted a new experiment that randomly samples different 8-shot demonstrations for distillation, and calculate the average metric values including performance improvement and distillation ratio. The results on GSM8K are shown as follows:
| # | Model         | # Token Original | # Token DGS | Ratio | Acc Original | Acc DGS |
|---|---------------|------------------|-------------|-------|--------------|---------|
| 1 | ChatGPT-0613  | 990              | 616         | *1.61 | 80.7         | 81.8    |
| 2 | ChatGPT-0925  | 1096             | 490         | *2.24 | 76.1         | 77.1    |
| 3 | ChatGPT-0925  | 1021             | 514         | *1.99 | 78.3         | 78.6    |
  
The average distillation ratio is ×1.95, and the average performance improvement is 0.8%. These two metrics are both consistent among repeated runs, thus showing that our DGS is robust enough against randomness in selected demonstrations.

Reviewer comment (Theory):
In the phrase "Such an emergent feature fundamentally", why is this called an 'emergent' feature? What is meant by this? There does not appear to be sufficient evidence and discussion to support the claim that this is an 'emergent' feature.

Author response:
We thank the reviewer for pointing out this statement which is not rigorous enough in the submission. By saying "emergent", we intuitively posit that the ability to "Effectively Incorporate" is more likely to be present in larger language models like ChatGPT compared with smaller models, since such an ability requires understanding and rephrasing of multi-hop complex reasoning statements. However, we did not compare with more language models of different sizes or conduct a quantitative analysis on this problem, so such a statement is indeed not rigorous enough. We apologize for our oversight and have removed this statement from the revised paper.

Reviewer comment (Experiments):
For your experimental setup, I have noticed that you test on **one** randomly sampled ICL prompt per task. I am just curious about the robustness of your utility in handling many other ICL prompts. That is to say, it would be better for showing some average results (performance, and compression ratio).

Author response:
We fully understand the reviewer's concern about the robustness of DGS given the randomness of sampled demonstrations to be distilled. To address this, we have conducted experiments that sample different random demonstrations for distillation and calculated average results including performance improvement and distillation ratio. The results on GSM8K are shown as follows:  
| # | Model         | # Token Original | # Token DGS | Ratio | Acc Original | Acc DGS |  
|---|---------------|------------------|-------------|-------|--------------|---------|  
| 1 | ChatGPT-0613  | 990              | 616         | *1.61 | 80.7         | 81.8    |  
| 2 | ChatGPT-0925  | 1096             | 490         | *2.24 | 76.1         | 77.1    |  
| 3 | ChatGPT-0925  | 1021             | 514         | *1.99 | 78.3         | 78.6    |  

  The average distillation ratio is 1.95, and the average performance improvement is 0.8%. These two results are both consistent among the repeated experiments shown above.

Reviewer comment (Evaluation):
It will definitely be better to include any kinds of **efficiency** discussions. If we aim at deploying your technique, how it costs?

Author response:
We also thank the reviewer for pointing out the necessity of efficiency discussion. We would like to discuss the problem of cost and efficiency in two aspects.  
1. First, we have conducted a new experiment to show that demonstration distillation for a given task by DGS can maintain the average cost within a reasonable budget. Specifically, we randomly select 8-shot demonstrations from GSM8K and distill them using the DGS method. During distillation, we record the number of times calling the distillist, generalist, and specialist respectively until the distillation is completed. We also record the total number of input and output tokens consumed in the whole procedure. The experiment is repeated 10 times and the final results are reported below. We have also updated the paper with this experiment in Appendix C.  
| No. | \# Distillist calls | \# Generalist calls | \# Specialist calls | \# Input tokens | \# Output tokens |  
|-----|------------|------------|------------|--------|--------|  
| 1   | 7          | 6          | 2          | 18011  | 5557   |  
| 2   | 3          | 2          | 2          | 8242   | 2630   |  
| 3   | 5          | 4          | 3          | 11085  | 3472   |  
| 4   | 3          | 2          | 2          | 7954   | 2377   |  
| 5   | 5          | 4          | 3          | 13368  | 4390   |  
| 6   | 2          | 1          | 1          | 5957   | 1256   |  
| 7   | 7          | 6          | 4          | 19635  | 5909   |  
| 8   | 5          | 4          | 1          | 11489  | 3160   |  
| 9   | 5          | 4          | 1          | 12485  | 4747   |  
| 10  | 3          | 2          | 1          | 8884   | 3356   |  
| **Mean** | **4.5 (±1.7)** | **3.5 (±1.7)** | **2 (±1.0)** | **11711 (±4391)** | **3685 (±1462)** |  

  From the data above we can estimate the average API usage cost. If we use OpenAI GPT-3.5-turbo-1106 API, then the average cost of the whole DGS procedure for a given task is (11711\*0.001 + 3685.4\*0.002) / 1000 = \$0.019. Such a cost is reasonable since our DGS is not a test-time method and only needs to be applied once to a given task.

  2. Second, it is natural to compare DGS with other finetuning-based methods aimed at extending the context limit for LLMs, under the discussion of reducing costs. We would like to point out that our DGS method is an API-only procedure that does not require any GPU resource or computational cost aside from a reasonable cost of API usage. This makes our method more user-friendly and easy to use especially for practitioners who do not have much experience in LLM deployment and training, compared with the potential cost of data engineering and training intricacies involved in model finetuning.

Reviewer comment (Novelty):
The innovation in this paper is lacking. Using large models to compress content and address the issue of excessively long text is a technique that has been applied for quite some time. For instance, techniques like summarization in langchain-github have been used to avoid exceeding maximum length limits.

Author response:
We fully understand the reviewer's concern about our paper's novelty compared with summarization techniques used in langchain. We think there exist fundamental differences between our DGS and langchain summarization, mainly in designing motivation and scope of application. First, our DGS method is specifically designed for "distilling" contexts instead of "compressing" contexts. The key difference between distillation and compression is that distillation is always performed and checked with regard to the specific question provided by the user, which is especially highlighted by the specialist. To be specific, the specialist checks if the distilled demonstrations presented by the distillist are over-distilled by prepending these demonstrations to the user-given question and testing if the LLM's response to the given question is still satisfactory. On the contrary, the techniques in langchain only focus on the simple and general task of summarizing a given document, without an effective checking procedure that is specially optimized for a few-shot demonstration setting or taking into consideration the effect of distilled demonstrations with respect to the user-provided question. Moreover, table 4 in Appendix B also shows some examples of key information loss in distilled few-shot demonstrations when the checking procedure is removed, which substantiates the importance of the completeness of our DGS structure.“”

Reviewer comment (Writing):
The description of the key aspects of this method is not precise enough, and I couldn't find a clear definition of "Punishment Score."

Author response:
We provide a definition of the Punishment Score in the prompt given to the generalist in Figure 7. Specifically, the punishment score is imposed by the generalist if the following criterion is not met: there exists one distilled demonstration whose answer part uses key information that is not provided in its question part but still derives the final answer, thus indicating over-distillation. Each ineligible demonstration under this criterion will increase the Punishment Score by 10 points.

Reviewer comment (Evaluation):
The lack of evidence for improvement in accuracy makes the explanations seem more like speculation. The compression that removes incorrect information appears similar to a CoT process, but is the test input compressed? If it is, it becomes difficult to verify if the improvement is due to changes in demonstrations because the test input questions have also changed.

Author response:
We would also like to point out that during the DGS process, the test question will not be involved and thus won't be distilled. The user-provided question used in DGS is a randomly sampled one from the same distribution as the demonstrations to be distilled (i.e. The training set). The whole distillation process is completed before the distilled demonstrations are prepended to the test question for inference and evaluation.

Reviewer comment (Writing):
Could you provide a clear definition of "Punishment Score"? I couldn't find the mathematical definition of this term. Is this value just evaluated by ChatGPT? How is its reliability ensured?

Author response:
We provide a definition of the Punishment Score in the prompt given to the generalist in Figure 7. Specifically, the punishment score is imposed by the generalist if the following criterion is not met: there exists one distilled demonstration whose answer part uses key information that is not provided in its question part but still derives the final answer, thus indicating over-distillation. Each ineligible demonstration under this criterion will increase the Punishment Score by 10 points.

Reviewer comment (Evaluation):
The rewriting seems like a CoT (Compression of Transformations) process that changes the way reasoning is done. It's reasonable to say that a more straight question has a higher accuracy for the model. However, if the final test questions are compressed, it becomes difficult to verify whether the improvement is due to changes in demonstrations. This is because the input questions have changed. For example, if a mathematical problem is rephrased by ChatGPT to make it clearer, it becomes easier to answer correctly. However, this doesn't prove that the improvement is due to the author's method or its connection to demonstrations. It may simply be the result of the question being rephrased, which is different from the claim made by the author regarding their contribution.

Author response:
We would also like to point out that during the DGS process, the test question will not be involved and thus won't be distilled. The user-provided question used in DGS is a randomly sampled one from the same distribution as the demonstrations to be distilled (i.e. The training set). The whole distillation process is completed before the distilled demonstrations are prepended to the test question for inference and evaluation.

Reviewer comment (Experiments):
The proposed DGS approach is powered using OpenAI's chatgpt. While chatgpt is a very popular and capable model, nonetheless it is a closed-source model. The authors should have investigated a few (1-3) open-source LLMs that could power the three LLM agents instead and check how the performance changes.

Author response:
## Weakness 1 (Question 1): Why not test DGS's performance with open-source LLMs?
We fully agree with the reviewer that it is beneficial to test DGS's performance with open-source LLMs so as to gain a more comprehensive picture of DGS's applicability. However, we would like to point out that the black-box property of DGS is also one of the most important considerations as we design its implementation. We specifically design DGS as an API-only procedure that does not require any GPU resource. This makes our method more convenient to use, especially for general practitioners who do not have much experience or abundant resources in LLM deployment. We deem such an API-only and user-friendly feature of DGS as one of its most valuable merits and therefore do not prioritize the testing with various open-source LLMs.

Reviewer comment (Experiments):
The proposed approach is motivated to help reduce computational and financial costs that are associated with using longer demonstrations in an ICL framework. However, using models like chatgpt to power the proposed approach can also end up being expensive to operate. The authors should have designed and included a small experiment describing the potential financial relief the proposed approach offers.

Author response:
## Weakness 2: How much financial relief does DGS offer, compared with other finetuning-based approaches to extending context-limit?
We fully understand the reviewer's concern about DGS's ability to reduce costs. We would like to address this concern in two aspects. 
  1. First, we have conducted a new experiment to show that demonstration distillation for a given task by DGS can maintain the average cost within a reasonable budget. Specifically, we randomly select 8-shot demonstrations from GSM8K and distill them using the DGS method. During distillation, we record the number of times calling the distillist, generalist, and specialist respectively until the distillation is completed. We also record the total number of input and output tokens consumed in the whole procedure. The experiment is repeated 10 times and the final results are reported below. We have also updated the paper with this experiment in Appendix C.

| No. | \# Distillist calls | \# Generalist calls | \# Specialist calls | \# Input  tokens| # Output tokens |
|-----|------------|------------|------------|--------|--------|
| 1   | 7          | 6          | 2          | 18011  | 5557   |
| 2   | 3          | 2          | 2          | 8242   | 2630   |
| 3   | 5          | 4          | 3          | 11085  | 3472   |
| 4   | 3          | 2          | 2          | 7954   | 2377   |
| 5   | 5          | 4          | 3          | 13368  | 4390   |
| 6   | 2          | 1          | 1          | 5957   | 1256   |
| 7   | 7          | 6          | 4          | 19635  | 5909   |
| 8   | 5          | 4          | 1          | 11489  | 3160   |
| 9   | 5          | 4          | 1          | 12485  | 4747   |
| 10  | 3          | 2          | 1          | 8884   | 3356   |
| **Mean** | **4.5 (±1.7)** | **3.5 (±1.7)** | **2 (±1.0)** | **11711 (±4391)** | **3685 (±1462)** |

  From the data above we can estimate the average API usage cost. If we use OpenAI GPT-3.5-turbo-1106 API, then the average cost of the whole DGS procedure for a given task is (11711\*0.001 + 3685.4\*0.002) \/ 1000 = \$0.019. Such a cost is reasonable since our DGS is not a test-time method and only needs to be applied once to a given task.

  2. Second, it is natural to compare DGS with other finetuning-based methods aimed at extending the context limit for LLMs, under the discussion of reducing costs. We would like to point out again that our DGS method is an API-only procedure that does not require any GPU resource or computational cost aside from a reasonable cost of API usage. This makes our method more user-friendly and easy to use especially for practitioners who do not have much experience in LLM deployment and training, compared with the potential cost of data engineering and training intricacies involved in model finetuning.

Reviewer comment (Experiments):
The proposed approach is only compared against AutoCompressor method despite including two other demonstration compression methods in the related work section.

Author response:
## Weakness 3 (Question 2): Why not compare DGS with demonstration compression methods other than AutoCompressor?
We appreciate the reviewer's suggestion to compare our method with additional compression methods referenced in our related work. However, we respectfully contend that a comparison with the two specific works cited (https://arxiv.org/pdf/2304.08467.pdfandhttps://arxiv.org/pdf/2307.06945.pdf) may not be strictly necessary for our study.
  - Regarding the paper "Learning to Compress Prompts with Gist Tokens," the authors have innovatively developed a technique for condensing short, instruction-like prompts into gist tokens. This approach, while effective for its intended purpose, seems less applicable to our goal of compressing longer demonstrations for in-context learning. Our focus is on distilling extensive contexts into shorter forms, which diverges from the gist token methodology.
  - On the other hand, the paper "In-context Autoencoder for Context Compression in a Large Language Model" introduces the In-context Autoencoder (ICAE), a novel model for context compression. This method shares similarities with the AutoCompressor, as both approaches aim to compress contexts into vectors of a fixed shape using pretrained language models. However, it's important to note that our distillation method, while akin to these compression techniques in reducing context lengths, has a distinct objective. Our method is specifically designed to tailor demonstrations to optimally address target questions, as opposed to seeking a universal compression solution. In light of these differences, we believe that a comparison between our DGS and the AutoCompressor sufficiently highlights the unique aspects of our approach compared to existing compression methods, and therefore consider further comparison with ICAE as not so necessary.

Reviewer comment (Experiments):
Can you test your approach using 1-2 open-source models and evaluate how that affects performance?

Author response:
## Weakness 1 (Question 1): Why not test DGS's performance with open-source LLMs?
We fully agree with the reviewer that it is beneficial to test DGS's performance with open-source LLMs so as to gain a more comprehensive picture of DGS's applicability. However, we would like to point out that the black-box property of DGS is also one of the most important considerations as we design its implementation. We specifically design DGS as an API-only procedure that does not require any GPU resource. This makes our method more convenient to use, especially for general practitioners who do not have much experience or abundant resources in LLM deployment. We deem such an API-only and user-friendly feature of DGS as one of its most valuable merits and therefore do not prioritize the testing with various open-source LLMs.

Reviewer comment (Experiments):
Can you include performance results using the demonstration compressions methods that were listed in the related work section?

Author response:
## Weakness 3 (Question 2): Why not compare DGS with demonstration compression methods other than AutoCompressor?
We appreciate the reviewer's suggestion to compare our method with additional compression methods referenced in our related work. However, we respectfully contend that a comparison with the two specific works cited (https://arxiv.org/pdf/2304.08467.pdfandhttps://arxiv.org/pdf/2307.06945.pdf) may not be strictly necessary for our study.
  - Regarding the paper "Learning to Compress Prompts with Gist Tokens," the authors have innovatively developed a technique for condensing short, instruction-like prompts into gist tokens. This approach, while effective for its intended purpose, seems less applicable to our goal of compressing longer demonstrations for in-context learning. Our focus is on distilling extensive contexts into shorter forms, which diverges from the gist token methodology.
  - On the other hand, the paper "In-context Autoencoder for Context Compression in a Large Language Model" introduces the In-context Autoencoder (ICAE), a novel model for context compression. This method shares similarities with the AutoCompressor, as both approaches aim to compress contexts into vectors of a fixed shape using pretrained language models. However, it's important to note that our distillation method, while akin to these compression techniques in reducing context lengths, has a distinct objective. Our method is specifically designed to tailor demonstrations to optimally address target questions, as opposed to seeking a universal compression solution. In light of these differences, we believe that a comparison between our DGS and the AutoCompressor sufficiently highlights the unique aspects of our approach compared to existing compression methods, and therefore consider further comparison with ICAE as not so necessary.

Reviewer comment (Experiments):
Compare your approach to other demonstration selection approaches that specifically look for valid but short demonstrations that can be used in an ICL framework. Look at the following papers: a) https://aclanthology.org/2023.findings-acl.273.pdf; b) https://arxiv.org/abs/2211.04486; c) https://arxiv.org/abs/2310.10707; d) https://arxiv.org/abs/2302.05698; e) https://arxiv.org/abs/2104.08786

Author response:
## Question 3: Why not compare DGS with other demonstration selection approaches that specifically look for valid but short demonstrations?
We appreciate the reviewer's insightful suggestion to compare DGS with other demonstration selection approaches that specifically look for short but effective demonstrations. However, we think it may not be very necessary to conduct such a comparison since demonstration selection techniques are in essence different from demonstration distillation techniques. To be more specific, demonstration selection focuses on the "quality" of a small fraction of demonstrations in the training set from which the LLM can learn most. In contrast, demonstration distillation does not care much about the quality of the demonstrations it receives and can distill and condense whatever groups of input demonstrations, to an extent where distilled demonstrations are shorter than most of the original demonstrations in the whole training set. Even if the demonstrations chosen by special selection techniques are on par with the distilled demonstrations in length, the comparison between them is still not fair enough, since the distilled ones are randomly selected and there may exist an inherent quality gap between their original versions and the selected short but effective ones. Out of such considerations, we did not compare DGS with demonstration selection approaches in our submitted paper.