Paper: Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR-2024)

Reviewer comment (Presentation):
The paper should reduce reliance on the Appendix for key details and improve presentation by sharing important information more up front. For instance, in the experiments, it should be clear both (a) what the candidate datasets are, and (b) what feature space/function is being used to calculate costs in OT. The different pre-processing steps different methods use on the candidate set should also be mentioned.

Author response:
**(i) There were several details that I think are important to share more up front.**

> "For instance, in the experiments, it should be clear both (a) what the candidate datasets are; (b) what feature space/function is being used to calculate costs in OT. I also think the different pre-processing steps different methods do to the candidate set should be mentioned. While having an extended related work in the Appendix is fine, it would nice if more of this content could be covered in the main."

**Re.:** **(i).** Thanks for the insightful feedback. The authors agree with the direction pointed out by the reviewer and are committed to making revisions accordingly. 

Currently, due to the many contents and volume of experiments, we provide a comprehensive presentation for all the relevant details–models, datasets, implementation, processing steps, etc. in Appendix C. We agree that it would provide a better presentation to include some key details right in the main paper. We are working toward that direction. Currently, we have added more details to the main text where the space allows. As mentioned in the previous response, we are currently working on revamping the experiment sections, condensing the current contents and making space for the revisions. It is our priority to include more key information in the main paper as suggested by the reviewer. We agree it should improve the readability of the paper and its overall quality.

Reviewer comment (Writing):
Some aspects of the notation need to be cleaned up (see Questions section for specifics).

Author response:
**(ii) Some aspects of the notation could also be cleaned up (see Questions below)**

**Re.:** We have revamped the main methodology section and remodeled it in the style of lemma/theorem with proof and remark in an effort to improve its clarity and general rigor.

We are working on simplifying the notation system used in this work and making it more intuitive and straightforward. We will make sure the mentioned notation issues are taken care of during this process. We wish this could further improve the readability of the manuscript, avoid potential confusion, and make it more accessible to broader readers.

Reviewer comment (Writing):
The All Domains baseline is mentioned on page 6 before being defined.

Author response:
**(iii) The All Domains baseline is mentioned on pg. 6 before being defined.**

**Re.:** Thanks for the catch. Apologies. We have fixed this and also checked other contents for inconsistencies.

Reviewer comment (Evaluation):
In the experiments in Sections 3.2–3.3, the improvements from GOT-D over previous baselines do not appear to be large in most cases (i.e., within standard deviations). The authors should comment on the significance of these results.

Author response:
**Weaknesses-2, Experiments: (i). Comment on the significance of performance improvements in Section 3.3/3.4**

**Re.:** The authors did realize this presentation issue not long after submitting the manuscript.

8 NLU tasks with a pre-defined domain (DAPT tasks, Section 3.3/Table 2, 3) are first presented in DAPT paper, where they show continued pre-training RoBERTa model for 2\~8B tokens on a target domain (equivalent to **20M\~80M** samples with 100-token length) could improve the performance on the tasks in this domain by **2.3%** on average (the improvement is less than 2% on 5 tasks). DSIR achieves an average of **1.56% improvements after continued pre-training on 25M samples.**

In this work, we show that favorable performance improvements can be established with a **much smaller** fine-tuning budget if the data is selected appropriately. We achieved an average of 1.13% improvement after fine-tuning with only 150k samples **(~0.5% of the data budget for continued pre-training) in Table 2 and a 0.76% improvement with 50k samples in low-resource scenarios in Table 3. It can be seen from the tables that the rate of performance improvements for GOT-D significantly **outpaces** that of baseline methods, highlighting the efficacy and data efficiency of the proposed method.

The results for 8 NLU tasks without a pre-defined domain (GLUE tasks Section 3.4/Table 4) are meant to be interpreted in a slightly different way. GLUE tasks are designed to measure the **general capability** of a language model. These tasks are specifically selected to measure very different aspects of the language model and it is hard to achieve a high score in all aspects at the same time.

We conducted this experiment as a test to see whether light fine-tuning can improve the scores on this kind of benchmark. DSIR achieves a **2%** improvement in GLUE score by curating pre-training data and training a BERT model from scratch for **51.2M** samples. **It is expected that performance improvements can be difficult.** As mentioned in the discussion in the paper, the proposed light pre-fine-tuning is more beneficial for tasks that benefit from certain task-relevant knowledge, rather than the ones relying on the fundamental capability of the model. It is an interesting result for us to see that light pre-fine-tuning with the proposed method achieves more than 1% performance improvement on the GLUE benchmark.

It is our responsibility not being able to provide sufficient discussion after the experiments to elaborate on the **takeaway** for each result.

**Besides, we have added a new set of experiments for pre-fine-tuning GPT models for zero-shot classification tasks, where the performance improvements of our method are 6.6%~13.9% after pre-fine-tuning with 40k samples.**

Reviewer comment (Experiments):
There seem to be several differences between DSIR and GOT-D beyond one doing distribution matching and the other finding the most informative examples given the pre-training distribution. Both involve preprocessing the candidate pool and use different feature representations to perform selection (DistillBERT vs. n-gram). More controlled ablations are needed to isolate what precisely causes GOT-D to perform better.

Author response:
**Weaknesses-2, Experiments: (ii). ...several differences between DSIR and GOT-D... both involve preprocessing the candidate pool in and use different feature representations to perform selection (i.e., DistillBERT v.s. n-gram). ...more controlled ablations of these other factors are important to ablate what precisely causes GOT-D to perform better.**

**Re.:** We agree with the reviewer that there exists several differences in processing steps and it is of interest to ablate on their effects to better understand this problem.

This has also been one of our concerns during the development of this work, and after careful discussions, we had decided not to change the processing pipelines since this is a key part of the main philosophy of each method. The method would have a different name if we alter its processing steps/embedding scheme, etc. For example, one of the main efforts of the DSIR paper is to show that the distributional distance on n-gram space is a valid proxy for downstream performance on target tasks so that they can use this simple feature to enable large-scale selection for pre-training data. Yet, it is also true that it is desirable to ablate the effect of these design choices when comparing it with other methods.

The implementation of DSIR is highly integrated and we are not able to replace its embedding scheme in a straightforward way. We conducted more controlled studies in the additional experiments. In the zero-shot classification task with GPT-2 XL, we ablate on the effect of choices of embedding space for computing OT distance. We tested embedding samples with distilled-BERT, sentence-transformer, and BERT-tokens. Lightweight and fast, the popular sentence-transformer uses a pre-trained all-MiniLM-L6-v2 model with 22M parameters as the backbone. It embeds up to 6 million samples/hours on a single GPU and is sometimes considered a 'default' option for sentence embedding in many NLP tasks. Token space isn't a proper embedding for OT (e.g., the distance on token space is not invariant to paraphrase). We are only listing it here for comparison. As can be seen from Table 17, the performance of sentence-transformer is mostly on par with distilledBERT. It suggests **the choice of embedding space might not be a critical part of our proposed data selection pipeline and any reasonable embedding space should work.**

**Besides, we also provide visualization of samples selected by each method to facilitate insights into the experiment results.** In the additional experiments with the “AG News” task where all the settings are best aligned, we conduct visualization of samples selected by each method and the distribution of target task data and pretraining/candidate data, offering insights on how the proposed method works differently from the distribution-matching approaches. As shown in Figure 16, the comparison shows a clear contrast. **Both DSIR and TAPT/c select samples that match the distribution of the target task data which has a high overlapping with the distribution of the candidate data where the model is already pre-trained on.** Thus, with such a small data budget, the information gain provided from pre-fine-tuning on these samples is naturally marginal. In contrast, **GOT-D selects predominately formal business news which is highly underrepresented in the candidate dataset but important for the target task.** Pre-fine-tuning the model with these samples provides a more direct benefit in aligning the model with the target tasks, **effectively validating the idea of this work.**

Reviewer comment (Experiments):
In the BERT experiments, it is unclear whether DSIR and DAPT/TAPT use the same pre-processing procedure to select relevant domains as described for GOT-D. It would be interesting to see what happens if this step is left out for GOT-D.

Author response:
**Weaknesses-2, Experiments: (iii). In the BERT experiments, do DSIR and DAPT/TAPT use the same pre-processing procedure to select relevant domains as described for GOT-D? ...what happens if this step is left out for GOT-D.**

**Re.:** The authors acknowledge the responsibility for the processing scheme not presented straightforwardly enough.

DSIR is designed for selecting data from all domains, which is one of its advertising advantages and also why it chooses the simpler n-gram feature space. DAPT/TAPT inherently requires a curated domain dataset, so its selection is always from the relevant domain dataset. Similar to the previous response, we are reluctant to change its processing steps. In the DSIR paper, the authors also conduct comparisons with DAPT under the same setting.

These BERT experiments are resource-intensive. A sizeable number of models need to be fine-tuned considering the number of tasks, baselines, data sizes, repetitions, and many ablations for design choices. Instead, we ablate the effect of this processing step in the additional experiments.

In the zero-shot classification task with GPT-2 XL, since the model is pre-trained on a single domain–OpenWebTextCorpus(OWTC),  **we use the same data as the candidate dataset and all baseline methods selected from the same pool of candidate data without additional processing.** The setting is well-aligned and consistent with assumptions in the development of our method. The proposed method achieves a performance gain of 13.9% after pre-fine-tuning with 40k samples, visibly outperforming DSIR and TAPT.

This is also the case for the NLG task with GPT-2 base in Section 3.2.

Reviewer comment (Evaluation):
It would be useful to show some qualitative examples of the kinds of examples GOT-D selects that are not emphasized in distribution matching approaches such as DSIR.

Author response:
**Weaknesses-2, Experiments: (iv). It would be nice to show some qualitative examples of what kinds of examples get selected by GOT-D but are not as emphasized in distribution matching approaches (e.g. DSIR).**

**Re.:** The authors agree that it will be desirable to direct compare the samples selected by each method and random selection from pre-training data. We added this to Appendix E.1.

In the additional experiments with the “AG News” task where all the settings are best aligned, we conduct visualization of samples selected by each method and the distribution of target task data and pretraining/candidate data, offering insights on how the proposed method works differently from the distribution-matching approaches. **As shown in Figure 16, the comparison shows a clear contrast. Both DSIR and TAPT/c select samples that match the distribution of the target task data which has a high overlapping with the distribution of the candidate data where the model is already pre-trained on.** Thus, with such a small data budget, the information gain provided from pre-fine-tuning on these samples is naturally marginal. In contrast, **GOT-D selects predominately formal business news which is highly underrepresented in the candidate dataset but important for the target task.** Pre-fine-tuning the model with these samples provides a more direct benefit in aligning the model with the target tasks, effectively validating the idea of this work.

Reviewer comment (Theory):
In the introduction, the claim *"Intuitively, fine-tuning a pre-trained model with such samples would boost its performance on the target dataset. We prove the validity of this intuition under certain assumptions, thereby setting our method on a solid theoretical foundation."* suggests there would be a formal theorem statement explicitly outlining assumptions and a guarantee. However, Section 2 instead describes and links together existing results in prose, making it unclear what has been newly proven.

Author response:
**Question-1: ...this suggested there would be a formal theorem statement in the text that explicitly outlines assumptions and a guarantee?**

**Re.:** The authors agree that it is important and needed to present the proposed method and its development in a more formal and rigorous manner.

**We have revamped the main methodology section (Section 2.3), replacing the previous derivation-with-elaboration with lemma/theorem with proof and remarks to improve its clarity and general rigor.**

Reviewer comment (Experiments):
In the toxicity reduction experiments, it is unclear whether any methods performed a second round of task-specific fine-tuning. If not, it would still be important to include a baseline that trains only on the initial 2.5K target examples. It would also be helpful to confirm whether GOT-D's improvements over other methods persist after doing a fine-tuning round on these examples.

Author response:
**Question-2: In the toxicity reduction experiments... no methods did the second round of task-specific fine-tuning... it is still important to include a baseline which trains on just the initial 2.5K target examples... to confirm that the improvements of GOT-D over other methods persists after doing the fine-tuning round on these examples**

**Re.:** Thanks for the suggestion. The NLG task in Section 3.2 as well as the additional experiments are  "zero-shot" as its fine-tuning is only conducted in an unsupervised/self-supervised manner. **The pre-fine-tuned model can be directly deployed to downstream tasks without further fine-tuning.** We consider the target data used for data selection as the “validation data”. **The validation data is usually considered identically distributed with the test data and cannot be used for training the model.** This setting is typical in data selection/valuation works.

As a comparison, these validation data are prepared in the same way as RTP–using Perspective API to evaluate the toxicity of OWTC samples and select a clean/toxic batch. Fine-tuning on the 2.5k clean batch might yield a similar result to RTP with 2.5k data, which will not be as good as pre-fine-tuning the model on the data selected by the proposed methods.

Reviewer comment (Writing):
The distinction between "pre fine-tuning" and "continued pre-training" is unclear. The paper suggests the same methods could be applied to both, with the main difference being the amount of data involved, but in some places "pre-training" is used interchangeably. For example, in Section 3.2 (pg. 7), the text states: *"This experiment involves two stages: pre-training over selected data and then fine-tuning over the downstream task."* Clarification of the motivation for introducing the new term would be helpful.

Author response:
**Question-2: As a general comment, the distinction between the terms "pre fine-tuning" and "continued pre-training" remains a bit unclear...What is the motivation for defining the new term as opposed to saying something akin to "continued pre-training with limited budget"?**

**Re.:** Thanks for the suggestion. This is actually a bit of a tricky issue that has been on the minds of the authors throughout. The simple reason is all the alternatives we can think of may potentially cause confusion in other contexts.

Yes, the proposed scheme is essentially continued pre-training with a (much) smaller budget. The tricky part is that continued pre-training is already very well-known to be associated with a scope that is orders of magnitudes more than ours. 

Continued pre-training provides techniques for **pre-trained model builders** to build separate pre-trained models for separate tasks given ample compute and data budgets. It is perceived to be part of the pre-trained model development stage as opposed to customization/fine-tuning, where our work provides tools and addresses considerations for **model users** who would like to leverage an existing pre-trained model and customize the model given a limited budget.

What is conducted in this work is more similar to fine-tuning but in an unsupervised/self-supervised manner. We think this idea of selecting pre-fine-tuning data from raw data (rather than curated labeled datasets) is relatively new, and being able to establish a visible performance margin with such a small budget is a novel result.

We thank the reviewer for the comment. We are still searching for a better term and considering all the options.

Reviewer comment (Writing):
Sometimes $D_S, D_R, etc.$ are used interchangeably to represent (finite) datasets and distributions, and this distinction should be made clearer.

Author response:
**Miscellaneous-1: It seems sometimes $D_S$, $D_R$, etc. are used interchangeably to represent (finite) datasets and distributions. Perhaps the authors can make this distinction clearer?**

**Re.:** We agree with the reviewer's suggestion. This is a slight abuse of notation that is introduced on Page 4 before Eq. (2). The intention is to avoid introducing additional notations for the ease of general readers. But it does come at a cost of mathematical clarity.

We are currently working on simplifying the notation systems to make it more direct and intuitive. We will keep this in mind and use separate notations for empirical distributions and sets of samples in an updated version.

Reviewer comment (Experiments):
The version of TAPT used appears to be the "curated-TAPT" described in Gururangan et al., rather than "TAPT" as typically defined (training on unlabeled end-task inputs). A clarifying note should be included, and ideally, results for both versions should be presented.

Author response:
**Miscellaneous-3: The version of TAPT here seems to be the "curated-TAPT" described in Gururangan et al. rather than "TAPT", which I've more typically seen as just training on the unlabeled end-task inputs.**

**Re.:** Thanks for the catch. The authors did overlook distinguishing the variations of this approach. Yes, what is being referred to in this manuscript should be addressed as "curated-TAPT". This is important and thanks for bringing it up. **We have changed all the references of TAPT to curated-TAPT (TAPT with a curated dataset, TAPT/c).**

Reviewer comment (Experiments):
The proposed approach shows only marginal improvement on a number of domain adaptation and GLUE benchmarks - less than 1% over prior approaches, and in some cases only 1% higher than baseline performance. Another concern with the evaluation is that it is only evaluated on small models (128M) that may be trained with more limited data. In contrast, it would be beneficial to evaluate whether using the pre-fine-tune set also leads to better performance on a larger model (1/7B+).

Author response:
**Weakness-1a: Performance improvements in Section 3.3/3.4 seem marginal.**

**Re.:** The authors did realize this presentation issue not long after submitting the manuscript.

8 NLU tasks with a pre-defined domain (DAPT tasks, Section 3.3/Table 2, 3) are first presented in DAPT paper, where they show continued pre-training RoBERTa model for 2\~8B tokens on a target domain (equivalent to **20M\~80M** samples with 100-token length) could improve the performance on the tasks in this domain by **2.3%** on average (the improvement is less than 2% on 5 tasks). DSIR achieves an average of **1.56% improvements after continued pre-training on 25M samples.**

In this work, we show that favorable performance improvements can be established with a **much smaller** fine-tuning budget if the data is selected appropriately. We achieved an average of 1.13% improvement after fine-tuning with only 150k samples **(~0.5% of the data budget for continued pre-training) in Table 2 and a 0.76% improvement with 50k samples in low-resource scenarios in Table 3. It can be seen from the tables that the rate of performance improvements for GOT-D significantly **outpaces** that of baseline methods, highlighting the efficacy and data efficiency of the proposed method.

The results for 8 NLU tasks without a pre-defined domain (GLUE tasks Section 3.4/Table 4) are meant to be interpreted in a slightly different way. GLUE tasks are designed to measure the **general capability** of a language model. These tasks are specifically selected to measure very different aspects of the language model and it is hard to achieve a high score in all aspects at the same time.

We conducted this experiment as a test to see whether light fine-tuning can improve the scores on this kind of benchmark. DSIR achieves a **2%** improvement in GLUE score by curating pre-training data and training a BERT model from scratch for **51.2M** samples. **It is expected that performance improvements can be difficult.** As mentioned in the discussion in the paper, the proposed light pre-fine-tuning is more beneficial for tasks that benefit from certain task-relevant knowledge, rather than the ones relying on the fundamental capability of the model. It is an interesting result for us to see that light pre-fine-tuning with the proposed method achieves more than 1% performance improvement on the GLUE benchmark.

It is our responsibility not being able to provide sufficient discussion after the experiments to elaborate on the **takeaway** for each result.

**Besides, we have added a new set of experiments for pre-fine-tuning GPT models for zero-shot classification tasks, where the performance improvements of our method are 6.6%~13.9% after pre-fine-tuning with 40k samples.**

Reviewer comment (Experiments):
Authors only conduct limited evaluation on the pre-fine-tuning dataset size at 50K and 150K. An experiment at different amounts of data from small data to continual pre-training with comparison to prior works can help to better motivate GOT-D as an approach for selecting pre-fine-tuning data.

Author response:
**Weakness-2: An experiment at different amounts of data from small data to continual pre-training with comparison to prior works can help to better motivate GOT-D as an approach for selecting pre-fine tuning data.**

**Re.:** The authors agree that the experiment mentioned by the reviewer would be desirable. Yet these LLM experiments take significant resources and can be considerably costly. For example, fine-tuning OpenAI's davinci-002 on a set of **100K short samples with a max length of 128 tokens using recommended settings with OpenAI's API costs $1,500.** A sizeable number of models need to be fine-tuned considering the number of tasks, baselines, data sizes, repetitions, and many ablations for design choices. Continued pre-training experiments are very difficult to conduct with an academic budget.

In light of the direction pointed out by the reviewer, we conducted **ablation studies on pre-fine-tuning data sizes** with our available resources. The results are presented in Table 16 at the end of Appendix. Performance improvements appear to comply with a **log-linear scaling law relationship** with pre-fine-tuning data size with the performance recording constant improvement every time the fine-tuning data budget is doubled.

Reviewer comment (Experiments):
Where does the data for pre-fine-tuning come from? In the main text, the authors suggest that it can come from the Pile, however GPT-2 is trained on WebText, and not necessarily subsets of the Pile. In this case, the experiments are adding more pre-training data where even in small amounts are expected to increase performance.

Author response:
**Question-1: Where does the data for pre-fine-tuning come from? ...GPT-2 is trained on WebText, and not necessarily subsets of the Pile.**

**Re.:** Thanks for providing the feedback. The authors acknowledge the responsibility for causing the potential misperception.

Due to the page limit of the main paper, the details of models/candidate datasets for each experiment are summarized in Appendix C.1/C.2. For GPT-2 base/XL that are pre-trained on a single domain–OpenWebTextCorpus(OWTC), **we use the same data as the candidate dataset for the NLG task in Section 3.2 as well as the additional experiments.** All baseline methods select from the same pool of candidate data. The setting is well-aligned and consistent with assumptions in the development of our method.

Thus, the selected data for pre-fine-tuning does not add anything new to the scope of the model's pre-training domain. The **performance gain** achieved on the downstream tasks can **reasonably support** the idea that "pre-fine-tuning with properly selected data better prepares the model for the downstream task."

Our replica of The Pile dataset consisting of data from 7 domains is used for NLU tasks in Section 3.3/3.4. with BERT as well as the additional experiments with GPT-neo, which is pre-trained on The Pile dataset.

Reviewer comment (Experiments):
What type of data is selected from the pre-training set? Is there a comparison between selected data vs. fine-tune data and random pre-train data?

Author response:
**Question-2: What type of data is selected from the pre-training set? Is there a comparison selected data vs. fine-tune data and random pre-train data?**

**Re.:** The authors agree that it will be desirable to direct compare the samples selected by each method and random selection from pre-training data. 

In the additional experiments with the “AG News” task where all the settings are best aligned, we **conduct visualization** of samples selected by each method and the distribution of target task data and pretraining/candidate data, offering insights on how the proposed method works differently from the distribution-matching approaches. As shown in Figure 16, the comparison shows a clear contrast. **Both DSIR and TAPT/c select samples that match the distribution of the target task data which has a high overlapping with the distribution of the candidate data** where the model is already pre-trained on. Thus, with such a small data budget, the information gain provided from pre-fine-tuning on these samples is naturally marginal. In contrast, GOT-D selects predominately formal business news which is **highly underrepresented in the candidate dataset but important for the target task**. Pre-fine-tuning the model with these samples provides **a more direct benefit** in aligning the model with the target tasks, effectively validating the idea of this work.

Reviewer comment (Experiments):
The baseline methods are limited. Although the related work discusses a batch of work for language data selection, only DSIR and DAPT (RTP on toxicity, TAPT on GLUE) are compared in the experiments. It will be more convincing to include more baseline methods or illustrate why some are excluded.

Author response:
**Re.:** **We picked the most representative baselines. Most methods are not applicable to the scale of this problem.** Some others are either overly heuristic and ad-hoc or have an unaffordable computing overhead.

Thanks for bringing up this interesting point! Most data selection methods discussed in the related work do not apply to the problem scale of this work. For example, data valuation/coreset methods often only scale to a few thousand samples. Some recent methods could potentially scale better at a cost of extreme computing overhead (e.g., [GIO] requires a cluster of servers for clustering) while the idea is essentially similar (matching distributions by explicitly minimizing KL-divergence.)

We included the most representative baselines relevant to the task, covering the most important methodologies for this problem.

- Distributional matching (DSIR)
- Nearest neighbors (curated-TAPT (TAPT with a curated domain dataset, TAPT/c))
- Curated domain dataset (DAPT)
- External knowledge/model (RTP)

To our knowledge, **this should cover existing principled approaches** toward this data selection problem.

As far as the authors are aware, there also exists a number of heuristic methods for curating data for LLMs such as filtering data for quality (with some simple criteria), diversity, length, etc. These methods are inherently ad-hoc and overly empirical where the effectiveness highly depends on many design choices/threshold settings. Since these are not principled approaches and they mostly underperform in general cases (as evaluated in [DSIR]), we left out these methods in this paper.

We appreciate this feedback from the reviewer and this discussion is helpful for revising our manuscript for a better presentation.


> [GIO] Dante Everaert and Christopher Potts. Gio: Gradient information optimization for training dataset
selection. arXiv preprint arXiv:2306.11670, 2023.

Reviewer comment (Evaluation):
The performance improvements, particularly in Tables 3 and 4, seem marginal, especially when taking the standard deviations into account.

Author response:
**Re.:** The authors did realize this presentation issue not long after submitting the manuscript.

8 NLU tasks with a pre-defined domain (DAPT tasks, Section 3.3/Table 2, 3) are first presented in DAPT paper, where they show continued pre-training RoBERTa model for 2\~8B tokens on a target domain (equivalent to **20M\~80M** samples with 100-token length) could improve the performance on the tasks in this domain by **2.3%** on average (the improvement is less than 2% on 5 tasks). DSIR achieves an average of **1.56% improvements after continued pre-training on 25M samples.**

In this work, we show that favorable performance improvements can be established with a **much smaller** fine-tuning budget if the data is selected appropriately. We achieved an average of 1.13% improvement after fine-tuning with only 150k samples **(~0.5% of the data budget for continued pre-training) in Table 2 and a 0.76% improvement with 50k samples in low-resource scenarios in Table 3. It can be seen from the tables that the rate of performance improvements for GOT-D significantly **outpaces** that of baseline methods, highlighting the efficacy and data efficiency of the proposed method.

The results for 8 NLU tasks without a pre-defined domain (GLUE tasks Section 3.4/Table 4) are meant to be interpreted in a slightly different way. GLUE tasks are designed to measure the **general capability** of a language model. These tasks are specifically selected to measure very different aspects of the language model and it is hard to achieve a high score in all aspects at the same time.

We conducted this experiment as a test to see whether light fine-tuning can improve the scores on this kind of benchmark. DSIR achieves a **2%** improvement in GLUE score by curating pre-training data and training a BERT model from scratch for **51.2M** samples. **It is expected that performance improvements can be difficult.** As mentioned in the discussion in the paper, the proposed light pre-fine-tuning is more beneficial for tasks that benefit from certain task-relevant knowledge, rather than the ones relying on the fundamental capability of the model. It is an interesting result for us to see that light pre-fine-tuning with the proposed method achieves more than 1% performance improvement on the GLUE benchmark.

It is our responsibility not being able to provide sufficient discussion after the experiments to elaborate on the **takeaway** for each result.

**Besides, we have added a new set of experiments for pre-fine-tuning GPT models for zero-shot classification tasks, where the performance improvements of our method are 6.6%~13.9% after pre-fine-tuning with 40k samples.**

Reviewer comment (Reproducibility):
To compute the OT distance, what’s the representation of the involved data points?

Author response:
**Re.:** In the original manuscript, OT is computed on embeddings from the **distilled-BERT** model.

In the additional experiments, we ablate on the effect of choices of embedding space for computing OT distance. We tested embedding samples with **distilled-BERT, sentence-transformer, and BERT-tokens**. Lightweight and fast, the popular sentence-transformer uses a pre-trained all-MiniLM-L6-v2 model with 22M parameters as the backbone. It embeds up to 6 million samples/hours on a single GPU and is sometimes considered a 'default' option for sentence embedding in many NLP tasks. Token space isn't a proper embedding for OT (e.g., the distance on token space is not invariant to paraphrase). We are only listing it here for comparison.

As can be seen from Table 16, the performance of sentence-transformer is mostly on par with distilled-BERT. **It suggests the choice of embedding space isn't a critical part of the data selection pipeline and any reasonable embedding space should work.**

Reviewer comment (Experiments):
Because the pre-training dataset is undisclosed, the authors resort to a proxy dataset that has a composition proximate. Is it possible to implement a synthetic task where the pre-training dataset is known to verify the proposed method?

Author response:
**Re.:**  GPT-2 is a perfect example as its training data is from a single domain–OpenWebTextCorpus(OWTC). In the NLG task in Section 3.2 and the additional experiments, **we use the same data as the candidate dataset** and all data selection methods (GOT-D, DSIR, TAPT/c) select from the same candidate dataset. This should provide a fair comparison. In both of these experiments, our methods demonstrate promising results.