Paper: Mo' Data Mo' Problems: How Data Composition Compromises Scaling Properties (ICLR-2024)

Reviewer comment (Novelty):
I see a notable weakness of this work in the novelty and the contribution. The paper seems to state claims that are more-or-less known in research community. So, even if there are no publications on the topic, e.g., the fact of having shift of distributions (and thus, lower performance) in the case you sample not from the target distribution is common knowledge and directly follows from ML grounds. So, such knowledges are also referred to as folklore. So, in my opinion, the paper contribution is narrowing down to experimental analysis, which is good but looks like an observational study without clear new insights (besides the ones expected by folklore knowledge). It is not enough for this venue. I would assume that such an analysis is a good illustration / help for students that study ML / statistics and might be published through some books on the topic.

Author response:
We thank the reviewer for raising their concerns; however, our contributions may have been fundamentally misunderstood by the reviewer. 

**Main Contribution**
> So, even if there are no publications on the topic, e.g., the fact of having shift of distributions (and thus, lower performance) in the case you sample not from the target distribution is common knowledge and directly follows from ML grounds. 

The reviewer is correct that the fact that “distribution shift between train and test datasets results in reduced performance” has been studied extensively in past work (See our references Sagawa et. al. in the main text and Appendix C.5 detailed related work on distribution shift). 

However, what is **not** known is (1) why distribution shift may arise as dataset size increases and (2) do the harms induced by distribution shift from scaling eclipse the benefits of scaling. This is an incredibly common scenario in the real world. For example, government resource allocation for social programs depends on projections from census data [1]. In smaller counties and states, the decision has to be made on whether adding more data is beneficial or harmful to achieve more accurate and equitable predictions. Moreover, formalizing **why** adding data does not yield better outcomes is important and it is a central contribution to our work. 
 
We hope our explanations have clarified the novelty of our contributions. As far as we are aware, we agree with the reviewer that there is no prior work that models the mechanisms that induce distribution shift when dataset size is scaled up. Moreover, our theoretical contributions are validated by our empirical results in different settings, and our practical metric bridges our theoretical model with empirically observed divergences. 
We are happy to continue answering any questions to help the reviewer better understand the contributions of our paper.

Reviewer comment (Experiments):
Another related weakness is the problem setup. The authors try to explore some effects when sampling not from the underlying distribution D in different ways knowing that they sample with shifts. What is the problem to be solved? If the practitioner face a situation, knowing D, then they will sample from it. If they have no access to D, then they try to sample some distribution such that it is most close to D. If they face two sources and expect that they have representation of D (they believe that their union is close to D), then they will sample randomly from the union. And etc. In any way, the practitioner knowing basics of ML, will attempt to be close to the best knowledge of D for them. It is hard for me to imagine situations described in the work, where a practitioner is aware of ML grounds and is increasing samples without carrying about the general population. I assume, it might be the case when this practitioner is working with ML tools without knowing ML grounds (so, in this case, a book on ML grounds might help). Overall, the work lack of clear problem statement: what is known for a practitioner, what is not, which decisions they can take, what are limitations.

Author response:
> What is the problem to be solved? If the practitioner face a situation, knowing D, then they will sample from it. If they have no access to D, then they try to sample some distribution such that it is most close to D.

Without access to D, how is it possible to know what distribution is closest to D? Our central question (Figure 1) is: “**How** does data composition impact model performance under scaling?”. We examine the situation where there is limited access to D (e.g. a very small sample). A problem arises when we have to decide whether adding data from some other distribution $D_{s_2}$, is beneficial or harmful with respect to overall accuracy, worst group accuracy, and group accuracy disparity.

Reviewer comment (Writing):
Sec.3.1 serves for me as support of absence of clear problem statement in the work. The authors write “While in reality the test set cannot be accessed, we assume we can use some part of the training set (e.g., the D_s_1 ) that is similar to the test distribution).” Seems that it means that the practitioner believes that D_s_1 IS the distribution D. So, when receiving D_s_2 they should take D_s_1 united with D_s_2 as D, or not considering D_s_2 at all.

Author response:
> Seems that it means that the practitioner believes that D_s_1 IS the distribution D. So, when receiving D_s_2 they should take D_s_1 united with D_s_2 as D, or not considering D_s_2 at all.

Suppose we are policymakers in South Dakota, $D_{s_1}$ is the dataset from our state, but $D_{s_2}$ is a very different state like California. If $D_{s_2}$ is not considered at all, $D_{s_1}$ might be too small and uniform to give adequate accuracy/fairness (e,g, only 2.6% of the population is Black [2] ). If we sample from the union of $D_{s_2}$ and $D_{s_1}$, the result would depend on the relationship between $D_{s_2}$ and $D_{s_1}$ (this is captured in our Mixture model Sec. 3). 

Crucially, we know that California is a different state but we do not know **how** it is different. In fact, the Mixture model (see Section 3), would sample from the union of South Dakota and California and we see that it yields worse results than just $D_{s_1}$ in the sequential model with a smaller n (Figure 3a vs Figure 4a). We also know that Louisiana could be as different from South Dakota as California is. However, adding data from Louisiana improves outcomes (Figure 9 vs Figure 3a). We identify Louisiana as a useful state to add using the heuristic we propose (Equation 1).

Reviewer comment (Writing):
In Section 4: “data composition on model outcomes as measured on a test set sampled exclusively from the initial dataset Source A (e.g., SD) – which we call the reference test set.” — I do not understand: at the beginning of the work it was stated that “test set” is the union. Here, it is stated that it is just D_d_1.

Author response:
> I do not understand: at the beginning of the work it was stated that “test set” is the union. Here, it is stated that it is just D_d_1.

In section 3, we state that in the data accumulation setting, the **training set** is a union of sources. Could the reviewer tell us specifically where we said the test set is the union?

Reviewer comment (Experiments):
**Simple Empirical Findings and Lack of Generalizability**: The experiments conducted as well as the results obtained are quite simple. Both the Sequential and Mixture model settings are quite trivial, and the obtained results are unsurprising to me. For instance, it seems intuitive that multi-source dataset scaling could lead to worse outcomes on a reference test set, and that it might lead to better generalization (Section 5). The only results that look at data scaling more deeply are located within Section 5.1, but still by themselves those cannot motivate this work. Furthermore, the results obtained (as well as the approach undertaken) are highly dataset dependent (for e.g., what if I sample the reference test set differently for Folktables? Or use a different state altogether?) This issue is also showcased via the accuracy results for both the Yelp and MIMIC-IV datasets under the Sequential paradigm (refer to Figure 6a and Figure 7a, respectively). These figures (for obvious reasons) show very different trends across both datasets for the same model.

Author response:
**Generalizability**
> Furthermore, the results obtained (as well as the approach undertaken) are highly dataset dependent (for e.g., what if I sample the reference test set differently for Folktables? Or use a different state altogether?) 

We agree that using a different state altogether might result in different results. However, the contribution of our work is not to claim that “data scaling benefits are **always** eclipsed by distribution shift” (this conclusion cannot be true if $D_{s_2}$ is sufficiently close to $D_{s_1}$). What we are trying to answer is “**How** does data composition impact model performance under scaling” (Figure 1 of main text). Our contribution is to give a principled theoretical model for data accumulation that is flexible to describe effects regardless of the dataset. Our empirical results show a **there exists** claim where we want to urge caution when adding data to illustrate that there are splits of datasets that lead to undesirable effects.

Reviewer comment (Experiments):
**Inconsistent Evaluation Across Datasets**: The experiments are mostly conducted on the Folktables dataset, with a few results for Yelp and MIMIC-IV. For consistency in evaluation, all the datasets should be used and conclusions can be drawn from the results more adequately. For instance, all the results for Section 5.1 (such as those on the generalized test set) consider only the Folktables dataset, and Yelp and MIMIC-IV are not considered. Furthermore, it is not always mentioned in the text when the other datasets are being used and what the motivation is to discard others, which can be confusing for readers.

Author response:
**Evaluation Across All Datasets**:
We include the sequential case for all datasets. For other experiments, we focus on the Folktables dataset to ensure our message is conveyed as clearly as possible since there are so many steps (e.g. Mix/Seq KL -> test train KL vs Acc -> excess KL vs acc drop -> data selection vs Acc).  We have additional results for the mixture setting that we can include in the supplementary materials. For Section 5 experiments, if there is a Figure or set of Figures that would motivate the reviewer to improve their rating of our work, we are happy to add them. 

**Conclusion**
We hope our response has clarified and alleviated the reviewer's concerns. If there are further concerns that can be addressed to help improve our work please do not hesitate to inform us. 


**References** 
[1] Fabris, A., Messina, S., Silvello, G., & Susto, G. A. (2022). Tackling documentation debt: a survey on algorithmic fairness datasets. In Equity and Access in Algorithms, Mechanisms, and Optimization (pp. 1-13). 

[2] Kadra, A., Lindauer, M., Hutter, F., & Grabocka, J. (2021). Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34, 23928-23941. 

[3] Shwartz-Ziv, R., & Armon, A. (2022). Tabular data: Deep learning is not all you need. Information Fusion, 81, 84-90.

Reviewer comment (Experiments):
**Limited Scope and Applicability**: The biggest drawback of the work is the mismatch in whether this work tackles a useful practical problem and its actual motivation. The original outlined motivations in the abstract and introduction imply that the paper will aim to provide more insights on data scaling in useful practical scenarios. However, the work only considers tabular data and very simple models (the most complex is the MLP). The focus on tabular datasets significantly narrows the paper's scope, especially considering that data scaling is a critical concern in large language models (LLMs) and other domains outside of tabular data (such as NLP and Vision). The paper fails to provide insights or implications for these broader and arguably more impactful areas (such as deep learning), limiting its relevance, scope and applicability. In its current form, I do not think the paper provides insights that are useful for a real-world practical application scenario.

Author response:
**LLMs and scope of work**: 
> The focus on tabular datasets significantly narrows the paper's scope, especially considering that data scaling is a critical concern in large language models (LLMs) and other domains outside of tabular data (such as NLP and Vision).

Indeed, we agree with the reviewer that running similar experiments on LLMs, and maybe also VLMs would increase the scope of our paper. However, due to resource constraints, pretraining LLMs from scratch with different data sources is currently outside of our capabilities and our scope. However, we emphasize that the folklore of more data being better is ubiquitous in machine learning even outside of the LLM setting. Furthermore, the question of how many EHR, loan approval, and recommendations datasets should be combined is important for practical decision-making. Our paper gives guidance in this common setting in terms of how data composition can be related to scaling matters.

Reviewer comment (Reproducibility):
It is not clear how long the model is re-trained with the new examples. This can help us better understand if the examples can’t be learned or if the model just did not have as many iterations to incorporate these new examples.

Author response:
We train from scratch at each dataset size presented in our plots. Thus, each training trajectory shows 5 models over 50 seeds (not 5 - typo in the main text) at for each size n.

Reviewer comment (Evaluation):
The implications of this work are not clear. In real-world settings, if there exists a dataset similar to one that we currently have but has high divergence, does it mean it should not be included in the analysis? Doesn’t not doing so restrict the model from better generalising? For example, Yelp reviews in MN vs SD.

Author response:
In the real world, if we knew a dataset is vastly different (e.g. loan applications for credit cards vs loan applications for mortgages), it does make sense to not include it. However, the question that arises is, what level of divergence would harm my model, and what level is acceptable? In all of our datasets, we are looking at the differences that might arise **within** a dataset. We show that adding data from states (near and far) with reasonable divergences (e.g. there is SOME shift) improves outcomes. Especially when the starting dataset is small (and undiverse) excluding data in the same dataset might not be the best choice. This is a very nuanced point and we will better explain it in our paper.

Reviewer comment (Evaluation):
It looks like adding more data reduces performance disparity between groups and in general helps the least performing group. Reducing disparity is perhaps indicating that the model is generalising better and getting more robust, and these should be considered good things.

Author response:
Reducing performance disparity by adding more data is indeed a desirable goal. We hope to show that by measuring not just overall accuracy but also disparity and worst group performance, understanding the consequences of data scaling is a multi-faceted pursuit and KL divergence as a lone metric is only just a first heuristic. We hope our work will inspire future work to develop further metrics for determining when to add data for properties like disparity and worst group robustness in particular.

Reviewer comment (Experiments):
How is the reference test set constructed? If it's in the appendix, it should be included in the main paper because it is consequential.

Author response:
The reference test set is constructed in our experiments from a very small initial training source $D_{S_1}$ (e.g. South Dakota). We make this assumption to simulate a setting where there is some limited access to the initial dataset of interest but it is then unclear if adding more data would help with outcomes on this initial dataset. We assume that the eventual deployment set might be different from the first source but is likely most similar to the first source.

Reviewer comment (Experiments):
How long do you retrain after adding new datasets?

Author response:
We train from scratch at each dataset size presented in our plots. Thus, each training trajectory shows 5 models over 50 seeds (not 5 - typo in the main text) at for each size n.

Reviewer comment (Experiments):
the authors did not check for data-based techniques used in active learning settings that can help identify data relevant to machine learning models

Author response:
We thank the reviewer for pointing out related work in active learning. We agree that it is important and we will include this in our work. We do highlight that our KL metric is distributional rather than for specific instances as described in equation 15 of Fu et. al. While Zhang et. al. takes a, notably different, model change approach to stop data addition, we believe it is still very wonderful prior work we will include to frame our contribution.

Reviewer comment (Novelty):
*Criteria for rejecting more data*: The problem posed by the authors resembles active learning and some specific data-based strategies. Furthermore, some research has been performed on active learning and stopping criteria. The authors may be interested in researching these areas. Here, we list two works they may find useful: (a) Fu, Yifan, Xingquan Zhu, and Bin Li. "A survey on instance selection for active learning." Knowledge and information systems 35 (2013): 249-283, and (b) Zhang, Yexun, et al. "Stopping criterion for active learning with model stability." ACM Transactions on Intelligent Systems and Technology (TIST) 9.2 (2017): 1-26.

Author response:
We thank the reviewer for pointing out related work in active learning. We agree that it is important and we will include this in our work. We do highlight that our KL metric is distributional rather than for specific instances as described in equation 15 of Fu et. al. While Zhang et. al. takes a, notably different, model change approach to stop data addition, we believe it is still very wonderful prior work we will include to frame our contribution.

Reviewer comment (Experiments):
*Experimental setup*: we consider the experiments valuable and valid. Nevertheless, the authors should consider enriching them with some scenarios where domain adaptation is used to mitigate distribution differences. The authors may be interested in the following work: Farahani, Abolfazl, et al. "A brief review of domain adaptation." Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020 (2021): 877-894.

Author response:
In our experimental setup, we specifically wanted to focus on data interventions rather than model interventions suggested by domain adaptation strategies. In our supplementary materials C.5 we discuss in detail our work in relation to domain adaptation in particular.

Reviewer comment (Evaluation):
*Results and analysis*: Do the authors venture some hypothesis as to why the XGB model is robust to data from different distributions, suffering a lower accuracy loss?

Author response:
Excellent question. While in our results we show that XGBoost does not suffer from an **accuracy drop**, the model still performs much worse than it would if more data from a more similar distribution were to be added. In Figures 11b and 11c (supplementary materials), we show that XGB continues to improve substantially when other sources are added like Louisiana.