Paper: A universal metric of dataset similarity for multi-source learning (ICLR-2024)

Reviewer comment (Theory):
The paper first constructs a metric between a pair of data samples using cosine similarity and Hellinger distance. However, the Hellinger distance is not well motivated. What's the key advantage of the Hellinger distance over the Wasserstein distance?

Author response:
We thank the reviewer for this observation. We recognize the need for a clearer motivation behind our choice of the Hellinger distance. Our primary reason for using the Hellinger distance is to leverage a bounded cost function that is admissible with the optimal transport framework. Specifically, the Wasserstein distance is a special case of the (f, Γ)-divergence family where the Γ cost function is bounded 1-Lipschitz functions (e.g., L1 or L2 norm) (*Birrell et al., JMLR, 2022*). However, the (f, Γ)-divergence family framework inherently allows for flexibility in choosing alternate  Γ functions provided it possesses certain properties. Given we use the Hellinger Distance to generate the cost map, and the Hellinger Distance uses a bounded, continuous function, it can be substituted. The advantage of this modification is that it shifts interpretation from assessing distribution differences in Euclidean geometry to comparing probability densities within the manifold. This is because the Hellinger distance can be interpreted as representing the shortest geodesic distance in the statistical manifold. This approach provides a meaningful way to compare label distributions, preserving the optimal transport interpretation, while ensuring boundedness and relevance in probability space. We now include this more detailed rationale for Hellinger distance in *Section 6.2*.

Reviewer comment (Theory):
The theoretical insights (Section 6) need improvements. The authors consider a scenario where "any pair of vectors drawn from two random and independent datasets". Such a scenario may not be representative enough because two different datasets may have overlaps. Also, the Hellinger distance is not discussed in the theoretical analysis.

Author response:
We thank the reviewer for this feedback. Our discussion focuses on scenarios involving pairs of vectors drawn from two random and independent datasets as this assumption allows us to produce our orthogonality bound. However, we state that this bound of orthogonality does not hold when datasets have overlap because we can no longer assume independence of the sample means and covariances (*Section S.3.1.4*). This bound and when it is violated motivates our primary reason for using the cosine similarity, i.e., that as datasets become increasingly less similar, more vectors are found to be orthogonal. Our empirical findings align with this argument. We have updated *Section 6.1* to make this assumption and when it is violated clearer.

Reviewer comment (Theory):
The first weakness is that this work claims privacy-preserving computation as a main contribution. However, this contribution is not clear, lacks any significant treatment of the techniques used in the main-text, and is also missing important analysis. On clearness, this work claims to enable "privacy-preserving" computation many times early in the paper (abstract, fourth main contribution, to name two). This is vague. Does this mean DP, SMPC, or something else? This only becomes clear on the fourth page of the paper when the work first mentions SMPC. Importantly, though, this SMPC contribution is rather limited. On treatment and analysis, it does not include computation analysis, a security proof, and appears to only use SMPC for the features (and not labels) as observed in Algorithm 1. This conflates the true security guarantees with what is provided as the work claims to provide a "privacy-preserving method for the metric". Further, this work also claims to provide a method that introduces no error, but the linked proof is actually a background (supplement S.1).

Author response:
Thank you for your valuable feedback. We realize that our initial presentation of the privacy-preserving methods was not as clear as it could have been. Our primary contribution lies in developing a cost metric that can be calculated in a privacy-preserving way with little computational overhead (see *Section S.5.4.3*). This is desirable for many practitioners in distributed settings where privacy constraints limit multi-source learning. We feel this is an important contribution as, to our knowledge, other dataset similarity measures do not support privacy-preserving calculation. In particular,  differential privacy can lead to considerable performance degradation when applied to complex models (*Bagdasaryan et al., NeurIPS, 2019*). In response to your feedback, we have revised the manuscript to more accurately reflect the scope of our contribution in this area. We also provide clearer guidance on where to find a comprehensive discussion of the methods.

We recognize that the treatment of SMPC in our manuscript does not include a security proof. Since our approach leverages existing methods, we omitted full handling of security guarantees as these have been shown elsewhere. We now make it clearer in the manuscript where such security guarantees can be found. 

Regarding computation analysis, we have provided this in *Section S.4.4.3* where we compare the computational cost for the dot product and comparison against the computational cost of Euclidean distance. This analysis demonstrates the feasibility and efficiency of our approach in practical scenarios. We have revised the manuscript accordingly.

We thank the reviewer for this comment. We initially provided a method for the features as this required sharing of sample-level information whereas label costs are calculated on summary statistics. However, we acknowledge that in the low sample size setting, summary statistics can also leak information. In response to your feedback, we have revised our approach to ensure that both features and labels are treated with appropriate privacy-preserving methods. For labels, we use a zero-Concentrated Differential Privacy (ρ-zCDP, an intermediate method between pure and approximate DP) method that enables release of the summary statistics (*Biswas et al., 2020, NeurIPS*) . This method was chosen as it is computationally light and has sample complexity similar to the non-privacy-preserving methods. We show that under a strong privacy guarantee, ρ=0.1 or ε-δ= 0.99,0.01, (*Near and Abuah, Programming Differential Privacy, 2021*) we  retain high accuracy in the label cost calculation as we are able to provide the true sample measures a priori (see *Figure S.5.2*). We have amended *Algorithm 1* and added *Section S.2.3* describing how we apply this method in more detail.

We thank the reviewer for catching this typo. The manuscript should point to *Section S.2.2* for a theoretical proof and *S.5.1* for the empirical comparison of the plaintext and ciphertext values showing perfect agreement.

Reviewer comment (Theory):
I'm a bit confused about the relation between data similarity and model utility. Intuitively I think the model utility should be improved the most when we add a dataset that is either too similar (e.g. apparently adding the same dataset would not help at all) or too different (e.g. if we invert all labels, the model might become garbage). But the paper seems to suggest that datasets should be as similar as possible, e.g. in the empirical evaluation Fig 1, and theoretical insights ("When the cosine similarity between x1 and x2 is close to zero, it implies a negligible change in w, leading to minimal improvement in loss"). This is a bit counter-intuitive to me.

Author response:
We thank the reviewer for their question. When we refer to datasets as 'similar,' we imply that they have similar distribution.  In our results we observe that more similar datasets improve performance and dissimilar datasets worsen performance. This finding aligns with the widely accepted notion that, under the scenario that test-sets are drawn from the same distributions, combining IID datasets improves model performance, but adding non-IID datasets may worsen performance (*Kairouz et al.,arXiv, 2019,  Lin et al., ICML, 2021*). 

We also explored scenarios involving datasets with inverted labels or with deliberately contrasting feature distributions per your point. We found that the cost is higher than what we would expect from two random datasets i.e.,> 0.5 and that model performance is no better than chance. We believe this is in keeping with your intuition *‘if we invert all labels, the model might become garbage’*. We have added this to the manuscript *Figure S.5*) 

***and theoretical insights When the cosine similarity between x1 and x2 is close to zero...***

In the theoretical section we are referring to the cosine of the angle between the vectors which ranges from [-1,1], *i.e.,* a score of 0 implies orthogonality. We then use (1-cosine of the angle) in our similarity calculation to ensure the cost is non-negative. We have amended the text to make this clearer.

Reviewer comment (Writing):
Some important concepts and settings need to be explained in more detail. For example, what is μ and Σ in the algorithm? I guess they are some Gaussian parameters but I don't understand what distribution we're talking about here. (And I presume the "u" in (5) is meant to be "μ"?) In general the intuition of the algorithm is not quite clear to me. I think it might be helpful if the authors can demonstrate a few similarity values for some simple cases, e.g. when D1 and D2 are the same, when one is a subset of the other, or when they're fresh samples drawn from the same distribution etc.

Author response:
We thank the reviewer for this observation. In the algorithm, μ and Σ represent the label mean and covariance matrix. As stated in Section 5. Proposed framework, we model labels as distributions over their features. We have amended the algorithm to clarify what μ and Σ represent. We have also amended u to μ in *eq.5*.

***demonstrate a few similarity values for some simple cases...***

We thank the reviewer for this question. We use the synthetic dataset to demonstrate various scenarios. As detailed in *Section S.4.1.1* and *Table S.1*, for the synthetic dataset, our approach involves drawing samples from two distinct distributions, varying how much of the two distributions the datasets share. This setup allows us to systematically explore and quantify the similarity under different conditions:

1. **Datasets drawn from the same distribution:** Both datasets are drawn from the same distribution. We obtain a score of 0.03 (i.e,. highly similar datasets) and find a large improvement in model performance from multi-source learning (MSL).

2. **Datasets drawn from different distributions**: Each dataset is  drawn from a distinct distribution. We obtain a score 0.5 (i.e., datasets are random to each other) and find that  performance is worse in MSL.

3. **Identical Datasets**: The datasets are totally identical. We obtain a score of 0  i.e., datasets can be mapped at no cost).

4. **Dataset is a subset of another**: We subsample (50%) of one dataset to produce the other dataset. We obtain a similarity score of 0.01 i.e., highly similar datasets.

Note points 3 and 4 are new analysis based on your suggestion and have been added to the manuscript in S4.1.1 and Table S.1.

Reviewer comment (Experiments):
In the experiments, how is the data partitioned to form datasets with different levels of similarities? I think it's important to examine whether these artificially created datasets reflect the real life scenario where slightly different datasets might be owned by different institutions who want to jointly train a model.

Author response:
To create datasets with varying levels of similarity, we employed different strategies depending on the dataset. *Section S4.1.1* provides a more detailed overview of the approach. As a summary:

1. **Datasets Without Natural Partition** (Credit, EMNIST, CIFAR-100): For datasets that do not have an inherent or natural partitioning, we implemented label and feature skew as described by *Hsieh et al., ICML, 2019*. These are established techniques for generating non-IID datasets in the federated learning literature.
2. **Datasets With Natural Partition (Weather, ISIC, IXITiny)**: 

     2a. Weather: We partitioned the data based on climate as described by *Malinin, et al., NeurIPS, 2021*.

     2b. ISIC and IXITiny: For these medical imaging datasets, we partition based on the site and equipment used for data collection as described by du Terrail, Ayed et al., NeurIPS, 2022 

***examine whether these artificially created datasets reflect the real life scenario***

We address this using the ISIC and IXITiny medical imaging datasets, which originate from different institutions and utilize different imaging machines. In *Section 7.2*, we provide a discussion for how our metric captures real-world differences in these datasets and matches intuition on which datasets are likely to be similar. Notably, our analysis with the ISIC dataset reveals an intriguing insight: the type of imaging machine plays a more critical role in dataset similarity than the geographical location of the sites. This finding underscores the practical relevance and applicability of our metric in real-world scenarios.