Paper: Data Distribution Valuation with Incentive Compatibility (ICLR-2024)

Reviewer comment (Theory):
A main idea in the authors' solution is to use the MMD $\hat d(D_i, D_N)$ estimated from samples to approximate the true MMD $d(P_i, P^*)$. This is basically a law of large number, where good estimation can be obtained if the number of samples is large enough. However, the motivation of data distribution valuation is exactly the lack of samples. As written in the Introduction, "data vendors offer a preview in the form of a sample dataset to prospective buyers. Buyers use these sample datasets to decide whether they wish to pay for a full dataset or data stream." The key tension is how the buyers can evaluate the value of a full dataset given access to only a limited-size preview dataset. **This is at odd with the authors' approach of using many samples to estimate the value of the distribuiton.**

Author response:
We hope to highlight that while the constraint (i.e., lack of samples) may appear to be at odds with the proposed approach (i.e., using samples to estimate the value), this is an important motivation for the specific design choice namely MMD due to its efficient sample complexity.

- [Better sample complexity.] The sample complexity of MMD (i.e., using sample MMD to estimate true MMD) is **efficient**, precisely $\mathcal{O}(1/ {\sqrt{m}})$ (where $m$ the size of the sample) and importantly, **independent of the dimension**. In contrast, the optimal transport (as discussed in Appendix B.1) has a sample complexity of $\mathcal{O} 1(1/n^{1/d})$ which suffers the *curse of dimensionality* (Genevay et al., 2019).

- [Faster empirical convergence.] The benefit of this efficient sample complexity is demonstrated in an empirical comparison in Appendix C.3.1 where the sample MMD converges more quickly (to the true MMD), making MMD an more practically appealing choice (than other divergence) under the constraint that only a limited-size sample is available.

Reviewer comment (Theory):
Another idea in the authors' solution is to use the average distribution $P_N$ of every one's distribution as the reference distribution $P^*$. This requires $P_N$ to be closed to $P^*$; in other words, the noisy parts $Q_i$ in everyone's distribution cancel out when averaged (Proposition 2). **This is a strong limitation.** As the authors point out (in page 17), it is possible that a majority of data providers are off from $P^*$ while only a few are close to $P^*$. In this case, the authors' solution does not work.

Author response:
We wish to clarify that Proposition 2 does _not_ require the assumption that $P_N$ is close to $P^*$. Instead, Proposition 2 shows the effect of the distance of $P_N$ from $P^*$. Providing a precise characterization of how the heterogeneity in the vendors (i.e., the $Q_i$'s and the $\epsilon_i$'s) affects the valuation function, is a contribution, as it has not been previously obtained: The same design choice (of using $P_N$ as a majority-based reference instead of $P^*$) has been previously adopted by Chen et al. (2020); Tay et al. (2022); Wei et al. (2021), as elaborated in Appendix B. The key distinction is that they do *not* provide the error guarantee (i.e., Proposition 2).

Indeed, because of the possibility that $P_N$ may be far from $P^*$, we have obtained results (in Appendix A.2.4) that are general and can be applicable to other alternative reference $P_{\text{ref}}$ (to $P_N$), should it become available by design or assumption. In this regard, these considerations do *not* have the limitation that $P_N$ must be close to $P^*$.

Reviewer comment (Theory):
In Section 6.2 (experiment about incentive compatibility), why does adding Gaussian noise ensure $d(\tilde P_{i'}, P^*) > d(P_{i'}, P^*)$?

Author response:
Intuitively, it is because the data (e.g., MNIST images as ground truth $P^*$) are *not* (isotropic) Gaussian noise, so adding (isotropic) Gaussian noise makes the data "worse" in terms of a higher MMD to the ground truth.

Reviewer comment (Novelty):
Maybe a better motivation/theme of this work should be "how to value heterogeneous datasets", instead of "how to value data distribution instead of dataset". As I argued in Weakness 1, the authors' approach is not about valuing data distributions using limited-size preview datasets; it is more about how to value heterogeneous datasets.

Author response:
We hope to clarify that our focus is to *characterize and precisely define a value for a data distribution*, motivated by the use-cases mentioned in Section 1. Our theoretical framework (built on the Huber model and the MMD) is proposed to enable theoretical analysis w.r.t. the definitions of the data distributions (e.g., Observation 1 and Eq. (2)). In other words, results such as Observation 1 and Eq. (2) are specifically targeting the valuation of data distribution.

Then, our proposed sample-based approach (along with the results including Proposition 1, Theorem 1 and Proposition 2) is to observe the practical constraint that the analytic expression of the sampling distribution (or data distribution) is __not attainable__ and to ensure our theoretical framework can still be implemented in practice, _without_ the analytic expressions of $P_i$'s. In particular, Proposition 1 and Theorem 1 are specific designs for implementations to obtain actionable policies to compare the values of the data distributions.

Reviewer comment (Theory):
[Quality] I find the setting studied in this paper slightly confusing. In particular, I am not sure if the author(s)' definition of IC is in line with my usual understanding: it looks like the definition of $\gamma$ -IC (Def. 1 and Cor. 1) depends on the choice of $\tilde{P}$ (instead of "for all $\tilde{P}$ in certain class"), which seems unusual. The approach in this paper is to propose a natural candidate for the metric, and then studies its IC property (as opposed to characterizing the class of IC designs) - there is nothing wrong with it, but I find it slightly misleading to refer to Cor. 1 as $\gamma$ -IC. I am quite certain that manipulating the dataset would quite often lead to better metrics. Moreover, the empirical results also seem insufficient in justifying the true IC (in the way I would define it) - the alternative distribution considered is overly specific and somewhat naive, and clearly cannot represent the class of all possible manipulations. Overall, I am not convinced of the correctness (or significance) of the main claim.

Author response:
[On quality/setting.]


>  I am quite certain that manipulating the dataset would quite often lead to better metrics.


Could the reviewer please confirm that by "manipulating the dataset leading to better metrics", it means the vendor performs some careful statistical modification of the dataset to improve its value?


> the alternative distribution considered is overly specific and somewhat naive, and clearly cannot represent the class of all possible manipulations

The "alternative distribution" is specified to be interpretable and intuitive, so that the empirical results can be understood and used to verify the theoretical results. A very complex "alternative distribution" can make the results and analysis difficult and thus not very useful in deriving insights.

We note that we do _not claim to represent the class of all possible manipulations_, in particular our empirical settings.

Reviewer comment (Writing):
I am curious why the problem is phrased "data valuation"? I see nothing about pricing in this work, and to me "evaluation" seems like a better fit?

Author response:
> I am curious why the problem is phrased "data valuation"? I see nothing about pricing in this work, and to me "evaluation" seems like a better fit?

We note that the term "data valuation" is a relatively common phrase in similar existing works (Ghorbani & Zou, 2019; Kwon & Zou, 2022, Amiri et al., 2022, Wang & Jia, 2023, Just et al., 2023, Wu et al., 2022, Bian et al. 2021, Jia et al. 2018; 2019, Yoon et al. 2020).

As described in Section 1, one primary motivating application of this work (and these above existing works) is to use the value of data for the pricing of data in data marketplaces such as Datarade, Snowflake (Chen et al., 2022) (reference in main paper) and (Agarwal et al., 2019, Sim et al., 2022) (references below).

Reviewer comment (Evaluation):
MMD as a valuation doesn’t reflect data distribution’s performance on a real decision problem / proper loss function. In contrast divergence-based valuations correspond to the value of dataset on a proper loss.

Author response:
> MMD as a valuation doesn't reflect data distribution's performance on a proper loss function.

In (Si et al., 2023), the opposite is argued that the MMD can be used to provide a proper scoring rule (i.e., proper loss), because the **_continuous ranked probability score_ (CRPS) is a proper scoring rule, which can be shown to be a form of MMD**. Specifically, in Section 2 (Background) of (Si et al., 2023):

> __A popular proper loss is the continuous ranked probability score (CRPS)__, defined for
two cumulative distribution functions (CDFs) $F$ and $G$ as $\text{CRPS}(F, G) = \int (F(y) − G(y))^2 \text{d}y$ ... The above CRPS can also be rewritten as an expectation relative to the distribution $F$:
\begin{equation}
 \text{CRPS}(F, y') =  -\frac{1}{2} \mathbb{E}_F |Y - Y'| + \mathbb{E}_F |Y -y'|
\end{equation}
where $Y, Y'$ are independent copies of a random variable distributed according to $F$.

And then in the last paragraph of the Section 2:

> A special case of IPMs is maximum mean discrepancy (MMD) (Gretton et al., 2008), in which $\mathcal{T} ={T : \Vert T \Vert_{\mathcal{H} } ≤ 1}$ is the set of functions with a bounded norm in a reproducing kernel Hilbert space (RKHS) with norm $\Vert \cdot \Vert_{\mathcal{H}}$; the __CRPS objective can be shown to be a form of MMD__ (Gretton et al., 2008).

In this regard, we believe that MMD is indeed connected to a proper loss function.

As to the other part of this comment:

> MMD as a valuation doesn't reflect data distribution's performance on a real decision problem.

Could the reviewer elaborate on the definition of _a real decision problem_ and its potential benefits?


>  In contrast divergence-based valuations correspond to the value of dataset on a proper loss.

A divergence-based valuation does __not necessarily correspond__ to the value of dataset on a proper loss, because the result recalled by Kimpara et al., (2023), namely Theorem D.0.1 (in their Appendix D) is specific to the __Bregman divergence__. For data valuation, some divergences such as $f$-divergence and optimal transport have been considered (as compared and discussed in our Appendix B.1). To the best our knowledge, these divergences are not directly equivalent to the Bregman divergence, so the result in (Kimpara et al., 2023) is not immediately applicable.

Reviewer comment (Novelty):
I suggest the author compare the results to previous results in information elicitation. Specifically, this following paper seems relevant. It studies elicitation with sample access to the distribution to be elicited. Suppose the principal has sample access to ground truth distribution, and sets it as a reference, it seems like the approach in this paper could be applied. 

    Frongillo, Rafael, Dhamma Kimpara, and Bo Waggoner. "Proper losses for discrete generative models."

Author response:
> I suggest the author compare the results to previous results in information elicitation. Specifically, this following paper seems relevant.


We thank the reviewer for the reference and will include in our revision. We wish to highlight the following __connection and distinctions__:

In (Kimpara et al., 2023), Definition 3.3 (_Proper divergence_) is a key component used, and indeed MMD is __a stritcly proper divergence__. This can be proven using Theorem 5 of (Gretton et al., 2012), which ensures $\text{MMD}(p,q) = 0 \iff p=q$, and the fact that $\text{MMD}(p,q) \geq 0$ because MMD is equivalently a norm in the Hilbert space.

The distinctions are in the _goal and a technical setting_ where the goal of Kimpara et al., (2023) is to __evaluate generative models__ in the __discrete__ setting while our goal is to __characterize the sampling distributions__ for data in the __continuous__ setting.

Note that we are citing the conference version of the paper, which has a different ordering of the authors from that reference (arXiv-version) raised by the reviewer.

> Suppose the principal has sample access to ground truth distribution, and sets it as a reference, it seems like the approach in this paper could be applied.

This corresponds to Proposition 1 in Section 4 where we make this assumption of having access to the ground truth distribution. We subsequently relax this assumption in Section 5.

Reviewer comment (Theory):
Suppose all N-1 distributions have the same deterministic bias, while only one distribution is the ground truth distribution. It seems like it is impossible in this case to discover or incentivize the most valuable dataset.

Author response:
> Suppose all N-1 distributions have the same deterministic bias, while only one distribution is the ground truth distribution. It seems like it is impossible in this case to discover or incentivize the most valuable dataset.

We wish to highlight that our theoretical result on incentivization aims to provide a precise characterization of how the incentive compatibility is dependent on the $n$ distributions ($1$ for each of $n$ vendors), instead of trying to achieving the exact incentive guarantee for all possible cases. In the case described by the reviewer, our analysis **continues to apply**, the effect is that the effectiveness of the incentivization for the vendor with the ground truth distribution would be lower, and will decrease as the deterministic bias increases.

Reviewer comment (Theory):
Impossibility results: as my comment 3 in Weakness, I wonder if the authors can prove impossibility results on discovering the truth. This can help justify the current approach in the paper.

Author response:
Q1 **[Impossibility results.]** While it may be appealing to obtain such an impossibility result on discovering the (ground) truth, we wish to highlight that our focus is _not_ to discover the ground truth, but rather to present an analysis, when the (ground) truth is unknown, to precisely characterize how the level of heterogeneity (as in Huber) or deterministic bias (as in the reviewer's previous comment) affects the degree of incentivization guarantee. As a contrast, an impossibility result categorically stating that under certain conditions, the exact incentivization is impossible, may not provide additional such insight.

That being said, we think an impossibility result is possible and worth exploring as a future direction. We will incorporate this comment in our revision.

Reviewer comment (Theory):
Following the previous point, if the authors believe learning the explicit distribution is not feasible, is it possible to construct estimators of KL divergence from samples and directly estimate KL divergence similarly as the approach in this paper?

Author response:
Q3 **[Estimating KL directly.]**
We wish to  highlight that our focus is *not* designing new estimators for an existing divergence (e.g., KL). Nevertheless, there are some techniques, such as the $k$-nearest neighbors ($k$NN) estimation for KL (Wang et al., 2008).

There are some disadvantages for adopting the KL and this sample-based estimation:
- (i) [**Direction of KL**] Since KL is not symmetric, the selection of the direction of KL is an additional question to consider during application;
- (ii) [**No finite sample guarantee**] The $k$NN estimation is only shown to provide asymptotic guarantees, instead of the **finite sample guarantee** for MMD (Lemma 1 in Appendix A.1), which is used to provide the theoretical results (e.g., Theorem 1). Furthermore, "the rate of convergence of this $k$NN estimator (for KL) can be arbitrarily slow depending on the distributions." (Section 1-B, Sriperumbudur et al., 2009);
- (iii) [**Difficult for high-dimensional data**] "The NN method becomes unreliable in a high-dimension" (Section IV, Wang et al., 2008) and "for increasing dimensionality of the data (in $\mathbb{R}^d$), the method (NN estimator) is increasingly difficult to implement." (Section 1-B, Sriperumbudur et al., 2009), while MMD is designed to be suitable to high-dimensional data (Gretton et al. 2008).

Reviewer comment (Writing):
Definition of IC is unclear to me, especially the part with wlog.

Author response:
Q4 **[Definition of IC.]**
Please refer to the general comment "Updated definition for incentive compatibility and theoretical results (in Appendix A.3)".

We hope our response has clarified your questions and helped improved your opinion of our work.