Paper: Incentivized Black-Box Model Sharing (ICLR-2024)

Reviewer comment (Evaluation):
Could you provide one specific example that motivates why individual rationality is chosen as one of your key metrics?

Author response:
Let's consider a scenario where three private hospitals (A, B, and C) want to share predictions based on their patient data to improve the predictive performance of their model. However, Hospital C has already invested significant resources into collecting a high-quality dataset (with a high data diversity), while Hospitals A and B have datasets that are noisy or of lower quality in comparison. If the ensemble predictions are naively shared among all three hospitals, Hospitals A and B would benefit more because they can significantly improve their models, while it may not benefit Hospital C at all (i.e., **no collaborative gain** in the form of either monetary or improved model) and its competitors are better off. In other words, hospital C did not improve its utility by collaborating (i.e., the individual rationality is *not* satisfied) and would be disincentivized to collaborate. Therefore, enforcing individual rationality can ensure Hospital C's collaboration by ensuring its utility will improve (or positive collaborative gain).

Additionally, a similar example is used to motivate individual rationality in the case of autonomous driving to achieve a win-win outcome (in the form of legislation that enforces individual rationality) for both consumers and car companies [1].

Therefore, individual rationality is one key incentive that we need to consider in collaboration.

[1] Karimireddy, S. P., Guo, W., & Jordan, M. I. (2022). Mechanisms that incentivize data sharing in federated learning. *arXiv preprint* arXiv:2207.04557.

Reviewer comment (Theory):
Why do you consider Shapley fairness as your main fairness notion? Any other fairness notions that might fit into your framework?

Author response:
Intuitively, the more one contributes, the more it should receive from collaboration. To formalize this intuition, there are many ways to do so. In our problem, we adopt the Shapley value as an interesting and meaningful way to achieve this because it is the unique solution that satisfies the four fairness properties (*efficiency*, *symmetry*, *dummy party*, and *linearity*) as in App. B.1.

- [**symmetry**] To ensure two equally contributing parties are equally recognized, symmetry ensures that two *parties with equal marginal contributions to any coalition in the collaboration receive the same collaborative gain*;

- [**dummy party**] To prevent *free-riders*, dummy party ensures that *parties with zero marginal contributions to any coalition receive no collaborative gain*.
    
    Additionally, it is shown by Sim et al. (2020) that Shapley value also satisfies some other fairness properties: strict desirability and strict monotonicity, which we have newly added to App. B.1. Hence, the Shapley value is a careful **design choice** in our work.

As additional justification, the Shapley value is a very widely used solution in previous data sharing works (Ghorbani & Zou, 2019; Jia et al., 2019a; Sim et al., 2020; Tay et al., 2022; Karimireddy et al., 2022;) as the notion of fairness.

> Any other fairness notions that might fit into your framework?

One other common fairness notion, known as *egalitarian*, aims at achieving equitable outcomes for all the parties (e.g., by rewarding all the parties equally). This does not satisfy the dummy party property and can cause the free-rider problem since even the non-contributing party is rewarded equally with the high-contributing parties.

Another common fairness notion (in the allocation of goods) is *envy-freeness*, which is not applicable here. Because it is typically used for rivalrous goods, but the rewards here (i.e., ensemble predictions) are non-rivalrous (since they can be replicated), and it is the reason for our proposed definition of fair replication game.

For discussions of different fairness notions, we will incorporate them in the revision.

Reviewer comment (Theory):
In Sec 5 you suggested that "We will later empirically show that the virtual regret $\epsilon$ is not needed and the strict IR is satisfied". Is this a purely empirical observation or do you believe stronger theoretical results can be established here?

Author response:
It is *mainly an empirical observation*, but we think *stronger theoretical results may be possible*. The reason is the derivations for our theoretical result on the virtual regret $\epsilon$ involve several inequalities (e.g., Hoeffding's inequality, the triangle inequality, the constraint of the growth function by the VC dimension, and the supremum inequality of the domain divergence measure, as in App. C.3.). The improvement of the tightness of one or more of these inequalities can potentially lead to a stronger theoretical result. We will include this comment in our revision.

Reviewer comment (Presentation):
My main complaint of the paper is that the overall presentation was pretty hard to follow, resulting in some confusion over few details of the paper. For instance, I’m a little confused about how the weights beta_{i,x}’s are set if the true label for point x is unknown. And also, it seems that there’s an assumption about the unique of the optimal ensemble weights.

Author response:
W1:
>I’m a little confused about how the weights $\beta_{i,x}$’s are set if the true label for point x is unknown.

Please kindly refer to our response below in Q1 and **App. B.2** for how $\beta_{i,x}$’s are set if the true label for point x is unknown.

>add more prose throughout the paper.

To avoid significant changes (listed below) in the rebuttal as instructed by the guideline in ICLR 2024 "*Area chairs and reviewers reserve the right to ignore changes that are significantly different from the original paper*", we will make the following changes in the revision:

(1) move the valuation part in section 6 to Appendix

(2) move some remarks in Appendix to the main paper to add more elaborations

(3) include more experiments with non-optimal (practical) ensemble methods in the main paper

Reviewer comment (Theory):
The paper describes once how the ensemble weights are set in 4.1. However, here it’s assumed the host actually knows the ground truth. So, is it just that in the very beginning where the host has access to a data set that’s held off, the host asks the clients to participate and find these weights in the very beginning and use these weights going forward? But more realistically, the host would want to query each party to provide predictions for points for which the true label is unknown. In those cases, how would one find these weights? Note that the way things are written, the weight beta_{i,x} is set differently for each point x, meaning one can’t estimate these beta_{i,x} differently for each x, if the true label for that y is not known, but rather set a weight beta_i that’s the same across all the points. This should still maintain proposition 1, as all the arguments are always averaged over the entire distribution D anyway.

Author response:
Q1:
>So, is it just that in the very beginning ... the host asks the clients to participate and find these weights in the very beginning and use these weights going forward?

No. In our setting, no matter what ensemble method is used, the host will query each party for each data point and continually determine the ensemble weights.

>But more realistically, the host would want to query each party to provide predictions for points for which the true label is unknown.

Your realistic consideration is correct. In fact, __in some of our experiments__, the host queries each party for each data point and continually determines the ensemble weights, where the true label is unknown.


>...for which the true label is unknown. In those cases, how would the host want find these weights?

We assume the host knows the ground truth **only when** we use the optimal ensemble method. As we have described in **App. B.2**, ensemble methods from previous works such as average ensemble (AVG), majority vote (MV), knowledge vote (KV), and multiplicative weight update (MWU) do **not** need the ground truth. Those ensemble methods can set the weight $\beta_{i,x}$ both **differently** for each party and each data $x$.

On the other hand, if the ground truth is available, we can observe a much stronger correlation (i.e., **-0.72**) between the average ensemble weight $\mathcal{V}\_i$ and the generalization error $L\_{\mathcal{D}}(h\_i)$ on MNIST in Table 2, compared with the correlation of MV (i.e., **-0.24**) and MWU (i.e., **-0.24**) on MNIST in table 1.

We conduct an **additional experiment** (update in App. D.1) and highlight that the requirement on the ground truth is **not** very restrictive, where the labeled data (i.e., ground truth) of size as small as 100 is sufficient to identify high-quality models, as shown in Fig. 6 in App. D.1. The negative correlation of size 100 is as good as that of size 4000. In practice, those weights estimated on the small labeled dataset can be potentially used for other unlabeled data.

Reviewer comment (Theory):
I think there’s an inherent assumption that the optimal weights beta’s are unique. Consider a following example where every party has the same exact model h. Then, the ensemble model will be the same no matter how the weights beta’s are set. In this case because everyone has the same model, one should be rewarded the same reward, meaning the beta’s should be uniform across every client. However, setting beta’s such that it places all its weight on a single model is also an optimal solution, which results in only that client receiving all the rewards. I think this is not just an artifact of this toy example, but if the data that each client has is pretty homogenous and resulting in similar overall loss, this can be very possible (assuming that as I described above the weights should be chosen not over (party i, point x) but rather over just the parties).

Author response:
Q2:
>Consider a following example where every party has the same exact model h ... In this case because everyone has the same model, one should be rewarded the same reward, meaning the beta’s should be uniform across every client.

In our experiment, the beta’s are indeed **uniform** across every client when they have the **same** exact model $h$, because we set the initial value of $\beta_{i,x}^\ast=1/n$ in solving linear optimization problem of the optimal ensemble. This implementation detail can be found in our provided code (in line 75 in the file "server/collaboration.py").

>I think this is not just an artifact of this toy example, but if the data that each client has is pretty homogenous and resulting in similar overall loss, this can be very possible.

Your understanding is correct. Translating your intuition of "pretty homogenous" data for $i\neq i'\in N$ into the following: if $h_i(x)$ and $h_{i'}(x)$ are "close", then we want $\beta_{i,x} \approx \beta_{i',x}$. To achieve this in implementation, we add the following regularization term $\sum_{\substack{i,i'\in N \\\ i\neq i'}} (|\beta_{i,x}-\beta_{i',x}|)^{(2-\||h_i(x)-h_{i'}(x)\||_1)}$ to the original linear optimization problem, as follows:

$
\underset{\beta_{i,x}^*\in [0,1],\forall i \in N.}{\text{minimize}}\ |f(x)-{\textstyle\sum_{i=1}^n\beta_{i,x}^*h_i(x)}|+\lambda \sum_{\substack{i,i'\in N \\\ i\neq i'}} (|\beta_{i,x}-\beta_{i',x}|)^{(2-\||h_i(x)-h_{i'}(x)\||_1)}
$

$\text{subject to}\ {\textstyle\sum_{i=1}^n\beta_{i,x}^*=1}.$

The hyperparameter 2 in $(2-\||h_i(x)-h_{i'}(x)\||_1)$ is because the L-1 norm of the difference is upper bounded by 2. Please kindly refer to App. B.2 for the detailed **updated** formulation.

Let's illustrate this concept with an example: Suppose we have a collaboration involving five parties. Here, each party shares a 3-class prediction vector $h_i(x)$ of the query data $x$. The ground truth for this data is $\mathbf{[1.0, 0.0, 0.0]}$. Refer to the table below for $h_i(x)$ values from each party. Specially, we impose that $h_1(x)$ and $h_{i\neq 1}(x)$ are "close".

| Party  | $h_i(x)$ | $\beta_{i,x}^\ast$ w/o the regularization | $\beta_{i,x}^\ast$ w/ the regularization  |
|---|:---:|:---:|:---:|
| 1 | [1.0, 0.0, 0.0] | 1.0  | 0.2016  |
| 2 | [0.9, 0.1, 0.0] | 0.0  | 0.1996  |
| 3 | [0.9, 0.1, 0.0] | 0.0  | 0.1996  |
| 4 | [0.9, 0.1, 0.0] | 0.0  | 0.1996  |
| 5 | [0.9, 0.1, 0.0] | 0.0  | 0.1996  |

We observe that $\beta_{1,x}^\ast$ of party $1$ is $1.0$ and $\beta_{i,x}^\ast$ of other parties are 0 when we **exclude** the regularization term, even though $h_i(x)$ and $h_{i'}(x)$ are "close". However, when we **include** the regularization term, we can then achieve $\beta_{i,x} \approx \beta_{i',x}$ for $i\neq i'\in N$ as desired.

Reviewer comment (Writing):
Limitations of the proposed method are not discussed clearly.

Author response:
We have added a section (App. A.1) discussing the limitations and will make this clearer in our revision.

For example, one limitation of our method is that it is developed for ensemble methods that take the form of weighted sum formulation (in Section 4.1), which does appear in several common ensemble methods (mentioned in Section 4.1).

Reviewer comment (Experiments):
Unclear applicability for practical ensemble methods: Average ensemble weight is uncorrelated with the objectives of the parties (Table 1), experiments are performed with an "ideal method" (Section 4.1).

Author response:
We indeed have experimental *results for non-ideal ensemble methods* (which are more practical) and have also *included additional experiments*.

- [Results for non-ideal ensemble methods.] The results are in Tables 4 and 5 (in App. D.1). We observe a **stronger correlation** between the average ensemble weight $\mathcal{V}\_{i}$  and the generalization error $L\_{\mathcal{D}}(h\_{i})$ with the practical ensemble methods (i.e., MV, KV, and MWU) as shown in Table 5 under the non-i.i.d. data setting, which we believe is more important because the setting is more realistic. If the ensemble is bad (e.g., AVG shown in Table 5), our method of valuation and allocation still works, but parties with better models may not be identified as such in the collaboration.

- [Additional experiments.] We included **additional results** in our paper, in Fig. 8 (in App. D.2.1), to demonstrate the accuracy gains w.r.t. different non-ideal (practical) ensemble methods used. In particular, we observe that *strict IR is always satisfied*.

    For the additional experiment on CIFAR-10, we will include it if it is complete before the end of the rebuttal; otherwise, we will include it in the revision.

As these different (non-ideal) ensemble methods produce predictions of varying qualities, we use the ideal (i.e., optimal ensemble) method to illustrate the effect of an ideal case of collaboration. The non-ideal ensemble methods might not be as effective in achieving fairness or IR as the ideal ensemble, but they can be applied nevertheless.

Reviewer comment (Presentation):
Presentation is dense, and was hard for me to follow. Many remarks which were very helpful to my understanding only appeared in Appendix A.

Author response:
Thank you for taking the time to carefully read our prepared appendix. It would be very helpful if the reviewer would let us know the specific remarks (currently in appendix) that would aid the understanding of the reader, if moved to the main paper. We will improve the presentation accordingly.

Reviewer comment (Theory):
Motivation: Under which conditions is the model incentive structure realistic, and the valuation assumption applicable? In the hospital example mentioned in Appendix A (Q2), it is reasonable to assume that every hospital has access to a data source $\mathcal{D}_i$ based on their local population, however it doesn’t seem intuitive to me that the hospital would desire a classifier that has good performance on a population $\mathcal{D}$ which is different than their own, and common to all other hospitals. Can you clarify this example, or give a different practical example where assumptions intuitively hold?

Author response:
The model incentive structure is realistic when the parties are **self-interested** (i.e., interested in accuracy gain and monetary compensation) and desire to perform well on the same target domain $\mathcal{D}$ (please refer to the first paragraph in Sec. 3 for the formal problem formulation).

The valuation assumption is applicable when the ensemble method follows the **weighted sum formulation** of $h_N(x) = \sum_{i=1}^n\beta_{i,x}h_i(x)$ (in Sec. 4.1).

>In the hospital example mentioned in Appendix A (Q2), it is reasonable to assume that every hospital has access to a data source $\mathcal{D}_i$ based on their local population, however it doesn’t seem intuitive to me that the hospital would desire a classifier that has good performance on a population $\mathcal{D}$ which is different than their own, and common to all other hospitals. Can you clarify this example, or give a different practical example where assumptions intuitively hold?

Consider the condition of a pandemic like COVID-19, that has an impact globally, affecting all hospitals and individuals. The data of different hospitals may have acquisition bias due to the different demographics of the patients, so the hospitals' source data $\mathcal{D}_i$ over COVID-19 variants would be heterogeneous. Individuals may be affected by other COVID-19 variants from other distributions in the future. In such a situation, each hospital is interested in developing generalizable classifiers to cater to all individuals by facilitating collaboration and would be dealing with the same population $\mathcal{D}$ [1].

Moreover, existing works on collaborative learning often adopt this same setting of learning from multiple domains to perform well on the same target domain (Chang et al., 2019; Lin et al., 2020; Feng et al., 2021).

We can also extend to different target domains in future work (as mentioned in Sec. 7) to design a collaboration that enables customized improvements (w.r.t. each $\mathcal{D}_i$).

[1] Peiffer-Smadja, N., Maatoug, R., Lescure, F. X., D’ortenzio, E., Pineau, J., & King, J. R. (2020). Machine learning for COVID-19 needs global collaboration and data-sharing. *Nature Machine Intelligence*, 2(6), 293-294.

Reviewer comment (Theory):
Price of fairness: If I understand correctly, it seems that the overall welfare of the parties ($\sum_i L_{\mathcal{D}}(h_i)$) would be maximized by sharing all target-dataset data $\{(x_t,h_N(x_t)\}_{t=1}^T$ with all parties. What are the shortcomings of this approach? How does its welfare compare to the mechanism presented in the paper?

Author response:
Yes, your understanding is correct.

> What are the shortcomings of this approach?

It can cause the **free-rider problem**, if we simply maximize the overall welfare regardless of the contributions. To elaborate, if there is a party that contributes nothing to the collaboration and yet still receives all the ensemble predictions (as every other party). This is **unfair** to the parties that make meaningful contributions and can thus disincentivize other parties (Sim et al., 2020).

> How does its welfare compare to the mechanism presented in the paper?

Under the case where all the parties have *no budget constraints* and make the maximal payment $p_i^\ast$ (see in the 7th line in the paragraph below Remark 1), our proposed mechanism can also **maximize the overall welfare**. In this case, **fairness** is still achieved by distributing the payment fairly, as the less-contributing parties will compensate the top contributors via monetary payments.

If there is any *budget constraint*, the maximum overall welfare may not be achieved. So, there is a price to maximize overall welfare. Which one to choose will depend on the problem setting.

Reviewer comment (Theory):
What is the relation between the objective $L_\mathcal{D}(h_i)$ and the utility $u_i$ presented in Theorem 1? Also, is it possible to quantify the relation between payment and accuracy increase for a given problem instance?

Author response:
The utility $u_i$ is the value that quantifies the received reward $(r_i + r_i^+)$ and the payoff $(p_i^+ - p_i)$. The generalization error $L_{\mathcal{D}}(h_i)$ can be reduced with incorporating ensemble predictions (of size $T_i$) realized from $(r_i + r_i^+)$.

From Proposition 2, $L_{\mathcal{D}}(h_i') \leq L_{\mathcal{D}}(h_i)+\epsilon_i$ and $\epsilon_i$ depends on $T_i:=(r_i+r_i^+)\times T$. The total reward $(r_i + r_i^+)$ in utility $u_i$ determines $T_i$, and affects the improved generalization error $L_{\mathcal{D}}(h_i')$. Specifically, if the payoff $p_i^+ - p_i$ is fixed, a larger $u_i$ could lead to smaller $L_{\mathcal{D}}(h_i')$.

>Is it possible to quantify the relation between payment and accuracy increase for a given problem instance?


Yes, and we present an *informal* idea: Note that any party $i$ receives ensemble predictions of size $T_i:=(r_i+r_i^+)\times T$. The payment $p_i$ is made to purchase the additional reward of value $r_i^+ = \frac{\mathcal{V}_N\times p_i}{\mathcal{V}_N - \phi_i}$, which is realized as additional samples of ensemble predictions of size $(r_i^+ \times T)$. Therefore, the larger payment $p_i$ that party $i$ makes, the more ensemble predictions it will receive.

From Proposition 2, $L_{\mathcal{D}}(h_i') \leq L_{\mathcal{D}}(h_i)+\epsilon_i$ and $\epsilon_i$ depends on $T_i$, where the payment $p_i$ is implicitly embedded in $T_i$. Therefore, a higher payment means a larger $T_i$, hence smaller $\epsilon_i$. If we use $L_{\mathcal{D}}(h_i)+\epsilon_i$ as an approximation of $L_{\mathcal{D}}(h_i')$, $\Delta\epsilon_i$ can represent the reduction in error, which is empirically **reflected as the increase in accuracy**. To **quantify the relation between payment and accuracy increase**, we write $\Delta\epsilon_i$ as a function of $p_i$ shown below:
\\[
\Delta\epsilon_i(p_i) = M  - 4 \sqrt{2d\log(2(r_iT+\frac{p_iT}{1 - \phi_i}+m_i+1))+2\log(\frac{8}{\delta})}\sqrt{\frac{\alpha_i^2}{r_iT+p_iT/(1 - \phi_i)}+\frac{(1-\alpha_i)^2}{m_i}}  
\\]

where
$M=4 \sqrt{2d\log(2(r_iT+m_i+1))+2\log(\frac{8}{\delta})}\sqrt{\frac{\alpha_i^2}{r_iT}+\frac{(1-\alpha_i)^2}{m_i}}$ is a constant. When $p_i=0$, $\Delta\epsilon_i(p_i)=0$. When $p_i$ is larger, $\Delta\epsilon_i(p_i)$ will be larger, and thus a higher accuracy.

This idea is empirically verified in an **additional empirical result** in Fig. 9 in App. D.2.1.

Reviewer comment (Theory):
Technical questions: What is the meaning of the notation $\hat{L}_{\mathcal{D}}(h,h_N)$ in Section 5.2? Is there an upper bound on the size of realized reward $T_i$?

Author response:
$\hat{L}\_{\mathcal{D}}(h,h\_N)$ is the empirical error of training with the ensemble predictions. Denote $U\_i=\\{x\_t, h_N(x\_t)\\}\_{t=1}^{T\_i}$ the set of ensemble predictions that party $i$ receives. By definition, $\hat{L}\_{\mathcal{D}}(h,h\_N) = \frac{1}{T\_i}\sum\_{x \in U\_i}\ell(h(x),h\_N(x))$.

There is an upper bound on the size of realized reward $T_i=(r_i+r_i^+)\times T$, where $T_i$ is always less or equal to the number of all ensemble predictions $T$. The size $T$ is specified by the size $|U|$ of the distillation dataset $U$.

Reviewer comment (Writing):
The main weakness is in the exposition - I was not able to understand the model. It seemed like the model and problem formulation were not comprehensively specified. The fact that there is an FAQ section on the model speaks to how the model is not completely clear.

Author response:
>The fact that there is an FAQ section on the model speaks to how the model is not completely clear.

Our provided Q\&A section in App. A.2 is meant to further supplement the main paper with more nuanced and specific details. In our revision, we will improve the exposition of our work to specify the problem formulation more clearly.

Reviewer comment (Evaluation):
How should we compare prediction error to monetary payments to "rewards" (samples of ensemble predictions)? (Do they use the same unit of measurement?)

Author response:
> How should we compare prediction error to monetary payments to "rewards" (samples of ensemble predictions)? (Do they use the same unit of measurement?)

The monetary payments (i.e., $p_i$) and the value $r_i$ of the rewards (of size $T_i$) can be viewed to use the same unit of measurement (i.e., in the same domain). This is by design so that the allocation mechanism (Theorem 1) enables a translation between ensemble predictions (i.e., the realized rewards $T_i$) and monetary payment. We note that this has not been previously achieved by existing works.
Please refer to Q4 in App. A where we illustrate how the monetary currency is projected into the positive real domain $\mathbb{R}_+$ to align with the reward domain.

When you say "compare prediction error to monetary payments", we assume you mean the relationship between the two. To elaborate more on the relationship: An increase in the monetary payment $p_i$ leads to a higher reward $r_i$, and thus a larger size for $T_i$, which can decrease the upper bound on the prediction error $L_{\mathcal{D}}(h_i')$:

- From Proposition 2, the upper bound of **prediction error** (i.e., $L_{\mathcal{D}}(h_i')$) depends on **payment** $p_i$, because $L_{\mathcal{D}}(h_i') \leq L_{\mathcal{D}}(h_i)+\epsilon_i$ and $\epsilon_i$ depends on $T_i$. To compare the prediction error to monetary payment with approximation, we investigate "the change in $\epsilon_i$" as a function of $p_i$ in a newly added empirical result (i.e., Fig. 9 in App. D.2.1), which provides an empirical quantification between the accuracy gain w.r.t. the increase in (monetary) payment.

Reviewer comment (Writing):
What is the formula for the utility of party i?

Author response:
>Relatedly, what is the formula for the utility of party i?

By **definition**, $u_i$ is the Shapley value to the linearly combined game of $G$ and $G^p$. By **derivation**, the explicit **formula** for the utility $u_i$ of party $i$ is given in Theorem 1: $u_i = r_i + r_i^+ + p_i^+ - p_i $, where 
- $r_i=\phi_i$,
- $r_i^+ = \frac{\mathcal{V}_N\times p_i}{\mathcal{V}_N - \phi_i}$,
- $p_i^+=\sum_{j \in N \setminus i }\frac{\phi_i \times p_j}{\mathcal{V}_N - \phi_j}$, and
- $p_i\in \mathbb{R}_+$ is determined by the party itself.

The utility $u_i$ represents the *overall gain* of party $i$ from the collaboration, as a sum of $i$'s total reward $(r_i + r_i^+)$ and payoff/net compensation $(p_i^+-p_i)$.

- $(r_i + r_i^+)$ represents the *value* of ensemble predictions that party $i$ receives from the collaboration,
- $p_i^+$ represents the value of the received monetary compensation from all other parties, and
- $p_i$ is the payment that party $i$ made to the host in exchange for additional rewards of the value $r_i^+$.

Our formula of the utility fully specifies all the terms. Note that these interpretations (of the utility) are provided in the main paper (above Theorem 1 and below Remark 1).

Reviewer comment (Theory):
Payments can be made from one party to another. Does each party, decide on their own, how much to pay to each other party, or is this transfer also specified as part of the mechanism? Does each party have a budget?

Author response:
> Payments can be made from one party to another. Does each party, decide on their own, how much to pay to each other party, or is this transfer also specified as part of the mechanism?

Each party decides on its own how much to pay *in total* (i.e., $p_i$) to the central host; the exact distribution of $p_i$ to each other party is **specified by our mechanism** (and managed by the central host) to achieve fairness: The proposed formula of $p_i^+=\sum_{j\in N\setminus \{i\}}\frac{\phi_i \times p_j}{\mathcal{V}_N - \phi_j}$ in Theorem 1 specifies this payment distribution from every party $j\in N\setminus \{i\}$ to party $i$.

> Does each party have a budget?

Our mechanism can achieve fairness whether or not a party has a budget constraint:

- [With budget constraint.] If party $i$ has a budget of $B_i$, it can freely choose payment on its own as long as $p_i \leq B_i$, and fairness is always achieved by the reward design and realization, as shown in App. C.6.

- [W/o budget constraint.] If party $i$ does not have any budget constraint, then it can make the maximal payment $p_i^\ast = (\phi^\ast -\phi_i)(\mathcal{V}_N-\phi_i)/\phi^\ast$ to receive all the reward of value $\mathcal{V}_N$, which is mentioned in the paragraph below Remark 1. In this case, fairness is achieved by fairly distributing the payments.

Reviewer comment (Experiments):
The model has two main parts, as described in Figure 5. Can we simply de-couple these two stages and study each part separately, or are there interactions that require studying them together? Just studying one aspect would make the paper simpler and more clear.

Author response:
> Can we simply de-couple these two stages and study each part separately, or are there interactions that require studying them together? Just studying one aspect would make the paper simpler and more clear.

We consider these two stages together (1) so that we can study the **overall gain** (i.e., $u_i$) of a party $i$ from the entire collaboration; and (2) because the $G^p$ depends on the outcome of $G$:

1. Considering the two stages together enables us to study the **overall gain** (i.e., $u_i$) of a party received from collaboration including the *reward* (i.e., $r_i + r_i^+$) and *payoff* (i.e., $p_i^+ - p_i$). Note that, in the paragraph above Theorem 1, we formally couple the two stages via two games, namely $G$ and $G^p$. The outcome of $G$ determines $r_i$, and the outcome of $G^p$ determines $r_i^+ - p_i + p_i^+$. With this coupling, we can directly determine the number for ensemble predictions $T_i=(r_i+r_i^+)\times T$ that party $i$ receives.

2. The fair replication game $G^p$ shown in Definition 5.1 depends on the outcome (i.e., $\\{\phi_i\\}_{i=1}^n$) from $G$, because the fairness in the monetary gain (i.e., $r_i^+ - p_i + p_i^+$) relies on the contribution measure $\phi_i$. To satisfy the Shapley fairness incentive (i.e., fairly allocating rewards and payoffs), it is necessary to calculate the Shapley value from the first stage. Hence, the two stages need to be coupled.

Let's consider a concrete scenario of private hospitals collaborating by sharing predictions of unlabeled medical data. Small hospitals with lower contributions (i.e., a model of lower quality/performance) can make payments to purchase more rewards (i.e., ensemble predictions) to improve their models. Hence, it is important for the payment (in exchange for rewards) to be carefully based on the outcome of the first stage so that these parties (i.e., small hospitals) are able to obtain a high utility (i.e., improvement in their local models' performance).

We will make these reasons (for considering the two stages) more explicit in our revision.