Paper: Incentive-Aware Federated Learning with Training-Time Model Rewards (ICLR-2024)

Reviewer comment (Evaluation):
The behavior of IAFL is not clearly explained in the experiments.

Author response:
>W1(a): "What is client contribution here" and "How is this set/varied across clients/time?"

To clarify the experimental settings, as described in Section 6.1, one way to measure client contribution is to "use standalone accuracies as a surrogate for the client contributions in FL". In this way, the client contributions vary across clients.

As an alternative way to measure client contribution, we also adopted CGSV (Xu et al., 2021) as the contribution measure in Appendix E.8. For CGSV, it not only considers variances of contribution values across clients, but also takes into account the changes occurring over time or iteration steps.

Reviewer comment (Experiments):
What is client contribution here? Is it number of local updates? How is this set / varied across clients / time?

Author response:
>W1(a): "What is client contribution here" and "How is this set/varied across clients/time?"

To clarify the experimental settings, as described in Section 6.1, one way to measure client contribution is to "use standalone accuracies as a surrogate for the client contributions in FL". In this way, the client contributions vary across clients.

As an alternative way to measure client contribution, we also adopted CGSV (Xu et al., 2021) as the contribution measure in Appendix E.8. For CGSV, it not only considers variances of contribution values across clients, but also takes into account the changes occurring over time or iteration steps.

Reviewer comment (Experiments):
How is incentivization determined? Do you compare a locally trained model to the (fully trained) server model?

Author response:
>W1(b): "How is incentivization determined? Do you compare a locally trained model to the (fully trained) server model?"

To clarify the metrics that we use to evaluate incentivization performance, we carefully utilize three distinct metrics covering different aspects of desired behaviors of IAFL in experiments. They are (1) incentivized participation rate (IPR), (2) correlation to contribution $\rho$, and (3) average/highest predictive performance. These three metrics are described in detail and thoroughly studied in Section 6.1. We do not compare to the fully trained server model because IAFL does not output any server global model.

As for the (fully trained) server model, it actually corresponds to the highest predictive performance (point 3 above). The tables of results are pushed to Table 5 of Appendix E.3 and Table 10 of Appendix E.9 due to space constraints. This is because the best client in our experimental setting receives the best model updates aggregated from all clients in all training iterations, therefore, the highest predictive performance corresponds to the (fully trained) server model.

Reviewer comment (Evaluation):
In section 6.1 ("We measure the performance of a model using the test loss and the test accuracy, denoting them as IPR_loss and IPR_accu, respectively."), this doesn't make sense, as IPR_accu (Table 1) appears to refer to the fraction of clients who are "incentived" to participate, and not an accuracy metric.

Author response:
>W2: About "IPR$_\text{accu}$"

we apologize for the confusion in the quoted sentence. We need to associate the sentence with the two sentences before it and hence refer the reviewer to the rephrased version here:

"We let IPR$\_\text{loss}$ be the percentage of clients receiving a model not worse than his standalone model in terms of the test loss. Likewise, We define IPR$\_\text{accu}$ but in terms of the test accuracy. Here, a standalone model refers to the model trained only with the client's local dataset. IPR therefore indicates IR (Definition 1)."

Reviewer comment (Evaluation):
Based on the results in Table 3, it is surprising to see that IAFL achieves much better accuracy than non-IR methods. However, shouldn't the other methods have an advantage when comparing raw accuracy, as they distribute a high-quality model to all clients without considering incentives? Clarification is needed on what exactly this accuracy metric refers to.

Author response:
>W3: About Table 3, "better accuracy than non-IR methods"

The “acc.” metric in Table 3 refers to “Predictive performance” explained in the corresponding paragraph in Section 6.1.

Now, we explain the results in Table 3. Other baseline methods yield low accuracies as they neglect the individual rationality (IR) of clients in their design in achieving fairness and personalization goals. As seen in the table below,

| Methods  | IPR$\_\text{accu}$ | acc. (highest) |
| -------- | --------- | ------- |
| FedAvg Finetune | 0.28 | 0.12 (0.16) |
| LG\-FedAvg      | 0.60 | 0.13 (0.15) |
| CGSV            | 0.00 | 0.06 (0.07) |
| Rank            | 0.94 | 0.24 (0.31) |
| IAFL            | 1.00 | 0.44 (0.53) |

the incentivized participation rates (IPR) achieved by baseline methods are low, indicating that the clients did not receive high-quality models (even lower than what they can achieve on their own), reflected as poor model performance in terms of predictive accuracies.

Reviewer comment (Theory):
Assuming contribution is participation rate, wouldn't there already be a disadvantage to partial participation if the server broadcasts an update (rather than the updated model) to the participating clients, as the local model could become desynchronized? Or does the paper assume the server is sending an updated model?

Author response:
>Q2: "partial participation" $\rightarrow$ "disadvantageous" since "the local model could become desynchronized?"

To clarify, we assume that clients are always available to receive **model updates** of varying quality commensurate to their contributions (not an updated model) from the server in all training iterations, and the participation rate only reflects the client's rate of computing and sending the local updates to the server. Therefore, the client model quality varies mainly because of the quality of the updates the server sends to the client depending on the client's contribution, rather than issues caused by synchronization.

In the event that it is impractical to assume the client's availability to receive model updates in every iteration, we can alternatively keep a database on the server side to store the cumulative model updates for clients until they become available.

Reviewer comment (Theory):
The incentive guarantee is weak in the sense that only a positive incentive is guaranteed, which might not be enough when the clients do suffer certain costs to contribute more to the center. When the cost is higher than the incentive, one may still not contribute 100% effort in the federated learning.

Author response:
We indeed only have a relative guarantee for the proposed mechanism instead of absolute ones, i.e., clients have more incentives to contribute to IAFL as compared to FL.

We assumed that your "positive incentive" refers to **the increase in utility**. In our paper, the utility is defined as "rewards minus costs". Therefore, our incentives design already takes cost $c_i$ of client $i$ into consideration. With that, incentivizing clients to contribute 100% effort is indeed the most ideal, but might not be achievable because costs are incurred by the clients to contribute more. A more practical perspective we have adopted in the paper is that a client will increase his contribution to a point where the gradient of utility is 0, i.e., the increment in reward balances the increment in cost. More specifically, in Proposition 1, we measure the gradient of the utility w.r.t. contribution $p_i$ to determine the client contribution level and show that a practical incentive guarantee with the consideration of client costs.

Reviewer comment (Theory):
Clients may have reasons to not contribute at full capacity, such as when contribution already exceeds the threshold parameter \( p_\mathsf{ceil} \).

Author response:
- About "$p_{ceil}$"

The reviewer's comments on $p_{ceil}$ is valid. It is worth noting that we have discussed the issue in the paragraph after Equation (1):
“One limitation of this design is that the clients lose incentive for $p_i > p_{ceil}$. The server can mitigate this by setting a high $p_{ceil}$ upper limit (to keep incentivizing high contributors) while setting a larger $\kappa$ (to even out the fractions $(p_i/p_{ceil})^{1-\kappa}$ towards 1 to motivate low contributors).” Please let us know whether the discussion is sufficient.

Reviewer comment (Presentation):
The authors should at least discuss the above limitations.

Author response:
- About "$p_{ceil}$"

The reviewer's comments on $p_{ceil}$ is valid. It is worth noting that we have discussed the issue in the paragraph after Equation (1):
“One limitation of this design is that the clients lose incentive for $p_i > p_{ceil}$. The server can mitigate this by setting a high $p_{ceil}$ upper limit (to keep incentivizing high contributors) while setting a larger $\kappa$ (to even out the fractions $(p_i/p_{ceil})^{1-\kappa}$ towards 1 to motivate low contributors).” Please let us know whether the discussion is sufficient.

- About "$\kappa$" and "$q$"

We would like to stress the importance of $\kappa$ and $q$ in the trade-off identified by the reviewer. We have discussed them in Section 6.1 of the main paper and we additionally have a dedicated section Appendix E.7 to discuss the **trade-off between model performance and client incentivization** (directly affected by $\kappa$ and $q$). In essence, decreasing $\kappa$ and $q$ results in stronger incentives. When the strength of incentives is stronger, we have higher incentivization performance (measured as IPR and $\rho$ defined in the paper) and lower client model accuracies (equivalent to higher loss of model accuracy).

In summary, we have established the trade-off in another form in the paper, which can relate to the loss of center model and strength of incentives trade-off. We will add this discussion to the revised paper.

Reviewer comment (Evaluation):
What is the tradeoff between the strength of incentive and the loss of (center) model accuracy?

Author response:
By "strength of incentives", we assume you mean the **incentivization performance** in the paper. As for the "loss of (center) model accuracy", we understand it as the loss of client v.s. the (center) model, reflected as client model accuracies.

Then, the two observations mentioned in the review align with our discussion in the paper in a slightly different form.

- About "$p_{ceil}$"

The reviewer's comments on $p_{ceil}$ is valid. It is worth noting that we have discussed the issue in the paragraph after Equation (1):
“One limitation of this design is that the clients lose incentive for $p_i > p_{ceil}$. The server can mitigate this by setting a high $p_{ceil}$ upper limit (to keep incentivizing high contributors) while setting a larger $\kappa$ (to even out the fractions $(p_i/p_{ceil})^{1-\kappa}$ towards 1 to motivate low contributors).” Please let us know whether the discussion is sufficient.

- About "$\kappa$" and "$q$"

We would like to stress the importance of $\kappa$ and $q$ in the trade-off identified by the reviewer. We have discussed them in Section 6.1 of the main paper and we additionally have a dedicated section Appendix E.7 to discuss the **trade-off between model performance and client incentivization** (directly affected by $\kappa$ and $q$). In essence, decreasing $\kappa$ and $q$ results in stronger incentives. When the strength of incentives is stronger, we have higher incentivization performance (measured as IPR and $\rho$ defined in the paper) and lower client model accuracies (equivalent to higher loss of model accuracy).

In summary, we have established the trade-off in another form in the paper, which can relate to the loss of center model and strength of incentives trade-off. We will add this discussion to the revised paper.

Reviewer comment (Evaluation):
Is there a fairness concern that for small clients, even if they contribute to their best, they still cannot receive a high quality model, yet the large clients only need to contribute above some threshold to receive the best model?

Author response:
There are two prevalent concepts of fairness in literature: (a) collaborative fairness [1] (i.e., contribute more get back more), and (b) equality [2]. To be fair, people from one of the camps will not view the other as fair.

In our paper, we have demonstrated results for the collaborative fairness view. Specifically, the contributions of clients are fairly evaluated by a contribution evaluation measure, e.g., CGSV (Xu et al., 2021), and then used to fairly reward the clients with a model that has performance commensurate with their contributions.

If the designer of the FL process views fairness more towards equality such that "if everyone puts in their best efforts, they should be treated equally", it is possible to use "individual efforts percentage" (e.g., how many percentages a client contributes to his/her best effort, as suggested by the reviewer) as a contribution evaluation measure (assuming that individual efforts of clients can be measured). This measure can be seamlessly combined with the IAFL framework, too. However, we would like to clarify that "individual efforts percentage" is not commonly used as a contribution measure in literature.

Thank you for the interesting insights and unique perspectives.

References:

[1] L. Lyu, X. Xu, Q. Wang, and H. Yu. Collaborative fairness in federated learning. In Q. Yang, L. Fan, and H. Yu, editors, Federated Learning, volume 12500 of Lecture Notes in Computer Science, pages 189–204. Springer, Cham, 2020.

[2] T. Li, M. Sanjabi, A. Beirami, and V. Smith. Fair resource allocation in federated learning. In Proc. ICLR, 2020.

Reviewer comment (Theory):
In Theorem 1 and Theorem 2, does convergence speed become faster as the number of agents $N$ grows?

Author response:
We greatly appreciate the reviewer's time and effort in providing constructive feedback on our paper, and we are encouraged by the high scores assigned to the soundness and presentation of our paper. We are also delighted to see that our scope of experiments is extensive.

We hope that our responses to your concerns as follows will further enhance your perception of our work.

>Q1: "Does convergence speed become faster as the number of agents $N$ grows?"

The simple answer is, adding clients that **"help than hurt"** is going to improve the convergence speed.

To elaborate, we first observe that it might be difficult to further simplify the bound because it has a dependence on every single client's contribution $\gamma_{m,t}'$ for $m \in [N]$. However, we can derive some insights on the convergence speed as $N$ grows.

The effect depends on the **contribution levels of the clients**.
We can look at the term $C_T = \frac{H_T}{N} \sum_{m=1}^N \sum_{t=1}^{T} \left(\frac{T+\alpha}{t+\alpha}\right)^2 \left(\frac{1}{\gamma_{m,t}'} + \frac{1}{\gamma_{i,t}'} -2 \right)$, when $N$ increases, the denominator becomes bigger but at the same time there are additional terms in the summation.

We can consider a case where we increase $N=n$ to $N=n+1$. Abstracting away the constants $H_T$, $T$ and $\alpha$ for simplicity, we can make a comparison for $C_T = \frac{1}{n} \sum_{m=1}^{n} \left(\frac{1}{\gamma_{m,t}'} + \frac{1}{\gamma_{i,t}'} - 2 \right)$ when $N=n$ against the case for $C_T = \frac{1}{n+1} \sum_{m=1}^{n+1} \left(\frac{1}{\gamma_{m,t}'} + \frac{1}{\gamma_{i,t}'} - 2\right)$ when $N=n+1$. Note that $\gamma_{i,t}'$ can also be treated as a constant here.

Informally speaking, we observe that when $N$ increases, the term $C_T$ generally decreases in its scale if the added client has contribution $\gamma_{m,t}'$ at least the **“average contribution”** of the existing clients $\frac{1}{n} \sum_{m=1}^{n} \gamma_{m,t}'$. Therefore, the decrease in $C_T$ tightens the performance bound in Theorem 1 and makes the convergence faster.

An exception that we can derive from the above is that when the added clients contribute badly with a very low $\gamma_{m,t}'$, it could slow down the convergence despite the increase in $N$. In practice, this implies that the model update provided by the added client is bad, harmful, or even adversarial, and is detected by the contribution evaluation measure. Theoretically, the convergence is adversely affected if we add "harmful" clients. However, we could easily implement client selection methods based on this insight on contribution evaluation to filter such clients during the training process to ensure faster convergence when $N$ grows.

Concluding, we hope our discussion above about the impact of $N$ on convergence speed will improve your opinion about our work.

Reviewer comment (Theory):
It would be helpful to simplify the bound and make the dependence on $N$ explicit.

Author response:
We greatly appreciate the reviewer's time and effort in providing constructive feedback on our paper, and we are encouraged by the high scores assigned to the soundness and presentation of our paper. We are also delighted to see that our scope of experiments is extensive.

We hope that our responses to your concerns as follows will further enhance your perception of our work.

>Q1: "Does convergence speed become faster as the number of agents $N$ grows?"

The simple answer is, adding clients that **"help than hurt"** is going to improve the convergence speed.

To elaborate, we first observe that it might be difficult to further simplify the bound because it has a dependence on every single client's contribution $\gamma_{m,t}'$ for $m \in [N]$. However, we can derive some insights on the convergence speed as $N$ grows.

The effect depends on the **contribution levels of the clients**.
We can look at the term $C_T = \frac{H_T}{N} \sum_{m=1}^N \sum_{t=1}^{T} \left(\frac{T+\alpha}{t+\alpha}\right)^2 \left(\frac{1}{\gamma_{m,t}'} + \frac{1}{\gamma_{i,t}'} -2 \right)$, when $N$ increases, the denominator becomes bigger but at the same time there are additional terms in the summation.

We can consider a case where we increase $N=n$ to $N=n+1$. Abstracting away the constants $H_T$, $T$ and $\alpha$ for simplicity, we can make a comparison for $C_T = \frac{1}{n} \sum_{m=1}^{n} \left(\frac{1}{\gamma_{m,t}'} + \frac{1}{\gamma_{i,t}'} - 2 \right)$ when $N=n$ against the case for $C_T = \frac{1}{n+1} \sum_{m=1}^{n+1} \left(\frac{1}{\gamma_{m,t}'} + \frac{1}{\gamma_{i,t}'} - 2\right)$ when $N=n+1$. Note that $\gamma_{i,t}'$ can also be treated as a constant here.

Informally speaking, we observe that when $N$ increases, the term $C_T$ generally decreases in its scale if the added client has contribution $\gamma_{m,t}'$ at least the **“average contribution”** of the existing clients $\frac{1}{n} \sum_{m=1}^{n} \gamma_{m,t}'$. Therefore, the decrease in $C_T$ tightens the performance bound in Theorem 1 and makes the convergence faster.

An exception that we can derive from the above is that when the added clients contribute badly with a very low $\gamma_{m,t}'$, it could slow down the convergence despite the increase in $N$. In practice, this implies that the model update provided by the added client is bad, harmful, or even adversarial, and is detected by the contribution evaluation measure. Theoretically, the convergence is adversely affected if we add "harmful" clients. However, we could easily implement client selection methods based on this insight on contribution evaluation to filter such clients during the training process to ensure faster convergence when $N$ grows.

Concluding, we hope our discussion above about the impact of $N$ on convergence speed will improve your opinion about our work.

Reviewer comment (Experiments):
Does adding more clients lead to a faster convergence rate?

Author response:
We greatly appreciate the reviewer's time and effort in providing constructive feedback on our paper, and we are encouraged by the high scores assigned to the soundness and presentation of our paper. We are also delighted to see that our scope of experiments is extensive.

We hope that our responses to your concerns as follows will further enhance your perception of our work.

>Q1: "Does convergence speed become faster as the number of agents $N$ grows?"

The simple answer is, adding clients that **"help than hurt"** is going to improve the convergence speed.

To elaborate, we first observe that it might be difficult to further simplify the bound because it has a dependence on every single client's contribution $\gamma_{m,t}'$ for $m \in [N]$. However, we can derive some insights on the convergence speed as $N$ grows.

The effect depends on the **contribution levels of the clients**.
We can look at the term $C_T = \frac{H_T}{N} \sum_{m=1}^N \sum_{t=1}^{T} \left(\frac{T+\alpha}{t+\alpha}\right)^2 \left(\frac{1}{\gamma_{m,t}'} + \frac{1}{\gamma_{i,t}'} -2 \right)$, when $N$ increases, the denominator becomes bigger but at the same time there are additional terms in the summation.

We can consider a case where we increase $N=n$ to $N=n+1$. Abstracting away the constants $H_T$, $T$ and $\alpha$ for simplicity, we can make a comparison for $C_T = \frac{1}{n} \sum_{m=1}^{n} \left(\frac{1}{\gamma_{m,t}'} + \frac{1}{\gamma_{i,t}'} - 2 \right)$ when $N=n$ against the case for $C_T = \frac{1}{n+1} \sum_{m=1}^{n+1} \left(\frac{1}{\gamma_{m,t}'} + \frac{1}{\gamma_{i,t}'} - 2\right)$ when $N=n+1$. Note that $\gamma_{i,t}'$ can also be treated as a constant here.

Informally speaking, we observe that when $N$ increases, the term $C_T$ generally decreases in its scale if the added client has contribution $\gamma_{m,t}'$ at least the **“average contribution”** of the existing clients $\frac{1}{n} \sum_{m=1}^{n} \gamma_{m,t}'$. Therefore, the decrease in $C_T$ tightens the performance bound in Theorem 1 and makes the convergence faster.

An exception that we can derive from the above is that when the added clients contribute badly with a very low $\gamma_{m,t}'$, it could slow down the convergence despite the increase in $N$. In practice, this implies that the model update provided by the added client is bad, harmful, or even adversarial, and is detected by the contribution evaluation measure. Theoretically, the convergence is adversely affected if we add "harmful" clients. However, we could easily implement client selection methods based on this insight on contribution evaluation to filter such clients during the training process to ensure faster convergence when $N$ grows.

Concluding, we hope our discussion above about the impact of $N$ on convergence speed will improve your opinion about our work.

Reviewer comment (Theory):
What it means for a client to be incentivized is not well-defined in this paper. From Proposition 1, it seems as long as the gradient of client $i$'s utility w.r.t. its contribution $p_{i}$ is higher than that under standard FL mechanism, we say the client $i$ is incentivized. It is not clear why we, as the designer of the mechanism, cares about whether the gradient of utility for each client is higher than what the client gets under a standard FL mechanism. Instead, a more natural goal is to incentivize the clients to contribute to FL using their full capacity in order to get the best learning outcome. In this regard, the intrinsic cost $c_{i}$ of each client also plays an important role, i.e., it is possible that the cost value is high, such that we end up with a negative gradient of the utility (contributing more leads to even lower utility). Therefore, a rational client will decide to contribute $p=0$ in this case, which affects the convergence of the FL process.

Author response:
>W1(a): "What it means for a client to be incentivized"

In game theory, incentives are typically based on the potential for gaining a benefit or circumventing a cost. Consequently, a rational client is incentivized to perform an action when there is a positive gain in utility for doing so. We formalize this as Individual Rationality (IR) in Definition 1, implying that clients are incentivized to participate in our IAFL scheme, because every client will receive a reward at least as good as what they can achieve on their own. IR is an established concept for player incentivization in game theory.

>W1(b): "A more natural goal is to incentivize the clients to contribute to FL using their full capacity"

Incentivizing clients to contribute to the "full capacity" for the best learning outcome is indeed the most ideal, like the reviewer has said. However, every client incurs costs to contribute and the actual contribution of clients depends on the interplay between rewards and costs for different contribution levels.

Hence, in our work, we propose the next achievable alternative: To incorporate the goal of **"incentivizing clients to contribute as much as possible"** on top of satisfying IR. To this end, we designed IAFL such that Theorem 1 and 2 are fulfilled. These theorems suggest that clients will be rewarded with better models if they contribute more. This will incentivize clients to further contribute to their full capacity for improved model performance received, of course, subject to their marginal utility increment in the presence of a cost $c_i$. Clients will contribute up till the marginal increment in reward still surpasses the associated cost.

This is **important to the FL designer** because higher contributions from clients (e.g., in the form of better gradient updates, more frequent participation, etc.) could potentially lead to improved model performance for all clients (i.e., a better learning outcome).

>W1(c): "The intrinsic cost $c_i$ of each client also plays an important role"

The reviewers' observations regarding the critical role of $c_i$ are indeed accurate. As specified by Proposition 1, in the presence of a cost $c_i$, our IAFL incentivizes more than the standard FL.

When $c_i$ is indeed high, our IAFL is still better than standard FL, since this cost is borne by both IAFL and standard FL. A client with a high cost $c_i$ will not participate in either IAFL or standard FL. Likewise, in the event of a strictly $p=0$ contribution, we can simply leave the client out of the collaboration to avoid detrimental effects on collaborative model training and convergence. Alternatively, the sharing parameter $\kappa$ presented in Equation (1) of the paper could be raised to enhance the convergence of the client models.

Therefore, this weakness is not exclusive to our framework, and notably, our IAFL even offers potential remedies. In practice, if a rational client finds it possible to benefit from standard FL, they will definitely benefit more from our framework. Hence, our framework is deemed reasonable.

Reviewer comment (Theory):
Can the authors elaborate on, in Theorem 1, why $o(1/\gamma_{i,T}^{\prime})$ suffices to make the convergence hold? Where did the $(T+\alpha)^{2}$ in the numerator go?

Author response:
>Q1: "Why $o(1/\gamma_{i,T}')$ suffices to make the convergence hold" and "where did the $(T+\alpha)^2$ go"


We are sorry that there is an unintentional typo that occurred while organizing the formula to make it more concise, the term $T^2$ was accidentally omitted.

Referring to the paragraph on "error propagation and non-convergence", instead of $(H_T/T)\sum_{t=1}^T 1/t^2=O(T)$, it should be $(H_T/T)\sum_{t=1}^T T^2/t^2=O(T)$.

With $H_T T^2/T = H_T T = O(T)$ and $1/{\gamma_{i,T}'}= o(1/T)$, they are jointly required for the convergence to 0. This typo does not affect any theoretical results in Theorem 1 or any subsequent theoretical results, and we hope to seek understanding from the reviewer for any unintentional confusion caused.