Paper: On Differentially Private Federated Linear Contextual Bandits (ICLR-2024)

Reviewer comment (Presentation):
There appears to be a spacing issue in the section discussing the tree-based mechanism, possibly resulting from the authors' use of the vspace command in LaTeX.

Author response:
**Placement of the concluding remarks and use of vspace:** Fixed both. The conclusion is now a separate section in our new version.

Reviewer comment (Novelty):
The novelty and inspiration of the problem itself could be further developed. The concept of silo-level Local Differential Privacy (LDP), while important and rigorous, may be considered relatively straightforward to formulate.

Author response:
**On our new results:** We agree with the reviewer that silo-level LDP and SDP are relatively straightforward to formulate since these notions are widely adopted in supervised learning problems. In this work, we adapt them to a sequential learning problem of linear contextual bandits.

However, in order to establsih our regret bounds, we have introduced two new results, since directly applying exsiting techniques or algorithms for supervised learning do not work in sequential learning.


- First, we point out that simply adding an additional shuffler between the agents and server to Algorithm 1 in Dubey & Pentland, 2020  is not sufficient to achieve "optimal" regret under SDP. Instead, the shuffle mechanism must be thoughtfully integrated into the tree-based algorithm to shuffle the p-sums rather than prefix sums (which is the case for standard shuffle mechanism).

- Second, we highlight that one cannot merely rely on existing privacy amplification lemmas to achieve SDP. We have derived a **new result** on privacy amplification for shuffling DP mechanisms where the dataset size is greater than one, as opposed to the standard results where the dataset size is equal to one.

Finally, for the regret analysis, our approach to bound the "local drift" under fixed-batch schedule is different from the approach under adaptive schedule.

Reviewer comment (Novelty):
The solutions presented in the paper largely draw upon existing results and algorithmic designs, rather than introducing fundamentally new insights or modifications to established findings, which limits the potential of this paper.

Author response:
**On our new results:** We agree with the reviewer that silo-level LDP and SDP are relatively straightforward to formulate since these notions are widely adopted in supervised learning problems. In this work, we adapt them to a sequential learning problem of linear contextual bandits.

However, in order to establsih our regret bounds, we have introduced two new results, since directly applying exsiting techniques or algorithms for supervised learning do not work in sequential learning.


- First, we point out that simply adding an additional shuffler between the agents and server to Algorithm 1 in Dubey & Pentland, 2020  is not sufficient to achieve "optimal" regret under SDP. Instead, the shuffle mechanism must be thoughtfully integrated into the tree-based algorithm to shuffle the p-sums rather than prefix sums (which is the case for standard shuffle mechanism).

- Second, we highlight that one cannot merely rely on existing privacy amplification lemmas to achieve SDP. We have derived a **new result** on privacy amplification for shuffling DP mechanisms where the dataset size is greater than one, as opposed to the standard results where the dataset size is equal to one.

Finally, for the regret analysis, our approach to bound the "local drift" under fixed-batch schedule is different from the approach under adaptive schedule.

Reviewer comment (Theory):
It appears that the algorithm design in this paper heavily relies on a binary-tree-based mechanism, which can only be applied to linear setting. Is it possible to generalize the algorithm or the analysis to nonlinear models, such as Generalized Linear Models (GLM)?

Author response:
Thanks for your positive evaluation of our paper! Regarding your sharp comment on the binary-tree-based algorithm, here is our clarification.

The key behind the binary-tree-based mechanism is the summation structure in parmater estimates. For linear models, the least-square estimate can be written in closed-form using two summations, one over bias vectors and another over covariance matrices. However, for GLM, the maximum likelihood estimate doesn't have a closed-form based on these summations. As a result, one needs to resort to other privacy mechanisms (e.g., objective perturbation), see [R1].

[R1] Chen, Xi, David Simchi-Levi, and Yining Wang. "Privacy-preserving dynamic personalized pricing with demand learning." Management Science 68, no. 7 (2022): 4878-4898.

Reviewer comment (Theory):
Related to the first point, it would be nice to add some discussions on whether the $M^{1/4}$ gap from centralized performance can be closed; otherwise, the significance of the result is hard to measure.

Author response:
**Close the gap under silo-level LDP:** Thanks for the keen comment. It turns out that the first step to obtain a lower bound for Federated LCBs under silo-level LDP is to establish a tight characterization of regret for single-agent LCBs under central JDP.

To see this, one can use a similar reduction as in [Lowy & Razaviyayn' 2021], where the authors derive a lower bound in the supervised learning setting under silo-level LDP. Specifically, for any silo-level LDP algorithm $\mathcal{A}$ with privacy guarantee $\epsilon$, one can first "virtually" shuffle all $MT$ user sequences and then apply $\mathcal{A}$, leading to a shuffled version $\mathcal{A}_s$. 
The shuffled version algorithm $\mathcal{A}_s$ has an SDP privacy guarantee of roughly $\epsilon/\sqrt{M}$. Since SDP implies central JDP in LCBs, one can conclude that $\mathcal{A}_s$ has a lower bound of $L_c(\epsilon/\sqrt{M})$, where  $L_c(\epsilon)$ 
denotes the regret lower bound for LCBs under central JDP with privacy $\epsilon'$. Since $\mathcal{A}$ and $\mathcal{A}'$ have same regret performance, this yields a regret lower bound $L_c(\epsilon/\sqrt{M})$ for $\mathcal{A}$ under silo-level LDP.

To our best knowledge, the existing lower bound for LCBs under central JDP is $\Omega(\sqrt{T} + 1/\epsilon')$ [R0], which implies a lower bound $L_c(\epsilon')=\Omega(\sqrt{MT} + 1/\epsilon')$ in the centralized setting (super-single agent). By the above argument, setting $\epsilon'=\epsilon/\sqrt{M}$, this implies a lower bound $\Omega(\sqrt{MT} + \sqrt{M}/\epsilon)$ under silo-level LDP, whereas our upper bound is $O(M^{3/4}\sqrt{T/\epsilon})$. That is, privacy cost in lower bound is only additive $\sqrt{M}/\epsilon$, whereas in our upper bound is a multiplicative $M^{1/4}/\sqrt{\epsilon}$. It is unclear to us which one of these is loose. Hence, whether the regret gap can be closed under silo-level LDP without resorting to SDP remains an open question.

 Now, if one can prove a lower bound $\Omega(\sqrt{T/\epsilon'})$ for single-agent LCBs under central JDP, then it would yield a lower bound $L_c(\epsilon') =\Omega(\sqrt{MT/\epsilon'})$ in the centralized setting, which would further imply a lower bound of $\Omega(M^{3/4}\sqrt{T/\epsilon})$ under silo-level LDP, and would close the regret gap.

 We have included this discussion in Appendix G.1. We believe our findings in this paper (e.g., identifying existing gaps and establishing new upper bounds) would motivate these interesting open questions.

[R0] Jiahao He, Jiheng Zhang, and Rachel Zhang. A reduction from linear contextual bandit lower bounds
to estimation lower bound, ICML'22

Reviewer comment (Experiments):
In terms of the DP techniques, I understand that there are many different choices and this work adopts two certain ones. It would be nice to clarify whether the adopted ones are necessary or if there are other feasible choices.

Author response:
**Other DP techniques:** One can indeed have the flexibility of choosing other DP techniques in our privacy protocol. In fact, this is the beauty of our proposed protocol. For example, one may replace our Vallina tree-based algorithm with other advanced techniques to improve constant factors in privacy parameters, e.g., a low-variance tree-based algorithm [R1] or some online matrix mechanisms [R2, R3]. For silo-level LDP, instead of Gaussian noise, one can use Wishart Noise as in [Sharif&Sheffet'18]. For SDP, instead of vector sum protocol, one can use some advanced SDP protocols, e.g., [R4, R5]

[R1] James Honaker. Efficient use of differentially private binary trees. Theory and Practice of
Differential Privacy (TPDP 2015), London, UK, 2015

[R2] Denisov, S., McMahan, H. B., Rush, J., Smith, A., & Guha Thakurta, A. (2022). Improved differential privacy for sgd via optimal private linear operators on adaptive streams. Advances in Neural Information Processing Systems, 35, 5910-5924.

[R3] Fichtenberger, H., Henzinger, M., & Upadhyay, J. (2022). Constant matters: Fine-grained Complexity of Differentially Private Continual Observation. arXiv preprint arXiv:2202.11205.


[R4] Ghazi, B., Kumar, R., Manurangsi, P., and Pagh, R. Private
counting from anonymous messages: Near-optimal accuracy with vanishing communication overhead. In International Conference on Machine Learning, pp. 3505–3514.
PMLR, 2020.

[R5] Balle, B., Bell, J., Gascon, A., and Nissim, K. Private summation in the multi-message shuffle model. In Proceedings of the 2020 ACM SIGSAC Conference on Computer
and Communications Security, pp. 657–676, 2020

Reviewer comment (Experiments):
The fixed batch size selection potentially can be improved. Although performing adaptive communication protocol is no longer feasible, it might still be a good choice to have the batch length exponentially growing (as in many low-switching bandit studies) instead of being a fixed one.

Author response:
**Exponentially growing batch:** Thanks for the comment. In contrast to standard MABs and linear bandits with stochastic contexts, it is not clear to us how to use exponentially growing batch schedule to derive meaningful regret bound in the setting of linear bandit with *adversarial contexts* and infinite actions (which is the setting of this paper). The only method we are aware of is the *rarely switching principle* (e.g. determinant trick) in [R1], which is the core idea behind adaptive communication in linear bandits, but not feasible in our case due to potential risk of privacy leakage. In fact, [R2] lists an open problem regarding the determinant trick under DP, even in the single-agent setting.

[R1] Abbasi-Yadkori, Y., Pál, D., & Szepesvári, C. "Improved algorithms for linear stochastic bandits". NeurIPS'11.

[R2] Chowdhury, Sayak Ray, and Xingyu Zhou. "Shuffle private linear contextual bandits." ICML 2022