Paper: Fixed-Budget Differentially Private Best Arm Identification (ICLR-2024)

Reviewer comment (Experiments):
The empirical evaluation is limited on a particular synthetic data instance.

Author response:
We acknowledge the limitation pointed out by the reviewer, and this is indeed an aspect that we are mindful of. The focus of our paper is primarily theoretical, aiming to establish and explore conceptual frameworks within differentially private BAI. Although the key intent of our work is to provide a strong theoretical foundation for future research and discussion, keeping in mind the importance of empirical validation, we benchmark the performance of our algorithm against existing non-private algorithms or the private adaptations thereof. We sincerely hope that the reviewer perceives this stance of ours optimistically.

Reviewer comment (Novelty):
Motivation to consider DP-BAI algorithm. For my understanding, DP is for preventing potential privacy risk when sharing statistics of a dataset. I would be great for the author to provide motivation to consider DP in bandit problem, especially on the best arm identification task.

Author response:
Our conceptualization of differential privacy within the context of bandit problems aligns closely with established literature [1,2,3,4,5,6], particularly drawing inspiration from the work of Nikolakakis et al. [3]. While the primary objective of our work is fixed-budget BAI, it is noteworthy that the objectives in other relevant studies center around either regret minimization or fixed-confidence BAI. Importantly, these alternative settings necessitate unique analytical methodologies, yielding results distinct in nature from our own.

The underlying principle of privacy in the aforementioned body of literature, including our work, may be articulated as follows: an alteration in one of the rewards received by the learning agent should not largely influence subsequent decisions regarding the pulling of arms. To illustrate this point further, certain works posit a scenario where each time step is associated with an independent "user"; pulling arm $i$ at time $t$ provides the agent with user $t$'s "score" of arm $i$. If the score from a preceding user significantly impacts subsequent arm selections by the agent, there exists a potential inference risk for later users regarding the (private) scores of their predecessors through observations of the arm selections. Such privacy leakage can be mitigated by the application of differential privacy mechanisms tailored for bandit scenarios. In the context of the BAI task, the "agent" can be perceived as a surveyor seeking to identify the optimal item from a set of $K$ items in a market survey, while the "users" can be likened to customers participating in the survey.

Reviewer comment (Writing):
Intuition on scheme (7). I had difficulty to make sense on the scheme (7) and would be thankful is author can provide explanations.

Author response:
We thank the reviewer for the comment. Scheme (7) defines the variable $s_p$, which is the pre-configured number of active arms in phase $p$. 
Following are some details of (7) assuming $K \gg d^2$. 
Note that for $p=1$, we have $s_1=h_0+g_0=K$, where $g_0=\Theta(d^2)$, i.e., there $K$ active arms in phase 1 (the default initialisation).
Then, for phase $p \in \{2,\ldots,M_1\}$, we have from (7) that
$$
s_{M_1}=g_0, \quad 
\forall p < M_1, ~ \frac{ s_{p+1}-g_0 }{ s_{p}-g_0 }=\frac{h_{p+1}}{h_{p}} \approx \frac{1}{\lambda} ,
$$
which means that if $s_p \gg g_0=\Theta(d^2)$ for some $p$, then the agent eliminates approximately $1-\frac{1}{\lambda}$ fraction of the current active arms at end of phase $p$. Therefore, from phase $1$ until phase $M_1$, the number of active arms reduces from $K$ to $\Theta(d^2)$, and the shrinkage rate is approximately $1-\frac{1}{\lambda}$ in a majority of these $M_1$ phases. Here, $\lambda$ is chosen to ensure that $M_1 = \Theta(\log d)$.
Lastly, for phase $p\in \{M_1+1,\ldots,M\}$, we have from (7) that $\frac{s_{p+1}}{s_p}\approx \frac{1}{2}$. That is, the agent eliminates approximately half of the active arms in each phase, and $M$ is chosen in such a way that $s_{M+1}=1$, i.e., only one active arm remains after $M$ phases. For clarity, we provide a numerical illustration of the above specifications in the newly created Appendix B of the revised manuscript.

Reviewer comment (Evaluation):
What makes the apparent dimension go down for the first few rounds?

Author response:
We thank the reviewer for the suggestion. We have now provided additional explanations in the newly created Appendix B of our revised paper (highlighted in blue).  Firstly, we would like to clarify that in the first few rounds (precisely $M_1$ rounds), the dimension may not go down as the number of arms is at least $\Theta(d^2)$. Secondly, as pointed out by the reviewer, "decreasing the span" is indeed critical in identifying the best arm. This is because "decreasing the span" increases the probability that the private empirical mean of the best arm surpasses that of others, as is shown in Lemma E.2 of the original manuscript ( or Lemma G.2 of the revised version). 

We remain open to elucidating any particular facet of our algorithm should the reviewer express a desire for further explanations.

Reviewer comment (Theory):
How does decreasing the span basis vectors of the arm space lead to convergence to the optimal arm?

Author response:
We thank the reviewer for the suggestion. We have now provided additional explanations in the newly created Appendix B of our revised paper (highlighted in blue).  Firstly, we would like to clarify that in the first few rounds (precisely $M_1$ rounds), the dimension may not go down as the number of arms is at least $\Theta(d^2)$. Secondly, as pointed out by the reviewer, "decreasing the span" is indeed critical in identifying the best arm. This is because "decreasing the span" increases the probability that the private empirical mean of the best arm surpasses that of others, as is shown in Lemma E.2 of the original manuscript ( or Lemma G.2 of the revised version). 

We remain open to elucidating any particular facet of our algorithm should the reviewer express a desire for further explanations.

Reviewer comment (Theory):
I am skeptical about their definition of DP since it is defined over length $T$ sequences. I would expect that in the online setting this definition would only be defined over a sample of sequence starting from the time when the reward sequences differ as done in joint differential privacy.

Author response:
We thank the reviewer for this question. Our definition of DP indeed aligns with your expectation of ``over a sample of sequence starting from the time when the reward sequences differ''. Intuitively, this is because the future rewards have no impact on the past sequence of arm pulls. Below are further details. 

Our definition of DP does not explicitly entail time steps; note that $X_{i,j}$ in Section 2 represents the $j$-th reward from arm $i$ (rather than the classical representation of reward from arm $i$ at time step $j$). In spite of the minor difference in the representation of rewards, our definition of DP is, in fact, equivalent to the notion of DP arising from the classical representation of rewards (which we call **$\varepsilon$-table-DP** borrowing the terminology from Azize et al. [1]). We prove this formally in the newly created Appendix C of the revised manuscript by introducing a **dual decision maker** who observes reward $X_{A_t,t}$ at time $t$ instead of $X_{A_t, N_{A_t, t}}$ as in our work. We then show that for any $\varepsilon$-DP policy $\pi$ of the decision maker as defined in our work, its dual decision maker's policy (or simply the dual policy) $D(\pi)$ meets the $\varepsilon$-table-DP criterion, and vice-versa (see Propositions C.4 and C.5 in the revised paper).  

We note here that table-DP is based on ``a sample of sequence starting from the time when the reward sequences differ'' highlighted by the reviewer; see Corollary C.3 of the newly created Appendix C for further details.

Reviewer comment (Writing):
I would like an intuitive explanation of why their algorithm works on a small example somewhere in the paper (maybe, in an appendix).

Author response:
We thank the reviewer for the suggestion. We have now provided additional explanations in the newly created Appendix B of our revised paper (highlighted in blue).  Firstly, we would like to clarify that in the first few rounds (precisely $M_1$ rounds), the dimension may not go down as the number of arms is at least $\Theta(d^2)$. Secondly, as pointed out by the reviewer, "decreasing the span" is indeed critical in identifying the best arm. This is because "decreasing the span" increases the probability that the private empirical mean of the best arm surpasses that of others, as is shown in Lemma E.2 of the original manuscript ( or Lemma G.2 of the revised version). 

We remain open to elucidating any particular facet of our algorithm should the reviewer express a desire for further explanations.

Reviewer comment (Theory):
Finally, the central idea of the paper is to use D-optimal design rather than G-optimal design. These ideas are theoretically equivalent (see Proposition~3 in Soare et al. NeurIPs 2014) but this paper demonstrates a dramatic performance improvement in their numerical experiments. What is the intuitive explanation for this?

Author response:
As pointed out by the reviewer, D-optimal design and G-optimal design are indeed theoretically equivalent. However, our Max-Det principle is rather different from D-optimal design. Note that $\gamma^*: \mathcal{A} \rightarrow [0,1]$ is a D-optimal design of vector set $\mathcal{A}$ if 
$$
\gamma^* = \arg\max_{\gamma} {\rm det}(V(\gamma)), 
\quad\mbox{where} \quad 
V(\gamma) := \sum_{a\in \mathcal{A}} \gamma(a) \\, aa^\top.
$$
There are two primary differences between D-optimal design and the Max-Det collection (defined in Definition 3.1). Firstly, the matrices of which the determinants are calculated are different. In D-optimal design, this matrix is formed by the weighted sum of some rank-one matrices of the form $aa^\top$ for $a \in \mathcal{A}$, whereas in Definition 3.1, this matrix is formed by stacking up a subset of vectors from $\mathcal{A}$ as its columns. Secondly, in D-optimal design, the weight of each vector (i.e., $\gamma^*(a)$ for $a\in \mathcal{A}$) may be any value in  $[0,1]$. In contrast, in Definition 3.1, this ``weight'' of each vector in $\mathcal{A}$ can be regarded as being $0$ or $1$, depending on whether or not it belongs to the Max-Det collection.

We appreciate the reviewer's question for providing us with the opportunity to clarify the differences between D-optimal design and our Max-Det principle.

Reviewer comment (Theory):
Page 4, Definition 3.1: Why is a Max-Det collection of $\mathcal{A}$ always linearly independent raises questions. Does this implicitly assume that $\mathcal{A}$ spans $\mathbb{R}^{d'}$?

Author response:
The reviewer is verily right in these two questions. We apologize for the typos, and have revised the above highlighted sentences in Section 3 (please see the parts in blue) accordingly. Briefly: (1) Yes, we only consider the case in which  $\mathcal{A}$ spans $\mathbb{R}^{d'}$. (2) Yes, your understanding of Line 8 is correct, and "randomly" was a typo appearing in the earlier description of Line 14; we have now rectified this in the revised version. We express our sincere gratitude to the reviewer for their careful reading of our manuscript.

Reviewer comment (Theory):
Can the authors provide some high-level intuition behind the choice of $g_i$ and $h_i$ as described in equations (5) and (6)?

Author response:
The rationale behind the choices of $g_i$ and $h_i$ is rooted in their role as auxiliary parameters governing the number of active arms in each phase, as outlined in (7). Adhering to the principle of successive elimination (SE) [1,2], our algorithm aims to eliminate a specific proportion of active arms in each of the $\Theta(\log d)$ phases. The determination of this proportion in each phase is crucial for achieving the overall performance guarantee. In essence, when the number of active arms in any given phase significantly exceeds $\Theta(d^2)$, the agent eliminates approximately $1-\frac{1}{\lambda}$ proportion of active arms; otherwise, it eliminates roughly half of the active arms. The threshold $\Theta(d^2)$ is pivotal in the Max-Det principle analysis. The parameter $\lambda$ in (4) is chosen to ensure that $\Theta(\log d)$ phases transpire before the number of active arms reduces to $\Theta(d^2)$ in cases where $K \gg d^2$. This leads to $g_i \approx g_{i-1}/2$ and $h_i \approx h_{i-1}/\lambda$ in equations (5) and (6), respectively. These specifications are numerically illustrated in the newly created Appendix B for clarity.

Reviewer comment (Novelty):
The paper should provide more discussion on the connections between this problem and Best Arm Identification (BAI) in the fixed-confidence regime.

Author response:
Thank you for the suggestion. Below are some comparisons between the different bandit settings.

(1) **Fixed-budget BAI and regret minimization**: While regret minimization inherently entails an exploration vs. exploitation dilemma, BAI (encompassing fixed-budget and fixed-confidence regimes) on the other hand is centered around pure exploration. As astutely demonstrated in the work of Zhong et al. [1], no algorithm can perform optimally for both of the above objectives simultaneously. Thus, for instance, an algorithm achieving the minimum probability of error in identifying the best arm with a fixed budget is inevitably accompanied by a larger regret value, and vice versa; for more details, see [1, Theorem 5.1]. Consequently, in the nuanced context of our work, which involves additional considerations such as differential privacy, a mere transposition of ideas from regret minimization proves insufficient to devise algorithms with near-optimal performance for the rather distinct objective of BAI.


(2) **Fixed-budget BAI and fixed-confidence BAI**: In the domain of pure exploration problems, two pivotal considerations, namely (a) the probability of error and (b) the stopping time required to ascertain the best arm, compete with one another. In the fixed-budget regime, a deterministic time (a.k.a. budget) $T$ is fixed, and the goal is to minimise the probability of error in determining the best arm after $T$ time steps. On the other hand, in the fixed-confidence regime, given a threshold $\delta \in (0,1)$ (a.k.a. confidence level), the goal is to minimise the expected stopping time subject to the error probability not exceeding $\delta$. 
A notable challenge in fixed-confidence scenarios lies in the determination of the expected stopping time, which is typically unknown to the agent. This is because the upper bound of the expected stopping time is, in general, a function of an unknown problem-specific ``hardness'' parameter, as evidenced in various works such as [2,3,4,5]. Consequently, establishing a straightforward relationship between $\delta$ in the fixed-confidence regime and the budget $T$ in the fixed-budget regime is substantially intricate. 

Furthermore, it is noteworthy that while the asymptotic complexity of fixed-confidence BAI has been comprehensively characterised in the asymptotic limit as $\delta \downarrow 0$ by Garivier and Kaufmann [2], a corresponding characterisation for fixed-budget BAI (where the asymptotics is as $T \to \infty$) remains an open problem, as articulated by Qin [6]. 
Hence, the transposition of ideas from fixed-confidence BAI into the realm of fixed-budget BAI appears nontrivial due to the above fundamental disparities.

Reviewer comment (Novelty):
The paper should provide more discussion on the connections between this problem and, more generally, Multi-Armed Bandits (MAB) under Differential Privacy (DP).

Author response:
Finally, we would like to emphasize that fixed-budget BAI remains a considerable challenge in multi-armed bandits, and continues to be actively investigated; see, for instance, the recent works [6,7,8,9,10]. To the best of our knowledge, our work is the first attempt towards incorporating DP considerations within the framework of fixed-budget BAI. The amalgamation of these two aspects, namely fixed-budget BAI and DP, introduces numerous analytical challenges. Despite these challenges, we provide a complete characterization of the exponent of the probability of error up to universal constants.

[1] Zhong, Z., Cheung, W. C., and Tan, V. (2023). Achieving the Pareto frontier of regret minimization and best arm identification in multi-armed bandits. Transactions on Machine Learning Research.

[2] Garivier, A., and Kaufmann, E. (2016, June). Optimal best arm identification with fixed confidence. In Conference on Learning Theory (pp. 998-1027). PMLR.

[3] Wang, P. A., Tzeng, R. C., and Proutiere, A. (2021). Fast pure exploration via Frank--Wolfe. Advances in Neural Information Processing Systems, 34, 5810-5821.

[4] Nikolakakis, K. E., Kalogerias, D. S., Sheffet, O., Sarwate, A. D. (2021). Quantile multi-armed bandits: Optimal best-arm identification and a differentially private scheme. IEEE Journal on Selected Areas in Information Theory, 2(2), 534-548.

[5] Hou, Y., Tan, V. Y., and Zhong, Z. (2022). Almost optimal variance-constrained best arm identification. IEEE Transactions on Information Theory, 69(4), 2603-2634.

[6] Qin, C. (2022, September). Open Problem: Optimal Best Arm Identification with Fixed-Budget. In Conference on Learning Theory (pp. 5650-5654). PMLR.

[7] Yang, J., and Tan, V. (2022). Minimax Optimal Fixed-Budget Best Arm Identification in Linear Bandits. Advances in Neural Information Processing Systems, 35, 12253-12266.

[8] Komiyama, J., Tsuchiya, T., and Honda, J. (2022). Minimax optimal algorithms for fixed-budget best arm identification. Advances in Neural Information Processing Systems, 35, 10393-10404.

[9] Barrier, A., Garivier, A., and Stoltz, G. (2023, February). On best-arm identification with a fixed budget in non-parametric multi-armed bandits. In International Conference on Algorithmic Learning Theory (pp. 136-181). PMLR.

[10] Lalitha, A. L., Kalantari, K., Ma, Y., Deoras, A., and Kveton, B. (2023, July). Fixed-Budget Best-Arm Identification with Heterogeneous Reward Variances. In Uncertainty in Artificial Intelligence (pp. 1164-1173). PMLR.

Reviewer comment (Novelty):
It is not very clear whether or not there exist deeper connections between this problem and the above topics, even if they are superficially different.

Author response:
Thank you for the suggestion. Below are some comparisons between the different bandit settings.

(1) **Fixed-budget BAI and regret minimization**: While regret minimization inherently entails an exploration vs. exploitation dilemma, BAI (encompassing fixed-budget and fixed-confidence regimes) on the other hand is centered around pure exploration. As astutely demonstrated in the work of Zhong et al. [1], no algorithm can perform optimally for both of the above objectives simultaneously. Thus, for instance, an algorithm achieving the minimum probability of error in identifying the best arm with a fixed budget is inevitably accompanied by a larger regret value, and vice versa; for more details, see [1, Theorem 5.1]. Consequently, in the nuanced context of our work, which involves additional considerations such as differential privacy, a mere transposition of ideas from regret minimization proves insufficient to devise algorithms with near-optimal performance for the rather distinct objective of BAI.


(2) **Fixed-budget BAI and fixed-confidence BAI**: In the domain of pure exploration problems, two pivotal considerations, namely (a) the probability of error and (b) the stopping time required to ascertain the best arm, compete with one another. In the fixed-budget regime, a deterministic time (a.k.a. budget) $T$ is fixed, and the goal is to minimise the probability of error in determining the best arm after $T$ time steps. On the other hand, in the fixed-confidence regime, given a threshold $\delta \in (0,1)$ (a.k.a. confidence level), the goal is to minimise the expected stopping time subject to the error probability not exceeding $\delta$. 
A notable challenge in fixed-confidence scenarios lies in the determination of the expected stopping time, which is typically unknown to the agent. This is because the upper bound of the expected stopping time is, in general, a function of an unknown problem-specific ``hardness'' parameter, as evidenced in various works such as [2,3,4,5]. Consequently, establishing a straightforward relationship between $\delta$ in the fixed-confidence regime and the budget $T$ in the fixed-budget regime is substantially intricate. 

Furthermore, it is noteworthy that while the asymptotic complexity of fixed-confidence BAI has been comprehensively characterised in the asymptotic limit as $\delta \downarrow 0$ by Garivier and Kaufmann [2], a corresponding characterisation for fixed-budget BAI (where the asymptotics is as $T \to \infty$) remains an open problem, as articulated by Qin [6]. 
Hence, the transposition of ideas from fixed-confidence BAI into the realm of fixed-budget BAI appears nontrivial due to the above fundamental disparities.