Paper: Variance-aware Regret Bounds for Stochastic Contextual Dueling Bandits (ICLR-2024)

Reviewer comment (Novelty):
I feel this paper is an extension to [1], with similar algorithm proposed and regret analysis. The main algorithm has identical structure with the SAVE proposed in [1] based on the SupLin methodology, and it is not new and novel for the variance-aware contextual bandit problem. Under the generalized linear contextual dueling bandit setting used in the paper, we can regard \( x_t - y_t \) as the contextual information, which would then be degeneralized to the ordinary generalized linear contextual bandit problem. I think the high similarity with the existing literature is the major weakness of this work.

Author response:
**Q1**: Comparison with SAVE in [1] and algorithms for generalized linear bandit problem.

**A1**: Compared with SAVE, the main difference is that we consider the problem in the generalized linear model and dueling bandit setting, which has not been studied in the literature before. Note that SAVE is for linear bandits, rather than generalized linear (dueling) bandits. The nonlinearity of the problem requires us to consider MLE instead of linear regression. Since the linear regression problem has a closed-form solution, the analysis of MLE is more difficult. 

Compared with the generalized linear bandit, there is a key difference. You cannot simply regard $\mathbf{x} _ t- \mathbf y _ t$ as the feature vector to reduce dueling bandits to multi-armed bandits. Note that the regret in dueling bandits is defined as $\text{Regret}$ $(T) = \sum _ {t} $ $(\mathbf x _ t+ \mathbf y _ t)^\top \boldsymbol{\theta}^*$ $- 2 \mathbf x ^ {*\top} \boldsymbol{\theta} ^ *$. In this definition, the regret depends on $\mathbf x _ t+ \mathbf y _ t$ rather than $\mathbf x _ t- \mathbf y _ t$, i.e., it aims at minimizing the regret caused by both arms. This distinction poses a challenge for dueling bandit algorithms, necessitating a design that differs from the generalized linear bandit algorithm.

----

1.We acknowledge that the analysis of MLE in the generalized linear model is well studied, but the variance-aware analysis of MLE for generalized linear bandits is limited, marking the novelty of our work. In addition, we provide a rigorous proof for the MLE estimator in Appendix D, which fixes an overlooked issue in previous MLE analysis of generalized linear models. This part is also new and a novel result. 

Furthermore, In Eqn (4.2), the only thing we use from [1] is the Freedom-type concentration inequality for self-normalized martingales, while the analysis for our estimator is still new, and is not deduced from [1].

----

2.We acknowledge that our algorithm shares some spirit with SAVE. However, SAVE is an algorithm for linear contextual bandits, and it is not immediately clear how to extend it to generalized linear bandits, let alone dueling bandits. For dueling bandits, we need to select two competing arms $\mathbf{x} _ t$ and $\mathbf{y} _ t$. We would like to point out that in Line 8, the arm selection involves both $\mathbf{x} _ t + \mathbf{y} _ t$ and $\mathbf{x} _ t-\mathbf{y} _ t$, which cannot be deduced from SAVE. As a comparison, in Line 8 of SAVE, the selected arm is $\mathbf{a} = \text{argmax} \_ \mathbf{a} \langle \mathbf {a}$ $, \hat\theta \rangle + \beta ||\mathbf a|| _ {\Sigma}$. The selected arm cannot simultaneously represent both for $\mathbf{x}_t + \mathbf{y}_t$ and $\mathbf{x}_t - \mathbf{y}_t$, and therefore it’s not deduced from SAVE.

The derivation in Appendix C method 3 is our contribution. While it may not seem that difficult to figure out, it represents a novel addition that has not been previously documented in the existing body of work.

Reviewer comment (Writing):
Title of subsection 4.3 contains a typo (sysmmetric → symmetric).

Author response:
**Q2**: Typos and minor issues:
title of subsection 4.3 (sysmmetric -> symmetric)
I may overlook. What is $\kappa$ in line 20 Algorithm 1 and Eqn. (4.3)?

**A2**: Thanks for pointing out the typos. $\kappa$ is the same as $\kappa_\mu$. We have revised them.

Reviewer comment (Writing):
I may overlook. What is \( \kappa \) in line 20 Algorithm 1 and Eqn. (4.3)?

Author response:
**Q2**: Typos and minor issues:
title of subsection 4.3 (sysmmetric -> symmetric)
I may overlook. What is $\kappa$ in line 20 Algorithm 1 and Eqn. (4.3)?

**A2**: Thanks for pointing out the typos. $\kappa$ is the same as $\kappa_\mu$. We have revised them.

Reviewer comment (Theory):
Is the variance of the noise \( \epsilon_t \) equal to \( p_t(1-p_t) \) under the Bernoulli setting? In that case I feel the variance would be fully dependent on the arms selected in each round.

Author response:
**Q3**: Is the variance of the noise equal to under the Bernoulli setting? In that case I feel the variance would be fully dependent on the arms selected in each round.


**A3**: Yes, the noise equals $p_t(1-p_t)$ under the Bernoulli setting, and it depends on the arm being selected in each round. We study the problem in a general case where the arm set is time-varying and therefore the variance of arms can vary with respect to time and arms. Therefore, the variance $p_t(1-p_t)$ in each round is influenced by both the historical data and the potentially adversarial arm set $\mathcal{A}_t$ in round $t$. Our result indicates that the regret of the problem should depend on the variance of the selected arms, which aligns with our intuition that comparing high-variance dueling arms pose a greater challenge than comparing lower-variance dueling arms.

Reviewer comment (Experiments):
The paper primarily conducts experiments on synthetic data to validate the proposed algorithm. While this is a common practice, the performance of the algorithm in real-world scenarios might differ. To strengthen the paper, the authors could include experiments on real-world datasets, particularly those related to the applications mentioned like ranking, recommendation systems, or any human-interactive system, ensuring the practicality and robustness of the algorithm in diverse settings.

Author response:
**A1**: To showcase the performance of our algorithms in a real-world setting, we have added a new experiment in Appendix B. We use EventTime dataset [1]. In this dataset, $K = 100$ historical events are compared in a pairwise fashion by crowd-sourced workers. The data contains binary response indicating which one of the events the worker thinks precedes the other. There is no side information 
$
    \mathcal A = \\{ \mathbf x_i, i \in [K] \\},
$
or the true parameter $ \theta^*$ readily available in the dataset.  Thus, we estimate them with pairwise comparison data. To achieve this, let $C_{ij}, i,j \in [K]$ be the number of times event $j$ precedes event $i$ labeled by the workers. The following MLE estimator is used:
$$
\mathop{\mathrm{argmax}} _  { \\{\mathbf{x} _ i \\}, \theta} \sum_{i \in [K]} \sum_{j \in [K]} C_{ij}
\log
(
    \sigma((\mathbf x _ i - \mathbf x _ j)^\top\theta)
).
$$
With the estimated $\mathcal A$ and $\theta^*$, it is then possible to simulate the interactive process. We can see that after about $2500$ rounds, our algorithm starts to outperform $\texttt{MaxInP}$ in terms of cumulative regret.

Reviewer comment (Novelty):
The concept of a layered design for bandit algorithms, which the authors adopt in the VACDB algorithm, has been previously explored in the literature. Layered or epoch-based approaches are widely used in bandit algorithms to balance exploration and exploitation in stochastic settings. The paper could benefit from a more thorough discussion on how the layered design in VACDB specifically contributes to or differs from existing approaches.

Author response:
**A2**: Thanks for your suggestion. We have provided a detailed discussion in Appendix A in our revision. Our approach is different in the elimination phase, where we construct our confidence set with a variance adaptive radius to determine the confidence radius (lines 10-12).  Therefore, besides the layer number, our algorithm also leverages the variance information in the exploration. Moreover, we construct a carefully chosen weight dependent on the layer  (line 15). It is used in the calculation of the regularized MLE estimator.

Reviewer comment (Experiments):
The VACDB algorithm relies on the computation of regularized MLE for parameter estimation. The computation of these estimators is known to grow in complexity with the number of iterations, potentially leading to scalability issues, especially in settings with a large number of arms, contexts, or iterations.

Author response:
**A3**:  The MLE estimator is widely used in the literature of generalized linear bandit and dueling bandits, such as [2] and [3]. Although the calculation can be expensive in some cases, there are some literally tractable solutions to that. The regularized MLE can be formulated as a finite-sum offline optimization problem. For many widely used models, such as the Bradley-Terry-Luce (BTL) model, the regularized MLE is a strongly convex and smooth optimization problem. We can solve it using accelerated gradient descent [4] and SVRG [5], both of which achieve a linear rate of convergence. This can mitigate the scalability issues caused by the increasing number of iterations. The regularized MLE estimator can also be solved by an online learning algorithm such as in [6,7].

Reviewer comment (Theory):
The appearance of $\sum_{t=1}^T \sigma_t$ indicates that even if we know $X_t$ for all arms and $\theta^*$, we may not know the value of the derived upper bound.

Author response:
Yes, the randomness of the variance is from the arm being chosen in each round, which further depends on historical data. The setting we study is quite general where the arm set is time-varying and therefore the variance of arms can vary with respect to time and arms. That’s why the regret depends on a random variable representing the variance of the chosen arms in each round. We view this as a strength rather than a weakness of our regret bound. 

When we restrict our setting to a special case where the variances for all the pairwise comparisons are identical, the randomness of $\sigma_t^2$ can be removed in the regret bound. But this is similar to the worst-case regret implied by our variance-aware regret.

Reviewer comment (Theory):
I wonder if it is possible to derive a lower bound for the problem. If not, may the author(s) clarify the analytical challenge?

Author response:
We study this problem in a general setting where the variance of instances can vary over time and across different arms. In this context, a variance-dependent lower bound is difficult and not found even in the literature on stochastic bandit settings. Deriving that lower bound stands as an open problem. The proof is challenging especially when the variance can change over time. For example, consider a scenario where initially there is no variance, but later, substantial variance emerges. In such cases, accurate early exploration could make correct policies, effectively mitigating the impact of later variance. However, the variance-dependent regret bound in the literature does not inherently possess this characteristic. In general, we are not sure if it is possible, and leave it as an interesting future work.

Although getting a lower bound in the general case is hard if not impossible, we can show our optimality in two special cases:

 (1) when we consider the worst case, the variance is upper bounded by $1/4$, our result matches the $d\sqrt{T}$ lower bound proved in [1]; and (2) when the preference is deterministic, our result is $\tilde O(d)$, which is also nearly optimal.

----
[1] Bengs et al., 2022, Stochastic contextual dueling bandits under linear stochastic transitivity models, ICML

Reviewer comment (Writing):
The paper is not self contained and requires reader to go through multiple papers for example in section 4.2.

Author response:
Thanks for your suggestion. In the main text, we have added Section 4.1 to give an overview of the algorithm and introduce the multi-layered structure in more detail. We have also added Appendix A to provide a detailed introduction to our algorithm and a comparison with previous works.