Paper: Dynamic Continuous Hyperparameter Tuning for Generalized Linear Contextual Bandits (ICLR-2024)

Reviewer comment (Evaluation):
Addressing this issue would need an additional layer of BOB, resulting in a three-layer algorithm structure. Are you sure that such an algorithm can run efficiently and can still have a good performance?

Author response:
In section 4.1, we mention that the choice of $H$ could be handled by introducing an extra layer for the completeness of theory, and we provide a detailed analysis in Appendix F with solid theoretical support. However, we don’t use this procedure in our CDT framework due to concerns about the efficiency of the three-layer structure in practice as well. As in Section 4.1, we illustrate that the choice of $H$ could be handled by introducing this extra layer **mostly for the completeness of theoretical analysis**.

Reviewer comment (Theory):
This approach may impose certain constraints on the total duration $T$, such as requiring $T$ to be larger than certain values.

Author response:
For the restriction on the value of $T$, our algorithm doesn’t have to assume the value of $T$ is larger than certain values in theory, and the warming-up time $T_1$ is always strictly less than $T$. For practice, it is noteworthy that bandit algorithms cannot achieve good performance if $T$ is small since they estimate the model parameters on the fly, not to mention the more difficult model selection task for bandits. From Figure 1, we can clearly observe that the cumulative regrets of our CDT framework are smaller after some warm-up iterations in most cases, which validates that our CDT could perform better than all existing baselines regardless of the magnitude of $T$.

Reviewer comment (Theory):
The theoretical results of the CDT algorithm only provide sublinear outcomes, and it seems like it can't recover the optimal results of the base algorithms. However, your experiments indicate that CDT is optimal, which feels somewhat hard to explain.

Author response:
Thank you for your insightful questions. It is widely known that there is a huge discrepancy between the theoretical analysis and practical performance for bandit algorithms. From Figure 1, we can see that the theoretically derived hyperparameters usually yield terrible or even linear cumulative regret for most algorithms while they achieve the optimal regret bound in theory, and this is because most theoretical optimal settings are way too conservative in practice. To obtain better practical performance, intuitively we should choose less conservative exploration hyperparameters, which will lead to worse regret bounds. This fact is the motivation why we study this intriguing practical problem: how to choose hyperparameters for bandit algorithms in practice to adapt to different environments with superior performance while some decent regret bounds can still hold. As we mention in our work, we mostly concentrate on the practical performances of our CDT framework in our paper. And according to our section 4.2 and our proof in Appendix G, it is very difficult and challenging to present the novel theoretical analysis of Theorem 4.2. Furthermore, the existing work on bandit hyperparameter tuning (e.g. "Syndicated bandits: A framework for auto-tuning hyper-parameters in contextual bandit algorithms") intrinsically can't achieve the best regret bound of the base algorithms without stringent assumptions. Specifically, Syndicated can only achieve the $\tilde O(T^{2/3})$ bound of regret with finite number of candidate values for each hyperparameter, and their regret bound will becomes infinitely large under our problem setting with continuous candidate sets. These facts also demonstrate the significance of our theoretical analysis, even though the focus of our work is mainly on practical performances.

Reviewer comment (Evaluation):
The author mentions in the contributions section that the algorithm presented in the paper is efficient. However, there is no comparison of running times in the experiments. This is quite crucial for bandit settings. Therefore, I would suggest that the author add a comparison of the algorithm's running time in the experiments.

Author response:
**In our work we put the running time of our method with all baselines in Appendix A.4.5** for the 10 cases corresponding to Figure 1 plots 1-10. For ease of your review, we copied the results from Appendix A.4.5 in the following tables. Furthermore, we reproduced the last 4 cases in Figure 1 (plots 11-14) these days and displayed the results for completeness:

LinUCB

|        | Simulation | Movielens |
|--------|------------|-----------|
| Theory | 2.11s      | 2.17s     |
| TL    | 2.61s      | 2.67s     |
| OP     | 2.60s      | 2.65s     |
| CDT    | 3.49s      | 3.39s     |
LinTS

|        | Simulation | Movielens |
|--------|------------|-----------|
| Theory | 2.19s      | 2.04s     |
| TL    | 2.67s      | 2.49s     |
| OP     | 2.62s      | 2.45s     |
| CDT     | 3.53s      | 3.46s     |

UCB-GLM

|        | Simulation | Movielens |
|--------|------------|-----------|
| Theory | 8.34s      | 7.89s     |
| TL    | 8.84s      | 8.44s     |
| OP     | 8.71s      | 8.35s     |
| CDT     | 9.57s      | 9.47s     |

GLM-TSL

|        | Simulation | Movielens |
|--------|------------|-----------|
| Theory | 305.28s      | 295.17s     |
| TL    | 305.98s      | 295.87s     |
| OP     | 306.02s      | 295.83s     |
| CDT     | 307.24s      | 296.72s     |

Laplace-TS

|        | Simulation | Movielens |
|--------|------------|-----------|
| Theory | 525.62s      | 502.89s     |
| TL    | 526.31s      | 503.45s     |
| OP     | 526.24s      | 503.51s     |
| CDT     | 527.15s      | 504.45s     |

GLOC

|        | Simulation | Movielens |
|--------|------------|-----------|
| Theory | 486.58s      | 474.87s     |
| TL    | 488.31s      | 476.45s     |
| OP     | 488.24s      | 475.51s     |
| CDT     | 490.15s      | 478.45s     |

SGD-TS

|        | Simulation | Movielens |
|--------|------------|-----------|
| Theory | 67.42s      | 62.09s     |
| TL    | 69.31s      | 63.85s     |
| OP     | 69.24s      | 64.51s     |
| CDT     | 71.15s      | 66.45s     |

From the above tables, we can see that all existing methods and our CDT can run very fast in practice, and our CDT is only slightly more expensive in computation (CDT only takes few more seconds). In addition, we can observe that the main computation time comes from the contextual bandit algorithm we tune on. Specifically, GLM-TSL requires much more running time than all other methods under different tuning methods. Therefore, our CDT significantly outperforms all existing baselines without increasing much computation.

Reviewer comment (Writing):
There is a typo in Notation: write \top as T.

Author response:
Thank you very much for your careful review. We will correct this typo in the revision.

Reviewer comment (Experiments):
The authors propose Algorithm 1 to zoom quickly into a good area in the parameter space. However, this introduces extra hyper-parameters in Algorithm 1 which is unclear to tune. The authors say that it is possible to mitigate this issue by adding another layer of EXP3, essentially treating the epoch size candidate in Alg. 1 as arms. Then how sensitive is the overall performance wrt the parameters of outer EXP3 layer?

Author response:
Thank you for your insightful questions. In Section 4.1, we illustrate that the choice of $H$ could be handled by introducing an extra layer **mostly for the completeness of theory**, and we provide a detailed analysis in Appendix F with solid theoretical support. 
We agree with the reviewer that in practice this extra outer layer may not bring many benefits since between each two outer EXP3 pulls we don’t have many iterations to explore the Lipschitz structure with restarts. However, in our CDT we don’t have to use this procedure, as we mention in the last paragraph of Section 4.1.

Reviewer comment (Experiments):
How do you tune Alg.1 wrt sub-gaussian parameter $\tau$ present in the noise of Eq.(2)?

Author response:
In practice, most studies would have the stochastic rewards bounded, e.g., in the interval (0,C) for some positive constant $C$, where we could naturally take this hyperparameter as $C/2$ when applying Algorithm 1 and Algorithm 2. Note we use this implementation in our experiments and it works very well as shown in our extensive simulations and real datasets.

We really appreciate your valuable insights. Please let us know if our response has sufficiently resolved your concern and improved your opinion of our work, and we are more than happy to engage in any further discussion with you.

Reviewer comment (Experiments):
The paper overlooks key benchmarks both theoretically and experimentally:  
- Regret Bound Balancing and Elimination for Model Selection in Bandits and RL.  
- Syndicated bandits: A framework for auto-tuning hyper-parameters in contextual bandit algorithms.

Author response:
Thank you very much for your insightful comments. For the work “Regret Bound Balancing and Elimination for Model Selection in Bandits and RL”, we will cite it in the revision of our paper, since it discusses how to choose the confidence radius of OFUL on the fly. However, we’d like to point out that this work mainly concentrates on the general model selection in bandits and reinforcement learning, and the theoretical regret bound deduced in this work also depends on the number of candidates $M$, which is infinitely large and hence becomes meaningless in our problem setting. For experiments, the algorithm proposed in this work is also inefficient since it has to update each active base candidate at every iteration, and the implementation of this algorithm also relies on some unspecified parameters such as $\kappa$. This implies that this algorithm could not be easily adapted to the hyperparameter tuning of any contextual linear bandit. Note in Remark B.1 in Appendix B, we illustrate why the general bandit model selection method can’t be used for hyperparameter tuning of bandit algorithms, and “Syndicated bandits: A framework for auto-tuning hyper-parameters in contextual bandit algorithms” is the only existing work with theoretical guarantees that considers exactly hyperparameter tuning of bandit algorithms.

For “Syndicated bandits: A framework for auto-tuning hyper-parameters in contextual bandit algorithms”. In theory, the general regret bound is of order $\tilde{O}(T^{2/3} + \sqrt{MT})$, where $M$ is the number of hyperparameters considered in the candidate set. Note that this regret bound also becomes futile in our problem setting since $M$ is infinitely large, and hence this regret bound becomes infinite. Note it is intrinsically much more difficult in theory for the infinite-arm case compared with the finite-arm case: for example, the minimax regret bound for multi-armed bandit is of order $\tilde{O}(\sqrt{T})$ while the minimax regret bound for the continuum-armed bandit in $[0,1]^d$ is of order $\tilde{O}(T^{(d+1)/(d+2)})$. This fact means that for continuum-armed bandit it is impossible to recover most of the results in multi-armed bandits in the same order. In experiments, we include the TL or Syndicated in all of our simulations and real datasets as an important baseline, and we can evidently observe the practical advantage of our proposed CDT over all existing baselines.

Reviewer comment (Theory):
The rationale behind the decomposition in Eq(2) suggests that “the bandit algorithm is likely to select similar arms if the hyperparameters chosen are close at round $t$.” This reasoning is not entirely convincing. When feature vectors are determined by either a non-oblivious or oblivious adversary, the claim doesn't hold up. Such an adversary can present arms where the hyperparameters are similar, yet the arms differ considerably. Hence, it's crucial for the authors to specify these conditions in the problem definition and offer a more plausible explanation for Eq(2).

Author response:
Thank you very much for your valuable questions. As in "Syndicated bandits: A framework for auto-tuning hyper-parameters in contextual bandit algorithms", in our work we also assume that the arms in the arm set $X_t$ are drawn IID from some unknown distribution. Therefore, without an oblivious adversary choosing our arm set, we can anticipate that the expected performance of the bandit algorithm should be similar with close hyperparameters given other conditions (F_t) fixed. We will emphasize this point in our problem setting in the revision.

Reviewer comment (Theory):
The regret bound presented in this paper is $\Omega(T^{2/3})$. This is less favorable than the bound provided in "Syndicated bandits: A framework for auto-tuning hyper-parameters in contextual bandit algorithms," which doesn't necessitate any stringent conditions.

Author response:
As we mention in our response to Weakness 1, in "Syndicated bandits: A framework for auto-tuning hyper-parameters in contextual bandit algorithms", the general regret bound is of order $\tilde{O}(T^{2/3} + \sqrt{MT})$. Their algorithm can only achieve the $\tilde{O}(\sqrt{T} + \sqrt{MT})$ and $\tilde{O}(T^{4/7}+ \sqrt{MT})$ bound of regret when there is **only one** hyperparameter to be tuned with some assumptions held. Specifically, to obtain the order $\tilde{O}(\sqrt{T} + \sqrt{MT})$, their work has to assume that the theoretical optimal exploration rate is smaller than any element in the candidate set. This assumption is stringent and unrealistic since it is widely known that the theoretical exploration rate is very large and conservative, and hence the elements in the candidate set should be smaller than the theoretical value in practice. On the other hand, we can observe that their regret bound also depends on the number of candidates $M$ in the order $\sqrt{MT}$, and $M$ becomes infinitely large under our problem setting. This fact implies that all their regret bounds are actually infinitely large and hence meaningless under our problem setting.

We sincerely value the time you've dedicated to reviewing our work. And we respectfully hope you could re-evaluate our work if your concerns have been decently addressed.