Paper: Only Pay for What Is Uncertain: Variance-Adaptive Thompson Sampling (ICLR-2024)

Reviewer comment (Theory):
Assuming Gaussian bandits would be impractical when facing realistic applications. As a complement, the analysis should discuss the cost of mis-modeling and show how to control this additional cost to obtain a meaningful regret bound. (In contrast, the experiments indeed consider distributions other than Gaussian.)

Author response:
**W1: Theory beyond Gaussian bandits**

A great question. One paper that analyzes the error of using Gaussian posteriors for other distributions is [Liu et al. (2022)](https://arxiv.org/pdf/2201.01902.pdf). Their results are quite pessimistic and lead to an additional $\varepsilon n$ term, where $\varepsilon$ is a measure of model misspecification. To make this term $O(\sqrt{n})$, an $O(1 / \sqrt{n})$ model misspecification would be needed. Since this is not practical, we opted for an empirical validation in Bernoulli and beta bandits.

**Re. W1:** We totally agree with you on W1 and we will add this discussion as a remark in the updated version of our paper.

Reviewer comment (Novelty):
The submission lacks a clarification that connects the novelty claimed in Contribution-(3) to the analysis in the appendix. This information would help evaluate and understand the technical contribution(s) of this submission.

Author response:
**W2: Novelty**

The novelty is in analyzing a variance-adaptive algorithm. We derive the first Bayes regret bound for such an algorithm. The bound captures the benefit of informative priors and is finite time. None of the prior bounds have either of these properties. This is discussed in the last paragraph of Section 5.2. We discuss the technical novelty in Q3 in **Common response**.

**Re. W2:** We have further elaborated on our technical novelties and challenges addressed in the proof of Theorem 2 in Q3 in **Common response**, and also responded to Reviewer Lfwz on the follow-up concerns. Please go over them.

Reviewer comment (Experiments):
The variance-dependent bounds (Theorems 1 and 2) would be better justified if we could see curves of the proposed methods that vary according to the change of the unknown parameters in experiments.

Author response:
**W3: Plot the variance-dependent bound to validate it**

***In Appendix D of the updated paper, we plot the regret of VarTS, the regret bound in Theorem 2, and also frequentist regret bounds.*** We observe three trends:

* The regret of VarTS decreases as the prior becomes more informative.
* The regret bound of VarTS decreases as the prior becomes more informative.
* Our regret bound becomes tighter than the frequentist bounds as the prior becomes more informative. This is because it depends on how informative the prior is. The expectation of the frequentist bounds does not depend on how informative the prior is.

Reviewer comment (Experiments):
The main paper categorizes the methods using Bayesian/frequentist approaches or the identical $\sigma$/distinct $\sigma_i$ settings. Could you please also reflect on these different baselines in the experiments, helping the reader to compare and contrast the differences between considering/not considering the prior information?

Author response:
**Q1: Categorization of the baselines**

We tried to cover all combinations of variance adaptive / not adaptive and Bayesian / frequentist bandit algorithms:

* Adapts to unknown variance: UCB1-Tuned, UCB-V, TS20, TS14
* Does not adapt to unknown variance: UCB1, Bernoulli TS, Gaussian TS
* Bayesian algorithm: Bernoulli TS, Gaussian TS, TS20, TS14
* Frequentist algorithm: UCB1, UCB1-Tuned, UCB-V

Reviewer comment (Theory):
As mentioned on page 5, the finite-time Bayes regret lower bound for Gaussian bandits remains unresolved, making the optimality of the Bayes regret bounds in Theorems 1 and 2 ambiguous. Furthermore, the current optimality discussion focuses on the order of $K, n$, but there should also be a discussion on its dependency on the prior and reward variances.

Author response:
**W1: Dependencies on prior and reward variances are not discussed**

They should be. For Theorem 1, read starting from "Second, it increases with variances" on page 5. For Theorem 2, read starting from "Further, a closer examination of $C$ reveals many similarities:" on page 7. Please let us know if you would like to discuss anything else.

**Known reward variances:** As mentioned in the last paragraph of Section 4.2, there are no finite-time $\Omega(\sqrt{K n})$ lower bounds for Bayes regret. The only known lower bound of this form is a $\Omega(\sqrt{K n})$ asymptotic lower bound of [Lattimore and Szepesvari. (2020)](https://tor-lattimore.com/downloads/book/book.pdf) (Theorem 35.1). We match it up to logarithmic factors when $\sigma_{0, i} = \sigma_i = 1$ for all $i \in [K]$. The lack of good lower bounds is a widely-recognized problem in Bayesian bandits. For instance, even the seminal works of Russo and Van Roy [(2014)](https://arxiv.org/abs/1301.2609) and [(2016)](https://www.jmlr.org/papers/volume17/14-087/14-087.pdf) that motivated all recent development do not provide any lower bound. Beyond a formal lower bound, there are other factors that indicate that the bound in Theorem 1 is of the right order:

* **Increasingly informative prior:** As $\sigma_{0, i} \to 0$ for all $i \in [K]$, the bound in Theorem 1 goes to $0$. In this case, Thompson sampling knows the bandit instance before it interacts with it and therefore has no regret.
* **Prior is not very informative:** When $\sigma_{0, i} \approx \sigma_i$ for all $i \in [K]$, the bound in Theorem 1 becomes $\tilde{O}\left(\sqrt{n \sum_{i = 1}^K \sigma_i^2} \right)$. This matches, up to logarithmic factors, gap-free regret bounds of frequentist bandit algorithms for heterogeneous reward variances.
* **Doubling of reward variances:** In the frequentist setting, gap-dependent and gap-free regret bounds would increase by multiplicative factors of $2$ and $\sqrt{2}$, respectively, when the reward variances double. The gap-free regret bound in Theorem 1 would also increase by a multiplicative factor of $\sqrt{2}$, up to logarithmic factors.

**Unknown reward variances:** When we discuss Theorem 2, in paragraph "Dependence on all parameters of interest and prior", we establish the following relation to Theorem 1:

* $\beta_{0, i} / (\alpha_{0, i} - 1)$ in Theorem 2 is analogous to $\sigma_i^2$ in Theorem 1.
* $\kappa_{0, i}$ in Theorem 2 is analogous to $\sigma_i^2 / \sigma_{0, i}^2$ in Theorem 1.

Based on this, we conjecture that the dependencies on the variance parameters in Theorem 2 are of the right order, as in the known reward variance case.

We hope that this answers your question. Please let us know if you have more questions.

Reviewer comment (Novelty):
The algorithm design for Gaussian bandits with unknown heterogeneous rewards seems as standard as the typical TS algorithm.

Author response:
**W2: Algorithm seems standard**

Algorithm 2 is standard posterior sampling for Gaussian-Gamma distribution, based on well-known formulas in [Murphy (2007)](https://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf). The novelty is in analyzing a variance-adaptive algorithm. We derive the first Bayes regret bound for such an algorithm. The bound captures the benefit of informative priors and is finite time. None of the prior bounds have either of these properties. This is discussed in the last paragraph of Section 5.2.

Reviewer comment (Theory):
Is it possible to derive finite-time prior-dependent regret bound for Bernoulli TS or Beta TS?

Author response:
**Q2: Do Bernoulli or beta TS have finite-time prior-dependent regret bounds?**

Bernoulli TS was analyzed in ranking problems by [Kveton et al. (2022)](https://proceedings.mlr.press/v151/kveton22a/kveton22a.pdf). Note that the Bernoulli distribution has a single parameter, which determines both its mean and variance. Beta TS is problematic because the beta distribution does not have a computationally tractable prior.

Reviewer comment (Experiments):
The paper omits some crucial related work such as: 
    (1) Minimax policies for adversarial and stochastic bandits;
    (2) Mots: minimax optimal thompson sampling;
    (3) A minimax and asymptotically optimal algorithm for stochastic bandits;
    (4) Prior-free and prior-dependent regret bounds for thompson sampling;
    (5) Thompson Sampling with less exploration is fast and optimal. It would be beneficial to see the baselines (2) (3) and (5) incorporated into the experiments.

Author response:
**W3: Paper omits some crucial related works**

We respectfully disagree. The objective of our paper was to design, analyze, and empirically evaluate an algorithm that can adapt to unknown reward variances. Therefore, we included all relevant baselines to variance adaptation. That being said, we agree that we should use the latest improvements of Thompson sampling. We reviewed (2), (3), and (5). (5) compared their algorithm to (2) and (3), and showed that it is generally superior. ***Therefore, we implemented (5) for both Bernoulli and Gaussian Thompson sampling (Section 6 of the updated paper).*** This significantly reduces the regret of Gaussian TS while keeping the regret of Bernoulli TS comparable. Apart from this, all other conclusions in Section 6 remain the same.

Reviewer comment (Theory):
In Bayesian analysis, regret bounds are typically derived using the Upper Confidence Bound (UCB) method, as illustrated in works like "Prior-free and prior-dependent regret bounds for Thompson Sampling." However, this paper presents an alternative approach. It raises the question of whether it is possible to obtain the regret bound through a UCB method.

Author response:
**Q1: Can we obtain a regret bound through a UCB-like analysis?**

We follow the classic Thompson sampling analysis of [Russo and Van Roy (2014)](https://djrusso.github.io/docs/Learning_to_Optimize.pdf). The key idea is to bound the per-round regret conditioned on history by high-probability confidence intervals. Then we add them up and get an upper bound on the $n$-round regret. This style of an analysis is also common in linear bandits.

Reviewer comment (Theory):
The variance aware regret bound depends on the summation of the variance across all the arms $\sum_{i=1}^K \sigma_i^2$. This can reduce the dependency of the number of arms $K$ to a variance-dependent result. However, the standard method to deal with large $K$ is to use function approximation. I feel confused about the relationship of these two methods. Can the result in this work be generalized to the bandit problem with function approximation?

Author response:
Please note that the goal of this paper was to design, analyze, and empirically evaluate an algorithm that can adapt to unknown reward variances in the standard bandit setting. Our work does not focus on a large number of arms $K$. Therefore, no function approximation is used. Please see Q2 in **Common response** for a generalization beyond Gaussian bandits.

Reviewer comment (Novelty):
Algorithm 2 is under the standard Thompson sampling framework, and the posterior sampling updates seem similar to previous works, for example, [Zhu & Tan., 2020]. Are there any novel techniques in the algorithm design, or do the new results come from the analysis?

Author response:
Algorithm 2 is standard posterior sampling for Gaussian-Gamma distribution, based on well-known formulas in [Murphy (2007)](https://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf). The novelty is in analyzing a variance-adaptive algorithm. We derive the first Bayes regret bound for such an algorithm. The bound captures the benefit of informative priors and is finite time. None of the prior bounds have either of these properties. This is discussed in the last paragraph of Section 5.2.

Reviewer comment (Theory):
There are some mistakes in the literature review. In the study of $d$-dimensional linear contextual bandits, they did not always keep the fixed variance across the arms. For example, in [Kim et al., 2022] and [Zhao et al.,2023] mentioned in the paper, the stochastic noise at time step $k$, $\epsilon_k$, is a random variable dependent on $\mathcal F_k = \sigma(x_1,\epsilon_1,\ldots,x_{k-1},\epsilon_{k-1}, x_k)$ with $\mathbb{E}[\epsilon_k | \mathcal F_k] = 0$ and $\mathbb{E}[\epsilon_k^2 | \mathcal F_k] = \sigma_k^2$. They do not assume the variances are fixed across arms.

Author response:
There seems to be a misunderstanding of what we wrote. We say on page 2 that "$\sigma_s^2$ is an unknown reward variance **in round $s$**". We mean that the reward variance depends on round $s$. However, it is the same for all arms in round $s$. In our work, each arm has its own reward variance, and this one is the same for all rounds. We are happy to incorporate any additional changes if the reviewer has a different opinion and can further elaborate.

Reviewer comment (Writing):
Some typos: should the bound in page 7 be $\tilde O(C\sqrt{n})$?

Author response:
$\tilde O(\sqrt{Cn})$ should be $\tilde O(C \sqrt n)$. Thanks for pointing this out.

Reviewer comment (Theory):
After reading what the authors mentioned in the response, I still believe that there is no need to make such assumptions on the fixed variance across the arms. The notation $\sigma _ k ^ 2$ means the variance in round $k$, which can change when the selected arms are different (otherwise, the conditional expectation on $\mathcal F_k = \sigma(x_1,\epsilon_1,\ldots,x_{k-1},\epsilon_{k-1}, x_k)$ is meaningless.) I agree that assuming the fixed variance across the arms is a common practice. But for completeness, I suggest the authors mention the possibility of dealing with more general cases.

Author response:
**Re. Further clarification on the discussion of Kim et al. (2022) and Zhao et al. (2023):** The main confusion could be in the notations used in their papers. Note that in these papers

* K is the time horizon and $k \in [K]$ denotes the $k$-th time step in the time horizon K. 
* We on the other hand use $K$ to denote the total number of arms and $k \in [K]$ is the indexing used to denote the $k$-th arm among the $[K]$ arms.

Now please read:

* Page 2 last paragraph of [Kim et al. (2023)](https://arxiv.org/pdf/2111.03289.pdf): "Let K be the total number of rounds ..... till Equation (2.1)". 

* Section 2.1 of [Zhao et al. (2023)](https://proceedings.mlr.press/v195/zhao23a/zhao23a.pdf): "Linear bandits. The linear bandit problem has ..... till the last sentence of Page 2".

Thus, in their notation $\sigma_k^2$ is the noise variance at time step $k$, which is same across all the arms, no matter which arm the algorithm selects in time step $k \in [K]$. To emphasize on our previous claim again,  these papers do not consider heterogeneous reward variance across arms -- unlike our setting -- where we assume distinct prior variance $\sigma_{0,i}$ as well as the distinct reward variance $\sigma_i$ for each distinct $i \in [K]$. Consequently, the nature of the results of these works are very different than ours, as they can not bring out the tradeoff between regret vs informative priors and variances (across the K arms), unlike our regret bounds in Theorem 1 and 2. Please let us know if you still have any confusion.

Reviewer comment (Novelty):
After careful consideration, I remain unconvinced about the paper's novelty, leading me to keep my scores.

Author response:
**Re. Novelty:** We have further elaborated on our technical novelties and challenges addressed in the proof of Theorem 2 in Q3 in **Common response**, please go over it and also our detailed proof of Theorem 2 in the Appendix (Page 15-20). Hope this clarifies the novelties of our analysis of Theorem 2.

* As promised, we will add a concise proof sketch of Theorem 2 in the updated version of our paper.  
* To emphasize again, as also clarified in Q3 in **Common response** above, our novelty not only lies in the proof of Theorem 2 but also in the implication of the final bound it reveals: We precisely obtained the ***First finite time regret bound for Bayesian bandits with unknown heterogeneous reward variances which captures the dependence of the regret on the degree of prior information and reward variances, unlike any other existing work.***

Please let us know if we can clarify anything further.

Reviewer comment (Theory):
Is it possible to deal with distributions other than the Gaussian-Gamma prior distribution?

Author response:
**Q1: Other reward distributions than Gaussian-Gamma**

Our algorithm design and analysis are structured enough to incorporate other models. We discuss this in Q2 in **Common response**.