Paper: Improved Regret Bounds in Stochastic Contextual Bandits with Graph Feedback (ICLR-2024)

Reviewer comment (Presentation):
The paper is not very well structured; it is not easy to understand the flow of the paper. In particular, although Table 1 and 2 capture quite well the main contributions, the involved notation are not introduced there but scattering across the paper (hence, readers need to revise these tables after finishing the paper to understand the notation). Similarly, while it is good to have a comparison with previous works in Section 1.2, it is hard to understand as no proper description of the contribution is introduced yet (for example, the beginning of page 4 discusses “first and second options” that has never been mentioned before). A simplified version of the main theorem before this section, for example, could facilitate the comprehension here.

Author response:
Thanks for suggestions. To help readers understand our work better, we add the name of graph parameters in the caption of tables. 
We also show the order sequence of this graph parameters to highlight our contributions. 
As for ``a simplified version of main theorem'', we will try to put it in our work if there exists additional space.
After all, the limited page requirement of ICLR forces us to put formal descriptions in the following sections.

Reviewer comment (Experiments):
It does not seem necessary to present ConstructExplorationSet with two options (or at least it lacks a major justification for the necessity of this). As I understand from page 6, “Option 2” is recommended to be used with an integration of empirical best arms from Option 1. Moreover, from Table 1, the major improvement (switching from bounds with independence numbers to bounds with dominating numbers) comes from Option 2. The experiments also show the superiority of this Option 2. I do not see why the authors cannot simply combine these two so-called options into one procedure.

Author response:
Our main contribution stems from the second option. 
However, the first option is not always useless, as in certain types of graphs like perfect graphs, 
it can outperform the second option. 
The two options also capture the trade-off of the exploration between the gaps and number of actions. 
Additionally, the first option can be served as a rough realization of the method in [1] as their regret order scales with the independence number.

Reviewer comment (Novelty):
The idea of the main algorithm is quite natural and obvious (applying IGW to a well-selected exploration set). Can the authors highlight further any novelty of this algorithm or whether the contribution comes more in the proof aspects?

Author response:
It requires non-trivial modifications of IGW technique. 
The time-varying hyperparameter $\gamma_t$ and the adaptive set $S_t$ are the crucial difference of original works. 
The original works focus on the upper bound so they only need an upper bound $|\mathcal{A}|$. 
In order to capture the influence of time-varying action sets, 
we have to bound the size of action sets in a more refined way. 
It requires us to define the policy space and prove lemmas in a more refined way. 

Here is the high-level intuition of our proof. 
For any policy $\pi \in \Psi(S_t)$, the regret of $\pi$ with respect to the greedy policy can be controlled by our lemma B.1. 
Secondly, since the sample size will increase as the learning processes, the regret of the greedy policy with respect to the optimal policy will become smaller, 
as shown in lemma B.2. 
Due to these two lemmas, 
the policy $\pi \in \Psi(S_t)$ can be controlled.

Reviewer comment (Experiments):
Do the authors choose to run experiments only in comparison with FALCON and not the one of Zhang et al. 2023 because the latter is unstable?

Author response:
We appreciate the Zhang's work[1], which firstly deals with the general function space, 
but their method need online regression oracle and works in semi-adversarial setting.
However, our method only requires offline regression oracle and works in stochastic setting. 
For one thing, the adversarial setting is more difficult than the stochastic setting.
It usually happens that the algorithms for adversarial settings will have worse regrets than those in the stochastic setting.
For another, the online oracle is more difficult to design than the offline oracle. 
Online oracle is sensitive to the order of input data sequence while offline oracle not.
Even for a fixed dataset, the input sequence will also affect the output of online oracles.
As an example, the OLS estimator is a valid offline oracle but not a valid online oracle.

To see our efforts, we refer the reviewers to option 1.
Roughly speaking, the option 1 can be viewed as the realization of this method,
though not fully identical. 
We try to find effective baselines and compare them in theory and simulations.

Reviewer comment (Theory):
Why do the authors prefer high-probability bound? Is it possible/easy to derive an expected regret result from Theorem 3.1?

Author response:
It can be converted to expected bounds by setting $\delta=1/T$.

Reviewer comment (Theory):
The presented framework uses undirected feedback graph. Can this be extended with directed ones? (note that Zhang et al. consider directed graph).

Author response:
This is an interesting work to extend our method to general graphs. 
However, as we admit in the conclusion section in Appendix D, 
we haven't find a way to deal with general graphs.

Reviewer comment (Writing):
Can we have a definition of the set S_t?

Author response:
The name of $S_t$ is called the exploration set.

Reviewer comment (Experiments):
**UCB-like approach**: The necessity for forced exploration is not clearly justified. The possibility of employing a UCB (Upper Confidence Bound) type scheme is not explored in depth. It would be beneficial if the authors could provide an explanation of the challenges or limitations associated with the UCB approach.

Author response:
It is widely known that UCB can achieve comparable performance with sampling method. 
However, it is unclear how one can build valid confidence upper bounds on general function space. 
It is also unknown whether one can truncate the action set as we do.
To point out the challenges, the previous works [1,2,3] about general function space all focus on IGW rather than UCB. 


[1] Practical contextual bandit with regression oracle
[2] Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability 
[3] Instance-dependent complexity of contextual bandits and reinforcement learning: A disagreement-based perspective

Reviewer comment (Theory):
Theorem 3.1's phrasing presents a contradiction. On one hand, it mentions that the "expectation of regret is taken w.r.t. all kinds of randomness," but then goes on to state the result is "with high probability." What is the specific randomness associated with the high probability argument, and how does it differ from the randomness tied to the expectation? Could you clarify this split?

Author response:
Thanks to point out our typos. The expectation is taken with respect to the randomness of algorithms.
If one wants to get the expected regret bounds, one can simply set $\delta=1/T$ and it only leads to additional $\delta T$ regrets, which is a constant.

Reviewer comment (Experiments):
It would be beneficial to have real-world examples that align with the setup described in the paper. Specifically, are there tangible instances where the function class and changing graphs over time can be observed?

Author response:
We haven't found any open dataset which fully fit our setting. 
Though we can conduct experiments in real-world datasets, what we obtain is a regret curve. 
To illustrate the basic property, we believe that the randomly generalized contexts and graphs are enough.

Reviewer comment (Experiments):
In the simulations section, is it feasible to use the baseline of algorithms from related works for comparison? This would offer a more holistic view of how the proposed methods stack up against existing solutions.

Author response:
For our mentioned works, their settings are not fully identical to us. 
[1] works in MAB with fixed graphs. 
[2] works for linear contextual graphs with fixed graphs. 
[3] deals with the general function space, but their method need online regression oracle and works in semi-adversarial setting.
However, our method only requires offline regression oracle and works in stochastic setting. 
Roughly speaking, the option 1 can be viewed as the realization of this method,
though not fully identical. 
To compare them effectively, we have to illustrate the theoretical properties of this work in Table 1.




[1] Reward maximization under uncertainty: leveraging side-observations on networks
[2] Contextual bandit with side-observations
[3] Practical contextual bandits with feedback graphs