Paper: Combinatorial Bandits for Maximum Value Reward Function under Value-Index Feedback (ICLR-2024)

Reviewer comment (Theory):
The most practically relevant setting for the proposed work seems to be the case with arbitrary distributions of arm outcomes with finite supports. Technical development in the paper starts from the simpler cases and gives the reader an expectation that the special cases are solved to provide intuition for the general case. Algorithm 4.1 looks like the pinnacle of the paper. However, is there any result (theoretical or experimental) related to this algorithm? If the regret bound in Theorem 3.4 still holds, putting proof of this would be nice. Moreover, simulations are concerned with the Bernoulli case. Verifying the performance of Algorithm 4.1 in simulations would be nice.

Author response:
Yes, a comparable regret upper bound as in Theorem 3.4 holds for Algorithm 4.1: For the $k$-MAX problem with max value-index feedback, under general case of arbitrary discrete distributions of arm outcomes with finite	support as defined in Section 2 and under assumption $\Delta_{\min}>0$, Algorithm 4.1 has the following distribution-dependent regret bound,
$$R(T)\leq c_1 \sum_{i=1}^n\left(\frac{k}{\Delta_{\min}^i} + \log\left(\frac{k}{\Delta_{\min}^i}+1\right)\right) s_i\log(T) +c_2 \sum_{i=1}^n s_i\left(\left(\log \left(\frac{ k}{\Delta_{\min}^i}+1\right)+1\right)\Delta_{\max}+1\right),$$
We give further comments of the claim in the appendix.

The current experimental setup is designed purposefully to show that the algorithm works for scenarios with high-risk high-reward items or stable-valued items with low reward. We will add another plot for items of categorical distributions to demonstrate that Algorithm 4.1 is an effective approach.

Reviewer comment (Theory):
Another weakness is related to the deterministic tie-breaking rule, which is proposed as a way to accommodate non-unique arm reward values. I wonder if there is an application in which this tie-breaking is in the control of the decision-maker instead of its environment. More about this is asked in the questions part of the review.

Author response:
Thanks for asking this good question. In the recommender system setting, deterministic tie-breaking models the case that the customer is always intrinsically looking for other clues and information when facing two similar items to make a selection in his/her head, instead of purely relying on a coin toss in his/her head. There could be a difference between the value feedback based on the choices that the platform offers and the internal values of the items that a user actually perceives, which rely on the granularity of the feedback scale. Think of four items A, B, C, and D given to the user on a five-star scale. The first time, A, B, and C are shown and the user favors A and rates it 4 stars. The second time, B, C, and D are shown and the user favors B and rates it also 4 stars. The user gives both A and B the same value feedback, but there could be some confounding factors that are not captured in the value feedback, which implies a deterministic tie-breaking when facing A and B at the same time. Considering random tie-breaking is perhaps only purely for mathematical curiosity or interest.

Reviewer comment (Presentation):
Other weaknesses are mostly related to presentation issues and pinpointing the difference between the estimators used compared to the ones in the semi-bandit feedback CMAB-T setting. Please see the questions section.

Author response:
$q_i^{\mu,S}$ in Definition 3.2 is a general definition for the triggering probability of arm $i$ under action $S$ and a set of base arms $\mathcal{B}$ with expectation values $\mu$, and the reward function depends on $\mu$ only. $q_i^{p,S}$ and $\tilde{q}_i^{p,S}$ are defined for our specific problem setting when the (extended) set of base arms $\mathcal{B}$ consists of the union of two sets $\mathcal{Z}$ and $\mathcal{V}$. $q_i^{p,S}$ is the triggering probability of arm $a_i^z$ under action $S$ and is dependent on $p$ only. $\tilde{q}_i^{p,S}$ is the triggering probability of arm $a_i^v$ under action $S$. We did not write $\tilde{q}_i^{p,v, S}$ as the triggering probability is also dependent on $p$ only.

Yes, as explained in section 3.3, the estimates of base arms are biased and this is one of the key challenges we have for our problem. As a result, the Event E2 does not directly apply. To tackle with this difficulty, we introduce the concept of arm equivalence such that we can restore the CMAB-T framework by using some replacement items $(p',v')$. We note that $p_{i,t} '= p_iv_i, v_{i,t}' = 1$
 when $v_i$ is unknown (first stage) and $p_{i,t} '= p_i, v_{i,t}' = v_i$ after observing $v_i$ (second stage). In this way, the estimates of base arms are unbiased in two stages. We then justify that using the replacement arms will not cause excessive regrets.

Reviewer comment (Theory):
The paper focuses on binary distributions of arm outcomes always including $0$ in their support, which may not be representative of many real-world scenarios. The paper could benefit from exploring more general distributions of arm outcomes. Even if the authors argue about adapting to more general discrete distributions with finite support, these are also quite limited in the applications of their techniques. Furthermore, no theoretical guarantee is provided for the latter distributions, and the proposed reduction is expected to introduce a linear dependence in the support sizes to the regret. This is therefore unfeasible and undesirable for distributions with large enough supports, let alone countably infinite (or even continuous) supports as per many common distributions.

Author response:
No, it is totally fine to have arms without 0 in its support. We did not impose this constraints for arm distributions. Please take another look at section 2 problem formulation on page 3. We require that $0 < \sum_{j=1}^{s_i} p_{i,j}\leq 1$. Note that when the right inequality holds with equal sign, $p_{i,0} = 0$ and the arm outcome does not include zero in the support. In the binary case, this simply means that we have a deterministic arm. In the general case, the equivalence relationship is not affected. For example, if $X_i$ takes value $0.1$ with probability $2/3$, and value $0.22$ with probability $1/3$, then by eqn.(4.1), we can represent $X_i$ by two random variables $X_{i,1}$ and $X_{i,2}$ such that $X_{i,1}$ takes value $0.1$ with probability $1$, and $X_{i,2}$ takes value $0.2$ with probability $1/3$ and value 0 with probability $2/3$. One can easily verify that $Y=\max\{X_{i,1}, X_{i,2}\}$ follows the distribution exactly the same as the original $X_i$, namely with probability $2/3$ $Y=0.1$ and with probability $1/3$ $Y=0.2$.

We also argue that our setting is not limited in the applications, as categorical ratings or ranks are commonly used in the real world. The general algorithm on discrete distributions with finite support does have theoretical guarantees. As we discussed in the main text, equipped with the equivalence relationship, we are able to treat arms with outcomes according to arbitrary distributions with finite supports as a set of arms with outcomes according to binary distributions and use our algorithm defined for the case of arms with binary distributions. Specifically, we have the following claim for regret upper bound,

For the $k$-MAX problem with max value-index feedback, under general case of arbitrary discrete distributions of arm outcomes with finitesupport as defined in Section 2 and under assumption $\Delta_{\min}>0$, Algorithm 4.1 has the following distribution-dependent regret bound,
$$R(T)\leq c_1 \sum_{i=1}^n\left(\frac{k}{\Delta_{\min}^i} + \log\left(\frac{k}{\Delta_{\min}^i}+1\right)\right) s_i\log(T) +c_2 \sum_{i=1}^n s_i\left(\left(\log \left(\frac{ k}{\Delta_{\min}^i}+1\right)+1\right)\Delta_{\max}+1\right)$$
We give further comments of the claim in the appendix.

We acknowledge that our current regret bound does depend on the support size of the discrete variables. We believe this does not limit the applicable scenarios of our algorithm. In many applications, the support size is relatively small. For application scenario where the support size is large, we would like to emphasize the distinction between the choices that the platform offers and the internal values of the items that a user actually perceives. In may cases, even though the support size may be infinite from the values offered by the platform (e.g. real-valued feedback), the actual value of the customer on an item may only have a few possible internal values.

We note the possibility of extending our result to allow for continuous reward distributions by discretizing continuous arm distributions. We do like to mention that when all the values of the variables are unknown, intuitively the exploration does seem to need to explore all possible support values of variables in order to exploit the best option, and this may suggest some inherent dependency on the support size or a tradeoff between discretization granularity and regret bound. This is indeed an intriguing research direction for future research	to study the upper and lower bound dependencies on the support size, but we believe it is out of the scope of the current paper, while our current paper already has proper technical innovations that cover a reasonable range of practical applications.

Reviewer comment (Experiments):
The baseline methods used in the numerical results are never explicitly described or properly referenced. It is therefore unclear how meaningful the experimental results are without understanding what assumptions are made in the other algorithms used in the comparison.

Author response:
We will explicitly cite the work for the baseline methods in the numerical section. Currently, we referred to the standard CUCB method at the beginning of section 2 literature review and in the appendix A. It is the previous work on the $k$-MAX bandit problem with semi-bandit feedback.

Reviewer comment (Theory):
In line 7 of Algorithm 3.1 it says the learner is able to observe value $v_{i^*}$ of $i^*$ while it might happen that no value is observed if all arms in $S_t$ for a certain round $t$ have realized value equal to $0$.

Author response:
Yes, this is possible. In this case, there is no winner and we do not update $v$. We will comment that we update $v$ values only when the winner value is non-zero.

Reviewer comment (Novelty):
Some of the main ideas adopted in this paper, while adjusted to fit a setting with simpler distributions but less informative feedback, seem to be taken directly from Chen et al. (2016a) without striking changes.

Author response:
This is one of the key reference papers. We take the offline oracles from Chen et al. (2016a); however, we argue that the assumptions, key ideas and methods of our paper are significantly different from Chen et al. (2016a). Our main challenge stems from the fact that Chen et al. (2016a) studied semi-bandit feedback while our problem has max value-index feedback. Unlike Chen et al. (2016a), our feedback is much weaker and we are not able to apply the SDCB algorithm that requires maintaining empirical distributions of the observed arm outcomes. We developed a variant of the CUCB algorithm to solve our problem. We established an equivalence relationship between general distributions of finite support and binary distributions. Our analysis rests on a novel idea of considering an extended set of base arms, associated with $v_1,\ldots, v_n$ and $p_1,\ldots, p_n$ values. The unknown values $v_1,\ldots,v_n$ cause significant technical challenges for our problem which we circumvent in our analysis. In the case of unknown value ordering, we cannot infer whether a base arm is triggered or not based on the winner value and the winner index only. We tackle this difficulty by introducing the concept of item equivalence. These are all new technical contributions to address issues that arise because of the different types of feedback we study.

Reviewer comment (Novelty):
There’s an uncited work that is relevant -- https://ojs.aaai.org/index.php/AAAI/article/view/16812 “DART” by the same authors of (Agarwal et al. (2021)), for picking K sized subsets with a non-linear rewards. The former does not have the same FSD assumption as the cited work and in terms of horizon $T$ dependence they get $\tilde{O}(\sqrt{T})$ problem independent upper regret bounds and lower bounds. They show that K-max for independent Bernoulli arms satisfies the assumptions in that paper.

Author response:
Thanks for pointing this out. We will cite this work in the related work section on full-bandit CMAB problems. We note that our $k$-MAX bandit setting does not satisfy the assumption 2 Good arms generate good actions in Agarwal et al. (2021). High-risk high reward item in our setting could be favored by the maximum reward. Consider $n=3$ and $k=2$ such that $X_1$ taking 0.9 with probability 0.2, $X_2$ taking 0.5 with probability 0.4 and $X_3$ taking 0.6 with probability 1. In this case, $E(X_1)<E(X_2)$ but $X_1$ is in the optimal set instead of $X_2$.

Reviewer comment (Theory):
Is the regret $R(T)$ formally defined? I didn’t see it in the problem formulation or anywhere before it is upper bounded in Theorem 3.4. In addition to being unambiguous about cumulative vs instantaneous, regret vs pseudo-regret, you mentioned approximation algorithms for the general offline problem and so it becomes ambiguous if you are working with regret or an $\alpha$-regret (and if so specifically a $\alpha=1-1/e$ regret or an $\alpha=1-\varepsilon$ regret).

Author response:
We truly apologize for the ambiguity caused. We have mistakenly removed one paragraph from the problem formulation section that formally defines the regret and the approximation oracle with the procedures. It is added back as a section in the appendix.

Reviewer comment (Reproducibility):
Relatedly, page 5 is the first place “approximation oracle” is mentioned once in the text before the Alg 3.1 pseudocode but what “approximation oracle” refers to is not formally described. That should be formally discussed in the problem setup or at least before pseudocode for the algorithm that calls it. Also, for completeness the procedures that are proposed to be used for an oracle should be formally described (appendix would be fine).

Author response:
We truly apologize for the ambiguity caused. We have mistakenly removed one paragraph from the problem formulation section that formally defines the regret and the approximation oracle with the procedures. It is added back as a section in the appendix.

Reviewer comment (Writing):
(Minor) I found the discussion in the text confusing about maintaining confidence bounds on not only $p_i$’s but also $v_i$’s, which are deterministic. Looking in the pseudocode for the $v_i$’s there aren’t any confidence bounds – instead the values are set optimistically to $1$ and (since the support is deterministic) updated exactly once on the first time that arm is the maximizer (if that occurs). Calling it a confidence bound is confusing in my opinion.

Author response:
The confidence radius of $v_i$ is defined as $\boldsymbol{1}\{\tilde{T}_i = 0\}$. It is defined in this way for notation convenience and consistency. Note that the value may never be revealed. We will make it explicit in Algorithm B.1 and link the pseudocode at the end of second paragraph of section 3.2.

Reviewer comment (Presentation):
(Minor) In Section 3.2 I would suggest linking to the pseudocode earlier, such as at end of second paragraph “Due to limited space, the pseudocode is presented in Algorithm B.1.”

Author response:
The confidence radius of $v_i$ is defined as $\boldsymbol{1}\{\tilde{T}_i = 0\}$. It is defined in this way for notation convenience and consistency. Note that the value may never be revealed. We will make it explicit in Algorithm B.1 and link the pseudocode at the end of second paragraph of section 3.2.

Reviewer comment (Theory):
I think it would be valuable to have a fuller discussion of the hardness for the offline setting and corresponding approximation methods (Bernoulli (which seems easy based on (Agarwal et al. (2021))’s DART results), non-Bernoulli binary (you mention can be solved exactly with dynamic programming, but are hardness results known for the non-Bernoulli binary case?), and general (you cite Chen et al. (2016a).) to give the reader a fuller intuition. E.g. explicitly state that the general offline problem (and non-Bernoulli binary offline problem too) is NP-hard; don’t just state that greedy methods can get an $\alpha$ approximation.

Author response:
Thanks for the good question. We give a full discussion of the hardness results. For the general offline $k$-MAX problem (Goel et al 2014), finding the exact optimal solution is NP-hard. (Chen et al 2016) shows that there exists a PTAS for the offline $k$-MAX problem such that for any constant $\epsilon>0$ there is a polynomial-time $(1-\epsilon)$-approximation algorithm for the offline $k$-MAX problem. The binary and Bernoulli $k$-MAX problems are not NP-hard. The binary $k$-MAX problem can be solved exactly using dynamic programming. The Bernoulli $k$-MAX problem can be solved exactly by ordering values and selecting the top $k$.

Reviewer comment (Experiments):
Semi-bandit CUCB is good to include; it would be good to also include Agarwal et al. (2021))’s DART since that has $\tilde{O}(\sqrt{T})$ guarantees under bandit feedback when the arms are Bernoulli. It would probably do worse since its assumptions might not hold for non-Bernoulli arms, but it might be more competitive than UCB.

Author response:
We will test the performance of the DART algorithm. We note that the current experimental setup is designed purposely to show that the algorithm works for scenarios with high-risk high-reward items or stable-valued items with low reward. As mentioned, this assumption does not hold in Agarwal et al. (2021)).

Reviewer comment (Experiments):
The distributions in the experiments are all binary? That should be mentioned in the main text.

Author response:
Thanks for pointing these out. We will clarify the experimental setup as suggested.

Reviewer comment (Experiments):
“Three different arm distributions representing different scenarios” is quite generic and suggests potentially distinct problems (such as modeling three different applications), but the ‘different scenarios’ are almost identical manually constructed examples with a single arm changed from scenarios 1 to 2 and then from 2 to 3. That is ok to do but the description in Section 5 should be revised to reflect both of those issues (manually designed and single arm changes from one scenario to another).

Author response:
Thanks for pointing these out. We will clarify the experimental setup as suggested.

Reviewer comment (Theory):
In the contributions, “Our work may be seen as a step towards solving full-bandit CMAB problems with non-linear reward functions under mild assumptions.” Can you elaborate on that (if that's kept in the main contributions)? The feedback model and modifications to CUCB appear specialized for the max reward function and thus the work does not seem extendable to other non-linear rewards.

Author response:
In our current work, we consider the expected maximum as the reward function. We acknowledge that the proof techniques are tailored to leverage properties of the expected maximum. We note that the idea of arm equivalence and using fictitious arms can be possibly applied to solve CMAB problems of other non-linear rewards. We will downplay a bit of this point of contribution.