Paper: Bandits Meet Mechanism Design to Combat Clickbait in Online Recommendation (ICLR-2024)

Reviewer comment (Theory):
I feel like it would be more proper and much more interesting if the problem could be modeled by adversarial bandit. Current setting assumes that arms decide their own strategies with knowledge of algorithm in advance (which seems strong to me), and the algorithm is a plain UCB estimating fixed $s$ and $\mu$. However, this problem has the intrinsic of an adversarial strategy selection, that can be well captured by adversarial bandit, in which case, arms dynamically select strategies based on history and thus omit the requirement of knowing algorithm in advance. More importantly, adversarial bandit constructs the interaction between arm strategy selection and learner recommendation/elimination, thus captures real trade-off between incentivizing all arms to be truthful and recommending only the best arms, while current model does not.

Author response:
**Adversarial Bandits**: We are not entirely sure that we understand your comment about adversarial bandits correctly. We believe that, while challenging, it will be possible to extend our work to the case where arms dynamically adapt their strategies (click-rates) to the learning algorithm. We took a first step in this direction with our experiments, in which we considered the case of “learning” arms (as modeled by gradient ascent) and showed that UCB-S still effectively incentivizes desirable arm behavior. However, theoretically analyzing such dynamics is out of the scope of this paper. Moreover, our algorithm UCB-S is not plain UCB (which is incentive-unaware), but---despite being simple in its design---addresses the main difficulty in the strategic click-bandit which is to *incentivize desirable NE under uncertainty* while minimizing regret. Moreover, we also want to stress that our work specifically highlights the trade-off between incentivizing *all* arms to be truthful and recommending only the best arms (see Theorem 5.2 and the discussion below the theorem).

Reviewer comment (Theory):
The proposed model assumes that arms are aware of the algorithm that the learner is going to run in advance. Is this realistic in practice? Further, it is assumed that arms will pick CTRs that form a NE in $NE(M)$, according to Theorem 5.2, does it imply that arms have to know $K$ and $T$ in advance? I recommend presenting how to pick CTRs to form a NE in $NE(M)$. If $K$, $T$, and algorithm $M$ are required to form a NE before running $M$, it appears that the assumption is too strong.

Author response:
*Modeling:*  
1. The assumption that the learner publicly commits to an algorithm is standard in the mechanism design literature and is giving the arms maximal knowledege which makes the problem only more difficult for the learner. There are various ways the arms could reach a NE. Prior knowledge of the algorithm $M$, $T$ and $\Delta_i$ would for example be sufficient (see footnote 4). The arms could also reach an equilibrium by repeatedly interacting with the mechanism over several epochs like we considered in our experiments.

Reviewer comment (Theory):
According to the description of Model 1, strategies $\bf{\it{s}}$ are picked in advance and stay fixed when running $M$, then, the goal of $M$ becomes maximizing learner's utility by estimating $\bf{\it{s}}$ and $\bf{\mu}$. In this case, why are we interested in strong and weak strategic regret, as they take into account the distribution of strategy selection while strategies are just unknown constants throughout the execution of $M$?

Author response:
*Modeling:*  
2. We believe that there might be a misconception about the goals of incentive-aware online learning. The goal of $M$ is *not* to simply minimize regret given a problem instance $s$ and $\mu$. Instead, $M$ must induce desirable equilibria among arms, which thereby allows $M$ to minimize regret. Under a given mechanism $M$ there can be several NE some of which may be more favourable than others. For example, a mechanism $M$ may induce two NE, the first of which is in desirable strategies, e.g., $(s^*(\mu_1), \dots, s^*(\mu_K))$, and the second in undeseriable strategies $(1, \dots, 1)$ (same goes for distributions over such strategy profiles). Under the first, desirable NE the algorithm would be able to achieve low regret, however, under the second, highly undesirable NE the regret would always be large. Hence, the weak regret is small whereas the strong regret is large. Please let us know if this remains unclear.

Reviewer comment (Reproducibility):
I did not see the explicit definition of $\hat{\mu}^t_i$. Is it an estimate of $\mu_i$? If yes, then it is non-trivial to explicitly state how to get $\hat{\mu}^t_i$ by observing $c_{t, i_t}$ and $r_{t, i_t}$ at each round, and I wonder if Step 4 in UCB-S maximizing $\bar{\mu}^t_i$ aligns with the goal of maximizing $u(\cdot)$.

Author response:
*Algorithm design:*  

1. Since the distribution of $r_{t,i}$ has mean $\mu_{i}$, we can directly estimate $\mu_i$ from the observations of the post-click rewards: $\hat \mu_i^t = \sum_{l=1}^t \frac{1_{\{i_l=i\}} c_{l,i} r_{l,i}}{m_t(i)}$.

Reviewer comment (Novelty):
The algorithm UCB-S only estimates strategies and eliminates bad arms that seem like clickbait, while arms are responsible to pick proper strategies that are less likely to get eliminated. Although it might be tricky for arms to pick strategies, it seems to me that the design of UCB-S is straightforward UCB.

Author response:
*Algorithm design:*  

2. UCB-S is an upper confidence bounds-based algorithm just like UCB, but extended by the screening rule. Despite its simplicity, analyzing the dynamics between strategic arms and the learning algorithm (UCB-S) is challenging and the problem of incentivizing arms under (strategy-)uncertainty while minimzing regret has not been studied before. For this reason, we also believe that keeping the mechansim design simple is highly desirable so that the idea and parts of the analysis can more easily be reused and extended.

Reviewer comment (Theory):
In Section 4, it is not surprising that $(s, \mu)-$Oracle has linear weak strategic regret, since weak strategic regret considers the strategy distribution $\bf{\sigma}$ while the oracle considers only a realization of $\bf{\sigma}$ (In fact, UCB-S also considers a realization of $\bf{\sigma}$ instead of $\bf{\sigma}$, so I doubt whether UCB-S achieves sublinear strong strategic regret as stated in Theorem 5.3).

Author response:
*Regret analysis:*  

1. We are not sure that we fully understand the question. Please note that the $(s, \mu)$-Oracle is the best response to the actual realization of $\sigma$, i.e., $R_{T}((s, \mu)\text{-Oracle}, \sigma) = E_{s \sim \sigma} [R_T((s, \mu)\text{-Oracle}, s)]$ and since $E[\min_x f(x,y)] \leq \min_x E[f(x, y)]$, it is always going to perform better than the best response to a mixed strategy (i.e., something like a $(\sigma, \mu)$-Oracle). We hope that this clarifies things.  
Regarding UCB-S, note that the regret analysis of UCB-S builds on Theorem 5.2 which characterizes *all arm strategies $s$ in the support of any NE $\sigma$*. Hence, we can ensure that any realization of $\sigma$ is sufficiently close to the desirable arm strategies, which allows us to bound the strong strategic regret of UCB-S.

Reviewer comment (Theory):
I cannot go through all the proofs, but I can hardly follow the proof of Proposition 4.1. In the first glance, it seems to me that the proof is self-contradictory: in Case 1, it is stated that $u(s'+\varepsilon, \mu_{i^*})<u^*_{j^*}$ with $\varepsilon>0, s'+\varepsilon\leq 1$ contradicts the continuity of $u(s, \mu)$ in $s$, while the proof of Case 2 is based on the statement $u(s'', \mu_{i^*})<u^*_{j^*}$ with $s''>s'$. Please double check the proof.

Author response:
*Regret analysis:*  

2. Please note that $u(s'+ \varepsilon, \mu_{i^*}) < \mu_{j^*}^*$ for some $\varepsilon > 0$ does not generally contradict the continuity of $u(s, \mu)$, but only in Case 1. In Case 1, we have $u(s', \mu_{i^*}) > u_{j^*}^*$ so that together with the above we get $u(s' +\varepsilon, \mu_{i^*}) < \mu_{j^*}^* < u(s', \mu_{i^*})$ for all $\varepsilon > 0$ and $s'+\varepsilon \leq 1$. This is a contradiction to the continuity of $u(s, \mu)$ in $s$ if $s' < 1$, and therefore $s' =1$ must hold in Case 1 as claimed. To make the proof easier to parse, we added an additional sentence that explains that the contradiction of continuity that was used stems from $u(s', \mu_{i^*}) > \mu_{j^*}^*$ in Case 1 and not from the statement $u(s'', \mu_{i^*}) < u_{j^*}^*$ for all $s'' > s'$ with $s'' \in [0,1]$, which is always true by definition of $s'$.

Reviewer comment (Theory):
Theorem 5.3 and Corollary 5.4 provide a problem-dependent and a problem-independent regret upper bound respectively. It is counterintuitive that the problem-dependent bound is larger than the problem-independent bound, as it is often the case that problem-dependent bound is smaller since it additionally takes into account distributional information.

Author response:
*Regret analysis:*  

3. The regret bounds in Theorem 5.3 (eq. (3)) and Corollary 5.4 are stated order-wise. We can upper bound the second term in equation (3) by an expression of order $\sqrt{KT \log(T)}$. This is then summarized in Corollary 5.4 with the first term in equation (3) to obtain the order $O(\sqrt{KT\log(T)})$ regret bound. In other words, the regret bound in Corollary 5.4 takes form $c_1 \sqrt{KT \log(T)}+ c_2 \sqrt{KT \log(T)}$ for some constants $c_1, c_2 > 0$. Hence, the bound of Corollary 5.4 is not necessarily better than that of Theorem 5.3. Please also note that problem-independent bounds can be better than problem-dependent bounds. For example, when $\Delta_i = o(1/T)$. In such cases, the dependence on problem-dependent quantities such as $\Delta_i$ leads to vacuous regret bounds, which is why obtaining a problem-independent bound is useful.

Reviewer comment (Theory):
As previously mentioned, UCB-S considers a realization of $\bf{\sigma}$ as a fixed unknown parameter, and performs straightforward UCB, so I feel like the problem-dependent regret upper bound in Theorem 5.3 can be tightened to order $\log T$. It is of interest to provide a problem-dependent lower bound to show the tightness of the upper bound.

Author response:
*Regret analysis:*  

4. We want to emphasize that UCB-S (and incentive-aware learning in general) does *not* treat the agent strategies $\sigma$ as a fixed parameter, but aims to incentivize desirable choices of $\sigma$. To do so, UCB-S actively incentivizes agents (arms) in contrast to standard UCB, which fails to incentivize desirable NE. It is also important to us to stress that the first term in Theorem 5.3 is *not* due to the “bandit loss”, i.e., the learning of the reward distributions and the selection of suboptimal arms, but instead due to the fact that the strategic arms exploit our uncertainty (see Theorem 5.2), which causes $\sqrt{KT\log(T)}$ regret. Please also refer to our discussion of the regret bound in the paragraph below Corollary 5.4.

Reviewer comment (Theory):
Why does the strategic agent behind an arm get to choose the CTR? Is that a realistic assumption? For example, how could a video creator on Youtube choose their CTR? They could direct effort into improving their content which could contribute to increasing CTR, but allowing the agent to choose the CTR as a fixed value seems like a fairly unrealistic assumption.

Author response:
Thank you for taking the time to read and review our paper. You can find our answers to the two questions you highlighted below. 

1. **CTR as a chosen quantity:** We agree that the case of arms freely choosing click-rates might not portray real-world scenarios. However, please note that when arms are limited in their control of the click-rates, we can expect to suffer less regret (as we remove some of their power to strategize). Still, an additional analysis would be required to make this statement precise. As you already hinted at, an interesting way to relax the mentioned assumption would be to assume that arms can *repeatedly* put effort towards changing their CTR, but there is no guarantee that this effort will have an effect every round (thereby limiting the arms' power/control). This would of course require additional modeling assumptions and the mechanism design and analysis would have to be carefully adapted to these changes. However, we believe that such a problem may still be theoretically tractable (depending on the specific modeling assumptions). In any case, this would also be an interesting problem to study empirically.

Reviewer comment (Theory):
The strategic aspect is interesting, but one limitation of the model is that the agents are in a sense “stuck” at their particular reward mean $\mu_i$. So an agent’s incentive is solely based on strategically choosing a CTR based on the mechanics of the utility function and learning algorithm. But you would expect that realistically an agent might also put in effort into improving $\mu_i$ (e.g., by improving or changing the quality/genre/tone of content they are producing in a recommender system setting). I feel like that’s an important component that would make this model more compelling, and studying the interplay between clickbait actions (strategically choosing $\mu_i$) and quality-improving actions would be very interesting.

Author response:
2. **Arms changing their mean reward:** Thank you for the interesting question which points at some exciting extensions to our work. Your question is also closely related to your other comment about modeling agent utilities. To give the arms more agency, we could, for instance, allow the arms to change their mean rewards each round at a cost. A simple, natural choice could be that the utility of arm $i$ in round $t$ is given by $v_{t,i} = \lambda_1 1_{\{i_t=i\}} c_{t,i} - \lambda_2 |\mu_{t,i}- \mu_{t-1,i}|$, i.e., arm $i$ has utility $\lambda_1$ for a click minus the cost $\lambda_2$ times distance for changing its mean reward.  We naturally expect the analysis of arm behavior to become more complex (arms now take an action every round). Moreover, in this case a detailed characterization of arm behavior could be very intriguing since we would expect an additional dependence on the parameters of the arm utilities (here, $\lambda_1$ and $\lambda_2$). We believe that there are many similar and interesting extensions in this direction, which can build on our work.

Reviewer comment (Theory):
I also think some more work and thought needs to be given on the formulation of agent utilities. The present results hold under some assumptions on utilities like Lipschitnzess and monotonicity of maximizers, and the utility is also based on a single-shot report of the strategically-chosen CTR which seems like a limiting assumption since the agents would probably change their behavior across time steps. But these aspects can be a good direction for future work and doesn’t diminish my opinion of the paper too much.

Author response:
2. **Arms changing their mean reward:** Thank you for the interesting question which points at some exciting extensions to our work. Your question is also closely related to your other comment about modeling agent utilities. To give the arms more agency, we could, for instance, allow the arms to change their mean rewards each round at a cost. A simple, natural choice could be that the utility of arm $i$ in round $t$ is given by $v_{t,i} = \lambda_1 1_{\{i_t=i\}} c_{t,i} - \lambda_2 |\mu_{t,i}- \mu_{t-1,i}|$, i.e., arm $i$ has utility $\lambda_1$ for a click minus the cost $\lambda_2$ times distance for changing its mean reward.  We naturally expect the analysis of arm behavior to become more complex (arms now take an action every round). Moreover, in this case a detailed characterization of arm behavior could be very intriguing since we would expect an additional dependence on the parameters of the arm utilities (here, $\lambda_1$ and $\lambda_2$). We believe that there are many similar and interesting extensions in this direction, which can build on our work.

Reviewer comment (Theory):
The problem setting where the arms can arbitrarily manipulate their click rates from 0 to 1 is an unrealistic assumption. In most real-world online recommendation systems, it is impossible for the item to get arbitrarily close to 1 click rate. There will be limits on the number of items that can appear on the webpage and there will be competition among the item providers, either through sponsors or genuinely improving the item's quality. Due to the limit, only some items will get high click rates but not all. I think the strategic MAB problem is more like an information elicitation problem than a strategic recommendation problem.

Author response:
**Arms ability to control click-rates:** Allowing the arms to arbitrarily select their click-rate results in a worst-case scenario from the perspective of the learner. If the arms had limited power, we would generally incur less regret. However, some adaptations to the mechanism and an additional analysis is required to make this statement precise. The case where each arm can only change their click-rate within a range (smaller than $[0,1]$) is an interesting direction for future work.

**Questions:**

2. One way to model such a setting would be to consider the case where the platform selects a *set* of arms each round and only one of the arms is clicked depending on their strategies. For instance, a natural click-model could be the MNL-model. In this case, the arms’ strategies no longer correspond exactly to their click-rate, but instead determine an arm’s click-rate *relative* to other arms (see also future work discussed in Appendix I).

Reviewer comment (Novelty):
It is not easy to understand how the strategic setting in this work is different from previous works in strategic MAB, the authors should consider adding a table with each work's setting, assumptions, solution existence/uniqueness, convergence, and regret bounds to help the readers better understand the novelty.

Author response:
**Comparison to other work:** Thank you for the suggestion. To the best of our knowledge, our paper is the first to study the situation where the agent strategies (and other parameters) are initially unobserved and must be learned from interaction while simultaneously incentivizing agents under uncertainty. It is therefore difficult to directly compare to other work along aspects like regret, and we found that the comparison of the regret to standard online learning problems such as MABs is the most insightful as this highlights the cost and challenges of mechanism design in online learning (see e.g. the discussion below Corollary 5.4).

Reviewer comment (Theory):
The motivation of the propose strategic click-bandit setting, modeling recommandation systems, is not quite convincing. Basically, vendors are willing to choose proper item descritions to improve its click-rate, but they are not able to controll the click rate. Especially in the motivated example of clickbait, such a behavior is definitely harm the click-rate in long time. However, in proposed the bandit setting, each arm chooses the click-rate arbitrarily and such a probability will be fixed forever.

Author response:
We agree that we study a simplified model in which we assume that each arm can freely choose their click-rate. However, this leads to a worst-case scenario from the perspective of the learner and when the arms are limited in their control over the click-rates we generally expect to incur less regret. Regarding clickbait and how clickbait influences click-rates, note that that the click-rate which the vendors can influence can be thought of as the click-rate of a new user without prior exposure to the specific item, hence, the click-rate wouldn't be necessarily harmed over time. Nevertheless, the extension to click-rates which change over time (whether caused by the arms or other sources) is an interesting direction for future work. The case where the arms can only change their click-rate within a limited range is also interesting.

Reviewer comment (Theory):
The theoretical analysis is based on the assumption that Nash equilibrium can be achieved by arms, which may be not realistic. On the other hand, the setting in the experiments that each arm optimize itself by gradient descend would be more realistic one, but maybe much harder to obtain theoretical result.

Author response:
We completely agree. For this reason we wanted to support the developed theory with our experiments which assume a natural model of gradually adapting arm behavior as modeled by gradient descent. We were happy to observe that our incentive design is also effective under such models of arm behavior and that our theoretical results are direclty corroborated by the experiments (e.g., in the experiments we also observe that the optimality gap of each arm governs their behavior/strategy). While challenging, theoretically analyzing such "learning" agents' behavior is definitely an interesting direction for future work but out of the scope of this paper.