Paper: Maximizing Benefits under Harm Constraints: A Generalized Linear Contextual Bandit Approach (ICLR-2024)

Reviewer comment (Experiments):
The experimental results only include synthetic studies and very minimal. Experimenting with real-world public datasets could be insightful for instance for the model misspecification accounts.

Author response:
**No real data experiment.**
  Since there is no real experiment conducted according to the algorithm, we cannot provide real data analysis.

Reviewer comment (Experiments):
How can a practitioner come up with $\theta$ and $\lambda$? these are hyperparameters of your algorithm and do not directly translate into their goals.

Author response:
**How can a practitioner come up with $\lambda$ and $\theta$?**
  We do not consider $\theta$ as a hyperparameter. $\theta$ is the upper bound of harm that practitioners want to force in the experiment. A major example of this work is clinical trials where the typical $\theta$ value can be 0.166 or 0.33. $\lambda$ will affect the probability of the event "chosen arm has harm $p >\theta$". So intuitively, if practitioners want this probability to be low, they can specify an arbitrarily large $\lambda$.

Reviewer comment (Presentation):
The graphs representing the average regret and average count of $p > \theta$ could benefit from representing also confidence intervals along with the mean.

Author response:
* **Confidence intervals along with the mean in the plots.**
Thank you for the suggestion. We will add confidence bands in the plots.

Reviewer comment (Novelty):
The comparison to related work in this paper is not sufficiently clear. In the section comparing this work to previous MOMAB papers, the authors claim that "this paper is the first to consider MOMAB in a contextual setting." However, it seems that this paper revolves around a single-objective problem, as in equation (1) of the paper. This approach doesn't involve the optimization of Pareto regret, which as far as I know is the central topic in prior MOMAB works. Thus I think it is not so fair to make direct comparsion to previous MOMAB works.

Author response:
Thank you for pointing this out. Although scalarization is a plausible approach to solve MOMAB, it is not the current focus of MOMAB (Pareto regret). Since originally we have 2 objectives, our work is related to MOMAB, and we would like to keep the MOMAB part in the literature review. But we have removed the claim and emphasized we do not work on the Pareto regret. We also think more deeply about why the Pareto regret may be not a good choice for us, see the response to Reviewer GyGV.

Reviewer comment (Theory):
The assumptions made in this paper is implicitly strong but not sufficiently clear: Theorem 1's results rely on the existence of a positive lower bound for $\lambda_{min}(\Sigma_1)$ and $\lambda_{min}(\Sigma_2)$. Implicitly, these bounds necessitate a diversity assumption concerning the distribution of context and action features. Such a strong assumption significantly simplifies the algorithm design of online decision-making. However, I was unable to locate this assumption clearly spelled out in Assumptions A1 to A3. It would be helpful if the authors could present such strong assumption in a more easily identifiable way.

Author response:
Thank you for your careful thoughts. We have added assumption A4 for $\Sigma_1$ and $\Sigma_2$ to be positive definite.

Reviewer comment (Writing):
There is an opaqueness in the modeling choices at various places in the paper. For example, the choice of \(u_{i}\) for arms — why are they scalar values, what is the physical interpretation, and why are they increasing across arms? Does that mean that arm \(1\) and arm \(K\) are best and worst or vice versa?

Author response:
* **The choice of $u_i$ for arms:** 
$u_1\leq u_2\leq\cdots \leq u_K$ is just an assumption for simplicity. Otherwise one may re-order the arms to achieve this rank. It does not mean arm 1 to $K$ are best to worst, or vice versa. Since there is context information $X$, the best arm will be different for different values of $X$. So the order of $u_k$ does not imply the optimality of arms in any sense. The physical meaning of $u_k$ will depend on the problem at hand. For example, in clinical trials, $u_k$ may mean the dose level ($u_k$ mg/ml of medicine) at treatment $k$ (see 1st paragraph of section 3). This notion of $u_k$ is also used in [3, 20] as shown in table 1.

Reviewer comment (Theory):
There is no explanation for why equation (5) is linear in \(u_k\) but equation (8) is quadratic. Is this coming from some existing models?

Author response:
* **Why is equation (5) linear but equation (8) quadratic? Is it coming from some existing models?** 
It stems from dose-repsonse models but we find it reasonable in general. We may assume the harm is increasing with the level of treatment (dose, for example), but benefit may not be monotone. So equation (5) for harm is chosen to be linear so that it is easy to make it monotonely increasing. Equation (8) is quadratic to allow increasing then decreasing behavior. The linear and quadratic functions are only examples of modeling choice, as explained below equation (8).

Reviewer comment (Writing):
Certain places seem to have missing definitions. For example, in equations (4), (5), (7), and (8) — what are the functions \(p\), \(q\), \(g\), \(h\), \(\zeta\), \(\xi\)? Why are these equations the way they are?

Author response:
**Missing definitions**
We have added more explanation in the paper.
$p$ and $q$ are the means of harm and reward, respectively. $g$ and $h$ are the inverses of link functions for the GLM, as explained below in equations (5) and (8). Some examples of link functions are given in 3 paragraphs below equation (5).
$\zeta$ and $\xi$ denote the systematic component of the GLM and their forms are defined in equations (5) and (8), respectively.

Reviewer comment (Theory):
There is no explanation or intuition as to why the upper bounds obtained in the paper could be tight. It would be good to have a quantification of the suboptimality of the upper bounds in terms of the dimensional dependence.

Author response:
**Missing lower bound**
We do not have a lower bound for the regret ready. However, the bound of rate $\sqrt{T\log T}$ matches the state-of-art upper bound for generalized linear contextual bandit problems in literature.

Reviewer comment (Writing):
The algorithm is not really \(\epsilon\)-greedy. Maybe change the name or reference it as \(\epsilon_t\)-greedy, since \(\epsilon\)-greedy would give the impression of permanent forced exploration based on static \(\epsilon\), which is not the case here.

Author response:
* $\epsilon_t$ greedy.
Thank you for your suggestion. We have changed the algorithm name to $\epsilon_t$ greedy. We did not name "varying coefficient" as "$\epsilon_t$-greedy" as all the methods under comparison are under the $\epsilon_t$-greedy framework, but different modeling choices.

Reviewer comment (Theory):
The complexity of each inner loop of the \(\epsilon\)-greedy algorithm is not discussed.

Author response:
* What is the complexity of each inner loop of the $\epsilon$-greedy algorithm?
We need to fit GLM in each inner loop for $t-1$ data points at round $t$. Using the notation of the paper, the dimension of covariate for harm model is $2d_1$ ($\Phi(X_t)$ is $d_1$-dimensional) and the dimension of the covariate for reward model is $3d_2$, then the complexity for fitting 2 GLM is $O(t(d_1^3 + d_2^3))$.

Reviewer comment (Novelty):
A suggestion is to rename the algorithm to "\(\epsilon\)-greedy" rather than "variable coefficient" (assuming they are the same).

Author response:
* $\epsilon_t$ greedy.
Thank you for your suggestion. We have changed the algorithm name to $\epsilon_t$ greedy. We did not name "varying coefficient" as "$\epsilon_t$-greedy" as all the methods under comparison are under the $\epsilon_t$-greedy framework, but different modeling choices.

Reviewer comment (Theory):
There is confusion about the connection with Multiobjective MAB. Can the framework not tackle the setup of reward and harm? What are the exact challenges in extending the MOMAB framework to encompass the current problem?

Author response:
**Connection with Multiobjective MAB**
Our scalarization formulation is from MOMAB, but with a specific focus on controlling harm, i.e., we penalize the harm probability when it exceeds the harm threshold. As for the popular Pareto formulation in MOMAB, it may not suit the needs of controlling harm very well. Consider a simple example, let $(p,q)$ denote a pair of mean harm $p$ and mean reward $q$, and there are totally two possible arms $(p,q) = (0.3, 0.7)$ and $(p,q) = (0.6, 0.8)$. Remember we prefer smaller $p$ and larger $q$. Then both arms $(0.3, 0.7)$ and $(0.6, 0.8)$ are in the Pareto optimal front, since they don't dominate each other. However, a harm of $p = 0.6$ may be unacceptable.