Paper: Multiobjective Stochastic Linear Bandits under Lexicographic Ordering (ICLR-2024)

Reviewer comment (Writing):
There seems to be a major typo among the whole writeup: do both $m$ and $d$ stand for the dimension of vectors? If so, the author(s) should consider to unify the notation. If not, may you clarify what do they mean individually?

Author response:
We investigate the multiobjective stochastic linear bandit problem. $m$ denotes the number of objectives, and $d$ is the dimension of arm vector. We will revise our paper to make the notation clear in the final version.

Reviewer comment (Theory):
As no lower bound is provided, is there still space to improve the algorithm? What is the difficulty to provide a lower bound? As Huyuk & Tekin (2021) provided a lower bound for the priority-based regret, is it possible to derive a lower bound on the general regret similarly? Why are we interested in the general regret?

Author response:
We are working hard to construct a lower bound for the problem investigated in our paper, thus we are unable to provide a definitive answer for this question now.

Reviewer comment (Novelty):
The STE$^2$LO and MTE$^2$LO algorithms seem to have similar structures to the SupLinRel and SupLinUCB algorithms.

Author response:
Although our algorithm MTE$^2$LO has a similar structure to SupLinRel and SupLinUCB, the purpose of applying this framework is completely different. SupLinRel and SupLinUCB utilize this sophisticated framework to guarantee independence among rewards, while we use this framework to ensure that the confidence intervals of the arm input to Algorithm 2 (LOAF) are bounded. The effectiveness of LOAF requires the confidence interval of the input arms to be bounded as we stated in Proposition 1.

Our analysis faces two challenges. The first challenge lies in proving Proposition 1, where we not only demonstrate that the novel arm filter LOAF can maintain the lexicographic optimal arm, but also provide a bound to measure the quality of the filtered arms. Mathematical induction is an important tool to prove Proposition~1. The second challenge is analyzing the multi-stage decision-making process (Step 4 to Step 15) in Algorithm 3, which delicately balance the exploration and exploitation. Here, we give a proof sketch for the analysis of multi-stage decision-making process and more details can be found in Appendix D. 

Let $\psi_s(T)$ be the set composed of the number of arms with confidence intervals larger than $2^{-s}$. The first step is to prove that $|\psi_s(T)|$ is bounded by $\tilde{O}(2^s\sqrt{\psi_s(T)})$, as shown in Lemma 4. Note that the arm selected in the $s$-th stage is filtered by LOAF in the $(s-1)$-th stage, thus Proposition~1 shows that the instantaneous regret of these arms is bounded by $\tilde{O}(2^{-(s-1)})$. The last step is to accumulate all instantaneous regret, $\sum_{s=1}^{S}2^{-(s-1)}\cdot2^{s}\sqrt{\psi_s(T)}=\tilde{O}\sqrt{T}$. Further discussion on these challenges will be provided in the final version.

Reviewer comment (Theory):
The two proposed algorithms seem to be adopted versions to the multi-objective setting. May the author(s) clarify what are the analytical challenge?

Author response:
Although our algorithm MTE$^2$LO has a similar structure to SupLinRel and SupLinUCB, the purpose of applying this framework is completely different. SupLinRel and SupLinUCB utilize this sophisticated framework to guarantee independence among rewards, while we use this framework to ensure that the confidence intervals of the arm input to Algorithm 2 (LOAF) are bounded. The effectiveness of LOAF requires the confidence interval of the input arms to be bounded as we stated in Proposition 1.

Our analysis faces two challenges. The first challenge lies in proving Proposition 1, where we not only demonstrate that the novel arm filter LOAF can maintain the lexicographic optimal arm, but also provide a bound to measure the quality of the filtered arms. Mathematical induction is an important tool to prove Proposition~1. The second challenge is analyzing the multi-stage decision-making process (Step 4 to Step 15) in Algorithm 3, which delicately balance the exploration and exploitation. Here, we give a proof sketch for the analysis of multi-stage decision-making process and more details can be found in Appendix D. 

Let $\psi_s(T)$ be the set composed of the number of arms with confidence intervals larger than $2^{-s}$. The first step is to prove that $|\psi_s(T)|$ is bounded by $\tilde{O}(2^s\sqrt{\psi_s(T)})$, as shown in Lemma 4. Note that the arm selected in the $s$-th stage is filtered by LOAF in the $(s-1)$-th stage, thus Proposition~1 shows that the instantaneous regret of these arms is bounded by $\tilde{O}(2^{-(s-1)})$. The last step is to accumulate all instantaneous regret, $\sum_{s=1}^{S}2^{-(s-1)}\cdot2^{s}\sqrt{\psi_s(T)}=\tilde{O}\sqrt{T}$. Further discussion on these challenges will be provided in the final version.

Reviewer comment (Theory):
The fundamental parameter $\lambda$ is assumed to be known in this work. I appreciate the elaboration of importance of $\lambda$ at the bottom of page 2. However, is it reasonable to assume that $\lambda$ is known?

Author response:
The dependence on $\lambda$ is indeed a limitation, as mentioned in the conclusion and future work section. However, in most multi-objective scenarios, it is possible to estimate $\lambda$. As we have discussed under Equation (5), $\lambda$ can be estimated by measuring how different objectives change as the decision varies. To further illustrate this, we provide a simple example with two objectives. 

The expected reward functions are denoted as $\mu^1(x)$ and $\mu^2(x)$, which is defined as $\mu^1(x)=1-\min_{p\in\\{0.4,0.8\\}}|x-p|$ and $\mu^2(x)=1-2|x-0.3|$ (Here we do not take a linear function so as to highlight the change rates of  expected reward functions). The optimal arms for the first objective are $\\{0.4, 0.8\\}$, and the lexicographic optimal arm is $0.4$. In this example, the value of $\lambda$ is 2, which is the ratio of the change rates of $\mu^1(x)$ and $\mu^2(x)$.

Another point worth mentioning is that it is not necessary to know the **precise** ratio of the change rates. Any value **greater** than the ratio of the change rates is sufficient to address the multi-objective problem. This is feasible because different objectives are related to each other in many applications, and historical data and expert knowledge can provide this greater value.

Reviewer comment (Writing):
The abstract states that 'This model has various real-world scenarios, including water resource planning and radiation treatment for cancer patients.' However, a more detailed description of real-life application is appreciated. I surmise that this may help me to understand why we can assume that $\lambda$ is known.

Author response:
We first provide a more detailed discription of two real-world applications metioned in our paper. One practical application is radiation treatment for patients, where the primary objective is target coverage and the secondary objective is the proximity of therapy to organs at risk (Jee et al., 2007). Another one is water resource planning for optimizing flood protection, supply shortage for irrigation, and electricity generation (Weber et al., 2002), where the water authorities legally mandate the prioritization of these three objectives and release policies over various time horizons. 

Based on the discussion in \textbf{Q6}, we only need to obtain an upper bound for the ratio of the change rates of different objectives. In above two applications, historical data and expert knowledge can provide this upper bound.

Reviewer comment (Writing):
In the example given in the first paragraph, what does 'click-conversion rate' mean?

Author response:
The click-conversion rate (CVR) refers to the percentage of visitors who perform a specific action on a website compared to the total number of visitors. For example,  CVR of e-commerce websites is the proportion of people who click and make purchases out of the total number of clicks.

Reviewer comment (Experiments):
Moreover, experiments with real-world data may make the efficiency of proposed algorithms more convincing.

Author response:
Thank you for raising this issue. Following the experiments of [1], we will do more experiments by incorporating the Yahoo! Webscope dataset R6A in the final version. R6A is a real-world dataset collected from a personalized news recommender system, whose first objective is the click-through rate (CTR) and the second is the average payment from advertisers.

*[1] Cem Tekin and Eralp Turgay. Multi-objective contextual multi-armed bandit with a dominant objective. IEEE Transactions on Signal Processing, 66(14):3799–3813, 2018.*

Reviewer comment (Presentation):
I do appreciate that the author(s) introduced the standard linear setting first and the multiobjective one after that. However, the notations are defined throughout the first 3 pages. I think that a notation table may help readers to find notations.

Author response:
We appreciate your constructive reviews. We will incorporate two tables for comparing existing works and presenting the notations in the final version of our paper.

Reviewer comment (Writing):
In the second paragraph of the introduction, it states that 'Therefore, if the evaluation criterion is Pareto regret, the agent can select any of the m objectives to optimize and ignore other objectives, which is unreasonable.' This statement is indeed strong. The author(s) may consider to explain this point a little bit more.

Author response:
Theorem~$4.1$ of Xu $\\&$ Klabjan (2023) states that Pareto regret is smaller than the regret of any objective $i\in[m]$. Thus, we can achieve the nearly optimal Pareto regret bound by applying the UCB strategy to only one of the $m$ objectives. However, the remaining $m-1$ objectives still suffer linear regret bounds $O(T)$.

Thank you for your constructive reviews. We will make a more detailed discussion about this point in the final version.

Reviewer comment (Writing):
Minor suggestion in the second paragraph of Section 3.1: 'We provide a formal definition of the chain relation to facilitate our presentation' may be a better expression than 'We give a formal definition of the chain relation to facilitate our presentation'.

Author response:
Thank you for your detailed reviews. We will revise our paper according to your suggestions.

Reviewer comment (Theory):
Related to the above question, consider the same $\theta$ values in my first example, and let $x_A = (1+\epsilon, 0)$, $\epsilon>0$ and $x_B = (1,1)$. Clearly, $x_A$ is the lexicographic optimal arm. For these arms, Equation 5 in the paper gives $1 \leq \lambda \epsilon$. Thus $\lambda \geq 1/\epsilon$. Let $\epsilon = T^{-1/3}$. According to Theorem 2, $R^2(T)$ of MTE$^2$LO becomes $\tilde{O}(T^{2/3})$. One should be able to construct similar examples for the case with more than two objectives. When the number of objectives increases, I expect the time order of the regret upper bound to get even worse since $i$th objective’s time order depends on $\lambda^{i-1}$.

Author response:
**Q4: One would expect that if the problem instance is adversarially chosen, then the algorithm proposed in the current work will not be able to achieve the claimed $\tilde{O}(\sqrt{T})$ upper bound.**

As we have discussed in **Q3**, $\lambda$ in the multi-objective bandit problem is a instance-dependent parameter similar to the number of arms $K$ in MAB and the feature dimension $d$ in SLB. To illustrate the adversarially chosen issue, we use the single objective SLB as an example. The upper regret bound of the single objective SLB is $\tilde{O}(d\sqrt{T})$. An adversary can select a decision set where $d$ becomes a function of $T$, resulting in a higher order dependence on $T$ rather than just $\sqrt{T}$. However, we cannot conclude that the upper regret bound of the single objective SLB is not $\tilde{O}(\sqrt{T})$.

The parameter $\lambda$ captures the complexity of identifying the lexicographic optimal arm, and it does not depend on $T$ for a fixed bandit problem. Therefore, $\tilde{O}(\lambda^{i-1}d\sqrt{T}/T)$ tends to zero at a rate of $\tilde{O}(1/\sqrt{T})$ as $T$ approaches infinity, which is the same rate as the single objective algorithm.

Reviewer comment (Novelty):
Paper fails to reasonably motivate why a simple algorithm such as mentioned below would not do better - If we have the lexicographic ordering as defined in the paper then why not apply a simple OFUL just for 1st objective and if there are multiple arms at the end of 1st OFUL then apply OFUL for second objective and so on? Would this not achieve similar to abbasi regret?

Author response:
OFUL selects the arm with the highest upper confidence bound. For the 1st objective, OFUL cannot guarantee that all arms with the highest expected rewards have the same highest upper confidence bound due to the uncertainty in the upper confidence bound. As a result, OFUL may only output a **single** arm and fail to consider the remaining $m-1$ objectives.

Reviewer comment (Theory):
Further the theorem's mentioned in the paper do not talk about the finiteness or infinite decision set $\mathcal{D}$, if the decision set is finite then the algorithm needs to be compared with that of SupLinUCB regret bounds, and if the decision set is infinite then how is it even possible to filter the arms based on chain relations?

Author response:
We present two algorithms to deal with the multi-objective bandits under lexicographic ordering. The first algorithm, STE$^2$LO, can only handle a finite decision set ($|\mathcal{D}_t|=K$), as we stated in Theorem 1. STE$^2$LO filters the arms by the chain relation. The second algorithm, MTE$^2$LO, is designed to handle the infinite decision set case. We design a novel filter called LOAF, which utilizes the intersection of scaling confidence intervals to filter the infinite arms. More details about the motivation for LOAF is explained at the beginning of Section 3.2.

Reviewer comment (Theory):
I fail to understand how is the new regret formulation different from the previous regret formulation with indicator? Because if the indicator function is false then that would inherently increase the regret of the objective for different $i$.

Author response:
The priority-based regret $\widehat{R}^i(T)$ of Hüyük $\\&$ Tekin (2021) is defined as follows:
$$
\widehat{R}\^i(T)=\sum\_{t=1}\^T\langle \theta\_\*\^i, x\_t\^\* - x\_t\rangle\mathbb{I}(\langle \theta\_\*\^j, x\_t\^\*\rangle = \langle \theta\_\*\^j, x\_t\rangle, 1\leq j\leq i-1), i=1,2,\dots,m
$$
where $x_t^\*$ denotes the optimal arm in $\mathcal{D}\_t$ according to the lexicographic order. Note that the priority-based regret $\widehat{R}^i(T)$ relies on the indicator function $\mathbb{I}(\cdot)$, which cannot accumulate the instantaneous gap $\langle \theta\_\*^i, x\_t^\* - x\_t\rangle$ for $i\geq 2$ if $\langle \theta_\*^1, x\_t\rangle < \langle \theta\_\*^1, x\_t^\*\rangle$. Thus, $\widehat{R}^i(T)$ can not accurately measure the performance of agent for $i\geq2$.

Our proposed regret $R^i(T)$ is a commonly used metric in the single-objective bandit field (Auer, 2002; Bubeck $\\&$ Cesa-Bianchi, 2012; Lattimore $\\&$ Szepesvári, 2020), and we simply extend it to $m$ objectives, such that
$$
R^i(T)=\sum\_{t=1}\^T\langle \theta_\*^i, x_t^\* - x_t\rangle, i=1,2,\dots,m.
$$
$R^i(T)$ removes the indicator function, enabling the accumulation of the rewards gap between the selected arm $x_t$ and the lexicographically optimal arm $x_t^*$ independently for all objectives $i=1,2,\ldots,m$. This makes it a more reliable metric for reflecting the algorithms' performance.

Reviewer comment (Novelty):
In section 2.2, it's mentioned that Lu et al., achieves Pareto regret bound that is optimal, doesn't Pareto optimal regret imply lexicographical optimal regret? How does this work go beyond this prior work?

Author response:
Theorem~$4.1$ of Xu $\\&$ Klabjan (2023) states that Pareto regret is smaller than the regret of any objective $i\in[m]$. Thus, we can achieve the nearly optimal Pareto regret bound by applying the UCB strategy to only one of the $m$ objectives. However, the remaining $m-1$ objectives still suffer linear regret bounds $O(T)$. Our proposed algorithm MTE$^2$LO achieves the nearly optimal regret bounds $\tilde{O}(d\sqrt{T})$ for all objectives simultaneously. The key innovations of our algorithm include the introduction of a new arm filter called LOAF and the implementation of a multi-stage decision-making strategy (Step 4 to Step 15 of MTE$^2$LO).

Reviewer comment (Theory):
There is little to no intuition on why there are 3 different phases in $MTE^2LO$.

Author response:
One important reason for dividing the decision-making operation into three parts is that the LOAF algorithm requires the width of the confidence interval for input arms to be no greater than $W$. MTE$^2$LO utilizes LOAF as a subroutine to filter arms. Therefore, MTE$^2$LO divides the decision-making operation into three parts in order to incorporate LOAF in the bounded confidence interval cases ("if" and "else" cases). To further strike a balance between exploration and exploitation, the decision-making process in each round is divided into $S$ stages, enabling the achievement of the nearly optimal regret bound $\tilde{O}(d\sqrt{T})$.

Reviewer comment (Writing):
On Page 5, what is $SCE^2LO$ — is it a typo for $STE^2LO$?

Author response:
Thank you for raising this issue. We will make a detailed revise to avoid typos in the final version.

Reviewer comment (Theory):
**Applying single objective MAB algorithms multiple times**: Would the authors provide arguments as to why it wouldn't be a good idea to just use a standard single objective MAB routine multiple times based on the objective priority order and solve the MOMAB setup? Even with a naive bound, regret of $\mathcal{O}(md\sqrt{T})$ would be achievable. This compared to the current regret bound in this paper $\mathcal{O}((\sum_{i}\lambda^i)d\sqrt{T})$ isn't clear to be a guaranteed improvement at first glance. I am willing to be completely wrong about this point.

Author response:
Applying a standard single objective MAB routine multiple times can only achieve a linear regret bound of $O(T)$. To illustrate this issue, we consider a 2-objective problem. Specifically, when applying a standard UCB strategy to the first objective, it outputs a **single** arm $\hat{x}_t^1$ that is promising for the first objective. However, the second objective is not taken into account at all, resulting in a linear regret bound $O(T)$ for the second objective. Therefore, we design a mechanism that can filter **a group of** arms $D_t^1$ from $D_t$, where $D_t^1$ are promising for the first objective. Then, we select the most promising arm for the second objective from $D_t^1$, which can guarantee sublinear regret bounds for both the first objective and second objective. The detailed procedure of our designed filter mechanism is presented in Algorithm 2, and its theoretical guarantee is provided by Proposition 1.

Reviewer comment (Theory):
**Intuition and necessity of $\lambda^i$**: Firstly, the paper keeps referencing $\lambda$ when technically it is $\lambda^i$ for each objective (please correct me if this is not the case). Secondly, $\lambda^i$ serves to establish some sort of regularity between the different objectives. I fail to see the need or necessity of doing so. If this is utterly needed, can you provide a counter-example of things going completely haywire in the absence of such regularity condition? Personally, I am just confused about the need for such a regularity. Hope the authors can shed some light on this.

Author response:
Our first algorithm, called STE$^2$LO, does not utilize $\lambda$ and can only achieve a regret bound of $\tilde{O}(T^{2/3})$, which is far from the nearly optimal regret bound of $\tilde{O}(\sqrt{T})$. In contrast, our second algorithm, MTE$^2$LO, incorporates the parameter $\lambda$ and improves the regret bound to $\tilde{O}(\sqrt{T})$. The main purpose of $\lambda$ is to scale the confidence interval, which prevents the absence of the optimal arm during the filtering process in the algorithm LOAF. For more details on the motivation behind adopting $\lambda$, please refer to the beginning of Section 3.2.

Reviewer comment (Theory):
Is forced exploration required, can we not employ UCB-like schemes that take care of exploration-exploitation inherently? Why was this three-part exploration-exploitation preferred over UCB like universal choice?

Author response:
To our knowledge, UCB-like strategies cannot solve the problems studied in our paper. One important reason for dividing the decision-making operation into three parts is that the LOAF algorithm requires the width of the confidence interval for input arms to be no greater than $W$. MTE$^2$LO utilizes LOAF as a subroutine to filter arms. Therefore, MTE$^2$LO divides the decision-making operation into three parts in order to incorporate LOAF in the bounded confidence interval cases ("if" and "else" cases). To further strike a balance between exploration and exploitation, the decision-making process in each round is divided into $S$ stages, enabling the achievement of the nearly optimal regret bound $\tilde{O}(d\sqrt{T})$.