Paper: Bandits with Replenishable Knapsacks: the Best of both Worlds (ICLR-2024)

Reviewer comment (Theory):
There are some implicit assumptions that are not well justified. Firstly, the authors say that B = \omega (T) which seems not practical as it is not evident in which real-world situations a direct linear relationship exists between the budget and the total number of rounds T. A constant budget will result in a non-sublinear final upper regret bound. Secondly, the range of the replenishable resource \beta is [-1,0] which indicates that such amount of source is close to the consumption cost of other action. This, in turn, suggests that the policy will frequently favor the empty action once the budget B is depleted. It would be beneficial to offer motivating real-world scenarios that illustrate the relevance and feasibility of such a resource range. Finally, in the example of the inventory management in Section 6.2, the reward action s is r_t(s) \in [-1,0] which is not consistent with the setting in Section 2. Please give a clear justification for this variation.

Author response:
**On the $B=\Omega(T)$ assumption:** We do *not* need to assume $B=\Omega(T)$. Indeed, one of the main contributions of our paper is to show that when $B=0$ and the replenishable factor is constant, we can still achieve $O(\sqrt{T})$ regret in the stochastic setting and constant competitive ratio in the adversarial setting.  Nonetheless, we agree with the reviewer that our results are meaningful only when the budget or the replenishment factor is ``large enough''. This is common to all the previous work on BwK in which the regret depends on $B/T$. We observe that this is not a limitation of our analysis but rather an **intrinsic feature of the problem**. Indeed, Balseiro and Gur (2019) provide a lower bound $\Omega(B/T)$ on the competitive ratio in the adversarial setting without replenishment. Hence, it is impossible to obtain a constant competitive ratio in general. 

**Constant budget:** The observation is correct only if there is no replenishment. If the replenishment factor $\beta$ is constant, we obtain meaningful regret bounds (e.g., a $O(\sqrt{T})$ in the stochastic setting). In the extreme case when the budget is constant and there is no replenishment our theorems hold but become meaningless. However, this is fine since **any** algorithm fails in this setting, as it does not have enough time to learn before depleting the budget (think about the case in which the initial budget is $1$ and it is depleted at the first round if an expensive action is played). We remark that this problem is present in **all** the algorithms for BwK.  For instance, the regret in BwK problems depends on $B/T$ (see, e.g., Castiglioni et al. (2022)) while for the case of bandits with replenishable knapsacks  Kumar and Kleinberg (2022) provide a regret bound that depends on the replenishment factor (as in our work).

**Regarding the assumption about the replenishment range:** this is without loss of generality since we can always scale the budget, costs, and replenishments to have replenishments $\beta$ in $[-1,0]$. So it might be the case that the costs and replenishments are similar, that costs are close to 1, and the replenishments are very small, or the opposite. Our algorithm can deal with all these scenarios. As is the case in previous works, this flexibility of the model and algorithm motivates the choice of keeping the problem at a high level of abstraction without the need to focus on specific applications.

**On inventory management:** In the problem's original formulation, the reward can be negative. However, this problem can be trivially solved by normalizing the rewards in [0,1]. Formally, we add 1 to the reward and divide it by 2. This does not affect the results in the stochastic setting. As mentioned in the paragraph ``Challenges of the adversarial setting'', rescaling and translation introduce some challenges in the adversarial settings, which we leave as an interesting open problem.

Reviewer comment (Novelty):
The Meta-Algorithm proposed in this paper appears to be a trivial extension of the one presented in "Online Learning with Knapsacks: the Best of Both Worlds" (2022). The main difference is that the policy does the empty action when the budget is smaller than one. Furthermore, in terms of theoretical analysis, although the concept of a replenishable resource is introduced, the proofs for Theorem 4.1 and Theorem 4.3 do not exhibit significant advancement when compared to the proofs for Theorem 6.1 and Theorem 7.1 of the 2022’s work. Please clarify the primary contribution compared with previous literature.

Author response:
**Relation with prior work:** From a high-level, many studies on BwK may appear similar in their approaches, yet they differ fundamentally in their analysis. For instance, ``Online Learning with Knapsacks: the Best of Both Worlds’’ (ICML 2022) employs almost the same algorithm of Immorlica et al. (FOCS 2019). This similarity is also present in subsequent works such as Slivkins et al. (COLT 2023), Castiglioni et al. (NeurIPS 2023). The reason why all these works might look similar at first glance is that they usually present a general primal-dual template, which is then instantiated with different subroutines. **The real contribution of these works, as in our case, lies in the specific subroutines and the new analysis of the framework.** 

We strongly believe that our techniques introduce several important innovations which distinguish them from the algorithms and analysis used in prior related works:
* Extending primal-dual methods to adversarial BwK with non-monotone resource consumption is highly non-trivial. A non-monotone constraint is more complex than monotone one, and our constraints are also hard (i.e., they must be satisfied at each round like any budget constraint). Our proofs crucially rely on the fact that the regret minimizers have sublinear regret in different sub-intervals (see proofs of Thm. 4.1 and Thm. 4.3). Most of the existing literature relies on standard regret minimization algorithms, which would prove inadequate in our setting.
* One of the key technical challenges is bounding the Lagrangian multipliers. This can be easily done in classical BwK problems (without replenishment) since a bound on the Slater’s parameter is available. Here, the replenishment factor is unknown and, therefore, the dual space cannot be bounded a priori. To circumvent this issue, we exploit no-adaptive-regret guarantees of the primal and dual regret minimizer. Dealing with such constraints requires different techniques and a new analysis, like the division of the rounds into three phases and a novel use of the dual regret minimizer with no-adaptive-regret.

Reviewer comment (Experiments):
The paper would benefit from the design of experiments to provide stronger empirical support for the theoretical findings.

Author response:
**On experiments:** Since we are the first to study an adversarial BwK model with replenishment, we would have no previous algorithms to employ as a meaningful baseline. All the existing algorithms would fail in the case $B=0$ (existing algorithms cannot even be initialized in this case). Conducting an experimental evaluation will certainly be valuable once there is a deeper understanding of adversarial BwK problems with replenishment and more algorithms are available.

Reviewer comment (Theory):
The authors argue that $\textit{in BwK problems constraints are required to be satisfied strictly at all rounds}$. However, when allowing negative $c_t$s, it is essentially allowing constraint violation in those rounds.

Author response:
There is a misunderstanding. We require that the **global** budget constraint must be satisfied at each round, i.e., the budget cannot be negative $B_t>0$ for all $t\in [T]$. This is not the case for general (soft) constraints that can be **globally** violated in some rounds, i.e., $c_t<B/T$ for all $t\in[T]$. Of course the two problems are related (but not equivalent) to satisfying the per-round consumption constraint  $c_t<B_0/T$ at each round.

Reviewer comment (Theory):
Any lower bound results?

Author response:
**Lower bounds:** Our setting strictly generalizes the classical (i.e., without replenishment) BwK model. In this setting, the lower bound is $\Omega(\sqrt{T})$ (as in standard bandit problems without constraints)  in the stochastic setting, and $1/\alpha$-competitive ratio (that in the BwK framework is equivalent to $B/T$) in this adversarial setting (see Balseiro and Gur (2019)). We match the lower bounds in both settings.

Reviewer comment (Theory):
Optimizing the coefficients in the regret bound? In bandits these are hidden in the big O term most of the times, but it could be pretty large in many cases, especially when T is not very large. Alternatively, without theoretical justification, some experiments showing that the bounds are empirically tight would be very helpful.

Author response:
**Optimizing coefficients:**  Most recent research on BwK primarily focuses on theoretical characterization instead of experimental evaluations. Also for this reason, we tried to prioritize clear result presentation over constant optimization. This helps to simplify the presentation of the analysis and results of the paper which, as noticed by other reviewers, is already quite complex.

Reviewer comment (Theory):
The theoretical results are not clearly discussed. For example, how tight are the theoretical results compared to lower bounds?

Author response:
**Lower bounds:** Our setting strictly generalizes the classical (i.e., without replenishment) BwK model. In this setting, the lower bound under stochastic inputs is $\Omega(\sqrt{T})$ (as in standard bandit problems without constraints), and $1/\alpha$-competitive ratio (that in the BwK framework is equivalent to $T/B$) in the adversarial case (see Balseiro and Gur (2019)). We match the lower bounds in both settings.

Reviewer comment (Theory):
From my aspect, "the best of both worlds" means that the proposed algorithm can achieve optimal theoretical guarantees in two different settings. Could the authors point out how their theoretical upper bounds match the lower bounds in these two settings? As BwRK is a new model, do we have such lower bounds?

Author response:
We thank the reviewer for their comment and we apologise if our response was not clear enough on this point.

Our algorithm guarantees are tight in both the stochastic and adversarial case. 
Indeed we achieve $\sqrt{T}$ regret in the stochastic case (which is obviously tight).
In the adversarial case our result essentially depends on only one parameter $\alpha$ and we achieve $\alpha$ fraction of the optimal reward. This is optimal in the worst case, since in the case without replenishment (where $\beta=0$ and $\alpha=\rho$) the lower bound is $\alpha$ as shown in Balseiro and Gur (2019). We will clarify this in the final version of the paper.

We agree that it would be interesting to further characterise the "interplay" between $\rho$ and $\beta$. We leave this as an interesting open question.

Reviewer comment (Theory):
It is not well discussed or stated how the negative cost could affect the problem. Is this new setup invalidate the existing algorithms, or it creates major challenges where we cannot simply use the existing algorithms or the regret analysis techniques?

Author response:
We thank the Reviewer for their comments. 

The presence of negative costs makes existing algorithms ineffective for the following two reasons. First, previous algorithms have regret upper bounds that depend on the ratio $T/B$ and, therefore, these guarantees are not meaningful when $B$ is small compared to $T$ (e.g. $B=O(1)$). Our algorithm works regardless of the initial budget $B$. It might seem that extending the classical algorithm by playing the void action when required is sufficient. In our paper, we show that this is not the case. For instance, we show that regret minimizers with weakly-adaptive regret guarantees are a crucial requirement for obtaining meaningful guarantees. 

Finally, it's important to note that although our algorithm and existing ones might appear similar due to their use of a primal-dual framework, the underlying analysis differs significantly. For example, our analysis divides the rounds into 3 separate phases, and our proofs crucially rely on the fact that the regret minimizers that we use are weakly adaptive and have sublinear regret in different sub-intervals (see the proofs of Theorem 4.1 and Theorem 4.3). Most existing literature relies on standard regret minimization algorithms, which would prove inadequate in this particular scenario.