Paper: Submodular Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
I'm not sure why this paper considers multiplicative approximations instead of regret/sample complexity, which are common in theory papers studying episodic MDPs.

Author response:
Thank you very much for the positive and valuable feedback. Multiplicative approximations are common for complexity analysis, especially in submodular optimization literature, so we used them. For future direction especially for statistical analysis, we will use regret/sample complexity types bounds.

Reviewer comment (Theory):
The optimal dependency on curvature remains unspecified. Whether Proposition 3 is (near-)optimal?

Author response:
Regarding optimal curvature dependency, a result exists for the simpler problem of set function optimization under cardinality constraints (Vondrak 2010), showing that $1-c$ is near optimal. Thanks, we now incorporated it in the paper.

Vondrak Jan, 2010 Submodularity and curvature: the optimal algorithm

Reviewer comment (Novelty):
Literature review: I agree with the motivation from diminishing returns, but a submodular reward design is not the only way to address that. For example, there is a blocking-bandit style framework that discourages repeated actions [1]. Maybe good to discuss why the submodular reward design is better.

Author response:
Thanks for sharing [1][2], both of them look very interesting. Firstly, about the reward design, we want to emphasize that submodularity is an equivalent characterization of diminishing returns and hence is very general as mentioned by other reviewers as well. Arguably a problem analogous to blocking bandits can be conceptualized with submodular functions as well. For instance, consider a coverage problem on state time $(v,t)$ pairs, where pulling an arm $v$ at time $t$ covers it for the next $\tau_v$ times and hence no reward until $t + \tau_v$ time. If the arm $v$ is pulled after $t + \tau_v$, it will receive a reward again. Thanks for raising this point, we now included a table in the appendix referenced in the related work section summarizing various applications captured using submodular functions. 

Secondly, about the differences with respect to the problem setting, we would like to emphasize to the reviewer that our focus is on MDPs i.e., we have states and need to satisfy transition constraints whereas both of these works ([1][2]) focus on bandits.

Throughout the entire paper, both in theory and experiments, we consider general MDPs. Only in the first half of section 5, we analyze a bandit case to motivate our SubPO algorithm with optimal approximation ratios and show interesting connections to generalize bandits. Although the bandits setting is not our focus, in order to connect with submodular bandits literature, we cite (Streeter & Golovin, 2008; Chen et al., 2017; Yue & Guestrin, 2011) in the last paragraph of related works. We are happy to further include [1].

Reviewer comment (Novelty):
I also encourage authors to survey more existing works that explore similar ideas with submodular reward design. For example, can the authors explain the difference between [2] in terms of the problem setting?  

[1] Basu et al., Blocking Bandits, NeurIPS 2019.  
[2] Chen et al., Contextual Combinatorial Multi-armed Bandits with Volatile Arms and Submodular Reward, NeurIPS 2018.

Author response:
Thanks for sharing [1][2], both of them look very interesting. Firstly, about the reward design, we want to emphasize that submodularity is an equivalent characterization of diminishing returns and hence is very general as mentioned by other reviewers as well. Arguably a problem analogous to blocking bandits can be conceptualized with submodular functions as well. For instance, consider a coverage problem on state time $(v,t)$ pairs, where pulling an arm $v$ at time $t$ covers it for the next $\tau_v$ times and hence no reward until $t + \tau_v$ time. If the arm $v$ is pulled after $t + \tau_v$, it will receive a reward again. Thanks for raising this point, we now included a table in the appendix referenced in the related work section summarizing various applications captured using submodular functions. 

Secondly, about the differences with respect to the problem setting, we would like to emphasize to the reviewer that our focus is on MDPs i.e., we have states and need to satisfy transition constraints whereas both of these works ([1][2]) focus on bandits.

Throughout the entire paper, both in theory and experiments, we consider general MDPs. Only in the first half of section 5, we analyze a bandit case to motivate our SubPO algorithm with optimal approximation ratios and show interesting connections to generalize bandits. Although the bandits setting is not our focus, in order to connect with submodular bandits literature, we cite (Streeter & Golovin, 2008; Chen et al., 2017; Yue & Guestrin, 2011) in the last paragraph of related works. We are happy to further include [1].

Reviewer comment (Presentation):
Suggestion: It looks slightly unnatural to have Section 4 (practical algorithms) in between hardness results (Section 3) and positive results (Section 5). A more natural flow would have been having the positive theoretical results first and then presenting practical algorithms, or at least having them connected.

Author response:
**Placement of sections:** Thank you for your suggestion. The positive results are for the proposed algorithm, SubPO,  and are therefore introduced in section 5 following the algorithmic details in section 4. Whereas the hardness result is for the framework (independent of the algorithm), hence presented in section 3 right after the framework is introduced in section 2. Presenting positive results for the algorithm prior to introducing the algorithm itself is difficult.

Reviewer comment (Novelty):
Overall, I feel that the framework is well-motivated for the "mathematical" purpose but less sound for the practical purpose or advancing the theory of RL.

Author response:
We would like to emphasize to the reviewer that we have multiple positive results.

Firstly, we consider a $\epsilon$-bandit SMDP which is a generalization of the bandit setting. Even under this simplified setting, it is intractable to achieve optimal value. However, SubPO achieves the information-theoretic optimal approximation ratio of $1-1/e$, which motivates a SuBPO-type approach as a sensible approach for Submodular RL.

**Secondly, we would also like to bring the other positive result to the reviewer's attention which is for general SMDPs (i.e, no restrictions on dynamics) in Proposition 3, where, based on the curvature ‘c’ of the submodular function, we can guarantee that policy $\pi$ from SubPO achieves $J(\pi) \geq (1-c)J(\pi^\star)$ where $\pi^\star$ is the optimal non-Markovian policy.**

Beyond this, giving any constant factor approximation in full generality is not possible as indicated by our hardness result in Theorem 1. Hence, the two positive results and the hardness result characterize the complete computational complexity of the proposed framework.  More importantly, the algorithm SubPO is applicable to full generality without any sort of assumptions and can be used to optimize experiment design, entropy exploration, coverage, etc. objectives as shown in the experiments.

Thank you very much for the valuable feedback and sharing your questions. Please let us know for any further suggestions. We hope that our response clarifies that our focus is on MDPs, the significance of our contributions to both RL theory and practical use cases, and convinces the reviewer that our paper warrants their strong acceptance.

Reviewer comment (Writing):
I do not understand what it means by $\pi$ is parameterized by $\pi^h(a)$ in Theorem 3. Does this mean $\pi$ does not depend on the state?

Author response:
Yes, $\pi^h(a)$ represents that the considered policies are state-independent and only depend on the horizons.

Reviewer comment (Theory):
The submission focuses on the "planning" side. Any thoughts on the "learning" side? (a.k.a., exploration and sample complexity)

Author response:
Different communities may use the “planning” word differently, we would like to emphasize that this work considers a general problem of learning with data (samples) from simulations as done in typical policy gradient algorithms e.g., Reinforce, TRPO, PPO, Soft actor-critic, etc. Currently, our theoretical analysis focuses on computational complexity, and we agree quantifying the statistical complexity (i.e., sample complexity) of SubPO is an interesting future direction. A motivation in this direction can be derived from Yuan, et al. 2022 but is an independent research work of its own.

Rui Yuan, Robert M. Gower, Alessandro Lazaric, A general sample complexity analysis of vanilla policy gradient, 2022

Reviewer comment (Presentation):
I think there can be a broader discussion of using submodular functions in reinforcement learning setups in the related work section. I am aware that the introduction also mentions some examples of submodular rewards, but I believe it is interesting enough to have its own paragraph in the related work.

Author response:
Thanks, we incorporated the suggestions. We agree that the modelling using submodular rewards deserves its own space.  We had a paragraph saying “Examples of submodular rewards” in section 2 (preliminaries and problem statement). Now, we included a table in the appendix (reference in related works) summarizing different tasks that can be conceptualized with submodular functions.

Reviewer comment (Novelty):
Are there other attempts of incorporating submodular functions to reinforcement learning problems?

Author response:
We agree with how the reviewer has signified the importance of the contributions. Indeed, there is no attempt with submodular functions in RL, the most general work in this direction still considers planning on graphs.

Reviewer comment (Novelty):
The idea behind the proposed algorithm, Submodulr Policy Optimization, is quite straightforward. It is just a relatively straightforward extension of the classical policy optimization algorithm.

Author response:
The key novelty in SubPO is to utilize the marginal gain decomposition for the submodular functions which allows for splitting the objective into per-step contributions that only depend on the history. We exploit this decomposition to obtain an unbiased gradient estimator with submodular (non-additive) reward functions (Theorem 2, proof in Appx C). In contrast to maximizing modular (state) reward in classical PG, the SubPO policy gradient theorem utilizes the effectiveness of greedily maximising marginal gains which is central to the literature on submodular maximization.

Reviewer comment (Theory):
The analysis in Section 5 seems to be very restricted. Could the authors provide a similar analysis in more general settings?

Author response:
Although $\epsilon$-bandit SMDP restricts the admissible MDPs, the notion nevertheless strictly generalizes the widely considered submodular bandit setting. The provable guarantee here motivates SuBPO as a sensible approach for Submodular RL since SubPO recovers the optimal approximation ratio of $1-1/e$ at least under this simplified SMDP.

**Furthermore, we do provide another result for general SMDPs (general dynamics) in Proposition 3, where, based on the curvature ‘c’ of the submodular function, we can guarantee that policy $\pi$ from SubPO achieves $J(\pi) \geq (1-c)J(\pi^\star)$ where $\pi^\star$ is the optimal non-Markovian policy.**

Beyond this, giving any constant factor approximation in full generality is not possible as established by our hardness result in theorem 1.

Reviewer comment (Presentation):
It is not clear to me why the authors chose to put the "Related Work" section between an analysis section and the experiment section. Probably the authors should put it after Introduction.

Author response:
We appreciate the reviewer's suggestion to bring related work up. However, there is no closely related work, as we address a novel setting and our approach is only distantly related to submodular optimization, convex RL and PG approaches. Instead, we kept it in the end to rather emphasize the connections of our method to different fields. We now included a forward reference to the related works in the introduction section and explained the reasoning for it.