Paper: Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks (ICLR-2024)

Reviewer comment (Novelty):
The reviewer is not well versed in recent theoretical/tabular RL exploration literature and cannot fully assess novelty and significance there, but in a deep RL context the work seems relevant yet very closely related to existing approaches such as goal-conditioned RL (particularly Hindsight Experience Replay) and, to some extent, automatic curriculum generation methods.

Author response:
We value the reviewer’s contextualization within the realm of deep RL. Indeed, multi-goal RL serves a motivating example of our proposed theoretical framework. HER, as proven to improve sample complexity, implicitly tackles a multitask RL problem, as articulated in Section 3. Specifically, Algorithm 1 encapsulates a form of HER within a multi-goal RL setting.

**Connection to HER and multi-goal RL:** In a multi-goal RL setting, all tasks share identical state space, action space, and transition function, with the only difference residing in the reward function. Algorithm 1 exemplifies this by sampling a trajectory on task $M$ through a mixture of all greedy policies from other tasks. Notably, this exploration strategy is akin to randomly selecting task $M'$ and collecting a trajectory on $M'$ using its own epsilon-greedy policy, followed by relabeling rewards to simulate a trajectory on $M$.

**Connection to curriculum learning:** Although Algorithm 1 does not explicitly implement curriculum learning by assigning preferences to tasks, an improvement could be made through adaptive task selection. If an algorithm with adaptive task selection achieves a sample complexity $\mathcal{C}$, Algorithm 1 should have a sample complexity of at most $\mathcal{C} |\mathcal{M}|^2$. Intuitively, if a curriculum selects tasks through an order of $M_1, \dots, M_T$, then at the each round, the algorithm uses the epsilon-greedy policy of MDP $M_{t-1}$ to explore $M_t$. This exploration is implicitly included in Algorithm 1 because Algorithm 1 explores all the MDPs in each round with the mixture of all epsilon-greedy policies.

This means that the sample-complexity of Algorithm 1 provides an upper bound on the sample complexity of underlying optimal task selection and in this way our theory provides some insights on the success of the curriculum learning. Designing a theoretically grounded algorithm for near-optimal adaptive task selection is a non-trivial improvement, and we defer this discussion to future work.

Reviewer comment (Novelty):
While the connections to existing methods like goal-conditioned RL and automatic curriculum generation are interesting and the theoretical analysis is not redundant with that prior work, the paper would be more interesting and have a stronger contribution if those connections were explored further. As it stands, the specific argument appears novel, but overlaps significantly with prior work.

Author response:
We thank reviewer for raising the score. We could not do an in-depth discussion on the connections to HER/Curriculum Learning due to the page limit in the rebuttal phase. We will improve this in the camera-ready version when we have an extra page. We will also discuss the future work that makes a more concrete connection between our theory and the real scenarios in Deep RL.

Reviewer comment (Writing):
While the argument is generally understandable, the paper has rough grammar and odd word choices in places, prompting a recommendation for a thorough editing pass to improve the language.

Author response:
**Question 1: While I generally follow the argument, this paper has some rough grammar and odd word choices in places. I'd recommend a thorough editing pass to improve the language.**

Response: We appreciate the observation regarding language issues and commit to a comprehensive editing pass to enhance the final version's clarity and coherence.

Reviewer comment (Theory):
Definition 1 is confusing. The reviewer questions whether C is a function of beta and delta, and is unsure why sample complexity does not depend on either the algorithm itself or the task(s) being learned. The following paragraph implies C is a function of the MDP, but this is not stated in the definition.

Author response:
**Question 3: Likewise, Definition 1 is a little confusing. C is a function of beta and delta? I'm confused as to why sample complexity doesn't depend on either the algorithm itself or the task(s) being learned.**

Response: We appreciate the clarification question. The sample complexity $C$ is indeed a function of $\beta$, $\delta$, the algorithm, and the environment $\mathcal{M}$. While we did not explicitly state the dependence on the algorithm and environment to maintain simplicity, we acknowledge the importance of this clarification.

Reviewer comment (Novelty):
The reviewer asks how Algorithm 1 differs from the cited Zhumabekov 2023 policy ensemble method, noting it seems like Algorithm 1 is essentially an ensemble of policies from which one is sampled for each episode.

Author response:
**Question 4: How does algorithm 1 differ from the cited Zhumabekov 2023 policy ensemble method?**

Response: While both Algorithm 1 and Zhumabekov 2023 share a similar high-level idea of ensembling policies, their implementation differs. In Algorithm 1, policies are mixed trajectory-wise, with one policy randomly selected for the entire episode. In contrast, Zhumabekov 2023 mixes policies on a step-wise basis for a continuous action space, selecting actions as a weighted average of actions chosen by different policies. We conjecture that this distinction impacts the worst-case sample complexity, with Algorithm 1 enjoying the guarantees outlined.

Reviewer comment (Novelty):
The reviewer asks how the idea of multitask myopic exploration differs from normal goal-directed RL methods like Hindsight Experience Replay. They note that the motivating example in Figure 1 seems roughly aligned with such methods and should share their limitations (e.g., large state spaces and low-dimensional manifolds of interesting/human-relevant tasks).

Author response:
**Question 5: How does this idea of multitask myopic exploration differ from normal goal-directed RL methods like hindsight experience replay?**

Response: As elucidated in the Weakness section, Algorithm 1 implicitly incorporates HER in a multi-goal setting. It converges with methods like HER, sharing a common approach to myopic exploration.

Reviewer comment (Writing):
The reviewer finds the term "myopic exploration gap" confusing, as the intuition makes sense but "gap" seems misnamed. They suggest something like "myopic exploration potential" might be clearer, since "gap" implies a comparison to another exploration algorithm. Although the term comes from previous literature, it remains confusing unless the reviewer is misunderstanding the definition.

Author response:
**Question 6: I find the term "myopic exploration gap" a little confusing. Maybe something like "myopic exploration potential?"**

Response: We agree with the suggested term "myopic exploration potential" and recognize its potential to more accurately capture the underlying intuition. Since "myopic exploration gap", we would like to keep the current name to avoid confusion for readers who know the previous literature.

Reviewer comment (Theory):
In the tabular case, multitask myopic exploration relies on coverage assumptions in the task space, but these are not tractable to assume in the deep RL case. The reviewer asks whether this factor came up in the BipedalWalker experiments and whether the assumptions held.

Author response:
**Question 8: In the tabular case, multitask myopic exploration relies on coverage assumptions. Did this factor come up in the BipedalWalker experiments? Do the assumptions hold?**

Response: We think that insights from linear MDPs may be relevant in deep RL, where a shared representation function is used for all tasks, an analog to the representation learning setting in deep RL. Our exploration of the embedding covariance matrix spectrum in Figure 2 aligns with findings from automatic curriculum learning task selection. Although not an exact translation because the real environment is not a linear MDP, the theoretical insights from linear MDPs offer valuable considerations for the deep RL setting.

Reviewer comment (Theory):
The only weakness of this paper in my opinion is that examples where multitask MEG is bounded are too restrictive. Definition 7 is a very strong requirement, and intuitively, diverse tasks can be defined more general. Moreover, the feature coverage assumption is additional since it is not needed for learning linear MDPs with strategic exploration.

Author response:
Response to weakness: The feature coverage assumption is an additional condition included for clarity in presentation. We acknowledge it is non-essential and discuss a relaxation of this assumption for the Tabular case. In this scenario, we construct a mirror MDP (refer to Appendix F.2), introducing an extra dependence of $\sqrt{SH}$.

Reviewer comment (Theory):
Is it possible to relax Definition 7?

Author response:
**Question 1: Is it possible to relax Definition 7?**

Response: We concede that Definition 7 is indeed restrictive. We might conjecture that we only need the $\theta$'s for the reward parameter is rank $d$. This is in fact not sufficient. We provide an example in Appendix D Figure 3. To have a lower bounded Multitask MEG, we require, at each $h$, the optimal policies of different tasks induce a full-rank covariance matrix of the feature vectors (refer to Lemma 8).

More generally, a thorough discussion of diversity for general function classes is available in Appendix G. The overarching concept involves constructing a set of tasks whose optimal policies induce occupancy measures that are $\epsilon$-dependent. The size of the largest set is quantified by the BE dimension. However, the technical challenge lies in the fact that current algorithms only guarantee finding a $\beta$-optimal solution, which doesn't necessarily ensure convergence to the induced occupancy measure of the true optimal policy. There can be many interesting future works towards this direction.

Reviewer comment (Theory):
The offline learning oracle solves  $f_1,...,f_h$ simultaneously. Can you do them sequentially and have similar guarantees?

Author response:
**Question 2: The offline learning oracle solves simultaneously. Can you do them sequentially and have similar guarantees?**

Response: By "solving them sequentially," if you mean updating the value function for one task at each round, this approach is indeed feasible. We can achieve this by selecting a task in each round and collecting an episode for that task using the mixture policy (where the greedy policy from other tasks is based on the most recent update). If tasks are chosen in a fixed order ($1, \dots, |\mathcal{M}|$), we obtain guarantees similar to those outlined in Algorithm 1. There is potential for an adaptive task selection approach, akin to curriculum learning, which we leave for further exploration in future work

Reviewer comment (Experiments):
It provides a possible explanation to explain the success of naive exploration in the case of multitask RL. However, it is hard to validate such explanation.

Author response:
We acknowledge the challenge of validating the theoretical explanation in real-world empirical studies. However, we present two key points supporting the validity of our explanation. Firstly, the HER (Hindsight Experience Replay) algorithm, a successful practical implementation, shares similarities with Algorithm 1. HER enhances exploration by relabeling sampled trajectories from other tasks, suggesting that multitask RL indeed improves the sample efficiency of myopic exploration. Secondly, drawing on the concept of diverse tasks in Linear MDP cases, we establish a connection to the Bipedal Walker environment, demonstrating that a diversity notion favoring a full rank structure of the covariance matrix of the embeddings is pivotal (as discussed in Section 6). Based on these observations, we firmly believe that our paper offers a valid perspective on explaining the success of myopic exploration.

Reviewer comment (Novelty):
The intuition of the proof is that, if we have a base policy class with good coverage, we are able to find the optimal policy by combining the base policies with naive exploration. However, it have been well known that exploration is simple when we have good coverage. Therefore, their contribution seems not novel enough.

Author response:
We appreciate the reviewer's comment and we would like to have some extra clarification on the coverage of a policy class. Typically, coverage assumptions are articulated for the behavior policy of an offline dataset, often with an all-policy or single-policy concentrability assumption (refer to [1]). While constructing a diverse task set for a discrete policy class with each policy representing the optimal policy of a task is straightforward, the challenge arises when the policy set is infinite. Constructing an epsilon-covering of the policy set may lead to an exponentially large task set. However, we demonstrate that for tabular and linear cases, a diverse task set can be constructed with only polynomial (SAH) or polynomial (dH) tasks, a departure from existing literature results.

Reviewer comment (Writing):
What does Def 3 mean in linear MDP?

Author response:
For a detailed understanding of Def 3 in linear MDP, we refer the reviewer to Lemma 8 in Appendix E.1, which states that the coverage condition in Def 3 could be expressed as a full rank condition of the feature covariance matrix. Low MEG in Def 3 in linear MDP intuitively implies that we should construct the task set “diverse” enough such that for all $h \in [H]$, the feature covariance matrix $\mathbb{E}_{\pi} \phi_h(s_h, a_h) \phi_h(s_h, a_h)^\top$ obtained by running the current exploration policy (mixture of epsilon greedy policies) is full rank.

Reviewer comment (Theory):
Can you provide an example where Def 3 holds and the coverage of the base policies is poor?

Author response:
Could the reviewer elaborate on "base policies"? Def 3 is not a condition. However, if the concern is whether low MEG holds even when the current greedy policies are non-optimal, we direct the reviewer to Figure 1, where low MEG consistently holds despite the non-optimality of the current greedy policies.