Paper: Efficient Multi-task Reinforcement Learning via Selective Behavior Sharing (ICLR-2024)

Reviewer comment (Evaluation):
The algorithm's design, which learns a distinct policy for each task, could potentially diminish sample efficiency.

Author response:
### Weakness C,D: Distinct policy for each task: 
We clarify that our algorithm’s design is not limited to distinct policies. In fact, the set of experiments in Section 6.2 show that QMP can be implemented with a single multi-head multi-task policy and that the behavior sharing advantages are complementary with parameter sharing advantages.

Furthermore, the choice of distinct v/s shared task policies depends on the task family. Evidently, parameter-sharing is not always optimal and could even hurt training over distinct policies (See Multistage Reacher results in Figure 5 and [3]). Since we consider task sets where different tasks may require conflicting behavior, we base our primary comparisons on distinct policy networks, but also provide results on parameter-sharing + QMP.

Reviewer comment (Evaluation):
The comparisons with baselines depicted in Figure 4 fail to demonstrate notable improvements conferred by the proposed algorithm.

Author response:
### Weakness F: Lack of notable improvements
We would like to note that reviewers QRge, Zv3o, and neqy all noted that the empirical results were comprehensive and demonstrated the effectiveness of QMP.  In our experiments, we chose a wide range of environments and tasks with shared and conflicting behaviors to demonstrate that our simple method of QMP improves sample efficiency and convergence across general multi-task sets, and did not design tasks specifically to favor our method which may have shown more dramatic differences. This is in line with the theoretical proofs in `[Appendix G]` showing that QMP guarantees a weak monotonic improvement over the baseline of no behavior sharing.

Reviewer comment (Theory):
Theoretically, the policy network is trained to give actions with the maximized Q-value. That is, $i = \arg\max_jQ_{i}(s, a_j)$ (Line 9 of Algorithm 1) should hold in most cases, which may make this key algorithm design trivial.

Author response:
### Question E: Should QMP always choose its own task policy?
`[Theorem G.2]` While the task policy $\pi_i$ is trained to maximize $Q_i$ over continuous action spaces, it always lags behind $Q_i$, which is itself constantly being updated with new rollouts over training. So the Q-filter can select a different policy if $pi_i$ is not fully optimized for the current state, and a different task policy suggests a better action. When this happens, Theorem G.2 guarantees performance improvement.

Reviewer comment (Experiments):
The baseline "Fully-Shared-Behaviors" cannot be viewed as a fair comparison, since the agent cannot identify which task it is dealing with. The task identifiers should also be part of the input. There are many research works in this area, such as [1-3] and the ones listed by the authors in Section 2. It would be beneficial to provide comparisons with these works as well.

Author response:
### Question F: “Fully-Shared-Behaviors” baseline
We use this baseline in section 6.1 specifically to answer “How does QMP compare to other forms of behavior sharing?”, by comparing against a policy that has the same behavior everywhere for all tasks, and therefore mask out the task identifiers to enable policies to learn shared behaviors across tasks.  This is intended as a behavior sharing comparison, not an analysis on parameter sharing in single policies which we do compare against in Section 6.2.

Reviewer comment (Theory):
The method, while simple, appears more heuristic in nature and lacks guarantees.

Author response:
### Theoretical guarantees
`[Appendix G]` Please refer to the combined response, where we add convergence and policy improvement proofs for the QMP Algorithm’s modification due to its mixture policy.

Reviewer comment (Presentation):
The discussion of related works is insufficient and comes across as disjointed and poorly structured.

Author response:
### Related Work Discussion
`[Section 2]` We added several relevant works on MTRL through shared representations and skill learning as a single task behavior sharing example.

Reviewer comment (Theory):
Lack of theoretical analysis on the convergence (rate) of QMP.

Author response:
`[Appendix G] Please refer to the combined response, where we add convergence and policy improvement proofs for the QMP Algorithm’s modification due to its mixture policy. We prove that QMP’s mixture policy induces a Bellman operator that is a contraction, and QMP further guarantees a weak monotonic policy improvement over its base algorithm, SAC.`

Reviewer comment (Theory):
Lack of intuition and detailed analysis on why the proposed method works (i.e., why could QMP make sharing desired behaviors among tasks possible? Please see Question 2 for my concern).

Author response:
`### Q1: Using Q-function to evaluate H-step behaviors
[Appendix H] Please refer to the combined response, where we justify (based on prior work, mathematical intuition, and empirical results) why temporally-extended behaviors can effectively obtain high-rewarding trajectories, even when the Q-switch is used to evaluate the mixture policy on the current action.`

Reviewer comment (Theory):
Why do you choose to roll out H steps instead of just one step after choosing one policy to collect data? QMP just uses the Q function under a particular state to choose the behavioral policy, which cannot guarantee that the selected behavior policy is helpful for the current task after stepping out of the considered state.

Author response:
`### Q1: Using Q-function to evaluate H-step behaviors
[Appendix H] Please refer to the combined response, where we justify (based on prior work, mathematical intuition, and empirical results) why temporally-extended behaviors can effectively obtain high-rewarding trajectories, even when the Q-switch is used to evaluate the mixture policy on the current action.`

Reviewer comment (Evaluation):
Will QMP cause undesired behaviors sharing? As I mentioned in Question 1, the selected behavioral policy will roll out for H(>1) steps, which may incur sub-optimal behaviors.

Author response:
`### Q2: Undesired Behavior Sharing?
We address this in detail in the combined response, but in summary, it is possible for QMP to cause undesired behavior sharing through the Q-filter but is able to recover quickly and self-correct due to our method design which only uses other task policies to *gather training data*. We see this in effect in Task 4 of Multistage Reacher which completely conflicts with the other tasks but QMP still performs as well as other baselines that do not share behaviors.`

Reviewer comment (Theory):
Why choosing the behavioral policy based on the learned Q-function to collect data will essentially share desired behaviors among tasks? I understand that using Q-function as a proxy may help select better actions. However, the Q-function may be biased and not learned well during training, which could even hurt the learning process.

Author response:
`### Q3: Concerns about bias in learned Q-function
The reviewer correctly points out that the learned Q-function may be biased, leading to QMP selecting sub-optimal policies. However, this is true for any type of Q-learning where a policy tries to pick actions to maximize a learned Q-function (methods based on SAC [1] and Deterministic Policy Gradient [2]). This generally does not hinder performance in any RL algorithm due to self-correction from online data collection. In fact, many performant Q-learning-based RL algorithms, learn policies directly to maximize a learned Q-function [2, 6, 7].  As explained in Section 4.2 in the paper, the learned Q-function is a critical component of many online RL algorithms [1,2] and has even shown to be a useful filter for high-quality training data [3,4,5]. Our results in conflicting tasks (see QMP vs. No-shared in Figure 8, Task 4) demonstrate that QMP does not suffer from Q-function bias any more than the RL algorithm it is built on top of.`

Reviewer comment (Theory):
Will QMP even slow down the training process? Say, the learned policy for the current task proposes an action, which is optimal but has an under-estimated Q function. Due to the biased Q function, QMP selects another policy to collect data, which chooses a sub-optimal action. Although the TD update will fix the estimated Q value of the sub-optimal action, it may be more efficient if we directly update the Q value of the optimal action, the thing that we really care about.

Author response:
`### Q4: Could QMP slow down the training progress?
As addressed in Q3, QMP is hindered by the same Q-function bias as most Q-learning algorithms but does not impact sample efficiency in any of the task sets we tried even when there are directly conflicting behaviors (see example in Q2).`

Reviewer comment (Theory):
Since the Q-switch evaluates each \( Q_1(s,a_j) \), it inherently assumes all tasks share the same state-action space, or at least assumes the same size of state-action dimension.

Author response:
### 1. Shared state-action dimension
This is a common assumption in many multi-task learning works [1-7]. Experiments on MT10 show that the state-action space does not need to be necessarily shared and only the dimensions are shared. Even specialized architectures like Perceiver that can take multimodal and diverse inputs are complementary to our work, if the Q-function is modeled in that way. So, the requirement of same state-action dimensionality is a consequence of the choice of Q-network, not a limitation of behavior-sharing.

Reviewer comment (Theory):
The reviewer noticed that the paper mentioned 'out-of-distribution action proposal' in Section 4.3; how does the agent know the action is out-of-distribution? Do you mean the action may be \([-10,10]\) while the action space is \([-1,1]\)?

Author response:
### 2. Out-of-distribution action proposal
Out-of-distribution means that the Q-switch, i.e., $Q_i$ may not yet be well-trained on (s, $a_j$), where the $a_j$ is another task’s proposal. In this case, the Q(s, $a_j$) value might be inaccurate. Even if the Q-switch erroneously selects this action $a_j$ to collect data, it would self-correct because by collecting data with $a_j$, the Q-switch would be more accurate on (s, $a_j$) now.

Reviewer comment (Reproducibility):
How to sample the data from \( \pi_i \) and QMP? Is there a specific fraction or equal sampling?

Author response:
### 4. Data Sampling Details
To train the policy and critic for task $i$, we sample uniformly from the data collected in the replay buffer by QMP’s mixture policy $\pi_i^{\text{mix}}$ without needing to do any specialized data balancing or priority sampling based on rewards.  This means that the data collected by QMP is helpful for training $\pi_i$ without additional implementation tricks.

Reviewer comment (Experiments):
The reviewer feels the comparison is unfair regarding the shared-parameters baseline. The key problem in MTRL is to address the interference when multiple tasks use one network to train the policy. Therefore, a lot of papers come up with different ideas, such as conflict gradient resolution (PcGrad, CAGrad) and pathway finding (soft modularization, T3S). However, this paper only compared to the basic Multi-head-SAC. While QMP has a designed criterion to select what behaviors to share and how to share. That's why the results in Figure 5 show that 'Parameters + Behaviors' performs worse than 'Behaviors Only'.

Author response:
### Baseline Parameter Sharing Method
In section 6.2, we use a widely used baseline parameter sharing method, instead of more sophisticated methods like the reviewer points out. This is because this experiment was not meant to compare parameter sharing v/s behavior sharing, but to show that they are **complementary** — which we will clarify in the text. Our goal is *not* to compare parameters-only v/s behaviors-only, because they are orthogonal and complementary ways to share knowledge in MTRL. Our goal is to show that in simultaneous MTRL, behavior-sharing can augment other ways of sharing.
- Particularly, in Maze Navigation, we see that Parameters-Only already improves significantly over Neither, which shows that methods like PCGrad are not needed in this environment. Our goal is to show that even in this case, Parameters + Behavior sharing further improves significantly over Parameters-Only.
- As the reviewer points out, in Multistage Reacher, because of the potential conflicts in tasks, parameters-only performs poorly than Neither. However, still Parameters + Behaviors improves over parameters-only. In this case, while combinations of QMP and more sophisticated parameter sharing may yield even better results. However, we believe that the multi-head policy we used is sufficient evidence to demonstrate that QMP is **complementary** with parameter sharing.

We hope this clarifies and addresses the concerns raised.

Reviewer comment (Novelty):
The literature review lacks related works such as [1-4].  
[1] Conflict-averse gradient descent for multi-task learning  
[2] Structure-Aware Transformer Policy for Inhomogeneous Multi-Task Reinforcement Learning  
[3] T3S: Improving Multi-Task Reinforcement Learning with Task-Specific Feature Selector and Scheduler  
[4] Provable benefit of multitask representation learning in reinforcement learning

Author response:
We thank you for your helpful feedback and pointing us to relevant works to include, that we have incorporated a discussion in the revision.