Paper:
Transfer reinforcement learning (RL) aims at improving learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under unknown diverse dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces.

Review:
The paper introduces a novel multi-source policy transfer problem, where we want to utilize policies from multiple source domains with different dynamics to improve the performance of the policy on our target dynamics. 
The paper addresses the problem by adaptively aggregating the deterministic actions produced by source policies to maximize the expected return in the target environment. The method further trains an auxiliary network to predict a residual to revise the predicted action when some source policies are not useful or even adversarial.
 In my understanding, the paper assumes that the source and target domain shared the same task (reward structure) but only differs in dynamics. Also, the two domains share similar state and action spaces since the policy accepts the target states and predict actions in the target environment. This may limit the usage of the method.
The paper proposes to use residual learning as an auxiliary to compensate for the sub-optimal expressiveness of the source policies, which is novel and interesting.
The paper performs experiments on multiple environments. But the source and target domains only vary in some parameters of the agents. The domain gap seems small for these experiments. The paper needs a metric to measure the domain gap between source and target dynamics and report how the domain gap influences the proposed method and the baselines according to the metric.
One of my major concerns is that the limitation of the method with the same state and action space of the source and target domains. Also, there is no theoretical or intuitive analysis of how large the domain gap can be. This problem can be impractical for real-world applications with restrict limitations.
Post-rebuttal:
My major concern is that the method is a naive combination of previous works and the paper is more like an engineering work. The method is also a weighted sum of source policies. There is no insight why the combination can work.
No assumption on source policies is given. That means I can get any random policy to learn a combination. This is like learning a policy from scratch by reinforcement learning. With better source policies, we can achieve better initialization for RL.
The paper is also related to hierarchical reinforcement learning, where the target reinforcement learning step is like building a high-level policy. 
The work still requires lots of steps to train in the target domain, which does not fit to the real application of transfer RL. We hope transfer learning can adapt to the target environment fast.