Paper:
In this paper, we consider the problem of learning a policy by observing numerous non-expert agents. Our goal is to extract a policy that, with high-confidence, acts better than the agents' average performance. Such a setting is important for real-world problems where expert data is scarce but non-expert data can easily be obtained, e.g. by crowdsourcing. Our approach is to pose this problem as safe policy improvement in reinforcement learning. First, we evaluate an average behavior policy and approximate its value function. Then, we develop a stochastic policy improvement algorithm that safely improves the average behavior. The primary advantages of our approach, termed Rerouted Behavior Improvement (RBI), over other safe learning methods are its stability in the presence of value estimation errors and the elimination of a policy search process. We demonstrate these advantages in the Taxi grid-world domain and in four games from the Atari learning environment.

Review:
The paper looks at learning a policy from multiple demonstrators which should also be safely improved by an reinforcement learning signal. They define the policy as a mixture of policies from the single demonstrators. The paper gives a new way to estimate the value function of each policy where the overall policy is defined as mixture of the single policies. The paper subsequently looks at the standard error of the value function estimation and then define the policy improvement step in the presence of value estimation error. The resulting reroute constraint for the policy improvement step is evaluated on the taxi toy task as well as on 4 different atari domains. 
This paper presents an interesting ideas which is also based on an exhaustive theoretical derivation. However, the paper is lacking clarity and motivation which makes it almost impossible to understand at the first pass. Moreover, the presented results are promising but not exhaustive and the resulting algorithm is also restricted to discrete action domains. More comments see below:
- The paper consists of 2 parts, the average behavior policy and its value function and the safe policy improvement step. The relation between these two parts are not clear. Is the policy improvement step only working if the policy is defined as in section 4 and the value function computed as in section 4? 
- Proposition 4.2 needs to be much better motivated and explained. It is totally unclear at this part of the paper why proposition 4.2 is used.
- Please explain why proposition 4.2 indicates that Q^D \approx Q^\beta
- The selection type of S in the taxi example is also unclear.
- How would we solve Equation 8 with continuous actions / parametrized policies \pi? Without this extension, the algorithm is quite restricted. 
- the figure captions need to be much more exhaustive. I am not sure I understand the x axis of Figure 4 (right). What iterations are shown here? We only do one improvement step of the behavior policy, without any resembling, is that right?
- Could we also use a similar policy update for policy improvement in reinforcement learning?
- Could you add an algorithm box for estimating the Q-function? Do we estimate every Q-function in isolation using MC estimates and then just use the weighted average?
- It would be interesting to also compare the value function learning method proposed in the paper in isolation to other value function learning methods such as DQN. while the presented method is simple (learn from MC estimates), this is also known to be very data inefficient.