Paper:
Our goal is to infer reward functions from demonstrations. In order to infer the correct reward function, we must account for the systematic ways in which the demonstrator is suboptimal. Prior work in inverse reinforcement learning can account for specific, known biases, but cannot handle demonstrators with unknown biases. In this work, we explore the idea of learning the demonstrator's planning algorithm (including their unknown biases), along with their reward function. What makes this challenging is that any demonstration could be explained either by positing a term in the reward function, or by positing a particular systematic bias. We explore what assumptions are sufficient for avoiding this impossibility result: either access to tasks with known rewards which enable estimating the planner separately, or that the demonstrator is sufficiently close to optimal that this can serve as a regularizer. In our exploration with synthetic models of human biases, we find that it is possible to adapt to different biases and perform better than assuming a fixed model of the demonstrator, such as Boltzmann rationality.

Review:
This paper has proposed algorithms for inferring reward functions from demonstrations with unknown biases. To achieve this, the authors have proposed to learn planners from demonstrations in multiple tasks via value iteration networks to learn the reward functions.
This paper has provided an excellent motivation of their work in Sections 1 & 2 with references being made to human behaviors and heuristics, though the authors can choose a more realistic running example that is less extreme than making orthogonal decisions/actions. The paper is well-written, up till Section 4. 
On the flip side, there does not seem to be any significant technical challenges, perhaps due to some of the assumptions that they have made. Like the authors have mentioned, I do find assumption 3 to be overly strong and restrictive, as empirically demonstrated in Section 5.2. Arguably, is it really weaker than that of noisy rationality? At this moment, it is difficult to overlook this, even though the authors have argued that it may not be as restrictive in the future when more sophisticated differentiable planners are developed.
The experimental results are not as convincing as I would have liked. In particular, Algorithm 2 (learning a demonstrator's model) does not seem to outperform that of assuming an optimal demonstrator for the noiseless case and a Boltzmann demonstrator for the noisy case (Fig. 3). This was also highlighted by the authors as well: "The learning methods tend to perform on par with the best of two choices." It begs the question whether  accounting for unknown systematic bias can indeed outperform the assumption of a particular inaccurate bias when we know a priori whether the demonstrations are noisy or not.
Other detailed comments are provided below:
I would have preferred that the authors present their technical formulations in Section 4 using the demonstrator's trajectories instead of policies.
The authors say that "In some cases, like they naive and sophisticated hyperbolic discounters, especially the noisy ones, the learning methods outperform both optimal and Boltzmann assumptions." But, Fig. 3 shows that Algorithm 2 does not perform better than either that of the optimal or Boltzmann demonstrator.
In Section 5.2, the authors have empirically demonstrated the poor approximate planning performance of VIN, as compared to an exact model the demonstrator. What then would its implications be on the adaptivity of Algorithms 1 and 2 to biases?
The following references on IRL with noisy demonstration trajectories would be relevant:
Benjamin Burchfiel, Carlo Tomasi, and Ronald Parr. Distance Minimization for Reward Learning from Scored Trajectories. In Proc. AAAI, 2016.
J. Zheng, S. Liu, and L. M. Ni. Robust Bayesian inverse reinforcement learning with sparse behavior noise. In Proc. AAAI, 2014.
Minor issues:
On page 4, the expression D : W × R -> S -> A -> [0, 1] can be more easily understood with the use of parentheses.
For Assumption 2b, you can italicize "some".
In the first paragraph of section 4.1, what are you summing over?
Line 3 of Algorithm 1: PI_W?
Page 7: For the learning the bias setting?
Page 7: figure 3 shows?
Page 7: they naive?
Page 7: so as long as?
Page 8: adaption?
Page 8: predicated?
Page 8: figure 4 shows?