Paper:
We compare the model-free reinforcement learning with the model-based approaches through the lens of the expressive power of neural networks for policies, Q-functions, and dynamics. We show, theoretically and empirically, that even for one-dimensional continuous state space, there are many MDPs whose optimal Q-functions and policies are much more complex than the dynamics. We hypothesize many real-world MDPs also have a similar property. For these MDPs, model-based planning is a favorable algorithm, because the resulting policies can approximate the optimal policy significantly better than a neural network parameterization can, and model-free or model-based policy optimization rely on policy parameterization. Motivated by the theory, we apply a simple multi-step model-based bootstrapping planner (BOOTS) to bootstrap a weak Q-function into a stronger policy. Empirical results show that applying BOOTS on top of modelbased or model-free policy optimization algorithms at the test time improves the performance on MuJoCo benchmark tasks.

Review:
This paper presents a mainly theoretical argument comparing the expressivity of model-free and model-based RL methods contrary to analysis in the past which usually relies on sample complexity. They construct a family of MDPs, where the true dynamics belong to a simple function class (in terms of the number of linear pieces needed to define the function), but the corresponding optimal Q-function belongs to a function class not necessarily expressible by a simple function. The paper then builds a similar case for randomly/semi-randomly generated MDPs. Finally, they propose to bootstrap the Q-function with n-step returns to boost the expressivity exponentially. 
I would lean towards accepting this paper, as this paper looks at an interesting problem and their analysis seems to be rigorous and valuable for the community to build upon. My questions/comments are as follows:
1. Previous work, for example [1], also talks about bootstrapping a neural net Q-function using an n-step approximation so as to improve the efficiency of a pure model-free algorithm. I think this paper should be cited.
2. In terms of the experiments, it is hard to understand the significance and connection to the theory. The theory talks about the expressive power of Q-functions, which suggests that we should look at only the asymptotic performance on these tasks, but most of the results are similar to MBPO or SAC in terms of asymptotic performance, although with a different learning speed, which could have to do with different factors.
3. This paper shows the existence and provides a constructive proof for a family of MDPs where expressing optimal Q-functions is exponentially harder than expressing dynamics. But how likely is such a setting to arise in MDPs in practice? For example, on the gym benchmarks, we would expect fairly not so complicated Q-functions -- although they might take longer to learn. 
4. There is also a divide between learning Q*, and learning a policy that optimizes Q* reasonably well. Further, the requirement for the Q-function is only to get relative ordering between Q-values for different actions at states visited by the optimal policy right, which might not be the same as the Q-function landscape across state-action pairs. So, I am not sure if the expressivity of the optimal Q-function, in general, is the right metric to answer this question, but I also completely agree that this is a good starting point. 
5. I think when applying RL to the current benchmarks or problems we have, the main problem could be linked to optimization of a neural-net Q-function via bootstrapping (in the sense of approximate dynamic programming) as compared to the expressive power of optimal Q-functions. But I agree that the problem being looked at in the paper would also exist.
References:
[1] Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control, Lowrey et.al.