Paper:
Value-based methods constitute a fundamental methodology in planning and deep reinforcement learning (RL). In this paper, we propose to exploit the underlying structures of the state-action value function, i.e., Q function, for both planning and deep RL. In particular, if the underlying system dynamics lead to some global structures of the Q function, one should be capable of inferring the function better by leveraging such structures. Specifically, we investigate the lowrank structure, which widely exists for big data matrices. We verify empirically the existence of low-rank Q functions in the context of control and deep RL tasks (Atari games). As our key contribution, by leveraging Matrix Estimation (ME) techniques, we propose a general framework to exploit the underlying low-rank structure in Q functions, leading to a more efficient planning procedure for classical control, and additionally, a simple scheme that can be applied to any value-based RL techniques to consistently achieve better performance on "low-rank" tasks. Extensive experiments on control tasks and Atari games confirm the efficacy of our approach.

Review:
Summary:
	This paper develops a method for taking advantage of structure in the value function to facilitate faster planning and learning. The key insight is that MDPs with low rank Q^* matrices can be solved more expediently using matrix estimation methods, both for classical dynamic programming methods (value iteration) and for learning in rich environments using recent model-free deep RL techniques. Thorough empirical analysis is conducted both for value iteration in tabular MDPs and for deep RL in rich environments. These experiments highlight new findings about the role the rank of the Q matrix plays in planning convergence and learning rates.
	I view this paper as containing several key contributions: first, the analysis on the role of Q rank in planning and learning---experiments conducted indicate that even complicated environments tend to have low rank Q matrices (when approximated). Highlighting the role of this rank and the corresponding empirical analysis estimating it in benchmark RL and control tasks is, to my knowledge, novel. Second, and perhaps the most significant contribution, is "Structured Value RL" (SV-RL), an easy-to-apply method that can be incorporated into many Q-based deep RL methods with little overhead. The empirical results are compelling: across three different variations of DQN-like architectures, the SV RL augmentation tends to improve learning. Presentation of results is rigorous, too, and provide strong evidence that the method works.
	As the paper mentions, theoretical analysis on the impact of Q-rank on dynamic programming (and perhaps learning) would be of great interest to the community. I take this analysis to be out of scope for this paper, but could see the work motivating future investigation into these questions.
Verdict: Overall, I take this paper to present many novel insights, establish solid motivation with good writing and examples, and offers compelling evidence about the strength of SV RL. I recommend accepting the paper.
Comments:
	C1: The visuals throughout the paper are helpful!
	C2: The paper is well written: the use of examples was effective in developing the motivation.
	C3: Section 2 is helpful for understanding the ideas developed in the paper. However, there are many well developed planning frameworks for MDPs that trade-off optimality with computational efficiency. It might be worth discussing some of these methods up front. For instance, Bounded Real-Time Dynamic Programming (McMahan et al. 2005) explicitly uses value function structure to improve planning speed, with performance guarantees, as does Focused RTDP (Smith and Simmons 2006). I don't take the computational complexity improvements of the proposed method to be the primary contribution, so just a brief discussion to contextualize the work against other planning literature would be helpful.
	C4: While the "rank" studied here is of a different form, some discussion of the Bellman Rank work (Jiang et al. 2017) might be useful for differentiating the two notions of "rank" at play, and how they are each used to expedite learning. The Bellman Rank is used as a measure of complexity of an MDP---Jiang et al. develop an RL algorithm that has sample complexity that depends on this measure. It is not strictly necessary, but I could see multiple uses of "rank" appearing in the RL literature as a means of exploiting structure for faster learning being confusing. If space (perhaps in the appendix if not), a sentence or two differentiating the two ideas might be helpful to readers. Additionally, the study of sparsity in value function representation was studied by Calandriello et al. 2014. If space permits, the paper might benefit from some discussion of the relation to this work.
Questions:
	Q1: In the inverted pendulum results, I am curious about the effect of the discretization on plan quality. Specifically: how were 2500 states and 1000 actions chosen? Were different orders of magnitude (for both values) considered? How did this impact SVP? Does the rank change as the discretization becomes more or less coarse? I don't think this is strictly critical for the paper, but a few sentences clarifying this point would be informative.
	Q2: Figure 4 provides nice insights into how to scale these ideas to deep RL. How were the four games chosen? Is there anything special that motivated their selection?
	Q3: Additionally, I am curious about whether the results from Figure 4 are the consequence of algorithmic decisions, rather than the environment. Is it possible to determine whether different value based methods (or different choices of hyperparameters) lead to different outcomes? For instance, I could imagine a more shallow network, or a tighter bottleneck, leading to Q evaluations that produce higher rank. 
Typos and Writing Suggestions:
[Abstract]
	- This sentence is quite long, and I had a hard time following it as a result: "As our key contribution, by leveraging...". Consider dividing into two sentences.
[Intro]
	- Oxford comma: "control, planning and reinforcement learning"::"control, planning, and reinforcement learning
	- "the structured dynamic"::"the structure in the dynamics"
	- "where much fewer samples"::"where fewer samples"
	- Consider rewording: "almost the same policy as the optimal one". Is it that the policies are in fact the same? Or that their values are close? Perhaps: "a policy with near optimal value".
	- When introducing Double DQN and Dueling DQN for the first time it would be appropriate to cite each (end of Section 1).
[Sec. 2: Warm Up]
	- "understand the structures"::"understand the structure"
	- "give a strong evidence for"::"provide evidence that"
	- "exploit the structures for"::"exploit structure in the value function for"
	- I think the italicized statement at the top of page 3 could be sharpened. The antecedent currently stating "why not" is quite a soft statement compared to the motivation the section develops. Consider changing: "...why not enforcing such a structure throughout the iterations?"::"...then enforcing such a structure throughout planning can improve the rate of convergence".
[Sec. 3: Structured ... Planning]
	- "even non-convex optimization approaches (..."::"even non-convex optimization approaches to solving this problem (..."
	- "offer a sounding foundation for future"::"offer a sound foundation for future"
[Sec. 4: Structured ... RL]
	- "Previously, we start by"::"Previously, we started by"
	- "which in deep scenarios"::"which in scenarios with large state spaces", or perhaps: "which in deep scenarios"::"which in scenarios where value function approximation is used"
References:
Calandriello, Daniele, Alessandro Lazaric, and Marcello Restelli. "Sparse multi-task reinforcement learning." Advances in Neural Information Processing Systems. 2014.
Jiang, Nan, et al. "Contextual decision processes with low Bellman rank are PAC-learnable." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
McMahan, H. Brendan, Maxim Likhachev, and Geoffrey J. Gordon. "Bounded real-time dynamic programming: RTDP with monotone upper bounds and performance guarantees." Proceedings of the 22nd international conference on Machine learning. ACM, 2005.
Smith, Trey, and Reid Simmons. "Focused real-time dynamic programming for MDPs: Squeezing more out of a heuristic." AAAI. 2006.