Paper:
Reinforcement learning (RL) has become an increasingly active area of research in recent years. Although there are many algorithms that allow an agent to solve tasks efficiently, they often ignore the possibility that prior experience related to the task at hand might be available. For many practical applications, it might be unfeasible for an agent to learn how to solve a task from scratch, given that it is generally a computationally expensive process; however, prior experience could be leveraged to make these problems tractable in practice. In this paper, we propose a framework for exploiting existing experience by learning reusable options. We show that after an agent learns policies for solving a small number of problems, we are able to use the trajectories generated from those policies to learn reusable options that allow an agent to quickly learn how to solve novel and related problems.

Review:
Summary:
The authors propose to learn reusable options to make use of prior information and claim to do so with minimal information from the user (such as # of options needed to solve the task, which options etc). The claim is that the agent is first able to learn a near-optimal policy for a small # of problems and then is able to solve a large # of tasks by such a learned policy.  The authors build on the idea that minimizing the number of decisions made by the agent results in discovering reusable options. The options are learned offline by learning to solve a small number of tasks. Their algorithm introduces one option at a time until introducing a new option doesn’t improve the objective further. The ideas are interesting, However, the paper as it stands is lacking in thorough evaluation.
Detailed comments:
The proposed approach offers two key contributions:
-an objective function to minimize the decision states
-incrementally constructing an option set that can be reused later, without the a priori specification of the # of options needed. 
The introduction is well written, however, given the intuitions behind the objective function; in some sense, the idea here is to minimize the decisions or terminations intuitively relates to terminating only at critical or bottleneck states. It would be useful to provide such motivation in the introduction. 
Intuitively the objective criterion is interesting. With a cursory look at the proofs, they seem fine, although I have to admit I have not looked in detail into the proofs. 
Paper writing could be significantly improved. Several points are not clear and need further clarification:
-The term near-optimal is mentioned several times, but it is not clear the policies are near-optimal with respect to what? The task or a set of tasks? 
-How does the proposed approach ensures that they are near-optimal? Please clarify.
-“We can obtain the estimate for equation 1 by averaging over a set of near-optimal trajectories” The aim as states is to learn options that are capable of generating near-optimal trajectories (by using a small # of terminations). The authors then say that “given a set of options, a policy over options, a near-optimal sample trajectory, we can calculate..” Where does the near-optimal sample trajectory come from? Please provide clarifications.
In experiments: FR rooms experiments are interesting, in the visualization of the option policies, do the figures here show the flattened policy of the options? What do the termination functions look like? 
Atari experiments are limited in nature in that they show only two games. Moreover, It is a bit confusing as to what is multi-task in the ATARI experiments. The authors mention the training of options and then talk about the results in the plots (4) show the training curves. However, they do not mention what are “novel tasks for Breakout/Amidar” in this context. 
Considering the proposed approach is closely related to the idea of selective terminations of options, it is natural to expect a comparison with Harb, 2018 and Hartyuanm 2019. The work could benefit by comparing with the aforementioned baselines. In particular, the visualization in 4b showing options learned in Amidar does not show much improvement from what was observed before in Harb, 2018.  
With the motivation of this paper, I am unable to convince myself about options being “reusable” for multi-task here. It would be very useful for the reader to clarify what “novel tasks” are here to appreciate what is learned. Looking deeper into the appendix, I understand that the authors “first learned a good performing policy with A3C for each game and sample 12 trajectories for training.” This is not at all clear in the main paper. Besides, what does it mean by a "good" policy? If we already have that, it is unclear what gains do we get from the proposed method.  
One obvious limitation here is that they also have a hard imposed constraint here is that the options cannot run for more than 20 time-steps in total, to make the objective function a suitable choice. 
Overall:		
An interesting objective function, Learn not only option set but also the number of options needed and incrementally learn new options. 
Paper writing does not convey clearly what are novel tasks and could be significantly improved.
Since the paper claims multi-task and mentions several lifelong learning works like [1], I was expecting rigorous baselines showing performance over multiple tasks. The experiments are lacking in that evidence except for four rooms domain, which is much simpler a domain. 
Near-optimal property is very much lacking the clarity to the best of my knowledge.
[1]Ammar, Haitham Bou, et al. "Online multi-task learning for policy gradient methods." International Conference on Machine Learning. 2014.