Paper:
Published as a conference paper at ICLR 2018 LEARNING TO MULTI-TASK BY ACTIVE SAMPLING
One of the long-standing challenges in Artificial Intelligence for learning goaldirected behavior is to build a single agent which can solve multiple tasks. Recent progress in multi-task learning for goal-directed sequential problems has been in the form of distillation based learning wherein a student network learns from multiple task-specific expert networks by mimicking the task-specific policies of the expert networks. While such approaches offer a promising solution to the multi-task learning problem, they require supervision from large expert networks which require extensive data and computation time for training. In this work, we propose an efficient multi-task learning framework which solves multiple goaldirected tasks in an on-line setup without the need for expert supervision. Our work uses active learning principles to achieve multi-task learning by sampling the harder tasks more than the easier ones. We propose three distinct models under our active sampling framework. An adaptive method with extremely competitive multi-tasking performance. A UCB-based meta-learner which casts the problem of picking the next task to train on as a multi-armed bandit problem. A meta-learning method that casts the next-task picking problem as a full Reinforcement Learning problem and uses actor critic methods for optimizing the multi-tasking performance directly. We demonstrate results in the Atari 2600 domain on seven multi-tasking instances: three 6-task instances, one 8-task instance, two 12-task instances and one 21-task instance.

Review:
The authors show empirically that formulating multitask RL itself as an active learning and ultimately as an RL problem can be very fruitful.  They design and explore several approaches  to the active learning (or active sampling) problem, from a basic 
change to the distribution to UCB to feature-based neural-network based RL. The domain is video games.   All proposed approaches beat the uniform sampling baselines and the more sophisticated approaches do better in the scenarios with more tasks (one multitask  problem had 21 tasks).
Pros:
- very promising results with an interesting active learning approach to multitask RL
- a number of approaches developed for the basic idea
- a variety of experiments, on challenging multiple task problems (up to 21 tasks/games)
- paper is overall well written/clear
Cons:
- Comparison only to a very basic baseline (i.e. uniform sampling)
Couldn't comparisons be made, in some way, to other multitask work?
Additional  comments:
- The assumption of the availability of a target score goes against
the motivation that one need not learn individual networks ..  authors
say instead one can use 'published' scores, but that only assumes
someone else has done the work (and furthermore, published it!).
The authors do have a section on eliminating the need by doubling an
estimate for each task) which makes this work more acceptable (shown
for 6 tasks or MT1, compared to baseline uniform sampling).
Clearly there is more to be done here for a future direction (could be
mentioned in future work section).
- The averaging metrics (geometric, harmonic vs arithmetic, whether
  or not to clip max score achieved) are somewhat interesting, but in
  the main paper, I think they are only used in section 6 (seems like
  a waste of space). Consider moving some of the results, on showing
  drawbacks of arithmetic mean with no clipping (table 5 in appendix E), from the appendix to
  the main paper.
- The can be several benefits to multitask learning, in particular
  time and/or space savings in learning new tasks via learning more
  general features. Sections 7.2 and 7.3 on specificity/generality of
  features were interesting.
--> Can the authors show that a trained network (via their multitask
    approached) learns significantly faster on a brand new game
    (that's similar to games already trained on), compared to learning from
    scratch?
--> How does the performance improve/degrade (or the variance), on the
    same set of tasks, if the different multitask instances (MT_i)
    formed a supersets hierarchy, ie if MT_2 contained all the
    tasks/games in MT_1, could training on MT_2 help average
    performance on the games in MT_1 ? Could go either way since the network
   has to allocate resources to learn other games too.  But is there a pattern?
- 'Figure 7.2' in section 7.2 refers to Figure 5.
- Can you motivate/discuss better why not providing the identity of a
  game as an input is an advantage? Why not explore both
  possibilities? what are the pros/cons? (section 3)