Paper:
Learning to Multitask
Multitask learning has shown promising performance in many applications and many multitask models have been proposed. In order to identify an effective multitask model for a given multitask problem, we propose a learning framework called Learning to MultiTask (L2MT). To achieve the goal, L2MT exploits historical multitask experience which is organized as a training set consisting of several tuples, each of which contains a multitask problem with multiple tasks, a multitask model, and the relative test error. Based on such training set, L2MT first uses a proposed layerwise graph neural network to learn task embeddings for all the tasks in a multitask problem and then learns an estimation function to estimate the relative test error based on task embeddings and the representation of the multitask model based on a unified formulation. Given a new multitask problem, the estimation function is used to identify a suitable multitask model. Experiments on benchmark datasets show the effectiveness of the proposed L2MT framework.

Review:
The authors of the manuscript consider the problem of selecting an information transfer algorithm in the multi-task setting. Instead of performing some kind of model selection, they propose a learning-based approach: based on historical MTL experience they build a mapping from MT training data and MT model into relative test performance improvement that using that MT model brings compared to solving each task in isolation. For a new, test, MTL problem they optimise the resulted function with respect to MT model to maximise the improvement.  Pros: This is a natural continuation of multi-task/learning to learn ideas with adding one more layer in the hierarchy  that has been mentioned in works of Baxter [7]. The authors' implementation of it relies on a nice unification of various MTL methods presented in section 2.  Cons: The practicality of the proposed L2MT seems rather limited to me - MT experiences used for training need to be very similar to the problems appearing at the testing stage. Also, data-embedding LGNN seem to require all experiences to be MTL problems with the same number of dimension. Consequently, in the experimental evaluation both training and test MTL problems come from the same dataset. This, in my opinion, greatly limits applicability of L2MT in realistic scenarios, especially since the number of training experiences has to be significant.  It would be interesting to see what happens if test and training problems are coming from different datasets, for example, by merging MIT-Indoor-Scene and Caltech256 experiments.  Experiments: - how parameters were selected for the baselines? - how MT models and relative test errors were obtained for the training stage of L2MT?  Additional comments: - I believe, in Theorem 4 one needs the MTL experiences to be sampled from a distribution, not individual tasks. Also, without a more direct bound on the Gaussian width of {E_i} the resulting bound is not particularly conclusive - the authors may want to add [1] to the related work, as it is based on a similar idea but for domain adaptation  [1] Transfer Learning via Learning to Transfer, Ying WEI, Yu Zhang, Junzhou Huang, Qiang Yang ICML'18  Update: based on the authors' response and other reviews I raise my score to 6