Paper:
Learning Multiple Tasks in Parallel with a Shared Annotator
We introduce a new multi-task framework, in which K online learners are sharing a single annotator with limited bandwidth. On each round, each of the K learners receives an input, and makes a prediction about the label of that input. Then, a shared (stochastic) mechanism decides which of the K inputs will be annotated. The learner that receives the feedback (label) may update its prediction rule, and then we proceed to the next round. We develop an online algorithm for multitask binary classification that learns in this setting, and bound its performance in the worst-case setting. Additionally, we show that our algorithm can be used to solve two bandits problems: contextual bandits, and dueling bandits with context, both allow to decouple exploration and exploitation. Empirical study with OCR data, vowel prediction (VJ project) and document classification, shows that our algorithm outperforms other algorithms, one of which uses uniform allocation, and essentially achieves more (accuracy) for the same labour of the annotator.

Review:
This paper studies the problem of online multi-task learning in which a single annotator with limited bandwidth is shared among tasks. An online algorithm is proposed and its performance is analyzed. Major Strengths:Online multitask learning is an important topic, and the field of multi-armed bandits has a rich history.   Seeking to leverage ideas and results from the latter to apply to the former domain will likely yield improved techniques, especially since for the multi-task learning problem some labels might not be observed.  Furthermore, this area is highly relevant to NIPS.Major Weaknesses:An important aspect to multi-task learning is that there is some dependency between the tasks so that there is utility between learning the tasks jointly.  In the work under review, no dependency is assumed.  This appears to be a significant departure from the main multitask learning literature with no apparent justification.  Without the ability to transfer knowledge between the tasks, it appears more as the multi-armed bandit analog in multi-task learning.  For instance, why can only one feature be annotated at a time though all features are allowed to be annotated?    In using Mechanical Turk, as suggested, for some applications the workers might not annotate labels for all features equally well.  Thus, only allowing a subset of features to be annotated while the rest cannot seems more relevant of a setting.  The article mentions it is a new multi-task framework, so it need not follow traditional assumptions, though more motivation and justification is needed for why the differences are important.One counter-example to the previous statement is Romera-Paredes et al. “Exploiting Unrelated Tasks in Multi-Task Learning” in AISTATS 2012.  There they use the knowledge that two tasks are unrelated to avoid overfitting by using the same features.  However, the work under review does not exploit the lack of dependence.This is less important, but to check the implication (lines 153-154) of Theorem 1, for the data analysis it would have been interesting to see how SHAMPO performs compared to the case when labels are available for all tasks.Minor notes:[25] “both allows”[209] “outputs multicalss"[347] remove comma