Paper:
Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using multitask learning. This saves computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives can compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We systematically study task cooperation and competition and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a timeaccuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks.

Review:
This paper works on the problem if training a set of networks to solve a set of tasks. The authors try to discover an optimal task split into the networks so that the test performances are maximized given a fixed testing resource budget. By default, this requires searching over the entire task combination space and is too slow. The authors propose two strategies for fast approximating the enumerative search. Experiments show their searched combinations give better performance in the fixed-budget testing setting than several alternatives.
+ This paper works on a new and interesting problem. Training more than one networks for a few tasks is definitely a valid idea in real applications and related to broad research fields.
+ The baseline setup is comprehensive. The difference between optimal, random, and worst clearly shows this problem worths effort for research.
- I believe this problem setup requires a larger task set. In the paper the authors manually picked 5 tasks. It seems straightforward for a human to manually group them together: segmentation and edge/ surface + depth/ keypoint for 3 networks. It is unclear how better a network can do than a human in one minute or should we expect learning the task split is better than manual design.
- Both technical contributions in Section 3.3 look straightforward. Given the good performance in Figure3, it is fine.
- I am confused by the comparison to Sener and Koltun. How do you change the inference budget for them? If it is changing the number of channels for a single network, I believe it can be improved more.
Overall I believe this is a good paper to open interesting research direction with solid baselines. I am happy to accept this paper and see more exciting future works in this direction.