Paper:
In the context of multi-task learning, neural networks with branched architectures have often been employed to jointly tackle the tasks at hand. Such ramified networks typically start with a number of shared layers, after which different tasks branch out into their own sequence of layers. Understandably, as the number of possible network configurations is combinatorially large, deciding what layers to share and where to branch out becomes cumbersome. Prior works have either relied on ad hoc methods to determine the level of layer sharing, which is suboptimal, or utilized neural architecture search techniques to establish the network design, which is considerably expensive. In this paper, we go beyond these limitations and propose a principled approach to automatically construct branched multi-task networks, by leveraging the employed tasks' affinities. Given a specific budget, i.e. number of learnable parameters, the proposed approach generates architectures, in which shallow layers are task-agnostic, whereas deeper ones gradually grow more task-specific. Extensive experimental analysis across numerous, diverse multi-tasking datasets shows that, for a given budget, our method consistently yields networks with the highest performance, while for a certain performance threshold it requires the least amount of learnable parameters.

Review:
This paper presents a method to infer multi-task networks architecture, more specifically to determine which part of the network should be shared among different tasks. The main idea is to first train a specific, standalone encoder/decoder network for each task. Subsequently, a task affinity factor is computed by looking at the similarity (or, more likely the dissimilarity) of an holdout set of images feature representations. Knowing these dissimilarities (based on RDM), one can cluster the tasks and make similar tasks share more of their layers. Computational budget can also be taken into consideration in the form of the number of parameters. Results on Cityscapes, Taskonomy, and CelebA datasets show, to some extent, improvements against the state of the art.
The paper is well written and addresses a common problem in multi-task learning. The experiments provided are extensive and cover most of the current multi-task learning methods and interesting problems. I especially like the idea of formalizing the dissimilarity between tasks using RDM. There are, however, a few key points that would need improvement.
First, except for CelebA, the experiments provided use ResNet50 with only 4 different "anchor point" in the network. In other words, the task at hand is limited to selecting the best sharing point between 4 choices. This is not wrong per se, but in my opinion, it does not tackle the main problem: what to do when brute force / exhaustive exploration cannot be fulfilled? CelebA provides a more complex case, but it also requires to change the method from an exhaustive search to a beam search (end of Sec. 3.2). Doing so get us back to a kind of greedy approach, precisely what was advocated against the paragraph before (in the discussion about Lu et al. 2017).
Second, the fact that task affinities are computed a priori leads to the following conclusion: "this allows us to determine the task clustering offline". While I agree that this could be useful, one has to keep in mind that compared to other methods, this one has to first train a network for _each_ task independently, which can take a long time.
Out of curiosity, did you consider other correlation coefficients than Spearman? Why use a rank-based method?
Overall, this is a nice paper, with adequate methodology. It is not groundbreaking, and most of the good results arise when we consider both performance and number of parameters, but it is interesting for the community nonetheless. I am not sure the impact would be very high, since it can probably be replaced by an architecture exhaustive search if the number of tasks and branching points in the network are low, but the formalism of the approach is welcomed.