Paper:
Revisiting Multi-Task Learning with ROCK: a Deep Residual Auxiliary Block for Visual Detection
Multi-Task Learning (MTL) is appealing for deep learning regularization. In this paper, we tackle a specific MTL context denoted as primary MTL, where the ultimate goal is to improve the performance of a given primary task by leveraging several other auxiliary tasks. Our main methodological contribution is to introduce ROCK, a new generic multi-modal fusion block for deep learning tailored to the primary MTL context. ROCK architecture is based on a residual connection, which makes forward prediction explicitly impacted by the intermediate auxiliary representations. The auxiliary predictor's architecture is also specifically designed to our primary MTL context, by incorporating intensive pooling operators for maximizing complementarity of intermediate representations. Extensive experiments on NYUv2 dataset (object detection with scene classification, depth prediction, and surface normal estimation as auxiliary tasks) validate the relevance of the approach and its superiority to flat MTL approaches. Our method outperforms state-of-the-art object detection models on NYUv2 dataset by a large margin, and is also able to handle large-scale heterogeneous inputs (real and synthetic images) with missing annotation modalities.

Review:
Main Idea: The work targets a special case of multi-task learning (MTL) referred to as primary MTL where the goal is to use auxiliary tasks to improve performance on a primary task. The primary task considered is object detection with scene classification, depth prediction, surface normal estimation being the auxiliary tasks.  The authors introduce a Residual Auxiliary Block (ROCK) which can be used to augment existing deep network-based object detection pipelines. ROCK is proposed as an alternative to the standard MTL approach used in object detection literature where a backbone feature extractor is followed by multiple task-specific heads e.g. classification, bounding box regression, and segmentation heads in MaskRCNN. The paper comprises of justification of various design choices in ROCK and ablation studies to quantify the benefit of those choices.   Strengths: - The authors do a good job of zooming in on a specific problem (primary MTL) that they are trying to solve. Writing and experiments are appropriately designed around that problem.  - Fig 2 summarizes the model well. - The writing is clear and Sec. 2 and 3 helps the reader understand the authors' motivation behind various design choices that went into ROCK.  - Ablations in Tables 1 and 3 demonstrate the benefits of using auxiliary annotations, encoders, and feature fusion in ROCK over the standard MTL baseline on the NYUv2 dataset.  - Table 4 tries to answer an interesting question - Is it better to have more images with one kind of annotations or fewer images with annotations for a wide variety of tasks? The numbers indicate that 70% of the data with multiple task annotations is sufficient to achieve performance comparable to a detector trained on 100% data without MTL.   Weaknesses: Writing - Primary MTL should be differentiated from alternative goals of MTL (such as improving the performance of all tasks or saving memory and computational cost by sharing computation in a single network) early on in the abstract and introduction - In the abstract, a point is made about residual connections in ROCK allowing auxiliary features to explicitly impact detection prediction. This needs to be contrasted with standard MTL where the impact is implicit.  - The claim about ROCK handling missing annotations modalities is unclear. The MLT dataset needs to be described in Sec 4.3.  - The introduction describes Transfer Learning (TL) and Fine-tuning (FT) as sequential MTL. I do not completely agree with this characterization. TL is a broader term for the phenomenon of learnings from one task benefitting another task. Fine-tuning is a sequential way of doing it. Standard MTL is a parallel means to the same end.  - The first 2 paragraphs of introduction read like literature review instead of directly motivating ROCK and the problem that it solves.  - Need to describe Flat MTL and include a diagram similar to Fig 2 that can be directly visually compared to ROCK.  - Fig 1 is not consistent with Fig 2. Fig 2 shows one encoder-decoder per auxiliary task whereas Fig 1 shows a single shared encoder-decoder for multiple tasks.  - L88-89 try to make the case that ROCK has similar complexity as the original model. I would be very surprised if this true because ROCK adds 8 conv layers, 3 pooling layers, and 1 fusion layer. Inference timings must be provided to make this claim.   Weaknesses: Experiments - Ablation showing the contribution of each auxiliary task on object detection performance in ROCK as well as in standard MTL. This can be done by comparing the full model (DNS) with models where one task is dropped out (DS, NS, DN).  - Cite MLT dataset in Table 2 caption and describe Geo, presumably some kind of geometric features.  - More details are needed for reproducibility like backbone architecture, activations between conv layers, number of channels, etc.   Weaknesses: Originality - While primary MTL setting considered in this work is very useful, especially in data-deficient domains, the main contribution of this work, the ROCK architecture, comes across as a little incremental.  Summary My current assessment is that in spite of the weaknesses, the model is simple, reasonably well motivated and shows decent performance gains over appropriate baselines. Hence I am currently leaning towards an accept with a rating of "6". I am not providing a higher rating as of now because the writing can be improved in several places, an ablation is missing, and the novelty is limited. My confidence score if 4 because I have only briefly looked at [19] and [35] which seem to be the most relevant literature.    --Final Review After Rebuttal--  The authors did present an ablation showing the effect of different tasks on multitask training as requested. A significant number of my comments were writing suggestions which I believe would improve the quality of the paper further. The authors have agreed to make appropriate modifications. In spite of some concern about novelty, I think this work is a valuable contribution towards an understanding of multitask learning. All reviewers seem to be more or less in agreement. Hence, I vote for an accept and have increased my rating to 7 (previously 6).