Paper:
Integrated Perception with Recurrent Multi-Task Neural Networks
Modern discriminative predictors have been shown to match natural intelligences in specific perceptual tasks in image classification, object and part detection, boundary extraction, etc. However, a major advantage that natural intelligences still have is that they work well for all perceptual problems together, solving them efficiently and coherently in an integrated manner. In order to capture some of these advantages in machine perception, we ask two questions: whether deep neural networks can learn universal image representations, useful not only for a single task but for all of them, and how the solutions to the different tasks can be integrated in this framework. We answer by proposing a new architecture, which we call multinet, in which not only deep image features are shared between tasks, but where tasks can interact in a recurrent manner by encoding the results of their analysis in a common shared representation of the data. In this manner, we show that the performance of individual tasks in standard benchmarks can be improved first by sharing features between them and then, more significantly, by integrating their solutions in the common representation.

Review:
This paper proposes multinet, a new integrated network in which not only deep image features are shared between tasks (representation sharing), but where tasks can interact in a recurrent manner by encoding the results of their analysis in a common shared representation of the data (output correlation). The proposed deep models are novel and seem technically sound, but the experimental evaluation may leave some room for improvement.
This paper is crystal clear and the main points are easily accessible. The key idea of integrated learning of representation sharing and output correlation is sound and well executed in the new architecture comprising CNNs, R-CNNs, RNNs and autoencoders.  My main concern is regarding the experimental evaluation. There is clear room for improvement: (1) the authors are encouraged to use the standard VOC 2012 dataset instead of the more obsolete VOC 2010/2007 datasets--this makes direct comparison of different methods possible; (2) the baseline methods (Independent and Multi-task in Table 1) are too simple to justify the effectiveness of the proposed method, and more recent work on multi-task deep learning should be compared. Note that, although this paper contrasts itself clearly from the literature, it does not mean that it is enough to evaluate the proposed method only against simple baselines.  The heterogeneous multi-task learning scenario is exactly the same with the state of the art work that won the MS COCO competition [8]. The multi-task network cascade (MNC) [8] is a sequential model that exploits the perception behavior of humans, i.e. from coarse-grained perception to fine-grained perception. I think MNC [8] is the most closely related method deserving empirical comparison (although MNC is sequential while the method of this paper is parallel).  Another baseline method could be MT-CNN [31], which addresses the different convergence rates of different networks in the multi-task learning setting. Extending the heterogeneous multi-task learning scenario [31] to the classification, localization, and part detection as this work is straight-forward (MT-CNN is also a parallel learning method as this work).  The final concern is on the model trainability. As multinet is a complex model comprising CNNs, R-CNNs, RNNs and autoencoders, I thus worry about the high difficulty in training the hybrid architecture. In particular, the autoencoders seem rather difficult: the back-propagation process (encoder) resorts to broadcast the outputs to the previous layers, which is quite ad hoc and requires more theoretical and empirical justification. I would expect to see what the authors can clarify in the rebuttal.  -- Update after rebuttal: The authors have provided some responses on why the relevant baselines I mentioned were not empirically evaluated. I think the responses are reasonable and thus keep my original rating unchanged.