Original: M ULTI -T ASK L EARNING BY A T OP -D OWN C ONTROL N ETWORK
Revised: M ULTI -T ASK L EARNING BY A T OP -D OWN C ONTROL N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As the range of tasks performed by a general vision system expands, executing multiple tasks accurately and efﬁciently in a single network has become an important and still open problem.
Revised: As the range of tasks performed by a general vision system expands, executing multiple tasks accurately and efﬁciently in a single network has become an important and still open problem.

Original: Recent computer vision approaches address this problem by branching networks, or by a channel-wise modulation of the network feature-maps with task speciﬁc vectors.
Revised: Recent computer vision approaches address this problem by branching networks, or by a channel-wise modulation of the network feature-maps with task speciﬁc vectors.

Original: We present a novel architecture that uses a dedicated top-down control network to modify the activation of all the units in the main recognition network in a manner that depends on the selected task, image content, and spatial location.
Revised: We present a novel architecture that uses a dedicated top-down control network to modify the activation of all the units in the main recognition network in a manner that depends on the selected task, image content, and spatial location.

Original: We show the effectiveness of our scheme by achieving signiﬁcantly better results than alternative state-of-the-art approaches on four datasets.
Revised: We show the effectiveness of our scheme by achieving signiﬁcantly better results than alternative state-of-the-art approaches on four datasets.

Original: We further demonstrate our advantages in terms of task selectivity, scaling the number of tasks and interpretability.
Revised: We further demonstrate our advantages in terms of task selectivity, scaling the number of tasks and interpretability.

Original: Code is supplied in the Supplementary material and will be publicly available.
Revised: Code is supplied in the Supplementary material and will be publicly available.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The goal of multi-task learning is to improve the learning efﬁciency and increase the prediction accuracy of multiple tasks learned and performed in a shared network.
Revised: The goal of multi-task learning is to improve the learning efﬁciency and increase the prediction accuracy of multiple tasks learned and performed in a shared network.

Original: In recent years, several types of architectures have been proposed to combine multiple tasks training and evaluation.
Revised: In recent years, several types of architectures have been proposed to combine multiple tasks training and evaluation.

Original: Most current schemes assume task-speciﬁc branches, on top of a shared backbone (Figure 1a) and use a weighted sum of tasks losses for training (Chen et al., 2017; Sener & Koltun, 2018).
Revised: Most current schemes assume task-speciﬁc branches, on top of a shared backbone (Figure 1a) and use a weighted sum of tasks losses for training (Chen et al., 2017; Sener & Koltun, 2018).

Original: Having a shared representation is more efﬁcient from the standpoint of memory and sample complexity (Zhao et al., 2018), but the performance of such schemes is highly dependent on the relative losses weights that cannot be easily determined without a “trial and error” search phase (Kendall et al., 2018).
Revised: Having a shared representation is more efﬁcient from the standpoint of memory and sample complexity (Zhao et al., 2018), but the performance of such schemes is highly dependent on the relative losses weights that cannot be easily determined without a “trial and error” search phase (Kendall et al., 2018).

Original: Another type of architecture (Zhao et al., 2018; Strezoski et al., 2019) uses task-speciﬁc vectors to modulate the feature-maps along a feed-forward network, in a channel-wise manner (Figure 1b).
Revised: Another type of architecture (Zhao et al., 2018; Strezoski et al., 2019) uses task-speciﬁc vectors to modulate the feature-maps along a feed-forward network, in a channel-wise manner (Figure 1b).

Original: Channel-wise modulation based architecture has been shown to decrease the destructive interference between conﬂicting gradients of different tasks (Zhao et al., 2018) and allowed Strezoski et al.
Revised: Channel-wise modulation based architecture has been shown to decrease the destructive interference between conﬂicting gradients of different tasks (Zhao et al., 2018) and allowed Strezoski et al.

Original: (2019) to scale the number of tasks without changing the network.
Revised: (2019) to scale the number of tasks without changing the network.

Original: Here, both training and evaluation use the single tasking paradigm: executing one task at a time, rather than getting responses to all the tasks in a single forward pass.
Revised: Here, both training and evaluation use the single tasking paradigm: executing one task at a time, rather than getting responses to all the tasks in a single forward pass.

Original: Executing one task at a time is also possible by integrating task-speciﬁc modules along the network (Maninis et al., 2019).
Revised: Executing one task at a time is also possible by integrating task-speciﬁc modules along the network (Maninis et al., 2019).

Original: A limitation of using task-speciﬁc modules (Maninis et al., 2019) or of using a ﬁxed number of branches (Strezoski et al., 2019), is that it may become difﬁcult to add additional tasks at a later time during the system life-time.
Revised: A limitation of using task-speciﬁc modules (Maninis et al., 2019) or of using a ﬁxed number of branches (Strezoski et al., 2019), is that it may become difﬁcult to add additional tasks at a later time during the system life-time.

Original: We propose a new type of architecture with no branching, which performs a single task at a time with no task-speciﬁc modules.
Revised: We propose a new type of architecture with no branching, which performs a single task at a time with no task-speciﬁc modules.

Original: Our model is trained to perform a set of tasks ( { t i } Ti =1 ) one task at a time.
Revised: Our model is trained to perform a set of tasks ( { t i } Ti =1 ) one task at a time.

Original: The model receives two inputs: the input image, and a learned vector that speciﬁes the selected task t k to perform.
Revised: The model receives two inputs: the input image, and a learned vector that speciﬁes the selected task t k to perform.

Original: It is constructed from two main parts (Figure 1c): a main recognition network that is common to all tasks, termed below BU2 (BU for bottom-up), and a control network that modiﬁes the feature-maps along BU2 in a manner that will compute a close approximation to the selected task t k .
Revised: It is constructed from two main parts (Figure 1c): a main recognition network that is common to all tasks, termed below BU2 (BU for bottom-up), and a control network that modiﬁes the feature-maps along BU2 in a manner that will compute a close approximation to the selected task t k .

Original: As detailed below, the control network itself is built from two components (Figure 1d): a top-down (TD) network that receives as inputs both a task vector as well as image information from a bottom-up stream termed BU1 (Figure 1d).
Revised: As detailed below, the control network itself is built from two components (Figure 1d): a top-down (TD) network that receives as inputs both a task vector as well as image information from a bottom-up stream termed BU1 (Figure 1d).

Original: As a result, the TD stream combines task information with image information, to control the individual units of the feature-maps along BU2.
Revised: As a result, the TD stream combines task information with image information, to control the individual units of the feature-maps along BU2.

Original: The modiﬁcation of units
Revised: The modiﬁcation of units

Original: [Figure]
Revised: [Figure]

Original: activity in BU2 therefore depends on the task to perform, the spatial location, and the image content extracted by BU1.
Revised: activity in BU2 therefore depends on the task to perform, the spatial location, and the image content extracted by BU1.

Original: As shown later, the task control by our approach becomes highly efﬁcient in the sense that the recognition network becomes tuned with high speciﬁcity to the selected task t k .
Revised: As shown later, the task control by our approach becomes highly efﬁcient in the sense that the recognition network becomes tuned with high speciﬁcity to the selected task t k .

Original: Our contributions are as follow:
Revised: Our contributions are as follow:

Original: a. Our new architecture is the ﬁrst to modulate a multi-task network as a function of the task, location (spatial-aware) and image content (image-aware).
Revised: a. Our new architecture is the ﬁrst to modulate a multi-task network as a function of the task, location (spatial-aware) and image content (image-aware).

Original: All this is achieved by a top-down stream propagating task, image and location information to lower levels of the bottom-up network.
Revised: All this is achieved by a top-down stream propagating task, image and location information to lower levels of the bottom-up network.

Original: b. Our scheme provides scalability with the number of tasks (no additional modules / branches per task) and interpretability (Localization of relevant objects at the end of the top-down stream).
Revised: b. Our scheme provides scalability with the number of tasks (no additional modules / branches per task) and interpretability (Localization of relevant objects at the end of the top-down stream).

Original: c. We show signiﬁcantly better results than other state-of-the-art methods on four datasets: MultiMNIST (Sener
Revised: c. We show signiﬁcantly better results than other state-of-the-art methods on four datasets: MultiMNIST (Sener

Original: Koltun
Revised: Koltun

Original: , CLEVR (
Revised: , CLEVR (

Original: Johnson et al., 2017), CELEB-A (Liu et al., 2015) and
Revised: Johnson et al., 2017), CELEB-A (Liu et al., 2015) and

Original: CUB-200 (Welinder et al., 2010).
Revised: CUB-200 (Welinder et al., 2010).

Original: Advantages are shown in both accuracy and effective learning.
Revised: Advantages are shown in both accuracy and effective learning.

Original: d. We introduce a new measure of task speciﬁcity, crucial for multi-tasking, and show the high task-selectivity of our scheme compared with alternatives.
Revised: d. We introduce a new measure of task speciﬁcity, crucial for multi-tasking, and show the high task-selectivity of our scheme compared with alternatives.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work draws ideas from the following research lines:
Revised: Our work draws ideas from the following research lines:

Original: Multiple Task Learning (MTL) Multi-task learning has been used in machine learning well before the revival of deep networks (Caruana, 1997).
Revised: Multiple Task Learning (MTL) Multi-task learning has been used in machine learning well before the revival of deep networks (Caruana, 1997).

Original: The success of deep neural networks in the performance of single tasks (e.g., in classiﬁcation, detection and segmentation) has revived the interest of the computer vision community in the subject (Kokkinos, 2017; He et al., 2017; Redmon & Farhadi, 2017).
Revised: The success of deep neural networks in the performance of single tasks (e.g., in classiﬁcation, detection and segmentation) has revived the interest of the computer vision community in the subject (Kokkinos, 2017; He et al., 2017; Redmon & Farhadi, 2017).

Original: Although our primary application area is computer vision, multi-task learning has also many applications in other ﬁelds like natural language processing (Hashimoto et al., 2016; Collobert &
Revised: Although our primary application area is computer vision, multi-task learning has also many applications in other ﬁelds like natural language processing (Hashimoto et al., 2016; Collobert &

Original: Weston, 2008) and even across modalities (Bilen & Vedaldi, 2016).
Revised: Weston, 2008) and even across modalities (Bilen & Vedaldi, 2016).

Original: Over the years, several types of architectures have been proposed in computer vision to combine the training and evaluation of multiple tasks.
Revised: Over the years, several types of architectures have been proposed in computer vision to combine the training and evaluation of multiple tasks.

Original: First works used several duplications (as many as the tasks) of the base network, with connections between them to pass useful information between the tasks (Misra et al., 2016; Rusu et al., 2016).
Revised: First works used several duplications (as many as the tasks) of the base network, with connections between them to pass useful information between the tasks (Misra et al., 2016; Rusu et al., 2016).

Original: These works do not share computations and cannot scale with the number of tasks.
Revised: These works do not share computations and cannot scale with the number of tasks.

Original: More recent architectures, which are in common practice these days, assume task-speciﬁc branches on top of a shared backbone, and use a weighted sum of losses to train them.
Revised: More recent architectures, which are in common practice these days, assume task-speciﬁc branches on top of a shared backbone, and use a weighted sum of losses to train them.

Original: The joint learning of several tasks has proven beneﬁcial in several cases (He et al., 2017), but can also decrease the accuracy of some of the tasks due to limited network capacity, the presence of uncorrelated gradients from the different tasks and different rates of learning (Kirillov et al., 2019).
Revised: The joint learning of several tasks has proven beneﬁcial in several cases (He et al., 2017), but can also decrease the accuracy of some of the tasks due to limited network capacity, the presence of uncorrelated gradients from the different tasks and different rates of learning (Kirillov et al., 2019).

Original: A naive implementation of multi-task learning requires careful calibration of the relative losses of the different tasks.
Revised: A naive implementation of multi-task learning requires careful calibration of the relative losses of the different tasks.

Original: To address these problem several methods have been proposed: ‘Grad et al., 2017) dynamically tunes gradient magnitudes over time to obtain similar rates of learning for the different tasks.
Revised: To address these problem several methods have been proposed: ‘Grad et al., 2017) dynamically tunes gradient magnitudes over time to obtain similar rates of learning for the different tasks.

Original: Kendall et al.
Revised: Kendall et al.

Original: (2018) uses a joint likelihood formulation to derive task weights based on the intrinsic uncertainty in each task.
Revised: (2018) uses a joint likelihood formulation to derive task weights based on the intrinsic uncertainty in each task.

Original: Sener & Koltun (2018) applies an adaptive weighting of the different tasks, to force a pareto optimal solution on the multi-task problem.
Revised: Sener & Koltun (2018) applies an adaptive weighting of the different tasks, to force a pareto optimal solution on the multi-task problem.

Original: Along an orthogonal line of research, other works suggested to add task-speciﬁc modules to be activated or deactivated during training and evaluation, depending on the task at hand.
Revised: Along an orthogonal line of research, other works suggested to add task-speciﬁc modules to be activated or deactivated during training and evaluation, depending on the task at hand.

Original: Liu et al.
Revised: Liu et al.

Original: (2019) suggests task speciﬁc attention networks in parallel to a shared recognition network.
Revised: (2019) suggests task speciﬁc attention networks in parallel to a shared recognition network.

Original: Maninis et al.
Revised: Maninis et al.

Original: (2019) suggests adding several types of low-weight task-speciﬁc modules (e.g., residual convolutional layers, squeeze and excitation (SE) blocks and batch normalization layers) along the recognition network.
Revised: (2019) suggests adding several types of low-weight task-speciﬁc modules (e.g., residual convolutional layers, squeeze and excitation (SE) blocks and batch normalization layers) along the recognition network.

Original: Note that the SE block essentially creates a modulation vector, to be channelwise multiplied with a feature-map.
Revised: Note that the SE block essentially creates a modulation vector, to be channelwise multiplied with a feature-map.

Original: Modulation vectors have been further used in Strezoski et al.
Revised: Modulation vectors have been further used in Strezoski et al.

Original: (2019) for a recognition application, in Cheung et al.
Revised: (2019) for a recognition application, in Cheung et al.

Original: for continual learning applications and in Zhao et al.
Revised: for continual learning applications and in Zhao et al.

Original: (2018) for a retrieval application and proved to decrease the destructive interference between tasks and the effect of catastrophic forgetting.
Revised: (2018) for a retrieval application and proved to decrease the destructive interference between tasks and the effect of catastrophic forgetting.

Original: Our design, in contrast, does not use a multi-branch architecture, nor task-speciﬁc modules.
Revised: Our design, in contrast, does not use a multi-branch architecture, nor task-speciﬁc modules.

Original: Our network is fully-shared between the different tasks.
Revised: Our network is fully-shared between the different tasks.

Original: Compared to Zhao et al.
Revised: Compared to Zhao et al.

Original: (2018), we modulate the feature-maps in the recognition network both channel-wise and spatial-wise, also depending on the speciﬁc image at hand.
Revised: (2018), we modulate the feature-maps in the recognition network both channel-wise and spatial-wise, also depending on the speciﬁc image at hand.

Original: Top-Down Modulation Networks Neuroscience research provides evidence for a top-down context, feedback and lateral processing in the primate visual pathway (Gazzaley & Nobre, 2012; Gilbert & Sigman, 2007; Lamme et al., 1998; Hopﬁnger et al., 2000; Pi ¨ ech et al., 2013; Zanto et al., 2010) where top-down signals modulate the neural activity of neurons in lower-order sensory or motor areas based on the current goals.
Revised: Top-Down Modulation Networks Neuroscience research provides evidence for a top-down context, feedback and lateral processing in the primate visual pathway (Gazzaley & Nobre, 2012; Gilbert & Sigman, 2007; Lamme et al., 1998; Hopﬁnger et al., 2000; Pi ¨ ech et al., 2013; Zanto et al., 2010) where top-down signals modulate the neural activity of neurons in lower-order sensory or motor areas based on the current goals.

Original: This may involve enhancement of task-relevant representations or suppression for task-irrelevant representations.
Revised: This may involve enhancement of task-relevant representations or suppression for task-irrelevant representations.

Original: This mechanism underlies humans ability to focus attention on task-relevant stimuli and ignore irrelevant distractions (Hopﬁnger et al., 2000; Pi ¨ ech et al., 2013; Zanto et al., 2010).
Revised: This mechanism underlies humans ability to focus attention on task-relevant stimuli and ignore irrelevant distractions (Hopﬁnger et al., 2000; Pi ¨ ech et al., 2013; Zanto et al., 2010).

Original: In this work, consistent with this general scheme, we suggest a model that uses top-down modulation in the scope of multi-task learning.
Revised: In this work, consistent with this general scheme, we suggest a model that uses top-down modulation in the scope of multi-task learning.

Original: Top down modulation networks with feedback, implemented as conv-nets, have been suggested by the computer vision community for some high level tasks (e.g., re-classiﬁcation (Cao et al., 2015), keypoints detection (Carreira et al., 2016;
Revised: Top down modulation networks with feedback, implemented as conv-nets, have been suggested by the computer vision community for some high level tasks (e.g., re-classiﬁcation (Cao et al., 2015), keypoints detection (Carreira et al., 2016;

Original: Newell et al., 2016), crowd counting (Sam & Babu, 2018), curriculum learning (Zamir et al., 2017), etc.)
Revised: Newell et al., 2016), crowd counting (Sam & Babu, 2018), curriculum learning (Zamir et al., 2017), etc.)

Original: and here we apply them to multi-task learning applications.
Revised: and here we apply them to multi-task learning applications.

Original:  A PPROACH
Revised:  A PPROACH

Original: A schematic illustration of our network is shown in Figures 1c and 2a, and explained further below.
Revised: A schematic illustration of our network is shown in Figures 1c and 2a, and explained further below.

Original: A control network is used in this scheme to control each of the units in the main recognition network (BU2) given the task and the current image.
Revised: A control network is used in this scheme to control each of the units in the main recognition network (BU2) given the task and the current image.

Original: In practice, this scheme is implemented by using three separate sub-networks (two of them identical) with lateral inter-connections.
Revised: In practice, this scheme is implemented by using three separate sub-networks (two of them identical) with lateral inter-connections.

Original: We next describe the network architecture and implementation in detail.
Revised: We next describe the network architecture and implementation in detail.

Original: Overall structure and information ﬂow Our model contains three essentially identical subnetworks (BU1, TD, BU2), with added lateral connections between them.
Revised: Overall structure and information ﬂow Our model contains three essentially identical subnetworks (BU1, TD, BU2), with added lateral connections between them.

Original: The networks BU1, BU2 share identical weights.
Revised: The networks BU1, BU2 share identical weights.

Original: The network receives two inputs: an input image (to both BU1, BU2), and a task speciﬁcation provided at the top of the TD network by a one-hot vector selecting one of k possible tasks.
Revised: The network receives two inputs: an input image (to both BU1, BU2), and a task speciﬁcation provided at the top of the TD network by a one-hot vector selecting one of k possible tasks.

Original: The processing ﬂows sequentially through BU1, TD (in a top-down direction), BU2, and the ﬁnal output is produced at the top of BU2.
Revised: The processing ﬂows sequentially through BU1, TD (in a top-down direction), BU2, and the ﬁnal output is produced at the top of BU2.

Original: In some cases, discussed below, we used an additional output (object locations) at the bottom of TD.
Revised: In some cases, discussed below, we used an additional output (object locations) at the bottom of TD.

Original: During the sequential processing BU1 ﬁrst creates an initial image representation.
Revised: During the sequential processing BU1 ﬁrst creates an initial image representation.

Original: The TD creates a learned representation of the selected task that converts the one-hot vector to a new form (task embedding), and propagates it down the layers.
Revised: The TD creates a learned representation of the selected task that converts the one-hot vector to a new form (task embedding), and propagates it down the layers.

Original: On the way down the TD stream also extracts relevant image information from the BU1 representation via the
Revised: On the way down the TD stream also extracts relevant image information from the BU1 representation via the

Original: BU1-TD lateral connections.
Revised: BU1-TD lateral connections.

Original: Finally, the TD stream controls the BU2 network, to apply the selected task to the input image via the TD-BU2 lateral connections.
Revised: Finally, the TD stream controls the BU2 network, to apply the selected task to the input image via the TD-BU2 lateral connections.

Original: Bottom-Up streams The BU streams use a standard backbone (such as Resnet, VGG, LeNet, etc.), which is usually subdivided into several stages followed by one or more fully-connected layers,
Revised: Bottom-Up streams The BU streams use a standard backbone (such as Resnet, VGG, LeNet, etc.), which is usually subdivided into several stages followed by one or more fully-connected layers,

Original: [Figure]
Revised: [Figure]

Original: including the ﬁnal classiﬁer.
Revised: including the ﬁnal classiﬁer.

Original: The lateral connections between streams are placed at the end of each stage, connecting between tensors of the same sizes, allowing element-wise modiﬁcations.
Revised: The lateral connections between streams are placed at the end of each stage, connecting between tensors of the same sizes, allowing element-wise modiﬁcations.

Original: Top-down stream The TD stream we use (unless stated otherwise) is a replica of the BU stream in terms of number of layers, type of layers (convolutional / residual) and number of channels in each layer.
Revised: Top-down stream The TD stream we use (unless stated otherwise) is a replica of the BU stream in terms of number of layers, type of layers (convolutional / residual) and number of channels in each layer.

Original: The downsampling layers (used in the BU stream) are replaced with upsampling layers (nearest neighbour interpolation layers).
Revised: The downsampling layers (used in the BU stream) are replaced with upsampling layers (nearest neighbour interpolation layers).

Original: This design allows us to immediately extend any given BU backbone to our scheme, and it gives good results in our comparison, but the optimal TD structure is subject to future studies.
Revised: This design allows us to immediately extend any given BU backbone to our scheme, and it gives good results in our comparison, but the optimal TD structure is subject to future studies.

Original: The TD stream has two inputs: the selected task at its top, and inputs from BU1 via lateral connections.
Revised: The TD stream has two inputs: the selected task at its top, and inputs from BU1 via lateral connections.

Original: The selected task is usually speciﬁed by a one-hot vector, which is transformed via learnable weights, into a learned task-representation tensor (called the task embedding, ‘Emb’ in
Revised: The selected task is usually speciﬁed by a one-hot vector, which is transformed via learnable weights, into a learned task-representation tensor (called the task embedding, ‘Emb’ in

Original: Figure 2a) that serves as an input to the TD stream.
Revised: Figure 2a) that serves as an input to the TD stream.

Original: lateral connections For the lateral connections, we experimented extensively with different types, and based on the results we selected two types of connections, one for BU1-to-TD, which is additive, the second for TD-to-BU2, which is multiplicative, shown in Figures 2b and 2c.
Revised: lateral connections For the lateral connections, we experimented extensively with different types, and based on the results we selected two types of connections, one for BU1-to-TD, which is additive, the second for TD-to-BU2, which is multiplicative, shown in Figures 2b and 2c.

Original: More details and ablation studies of the lateral connections are given in the Supplementary.
Revised: More details and ablation studies of the lateral connections are given in the Supplementary.

Original: Auxiliary losses The use of three sub-networks (BU1, TD, BU2) suggest the natural use of auxiliary losses at the end of the BU1 or TD streams.
Revised: Auxiliary losses The use of three sub-networks (BU1, TD, BU2) suggest the natural use of auxiliary losses at the end of the BU1 or TD streams.

Original: In the scope of multi-task learning, the TD auxiliary loss can be used to train the extraction of useful spatial information such as the detection of task-relevant objects.
Revised: In the scope of multi-task learning, the TD auxiliary loss can be used to train the extraction of useful spatial information such as the detection of task-relevant objects.

Original: This issue is further discussed in Section 4.2 where we demonstrate the use of a localization loss in the last TD feature-map.
Revised: This issue is further discussed in Section 4.2 where we demonstrate the use of a localization loss in the last TD feature-map.

Original: We show that applying a localization loss allows us to obtain a task-dependent spatial map in inference time, helping interpretability by locating objects of interest.
Revised: We show that applying a localization loss allows us to obtain a task-dependent spatial map in inference time, helping interpretability by locating objects of interest.

Original: Training & evaluation During training, the learning optimizes all the weights along the BU and
Revised: Training & evaluation During training, the learning optimizes all the weights along the BU and

Original: TD streams, shared by all tasks, as well as the task speciﬁc embedding parameters.
Revised: TD streams, shared by all tasks, as well as the task speciﬁc embedding parameters.

Original: Learning uses a standard backpropagation, as the full model forms an end-to-end trainable model.
Revised: Learning uses a standard backpropagation, as the full model forms an end-to-end trainable model.

Original: In training time, the network is supplied with an input image and a selected task, drawn at random from the different tasks.
Revised: In training time, the network is supplied with an input image and a selected task, drawn at random from the different tasks.

Original: During testing, the different tasks are applied sequentially to each test image.
Revised: During testing, the different tasks are applied sequentially to each test image.

Original: We used in our implementation shared weights between BU1 and BU2 streams.
Revised: We used in our implementation shared weights between BU1 and BU2 streams.

Original: The main motivation for this design was to allow in future applications a multi-cycle use of the model, by using the BU and
Revised: The main motivation for this design was to allow in future applications a multi-cycle use of the model, by using the BU and

Original: TD streams iteratively.
Revised: TD streams iteratively.

Original: With this broader goal in mind, our scheme can also be seen as an unfolded version of a BU-TD recurrent network for one and a half cycles, which is the minimal cycles that allow an image-aware modiﬁcation process.
Revised: With this broader goal in mind, our scheme can also be seen as an unfolded version of a BU-TD recurrent network for one and a half cycles, which is the minimal cycles that allow an image-aware modiﬁcation process.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We validated our approach on four datasets (Multi-MNIST, CLEVR, CELEB-A and CUB-200) with tasks ranging from low-level (e.g., colors) to higher-level (e.g., CLEVR conﬁgurations, facial features) recognition, and from simple (e.g., classiﬁcation by location) to more complex tasks (e.g., classiﬁcation by combined attributes and spatial relations).
Revised: We validated our approach on four datasets (Multi-MNIST, CLEVR, CELEB-A and CUB-200) with tasks ranging from low-level (e.g., colors) to higher-level (e.g., CLEVR conﬁgurations, facial features) recognition, and from simple (e.g., classiﬁcation by location) to more complex tasks (e.g., classiﬁcation by combined attributes and spatial relations).

Original: We compared our approach to alternatives in terms of prediction accuracy, scaling to a larger number of tasks and task selectivity.
Revised: We compared our approach to alternatives in terms of prediction accuracy, scaling to a larger number of tasks and task selectivity.

Original: 4. DATASETS & TASKS
Revised: 4. DATASETS & TASKS

Original: Multi-MNIST (Sabour et al., 2017) is a version of the MNIST dataset in which multiple MNIST images are placed on a grid with some spatial overlaps, previously used in Sener & Koltun (2018).
Revised: Multi-MNIST (Sabour et al., 2017) is a version of the MNIST dataset in which multiple MNIST images are placed on a grid with some spatial overlaps, previously used in Sener & Koltun (2018).

Original: We used 2x1, 2x2, 3x3 grids; several training examples are shown in Figure 1e.
Revised: We used 2x1, 2x2, 3x3 grids; several training examples are shown in Figure 1e.

Original: We used Multi-MNIST to test performance on two sets of tasks: (a) recognizing a digit at a selected location (‘by loc’, e.g., to recognize the digit in the upper-right location) and (b) recognizing a digit which is right to another digit (‘by ref’, e.g., to recognize the digit to the right of the digit ‘7’).
Revised: We used Multi-MNIST to test performance on two sets of tasks: (a) recognizing a digit at a selected location (‘by loc’, e.g., to recognize the digit in the upper-right location) and (b) recognizing a digit which is right to another digit (‘by ref’, e.g., to recognize the digit to the right of the digit ‘7’).

Original: CLEVR is a synthetic dataset, consisting of 70K training images and 15K validation images.
Revised: CLEVR is a synthetic dataset, consisting of 70K training images and 15K validation images.

Original: The dataset includes images of simple 3D objects, with multiple attributes (shape, size, color and material) together with corresponding (question-answer) pairs.
Revised: The dataset includes images of simple 3D objects, with multiple attributes (shape, size, color and material) together with corresponding (question-answer) pairs.

Original: We used the CLEVR dataset to test performance on sets of ‘by ref’ tasks, scaling the number of tasks up to 1645, with a ﬁxed model size.
Revised: We used the CLEVR dataset to test performance on sets of ‘by ref’ tasks, scaling the number of tasks up to 1645, with a ﬁxed model size.

Original: We created multiple tasks by randomly choosing 40, 80, 160 and 1645 queries about an attribute of an object to the (left, right, up, down) of a referred object.
Revised: We created multiple tasks by randomly choosing 40, 80, 160 and 1645 queries about an attribute of an object to the (left, right, up, down) of a referred object.

Original: An example task is: “What is the color of the object to the left of the metal cylinder?”
Revised: An example task is: “What is the color of the object to the left of the metal cylinder?”

Original: (metallic cylinder is the referred object).
Revised: (metallic cylinder is the referred object).

Original: CELEB-A is a set of real-world celeb face images, intensively used in the scope of MTL (e.g.,
Revised: CELEB-A is a set of real-world celeb face images, intensively used in the scope of MTL (e.g.,

Original: Sener & Koltun (2018), Strezoski et al.
Revised: Sener & Koltun (2018), Strezoski et al.

Original: on attribute classiﬁcation tasks.
Revised: on attribute classiﬁcation tasks.

Original: The dataset consists of 200K images with binary annotations on 40 face attributes related to expression, facial parts, etc.
Revised: The dataset consists of 200K images with binary annotations on 40 face attributes related to expression, facial parts, etc.

Original: CUB-200 is a ﬁne-grained recognition dataset that provides 11,788 bird images of 200 bird species, previously used in (Strezoski et al., 2019).
Revised: CUB-200 is a ﬁne-grained recognition dataset that provides 11,788 bird images of 200 bird species, previously used in (Strezoski et al., 2019).

Original: We used CUB-200 to test performance on real-world images with low-level features, and to demonstrate our use of interpretability.
Revised: We used CUB-200 to test performance on real-world images with low-level features, and to demonstrate our use of interpretability.

Original: An example task is to recognize the color of the bird’s crown.
Revised: An example task is to recognize the color of the bird’s crown.

Original: The datasets and corresponding tasks are illustrated in Figure 1e and further discussed in the Supplementary material.
Revised: The datasets and corresponding tasks are illustrated in Figure 1e and further discussed in the Supplementary material.

Original: 4. IMPLEMENTATION DETAILS
Revised: 4. IMPLEMENTATION DETAILS

Original: [Table]
Revised: [Table]

Original: We performed ﬁve randomly initialized training runs for each of our experiments, and present average accuracy and standard deviation.
Revised: We performed ﬁve randomly initialized training runs for each of our experiments, and present average accuracy and standard deviation.

Original: We used LeNet, VGG-11, VGG-7 and resnet-18, as our BU backbone architectures for the Multi-MNIST, CLEVR, CELEB-A and CUB200 experiments respectively.
Revised: We used LeNet, VGG-11, VGG-7 and resnet-18, as our BU backbone architectures for the Multi-MNIST, CLEVR, CELEB-A and CUB200 experiments respectively.

Original: We used the Adam optimizer, and performed learning rate search over { 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 } on a small validation set; the main hyperparameters (number of epochs, learning rate and batch size) are shown in Table 1.
Revised: We used the Adam optimizer, and performed learning rate search over { 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 } on a small validation set; the main hyperparameters (number of epochs, learning rate and batch size) are shown in Table 1.

Original: We trained the full model end-to-end, using cross entropy loss at the end of BU2.
Revised: We trained the full model end-to-end, using cross entropy loss at the end of BU2.

Original: In some of the experiments of CLEVR and CUB-200 datasets we added an auxiliary loss at the end of the TD stream.
Revised: In some of the experiments of CLEVR and CUB-200 datasets we added an auxiliary loss at the end of the TD stream.

Original: The target in this case is a 224x224 mask, where a single pixel, blurred by a Gaussian kernel (s.d. 3 pixels) was labeled as the target location.
Revised: The target in this case is a 224x224 mask, where a single pixel, blurred by a Gaussian kernel (s.d. 3 pixels) was labeled as the target location.

Original: Training one task at a time, we minimized the cross-entropy loss over the 224x224 image at the end of the TD softmax output (which encourages a small detected area), for each visible ground-truth annotated object or part.
Revised: Training one task at a time, we minimized the cross-entropy loss over the 224x224 image at the end of the TD softmax output (which encourages a small detected area), for each visible ground-truth annotated object or part.

Original: This auxiliary loss, allows us, at inference, to create task-dependent spatial maps of detected objects; examples of interest are shown in ﬁgure 4 and in the Supplementary material.
Revised: This auxiliary loss, allows us, at inference, to create task-dependent spatial maps of detected objects; examples of interest are shown in ﬁgure 4 and in the Supplementary material.

Original: For a fair comparison, we also trained another version of the channel modulation architecture with an additional regression loss, calculated by a FC layer at the top of the network, using the same ground truth annotations.
Revised: For a fair comparison, we also trained another version of the channel modulation architecture with an additional regression loss, calculated by a FC layer at the top of the network, using the same ground truth annotations.

Original: Database details, full architecture description, more hyperparameters and an analysis of the number of parameters in the architectures can be found in the Supplementary material.
Revised: Database details, full architecture description, more hyperparameters and an analysis of the number of parameters in the architectures can be found in the Supplementary material.

Original: [Table]
Revised: [Table]

Original: 4. COMPARISONS :
Revised: 4. COMPARISONS :

Original: We compared our method with existing alternatives from three main approaches (listed in Table 2): (i) A ‘Single task’ approach, where each task is performed by its own network, (ii) a ‘Multi-branched’ approach, where the sum of the individual losses is minimized (in ‘Uniform scaling’ the losses are equally weighted, whereas in ‘mult-obj-opt’ (Sener & Koltun, 2018), ‘kendall’ (Kendall et al., 2018) and ‘grad-norm’ (Chen et al., 2017)
Revised: We compared our method with existing alternatives from three main approaches (listed in Table 2): (i) A ‘Single task’ approach, where each task is performed by its own network, (ii) a ‘Multi-branched’ approach, where the sum of the individual losses is minimized (in ‘Uniform scaling’ the losses are equally weighted, whereas in ‘mult-obj-opt’ (Sener & Koltun, 2018), ‘kendall’ (Kendall et al., 2018) and ‘grad-norm’ (Chen et al., 2017)

Original: the weights are dynamically tuned) and (iii) a ‘Modulation’ approach, where channel-wise vectors modulate the recognition network in several of the net’s stages (‘ch-mod’ (Zhao et al., 2018) uses learnable weights and ‘task-routing’
Revised: the weights are dynamically tuned) and (iii) a ‘Modulation’ approach, where channel-wise vectors modulate the recognition network in several of the net’s stages (‘ch-mod’ (Zhao et al., 2018) uses learnable weights and ‘task-routing’

Original: (Strezoski et al., 2019) uses constant binary weights).
Revised: (Strezoski et al., 2019) uses constant binary weights).

Original: For a fair comparison with our approach we placed the task embeddings (‘modulation’ approach) and the lateral connections from the TD (‘ControlNet, ours’ approach) in the same recognition network locations followed by a single branch for ﬁnal recognition (the original implementation of ‘task-routing’ uses multiple branches).
Revised: For a fair comparison with our approach we placed the task embeddings (‘modulation’ approach) and the lateral connections from the TD (‘ControlNet, ours’ approach) in the same recognition network locations followed by a single branch for ﬁnal recognition (the original implementation of ‘task-routing’ uses multiple branches).

Original: 4. RESULTS
Revised: 4. RESULTS

Original: Table 2 summarizes our results on the Multi-MNIST experiment with 2, 4 and 9 digits.
Revised: Table 2 summarizes our results on the Multi-MNIST experiment with 2, 4 and 9 digits.

Original: We show the average accuracy of all tasks based on 5 experiments for each row.
Revised: We show the average accuracy of all tasks based on 5 experiments for each row.

Original: The ‘ # P ’ column shows the number of parameters as a multiplier of the number of parameter in a standard LeNet architecture.
Revised: The ‘ # P ’ column shows the number of parameters as a multiplier of the number of parameter in a standard LeNet architecture.

Original: Detailed results with standard deviations and number of parameters are speciﬁed in the Supplementary material.
Revised: Detailed results with standard deviations and number of parameters are speciﬁed in the Supplementary material.

Original: Our method achieves signiﬁcantly better results than the other approaches, even compared with the single-task baseline.
Revised: Our method achieves signiﬁcantly better results than the other approaches, even compared with the single-task baseline.

Original: Scaling the number of tasks increases the accuracy gap almost without additional parameters.
Revised: Scaling the number of tasks increases the accuracy gap almost without additional parameters.

Original: The ‘by ref’ test (10 tasks) proved to be more difﬁcult (lower accuracies), and shows a signiﬁcant gap between our approach and other methods.
Revised: The ‘by ref’ test (10 tasks) proved to be more difﬁcult (lower accuracies), and shows a signiﬁcant gap between our approach and other methods.

Original: Extending the channel-modulation scheme to a wider backbone (‘ch-mod(extended)’ in the table, with 15 and 25 channels featuremaps) with roughly the same number of parameters as in our scheme, maintains a large accuracy gap.
Revised: Extending the channel-modulation scheme to a wider backbone (‘ch-mod(extended)’ in the table, with 15 and 25 channels featuremaps) with roughly the same number of parameters as in our scheme, maintains a large accuracy gap.

Original: Our quantitative results for the CLEVR, CELEB-A and CUB-200 experiments are summarized in Table 3.
Revised: Our quantitative results for the CLEVR, CELEB-A and CUB-200 experiments are summarized in Table 3.

Original: The ‘ # P ’ column shows the number of parameters as a multiplier of the number of parameter used in the corresponding BU architecture.
Revised: The ‘ # P ’ column shows the number of parameters as a multiplier of the number of parameter used in the corresponding BU architecture.

Original: Experiments that used localization ground-truth data are indicated with √ on the ‘loc’ column.
Revised: Experiments that used localization ground-truth data are indicated with √ on the ‘loc’ column.

Original: The results show better accuracy of our scheme compared to all baselines.
Revised: The results show better accuracy of our scheme compared to all baselines.

Original: Table 3a shows that our results on CLEVR, 40 tasks setting, surpass other methods by a signiﬁcant margin.
Revised: Table 3a shows that our results on CLEVR, 40 tasks setting, surpass other methods by a signiﬁcant margin.

Original: Our advantage in the CELEB-A and CUB-200 experiments is smaller than in the former tests, possibly due to database biases (e.g., color of different bird’s parts are highly correlated) and the diverse appearance of relevant attributes in these databases
Revised: Our advantage in the CELEB-A and CUB-200 experiments is smaller than in the former tests, possibly due to database biases (e.g., color of different bird’s parts are highly correlated) and the diverse appearance of relevant attributes in these databases

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. E XPERIMENTS DISCUSSION
Revised: 4. E XPERIMENTS DISCUSSION

Original: We discuss below additional aspects of the experiments and general conclusions.
Revised: We discuss below additional aspects of the experiments and general conclusions.

Original: Accuracy vs. model size tradeoff Figure 3 demonstrates the average accuracy of the 9-class experiment as a function of the number of parameters in four types of architectures; top-left is better.
Revised: Accuracy vs. model size tradeoff Figure 3 demonstrates the average accuracy of the 9-class experiment as a function of the number of parameters in four types of architectures; top-left is better.

Original: Large markers in the ﬁgure correspond to the ﬁxed architectures used in the experiments above.
Revised: Large markers in the ﬁgure correspond to the ﬁxed architectures used in the experiments above.

Original: Within each family, parameters can be changed by changing (uniformly) the number of channels along the network.
Revised: Within each family, parameters can be changed by changing (uniformly) the number of channels along the network.

Original: Small markers correspond to modiﬁed network sizes: wider LeNet architectures (termed above ‘ch-mod(extended)’), or to reduced TD implementations (using TD streams with 1, 4 or 6 channels along its feature-maps).
Revised: Small markers correspond to modiﬁed network sizes: wider LeNet architectures (termed above ‘ch-mod(extended)’), or to reduced TD implementations (using TD streams with 1, 4 or 6 channels along its feature-maps).

Original: Exact design choices are summarized in the Supplementary material.
Revised: Exact design choices are summarized in the Supplementary material.

Original: Our control network architectures correspond to the highest (red) curve in the plot, indicating higher performance for a similar number of parameters.
Revised: Our control network architectures correspond to the highest (red) curve in the plot, indicating higher performance for a similar number of parameters.

Original: A similar comparison for the CLEVR dataset is reported in the Supplementary material.
Revised: A similar comparison for the CLEVR dataset is reported in the Supplementary material.

Original: Scaling the number of tasks Table 2 above shows the accuracies of the Multi-MNIST experiment with the 2, 4 and 9 tasks datasets.
Revised: Scaling the number of tasks Table 2 above shows the accuracies of the Multi-MNIST experiment with the 2, 4 and 9 tasks datasets.

Original: Note that as the number of digits increases, the task also increases in difﬁculty due to the digits overlap.
Revised: Note that as the number of digits increases, the task also increases in difﬁculty due to the digits overlap.

Original: Increasing the number of tasks increases the accuracy gap compared with alternative models with a similar or even larger number of parameters.
Revised: Increasing the number of tasks increases the accuracy gap compared with alternative models with a similar or even larger number of parameters.

Original: Table 4a below shows the accuracies of gradually increasing the number of tasks up to 1645 on the
Revised: Table 6a below shows the accuracies of gradually increasing the number of tasks up to 1645 on the

Original: CLEVR dataset.
Revised: CLEVR dataset.

Original: Compared with alternatives, our results are scalable with the number of tasks, with a smaller decrease in performance for 1645 tasks.
Revised: Compared with alternatives, our results are scalable with the number of tasks, with a smaller decrease in performance for 1645 tasks.

Original: The uniform scaling approach cannot scale abovetasks due to memory restrictions, and did not take part in this experiment.
Revised: The uniform scaling approach cannot scale abovetasks due to memory restrictions, and did not take part in this experiment.

Original: Overall, the results show that our scheme deals better in performing multiple tasks in limited capacity, the performance gap increases with the number of tasks and complexity and the scheme beneﬁts from the contribution of spatial information and image content. 
Revised: Overall, the results show that our scheme deals better in performing multiple tasks in limited capacity, the performance gap increases with the number of tasks and complexity and the scheme beneﬁts from the contribution of spatial information and image content. [Table] Adding Tasks

Revised: A general question of interest in mutli-task learning is whether the tasks need to be pre-deﬁned, compared with the possibility of adding at least some tasks at a later stage to an already trained model.

Original: Ablations and the use of image content We compared our scheme, which uses image information (via BU1) and performs full tensor element-wise modiﬁcation of the feature maps (of BU2) to the channel modulation scheme (which performs channel modulation with no image content information) with roughly the same number of parameters, shown in  Table 4b, top row (ch-mod (extended)).
Revised: We used in this task an extension of the M-mnist ‘byref’ experiment. Speciﬁcally, our architecture has been trained and evaluated on the deﬁned ‘by ref’ tasks, excluding the task involving the digit ‘9’ (9 tasks in all). We then extended the embedding layer and trained it, while keeping the rest of the model ﬁxed, on the new task examples. Table 4 shows the results.

Revised: The obtained accuracy for the added digit ‘9’ task are 64 .

Revised: 68% , other tasks mean accuracy remains unchanged ( 74 . 89% ).

Revised: The new task accuracy is lower than the mean, but shows signiﬁcant learning, compared with the pre-trained accuracy of 16.3 % .

Revised: The accuracy of all other tasks is unaffected (avoiding ‘catastrophic forgetting’) without requiring any further training of the previous tasks.

Revised: Tasks heterogeneity

Revised: Our model performs multiple tasks in the same network by applying activation-modiﬁcations according to the task and to the image at hand.

Revised: A question of interest is whether the set of tasks is limited to a homogenous set, such as similar classiﬁcation tasks (as in the M-MNIST experiment), or extends to more heterogeneous tasks.

Revised: In particular, performing both recognition and pixel-labeling tasks in the same network using task selection schemes (e.g., channel modulation, task-routing, ours) has not been studied in the past.

Revised: [Table]

Revised: We extended our proposed architecture to perform both recognition tasks (producing class labels) and segmentation tasks (producing a spatial map), guided by the TD instruction provided to the network.

Revised: The instruction is composed of two parts: selecting a digit, either by location or by reference (the ‘by loc’ and the ‘by ref’ vector in the M-MNIST experiments above), and then applying either recognition or segmentation, using a two slots one-hot-vector (results in Table 5 row 4).

Revised: We compared this task selection with three alternatives.

Revised: The ﬁrst includes no instruction, implemented with an individual branch for each task (top row, multi-branched architecture).

Revised: The second performs both segmentation and recognition together, implemented with two simultaneously executed branches (second row in the table).

Revised: The ﬁnal compares with channel modulation (row 3).

Revised: To deal with both recognition and segmentation tasks we used two separate output branches (one producing one of 10 class labels, the other producing a 28*28 map) rather than a single one.

Revised: The branches were used according to the task instruction: using one at a time when classiﬁcation/segmentation instruction is given (rows 3, 4), or using both branches simultaneously when no such instruction is provided (row 2).

Revised: Results show that our model with the selected branch (row 4) performs better than all alternatives.

Revised: A direct comparison with the alternative of using the two branches together, as in standard (un-instructed) branching models (second row), reveals that the selected branch increases in performance, and at the same time, the results of the un-selected branch signiﬁcantly decreases in performance, by 10-30 % .

Revised: Ablations and the use of image content We compared our scheme, which uses image information (via BU1) and performs full tensor element-wise modiﬁcation of the feature maps (of BU2) to the channel modulation scheme (which performs channel modulation with no image content information) with roughly the same number of parameters, shown in Table 6b, top row (ch-mod (extended)).

Original: We also tested two ablations of our network: one without the BU1 stream, removing the image-content contribution (TD, second row) and second without the TD stream, where the task is supplied to both
Revised: We also tested two ablations of our network: one without the BU1 stream, removing the image-content contribution (TD, second row) and second without the TD stream, where the task is supplied to both

Original: BU streams, concatenated to the image data (BU, third row, details in Supplementary).
Revised: BU streams, concatenated to the image data (BU, third row, details in Supplementary).

Original: We conducted the experiments on two sets of tasks applied to the 9-location MNIST: classiﬁcation by location (‘by loc’) and classiﬁcation by reference (‘by ref’), and on two sets of tasks used in the CLEVR dataset (40 and 1645 tasks), which are inherently a ‘by ref’ tasks, using reference objects.
Revised: We conducted the experiments on two sets of tasks applied to the 9-location MNIST: classiﬁcation by location (‘by loc’) and classiﬁcation by reference (‘by ref’), and on two sets of tasks used in the CLEVR dataset (40 and 1645 tasks), which are inherently a ‘by ref’ tasks, using reference objects.

Original: Table 4b below shows that ControlNet achieves signiﬁcantly better results than its ablations, with a gap that increases as the complexity or the number of tasks increase.
Revised: Table 6b below shows that ControlNet achieves signiﬁcantly better results than its ablations, with a gap that increases as the complexity or the number of tasks increase.

Original: The comparison between our model (4th row) and our ablated model (2nd row), which does not use the image content in the modiﬁcation process, shows the signiﬁcant advantage of using image content information.
Revised: The comparison between our model (4th row) and our ablated model (2nd row), which does not use the image content in the modiﬁcation process, shows the signiﬁcant advantage of using image content information.

Original: The addition of BU1 (which shares weights with BU2) improves the accuracy substantially with almost no additional parameters.
Revised: The addition of BU1 (which shares weights with BU2) improves the accuracy substantially with almost no additional parameters.

Original: Task Selectivity Task-selectivity is likely to be a crucial property of architectures that execute one task at a time (such as ch-mod, ours) in order to fully utilize the network resources for the selected task.
Revised: Task Selectivity Task-selectivity is likely to be a crucial property of architectures that execute one task at a time (such as ch-mod, ours) in order to fully utilize the network resources for the selected task.

Original: We deﬁned a task-selectivity measure by comparing the prediction accuracy of the model for the selected task vs. non-selected tasks.To make this comparison, we trained readout heads to [Table] [Figure]
Revised: We deﬁned a task-selectivity measure by comparing the prediction accuracy of the model for the selected task vs. non-selected tasks.

Original: [Table] [Table] [Figure] predict from the ﬁnal representation produced by the BU2 stream not just the selected task, but all possible tasks.
Revised: To make this comparison, we trained readout heads to predict from the ﬁnal representation produced by the BU2 stream not just the selected task, but all possible tasks.

Original: For example, in the 4-digits MNIST, we trained four readout branches on the top of the ﬁnal representation (trained to produce the selected location) to predict the digit identities at all four locations.
Revised: For example, in the 4-digits MNIST, we trained four readout branches on the top of the ﬁnal representation (trained to produce the selected location) to predict the digit identities at all four locations.

Original: We deﬁne task-selectivity by the ratio between the accuracy for the selected task (above chance level) and the average accuracy of the non-selected tasks (above chance level).
Revised: We deﬁne task-selectivity by the ratio between the accuracy for the selected task (above chance level) and the average accuracy of the non-selected tasks (above chance level).

Original: Detailed results are shown in Figure 4a for the multi-MNIST, 4 tasks setting (top) and 9 tasks-setting (bottom).The results show over 90% accuracy for the selected task branch (along the diagonal), and close to chance-level accuracies for all other branches.
Revised: Detailed results are shown in Figure 4a for the multi-MNIST, 4 tasks setting (top) and 9 tasks-setting (bottom).The results show over 90% accuracy for the selected task branch (along the diagonal), and close to chance-level accuracies for all other branches.

Original: Figure 4b summarizes the results of the same experiment for the channel modulation architecture, which shows less selectivity.
Revised: Figure 4b summarizes the results of the same experiment for the channel modulation architecture, which shows less selectivity.

Original: The corresponding selectivity indices for the 4-class case are 26.5 and 8.25 and for the 9-class case 37.23 and 7.97 for our model and channel-modulation respectively. 
Revised: The corresponding selectivity indices for the 4-class case are 26.5 and 8.25 and for the 9-class case 37.23 and 7.97 for our model and channel-modulation respectively. The higher selectivity index of our method is likely [Table]

Original: The higher selectivity index of our method is likely to be related to the increased gap in performance with respect to alternative models in Tables 2 and 3.
Revised: [Table] [Table] [Figure] to be related to the increased gap in performance with respect to alternative models in Tables 2 and 3.

Revised: Results in the section on tasks heterogeneity further show that even for heterogeneous tasks, task selection increases the performance of the selected task at the expense of the non-selected tasks.

Original: Task-dependent spatial maps Using a task dependent localization loss at the end of the TD stream in train time allows us to obtain task-dependent spatial maps in inference time, helping interpretability of the result by locating intermediate objects of interest.
Revised: Task-dependent spatial maps Using a task dependent localization loss at the end of the TD stream in train time allows us to obtain task-dependent spatial maps in inference time, helping interpretability of the result by locating intermediate objects of interest.

Original: Figures 4c and 4d demonstrates the location maps produced by our architecture at inference time.
Revised: Figures 4c and 4d demonstrates the location maps produced by our architecture at inference time.

Original: In both examples, the predicted mask/object is well localized (on the crown of the bird or on the object below the small metal object) and the attribute (color or shape) is correctly predicted.
Revised: In both examples, the predicted mask/object is well localized (on the crown of the bird or on the object below the small metal object) and the attribute (color or shape) is correctly predicted.

Original: In case of wrong result, it is possible to examine whether the error in the attribute was associated with mis-detecting the relevant part.
Revised: In case of wrong result, it is possible to examine whether the error in the attribute was associated with mis-detecting the relevant part.

Original: Additional examples of interest and failure cases are shown in the Supplementary material.
Revised: Additional examples of interest and failure cases are shown in the Supplementary material.

Original:  S UMMARY
Revised:  S UMMARY

Original: We described an architecture for multi-task learning, which is qualitatively different from previous models, primarily by the use of the BU1 stream and a TD convolutional stream, which controls the ﬁnal BU2 stream as a function of the selected task, location and image content.
Revised: We described an architecture for multi-task learning, which is qualitatively different from previous models, primarily by the use of the BU1 stream and a TD convolutional stream, which controls the ﬁnal BU2 stream as a function of the selected task, location and image content.

Original: We tested our network on four different datasets, showing improvements in accuracy compared with other schemes, along with scaling the number of tasks with minimal effect on performance, and helping interpretability by pointing to relevant image locations.
Revised: We tested our network on four different datasets, showing improvements in accuracy compared with other schemes, along with scaling the number of tasks with minimal effect on performance, and helping interpretability by pointing to relevant image locations.

Original: Comparisons show higher task-selectivity of our scheme, which may explain at least in part its improved performance.
Revised: Comparisons show higher task-selectivity of our scheme, which may explain at least in part its improved performance.

Original: More generally, multiple-task learning algorithms are likely to become increasingly relevant, since general vision systems need to deal with a broad range of tasks, and executing them efﬁciently in a single network is still an open problem.
Revised: More generally, multiple-task learning algorithms are likely to become increasingly relevant, since general vision systems need to deal with a broad range of tasks, and executing them efﬁciently in a single network is still an open problem.

Original: Our task-dependent TD control network is a promising direction in this ﬁeld in terms of accuracy and scalability.
Revised: Our task-dependent TD control network is a promising direction in this ﬁeld in terms of accuracy and scalability.

Original: In future work we plan to adapt our architecture to a wider range of applications (e.g., segmentation, images generation) and to a wider range of architectures with higher capacity.
Revised: In future work we plan to adapt our architecture to a wider range of applications (e.g., scene understanding, images generation), to a wider range of architectures with higher capacity and to examine the possible combination of branching strategy with our TD task selection approach, extending our heterogeneous tasks example.

Original: Our architecture is also potentially useful for problems such as online learning, domain adaptation and catastrophic forgetting and we plan to further explore these still open problems in the future.
Revised: Our architecture is also potentially useful for problems such as online learning, domain adaptation and catastrophic forgetting, demonstrated in part by our example of adding a new task to an already trained model, and we plan to further explore these central problems in the future.