G ENERATIVE M ODELING FOR M ULTI - TASK V ISUAL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative modeling has recently shown great promise in computer vision, but it has mostly focused on synthesizing visually realistic images.
In this paper, motivated by multi-task learning of shareable feature representations, we consider a novel problem of learning a shared generative model that is useful across various visual perception tasks.
Correspondingly, we propose a general multi-task oriented generative modeling (MGM) framework, by coupling a discriminative multi-task network with a generative network.
While it is challenging to synthesize both RGB images and pixel-level annotations in multi-task scenarios, our framework enables us to use synthesized images paired with only weak annotations ( i.e ., image-level scene labels) to facilitate multiple visual tasks.
Experimental evaluation on challenging multi-task benchmarks, including NYUv2 and Taskonomy, demonstrates that our MGM framework improves the performance of all the tasks by large margins, especially in the low-data regimes, and our model consistently outperforms state-of-the-art multi-task approaches.
I NTRODUCTION
Seeing with the mind’s eye — creating internal images of objects and scenes not actually present to the senses, is perhaps one of the hallmarks in human cognition (Pelaprat & Cole, 2011).
For humans, this visual imagination integrates learning experience and facilitates learning by solving different problems (Egan, 2014; Pearson, 2019; Pelaprat & Cole, 2011; Egan, 1989).
Inspired by such ability, there has been increasing interest in building generative models that can synthesize images (Goodfellow et al., 2014).
Yet, most of the effort has focused on generating visually realistic images (Brock et al., 2018; Zhang et al., 2019), which are still far from useful for machine perception tasks (Shmelkov et al., 2018; Borji, 2019; Wu et al., 2016).
Even though recent work has started improving the “usefulness” of synthesized images, this line of investigation is often limited to a single speciﬁc task (Nguyen-Phuoc et al., 2018; Sitzmann et al., 2019; Zhu et al., 2018; Souly et al., 2017).
Could we guide generative models to beneﬁt multiple visual tasks?
While similar spirits of shared feature representations have been widely studied as multi-task learning or meta-learning (Finn et al., 2017; Zamir et al., 2018), here we take a different perspective — learning a shared generative model across various tasks (as illustrated in Figure 1).
Leveraging multiple tasks allows us to capture the underlying image generation mechanism for more comprehensive object and scene understanding than being done within individual tasks.
Taking simultaneous semantic segmentation, depth estimation, and surface normal prediction as an example (Figure 1), successful generative modeling requires understanding not only the semantics but also the 3D geometric structure and physical property of the input image.
Meanwhile, a learned generative model facilitates the ﬂow of knowledge across tasks, so that they beneﬁt one another.
For instance, the synthesized images provide meaningful variations in existing images and could work as additional training data to build better task-speciﬁc models.
These variations are especially critical when the data is limited.
This paper thus explores multi-task oriented generative modeling (MGM), by coupling a discriminative multi-task network with a generative network.
To make them cooperate with each other, a straightforward solution would be to synthesize both RGB images and corresponding pixel-level annotations ( e.g ., pixel-wise class labels for semantic segmentation and depth map for normal estimation).
In the single task scenario, existing work trains a separate generative model to synthesize paired pixel-level labeled data (Sandfort et al., 2019; Choi et al., 2019) and produce an augmented set.
However, these models are still highly task-dependant, and extending them to multi-task scenarios becomes difﬁcult.
A natural question then is: Do we actually need to synthesize paired image and multi-annotation data to be useful for multi-task visual learning?
[Figure]
Our MGM addresses this question by proposing a general framework that uses synthesized images paired with only weak annotations ( i.e ., image-level scene labels) to facilitate multiple visual tasks.
Our key insight is to introduce auxiliary discriminative tasks that (i) only require image-level annotation or no annotation, and (ii) correlate with the original multiple tasks of interest.
To this end, as additional components of the discriminative multi-task network, we introduce a reﬁnement network and a self-supervision network that satisﬁes these properties.
Through joint training, the discriminative network explicitly guides the image synthesis process.
The generative network also contributes to further reﬁning the shared feature representation.
Meanwhile, the synthesized images of the generative network are used as additional training data for the discriminative network.
In more detail, the reﬁnement network performs scene classiﬁcation on the basis of the multitask network predictions, which requires only scene labels.
The self-supervision network can be operationalized on both real and synthesized images without reliance on annotations.
With these two modules, our MGM is able to learn from both (pixel-wise) fully-annotated real images and synthesized (image-level) weakly labeled images.
We instantiate MGM with the state-of-the-art encoder-decoder based multi-task network (Zamir et al., 2018), self-attention GAN (Zhang et al., 2019), and contrastive learning-based self-supervision network (Chen et al., 2020).
Note that our framework is agnostic to the choice of these model components .
We evaluate our approach on standard multi-task benchmarks, including the NYUv2 (Nathan Silberman & Fergus, 2012) and Taskonomy (Zamir et al., 2018) datasets.
Consistent with the previous works (Sun et al., 2019; Standley et al., 2020), we focus on three tasks of great practical importance: semantic segmentation, depth estimation, and normal prediction.
The evaluation shows that: (1) our
MGM consistently outperforms state-of-the-art multi-task approaches by large margins, especially in the low-data regime.
(2) With the increasing of generated samples, the performance of MGM consistently improves and it also almost reaches the performance upper-bound that trains with weakly annotated real images.
(3) Our model can be extended to more visual tasks.
P ILOT S TUDY
This pilot study provides an initial experimentation, which shows the importance of our proposed problem (multi-task oriented generative modeling) and motivates the development of our method.
Speciﬁcally, we show that directly using images synthesized by an off-the-shelf generative model that is trained with the realistic objective is not helpful for downstream pixel-level perception tasks .
Experimental Design: For ease of analysis, here we focus on a single task – semantic segmentation, and use the Tiny-Taskonomy dataset (Zamir et al., 2018).
The dataset split and evaluation metric are the same as our main experiments (See Section 4 for details).
We train a self-attention GAN (Zhang et al., 2019) on Tiny-Taskonomy, and use it to generate the same number of synthesized images as the real images to augment the training set.
How to generate pixel-level annotations?
One remaining question is how to generate pixel-level annotations for these synthesized images produced by the self-attention GAN.
While prior work has explored synthesizing both images and their pixel-level annotations for some speciﬁc tasks (Sandfort et al., 2019; Choi et al., 2019), these annotations are still not reliable.
For ease of analysis, in this study, we factor out the effect of annotations and assume that we have an oracle annotator .
We use the annotator from Taskonomy (Zamir et al., 2018), which is a state-of-the-art large fully supervised semantic segmentation network.
In fact, the ground-truth of semantic segmentation on Taskonomy is produced as the output of this network rather than labeled by humans.
By doing so, we ensure that the annotations of the synthesized images are “accurate”.
Comparisons: Single-Task (ST) model is our baseline which follows the architecture of Taskonomy single task network.
ST is trained on real images only.
ST G is the ST model trained on the augmented set.
Table 1 reports the results of these two models, and ST G is worse than ST .
[Figure]
There are four main components in the framework: Multi-task network to address the target multiple pixel-level prediction tasks; self-supervision network to facilitate representation learning using images without any annotation; reﬁnement network to perform scene classiﬁcation using weak annotation; image generation network to synthesize useful images that beneﬁt multiple tasks.
[Figure]
Does synthesized images help downstream tasks?
From Table 1, the answer is NO .
Even though the images are synthesized by one of the state-of-the-art generative models and are labeled by the oracle annotator, they still cannot beneﬁt the downstream task.
This is probably because these images are synthesized off the shelf without “knowing” the downstream task.
Our key insight then is that we need to explicitly use the downstream task objective to guide the image synthesis process.
Moreover, here we focused on a single task and assumed that we had the oracle annotations.
However, an oracle annotator is difﬁcult to obtain in practice, especially for multiple tasks.
Also, existing work cannot synthesize paired images and pixel-level annotations for multiple tasks (Sandfort et al., 2019; Choi et al., 2019).
To overcome these challenges, in what follows we demonstrate how to facilitate multiple visual tasks with synthesize images that only need image-level scene labels.
M ETHOD
We propose multi-task oriented generative modeling (MGM) to leverage generative networks for multi-task visual learning, as summarized in Figure 2.
In this section, we ﬁrst formalize the novel problem setting of MGM.
Then, we explain the general framework and an instantiation of the MGM model with state-of-the-art multi-task learning and image generation approaches.
Finally, we discuss the detailed training strategy for the framework.
3. P ROBLEM S ETTING
Multi-task discriminative learning:
Given a set of n visual tasks T = { T 1 , T 2 , · · · , T n } , we aim to learn a discriminative multi-task model M that is able to address all of these tasks simultaneously:
M ( x ) → (cid:98) y
= ( (cid:98) y 1 , (cid:98)
y 2 , · · · , (cid:98) y n ) , where x is an input image and (cid:98)
y
i is the prediction for task T i .
Here we focus on the type of per-pixel level prediction tasks ( e.g ., semantic segmentation or depth estimation).
We treat image classiﬁcation as a special task, which provides global semantic description ( i.e ., scene labels) of images and only requires image-level category annotation c .
Therefore, the set of fully annotated real data is denoted as S real = { ( x j , y 1 j , y 2 j , · · · , y nj , c j ) } .
Generative learning: Meanwhile, we aim to learn a generative model G that produces a set of synthesized data but with only corresponding image-level scene labels (weak annotation): G ( c, z ) → (cid:101) x , where z is a random input, and (cid:101) x is a synthesized image.
The scene label of (cid:101) x is denoted as (cid:101) c
= c .
We denote the set of synthesized images and their corresponding scene labels as (cid:101)
S syn =
{ ( (cid:101) x k , (cid:101) c k ) } .
Cooperation between discriminative and generative learning: Our objective is that the discriminative model M and the generative model G cooperate with each other to improve the performance on the multiple visual tasks T .
During the whole process, the full model only gets access to the real fully-labeled data S real , then the generative network is trained to produce the synthesized set (cid:101) S syn .
Finally, M effectively learns from both S real and (cid:101)
S syn .
Note that, unlike most of the existing work on image generation (Brock et al., 2018; Zhang et al., 2019), we do not focus on the visual realism of the synthesized images (cid:101) x .
Instead, we hope G to capture the underlying mechanism that beneﬁts M .
3. F RAMEWORK AND A RCHITECTURE
Figure 2 shows the architecture of our proposed MGM framework.
It contains four components: the main multi-task discriminative network M , the image generation network G , the reﬁnement network
R , and the self-supervision network.
By introducing the reﬁnement network and the self-supervision network, the full model can leverage both fully-labeled real images and weakly-labeled synthesized images to facilitate the learning of latent feature representation.
These two networks thus allow M and R to better cooperate with each other.
Notice that our MGM is a model-agnostic framework, and here we instantiate its components with state-of-the-art models.
In the ablation study (Sec. 4.3) and the supplementary material, we show that our MGM works well with different choices of the model components.
Multi-task Network (M): The multi-task network aims to make predictions for multiple target tasks based on an input image.
Consistent with the most recent work on multi-task learning, we instantiate an encoder-decoder based architecture (Zamir et al., 2018;
Zhang et al., 2019; Sun et al., 2019).
Considering the trade-off between model complexity and performance, we use a shared encoder
E to extract features from input images, and individual decoders for each target task.
We adopt a
ResNet-18 (He et al., 2016) for the encoder and symmetric transposed decoders following Zamir et al.
For each task, we have its own loss function to update the corresponding decoder and the shared encoder.
Image Generation Network (G): The generative model G is a variant of generative adversarial networks (GANs).
We include the generator in our framework, but this module also has a discriminator during its own training.
G takes as input a latent vector z and a category label c , and synthesizes an image belonging to category c .
Considering the trade-off between performance and training cost, we instantiate G with self-attention generative adversarial network (SAGAN) (Zhang et al., 2019).
We achieve conditional image generation by applying conditional batch normalization (CBN) layers (De Vries et al., 2017):
[Equation]
where f i,c,h,w is an extracted c -channel 2D feature for the i -th sample, and (cid:15) is a small value to avoid collapse.
γ c and β c are two parameters to control the mean and variance of the normalization, which are learned by the model for each class.
We use hinge loss for the adversarial training.
Notice that the proposed framework is ﬂexible with different generative models, and we also show the effectiveness of using DCGAN (Radford et al., 2015) in the supplementary.
Reﬁnement Network (R):
As one of our key contributions, we introduce the reﬁnement network
R to further reﬁne the shared representation using the global scene category labels.
R takes the predictions of the multi-task network as input and predicts the category label of the input image.
Importantly, because R only requires category labels, it can be effortlessly operationalized on the “weakly-annotated” synthesized images.
Meanwhile, R also enforces the semantic consistency of the synthesized images with G .
We apply an algorithm inspired by Expectation-Maximum (EM)
(Dempster et al., 1977) to train the reﬁnement network R .
For the fully-annotated real images ( x, y , c ) , we use the scene classiﬁcation loss to update R and reﬁne the encoder E in the multi-task network M .
Then for the synthesized images ( (cid:101) x, (cid:101) c ) , since their multi-task predictions produced by M might not be reliable, we only reﬁne E with R frozen using the scene classiﬁcation loss.
Through reﬁning the share feature representation with the synthesized images, this process also provides implicit guidance to the image generation network.
More speciﬁcally, we model the whole multi-task network and reﬁnement network as a joint probability graph:
[Equation]
[Figure]
[Table] [Table]
[Table]
; for epoch ← 1 to max epoch do Split S real into minibatches with size N : S mini ; for ( x, y , c ) ∈ S mini do ˆ y =
M ( x )
L
[Equation] [Equation]
→ update M ; ˆ
[Equation]
Use L
GAN to train
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
end end where x is an input image, y is the vector of multi-task predictions, c is the scene label, θ is the vector of parameters of the multi-task network, and θ (cid:48) is the vector of parameters of the reﬁnement network.
The parameters θ and θ (cid:48) are learned to maximize the joint probability.
For data samples in S real , we maximize the joint probability and update θ (cid:48) to train the reﬁnement network.
[Equation] [Equation]
For data samples in (cid:101) S syn , we update the parameters of M ( θ ) in an EM-like manner.
During the E step, we estimate the latent multi-task ground-truth by:
[Equation]
Then for the M step, we back-propagate the error between y † and (cid:98) y (the multi-task predictions) to the multi-task encoder.
[Equation]
We use cross-entropy as the classiﬁcation loss function.
Self-supervision Network: The self-supervision network facilitates the representation learning of the encoder E by performing self-supervised learning tasks on images without any annotation so that can be operationalized on both real and synthesized images.
We modify SimCLR (Chen et al., 2020), one of the state-of-the-art approaches, as our self-supervision network.
This network contains an additional embedding network E self , working on the output of the multi-task encoder E , to obtain a 1D latent feature of the input image: µ = E self ( E ( x )) .
Then, it performs contrastive learning with these latent vectors.
Speciﬁcally, given a minibatch of N images, this network ﬁrst randomly samples two transformed views of each source image as augmented images (See supplementary material for the detailed transformations), resulting in 2 N augmented images.
For each augmented image, there is only one pair of positive augmented examples from the same source image, and other 2( N − 1) negative pairs.
Then the network jointly minimizes the distance of positive pairs and maximizes the distance of negative pairs in the latent space, through the normalized
[Table] [Table]
temperature-scaled cross-entropy ( NT-Xent )) loss (Chen et al., 2020):
[Equation]
where (cid:96) i,j is the NT-Xent loss for a positive pair of examples in the latent space ( µ i , µ j ) .
1 [ k (cid:54) =
i ] ∈ 0 ,is an indicator function evaluating to 1 if k (cid:54)
=
i , and τ is a temperature parameter.
dis ( µ i , µ j ) is a distance function, and we use cosine distance following Chen et al.
This loss is further back-propagated to reﬁne the multi-task encoder E .
Notice that other types of self-supervised tasks are applicable as well.
To demonstrate this, in Sec.
4.3 we also report the result with another task — image reconstruction.
3. I NTERACTION A MONG N ETWORKS
Cooperation Through Joint Training: We propose a simple but effective joint training algorithm shown in Figure 3.
The image generation network G takes the transferred feature representation of the multi-task encoder E , added with some Gaussian noise, as the latent input z to conduct conditional image generation.
Hence, the generation network obtains additional, explicit guidance ( i.e ., extra effective features) from the multi-task network to facilitate the generation of “better images”— images that may not look more realistic but are more useful for the multiple target tasks.
Then, the generation error of G will be back-propagated to E to further reﬁne the shared representation.
This process can be also viewed as introducing image generation as an additional task in the multi-task learning framework.
Training Procedure: We describe the procedure in Algorithm 1 and further explain it in the supplementary material.
E XPERIMENTS
To evaluate our proposed MGM model and investigate the impact of each component, we conduct a variety of experiments on two standard multi-task learning datasets.
We also perform detailed analysis and ablation studies.
4. D ATASETS AND C OMPARED M ETHODS
Datasets: Following the work of Sun et al.
(2019) and Standley et al.
(2020), we mainly focus on three representative visual tasks in the main experiments: semantic segmentation (SS), surface normal prediction (SN), and depth estimation (DE).
At the end of this section, we will show that our approach is scalable to an additional number of tasks.
We evaluate all the models on two widely-
[Figure]
benchmarked datasets: NYUv2 (Nathan Silberman & Fergus, 2012; Eigen & Fergus, 2015) and
Tiny-Taskonomy (Zamir et al., 2018).
See Sec.
B in the appendix for more details.
Compared Methods: We mainly focus on our comparison with two state-of-the-art discriminative baselines: Single-Task (ST) model follows the architecture of Taskonomy single task network (Zamir et al., 2018), and address each task individually; Multi-Task (MT) model refers to the sub-network for the three tasks of interest in Standley et al.
These two baselines can be viewed as using our multi-task network without the proposed reﬁnement, self-supervision, and generation networks.
Note that our work is the ﬁrst that introduces generative modeling for multi-task learning, and there is no existing baseline in this direction .
Our MGM is the full model trained with both fully-labeled real data and weakly-labeled synthesized data, which are produced by the generation network through joint training.
In addition, to further validate the effectiveness of our MGM model, we consider its variant model MGM r that is trained with both fully and weakly labeled real data.
MGM r is used to show the performance upper bound in the semi-supervised learning scenario, where the synthesized images are replaced by the real images in the dataset.
The resolution is set to 128 for all the experiments.
For all the compared methods, we use a ResNet-18 like architecture to build the encoder and use the standard decoder architecture of
Taskonomy (Zamir et al., 2018).
Data Settings: We conduct experiments with three different data settings: (1) 100% data setting; (2) 50% data setting; and (3) 25% data setting.
For each setting, we use 100%, 50%, or 25% of the entire labeled training set to train the model.
For MGM r , we add another 50% or 25% of weaklylabeled real data in the last two settings.
For MGM , we include the same number of weakly-labeled synthesized data in all three settings.
Evaluation Metrics: For NYUv2, following the metrics in Eigen & Fergus (2015); Sun et al.
(2019), we measure the mean Intersection-Over-Union (mIOU) for the semantic segmentation task, the mean
Absolute Error (mABSE) for the depth estimation task, and the mean Angular Distance (mAD) for the surface normal estimation task.
For Tiny-Taskonomy, we follow the evaluation metrics of previous work (Zamir et al., 2018; Standley et al., 2020; Sun et al., 2019) and report the averaged loss values on the test set.
Implementation Details: See appendix Sec.
A for the training details and the sensitivity of the hyper-parameters.
4. M AIN R ESULTS
Quantitative Results: We run all the models for 5 times and report the averaged results and the standard deviation on the two datasets in Table 2.
From this table, we have the following key observations that support the effectiveness of our approach.
Existing discriminative multi-task learning approaches may not consistently beneﬁt all the three individual tasks.
However, our MGM consistently and signiﬁcantly outperforms both the single-task and multi-task baselines across all the scenarios.
(2) By using weakly-labeled synthesized data, the results of our model in the 50% data setting are even better than those of baselines in the 100% data setting.
(3) More interesting, the performance of our MGM is close to MGM r , which indicates that our synthesized images are comparably useful as real images for improving multiple visual perception tasks.
(4) The performance gap between the two models is especially minimal in the 25% labeled data setting, suggesting that our proposed MGM model is, in particular, helpful with limited data.
Qualitative Results: We also visualize the prediction results on the three tasks for ST, MT, and
MGM in the 50% data setting in Figure 4.
While obvious defects can be found for all the baselines, the results of our proposed method are quite close to the ground-truth.
How Does Generative Modeling Beneﬁt Multi-tasks?
To have a better understanding of how the generative modeling and joint learning mechanism beneﬁt multi-task visual learning, we also
[Table]
The reﬁnement network works better for surface normal; the self-supervision network works better consider two variants of our MGM model and evaluate their performance.
MGM / G is the MGM model trained with S real only (without generative modeling), which shows the performance of our proposed multi-task learning framework in general (with the help from the auxiliary reﬁnement and self-supervision networks), and helps to understand the gain of leveraging generative modeling.
MGM / j is trained with synthesized images produced by a pre-trained SAGAN without the joint training mechanism.
Table 3 shows the results on the two datasets.
Combining the results of Tables 3 and 2, we ﬁnd: (1) MGM outperforms both ST and MT baseline even without generative modeling, indicating the beneﬁt of the self-supervised task and the reﬁnement network.
(2) By introducing synthesized images that are trained separately, the multi-task performance slightly improves, which shows the effectiveness of involving generative modeling into multi-tasks, under the assistance of our reﬁnement and self-supervision networks.
The joint learning mechanism further improves the cooperation between generative modeling and discriminative learning, thus enabling the generative model to better facilitate multi-task visual learning.
4. A BLATION S TUDY
For all the experiments in this section, models are trained in the 50% data setting, unless speciﬁcally mentioned.
Impact of Self-supervision Task and Reﬁnement Network: Two important components of the proposed framework are the self-supervision task and the reﬁnement network.
We evaluate their impact individually in Table 4.
MGM / self is the model trained without the self-supervision task;
MGM / reﬁne is the model without the reﬁnement network; for MGM recon , we replace the SimCLR based self-supervision method with a weaker reconstruction task, and use Mean Square Error as the loss function.
We could see that the reﬁnement network works better for the surface normal task, and the selfsupervision task works better for the semantic segmentation task; they are complementary to each other, and combining them generally achieves the best performance.
In addition, the model could still gain some beneﬁt even when we use some weak self-supervision tasks like reconstruction, which indicates the generalizability and robustness of our MGM model.
Number of Synthesized Images vs. Real images: From the previous results, we have found that the synthesized images could beneﬁt the target multi-tasks in a way similar to weakly labeled real images.
To further investigate the impact of the number of synthesized images, we vary if from 25% to 125% during multi-task training on NYUv2 in the 25% real data setting.
Figure 5 summarizes the result.
First, we can see that the performance gap between MGM / j (without joint training) and
MGM becomes larger for a higher ratio of weakly labeled data, which indicates the importance of our joint learning mechanism.
More importantly , while the real images are constrained in number due to the human collection effort, our generation network is able to synthesize unlimited amounts of images.
This is demonstrated in the comparison between MGM r (with real images) and MGM: the performance of our MGM keeps improving with respect to the number of synthesized images, achieving results almost comparable to that of MGM r when MGM r uses all the available weakly labeled real images.
We also provide ablation studies on the generalizability, impact of parameters, and the resolution of images in Sec.
D in the appendix.
4. E XTENSION
Experiments with More Tasks: MGM is also ﬂexible and scalable with different tasks.
In addition to the three tasks addressed in the main experiments, here we add three extra tasks: Edge Texture (ET), Reshading (Re), and Principal Curvature (PC), leading to six tasks in total.
We evaluate the performance of all the compared models on Tiny-taskonomy in the 50% data setting, and report the mean test loss for all the tasks.
The result is reported in Table 5.
Again, our proposed method still outperforms state-of-the-art baselines.
[Table]
R ELATED W ORK
Multi-task Learning and Task Relationship: Multi-task learning (MTL) aims to leverage information coming from signals of related tasks so that each individual task can gain beneﬁt (Doersch &
Zisserman, 2017).
Ruder (2017) identiﬁes that most recent works use two clusters of strategies for
MTL: hard parameter sharing techniques (Kokkinos, 2017; Doersch & Zisserman, 2017; Pentina &
Lampert, 2017) and soft parameter sharing techniques (Misra et al., 2016; Sener & Koltun, 2018;
Chen et al., 2018b).
These strategies have achieved good performance for MTL with similar tasks.
Researchers have also carefully studied the task relationships among different tasks to make the best cooperations among them.
Taskonomy exploits the relationships among various visual tasks to beneﬁt the transfer or multi-task learning (Zamir et al., 2018).
Standley et al.
(2020) considers task cooperation and competition, and proposes a method to assign tasks to a few neural networks to balance all of them.
Some other following works also explores task relationships among different types of tasks (Sun et al., 2019; Zamir et al., 2020; Armeni et al., 2019; Pal & Balasubramanian, 2019; Yeo et al., 2021; Wallace et al., 2021).
These works only consider MTL with discriminative tasks.
In comparison, we ﬁrst introduce generative modeling to multi-task visual learning.
Generative Modeling for Visual Learning:
Besides the initial goal of synthesizing realistic images, some recent work has explored the potential to leverage generative models to synthesize “usefull” images for other visual tasks (Shorten & Khoshgoftaar, 2019).
The most straightforward way is to generate images and the corresponding annotations as data augmentation for the target visual task (Bao et al., 2021;
Sandfort et al., 2019; Choi et al., 2019; Wang et al., 2018).
Another strategy to leverage generative models is through well-designed error feedback or adversarial training (Luc et al., 2016; CS Kumar et al., 2018; Mustikovela et al., 2020).
There have been works that apply generative models for different visual tasks including classiﬁcation (Zhan et al., 2018; Frid-Adar et al., 2018; Zhu et al., 2018), semantic segmentation (Luc et al., 2016) and depth estimation (Pilzer et al., 2018; Aleotti et al., 2018). Among them, Souly et al.
(2017) also proposes a semi-supervised framework to leverage generative models for semantic segmentation through adversarial training.
Different from these methods, MGM is applicable to various multiple visual tasks and different generative networks.
Reduced-Supervision Methods: Recent works take advantage of weakly labeled data by assigning some self-created labels ( e.g .colorization, rotation, reconstruction) (Noroozi & Favaro, 2016; Noroozi et al., 2017; Chen et al., 2020; Dosovitskiy et al., 2014; Pathak et al., 2016).
Similar self-supervised techniques have been proved useful for multi-task learning (Liu et al., 2008; Ren & Jae Lee, 2018;
Doersch & Zisserman, 2017; Lee et al., 2019).
Among these techniques, a famous one is the
Expectation-Maximization (EM) algorithm
(Dempster et al., 1977; Papandreou et al., 2015), which leverages the information of weakly or unlabelled data by iteratively estimating and reﬁning their labels.
We adopt a similar spirit and introduce the reﬁnement network for MGM framework.
C ONCLUSION
This paper proposes to introduce generative modeling for multi-task visual learning.
The main challenge is that current generative models almost cannot synthesize both RGB images and pixel-level annotations in multi-task scenarios.
We address this problem by proposing multi-task oriented generative modeling (MGM) framework equipped with the self-supervision network and the reﬁnement network, which enable us to take advantage of synthesized images paired with image-level scene labels to facilitate multiple visual tasks.
Experimental results indicate our MGM model consistently outperforms state-of-the-art multi-task approaches.
A A DDITIONAL D ETAILS OF M ODEL A RCHITECTURE
Multi-task Network: The multi-task network contains a shared encoder network and separate decoder networks for target tasks.
We use a ResNet-18 (He et al., 2016) as the encoder network, and its architecture follows the standard Pytorch implementation 1 .
We only change the size of the features of each layer group from [64 , 128 , 256 , 512] to [48 , 96 , 192 , 360] , so as to better address our
GPU memory constraints.
The size of the ﬁnal feature representation is (360 , 8 , 8) .
For the decoder network, we use the same architecture as in Taskonomy (Zamir et al., 2018).
Self-supervision Network: We adopt SimCLR (Chen et al., 2020) as our self-supervision network.
SimCLR is one of the state-of-the-art self-supervised learning approaches based on instance-level discrimination tasks.
Following (Chen et al., 2020), we randomly apply 5 types of transformations on a source image to obtain an augmented image.
These transformations are as follows: (1) random resizing and cropping followed by resizing back to the original size; (2) random horizontal ﬂipping with probability of 0.5; (3) random color jittering with probability of 0.5; (4) random transformation of
RGB images to gray-scale images with probability of 0.2; (5) random Gaussian blur with probability of 0.5.
The shape of the transformed latent feature, which is used to perform contrastive learning, is (128 , ) .
Reﬁnement Network: The reﬁnement network takes as input the prediction results of the multitask network.
For each individual prediction, we apply a ResNet-
(He et al., 2016) as the reﬁnement encoder to extract the features.
The feature dimension of each layer group of the reﬁnement encoder is the same as the multi-task encoder network.
Then we concatenate all the features together and apply a fully-connected layer with the hidden size of 128 to obtain the ﬁnal scene class prediction.
Image Generation Network: We have instantiated the image generation network with two widely used generative networks: self-attention GAN (SAGAN) 2 (Zhang et al., 2019) in the main paper and deep convolutional GAN (DCGAN)
(Radford et al., 2015) in Section D.5 of this document.
For DCGAN, we change the original batch-normalization layers to conditional batch normalization layers (De Vries et al., 2017) to allow conditional image generation.
For the additional embedding layer used for joint learning, we use a simple global averaged pooling layer followed by a dense layer.
e.g . B I MPLEMENTATION D ETAILS
Data Processing: For the NYUv2 (Nathan Silberman & Fergus, 2012) dataset, following Sun et al.
(2019) we resize and normalize the RGB images to ( − 1 , 1) , standardize the normal ground-truth to (0 , 1) , and do not normalize the depth.
For the Taskonomy dataset (Zamir et al., 2018), we follow the standard data normalization in Zamir et al.
Additional Implementation Details: We use Adam (Kingma & Ba, 2014) optimizer for all the models.
The learning rates are set to 0.001 for the multi-task, self-supervision, and reﬁnement networks, 0.0001 for the SAGAN generator, and 0.0004 for the SAGAN discriminator.
The batch size is set to 32.
We use
Due to the different converge time for the different modules, we use a three-stage strategy to perform joint training: (1) We ﬁrst train the multi-task network separately with fully labeled real data; (2)
We then freeze the multi-task network, and train the image generation network and the embedding network separately ; (3) We do joint training with the whole network using both fully labeled real data and weakly labeled sythesized data.
During the pre-training process of SAGAN, we set the batch size to 128 to train a better model following Zhang et al.
Then, for the joint training, we use a batch size of 32 for all the sub-networks.
Additionally, for the same minibatch of data, we update the image generation network 2 times iteratively during stage (3).
Training Procedure: We summarized the training procedure in Algorithm 1 in the main paper.
Here we further explain the training procedure in more details.
Given a minibatch of data in S real , we conduct the following training procedure.
For the input images x , we predict ˆ y =
M ( x ) , and then use the task-speciﬁc losses between y and ˆ y to update the multi-task network M . 2.
We predict the scene labels by ˆ c
= R (ˆ y ) , and update the reﬁnement network R and the multi-task encoding network E using the cross-entropy loss between c and ˆ c .
We randomly sample pairs of augmented images, process them with the self-supervision network, and then update the self-supervision network and the multi-task encoder E with the NT-Xent loss in Eqn.
We train the image generation network G through adversarial training with ( x, c ) , and back-propagate the adversarial error and update E at the same time.
We sample another minibatch of synthesized data ( (cid:101) x, (cid:101) c ) , and use these data to update E by performing both the EM-like algorithm described in Section 3.2 (main paper) with R and the self-supervised learning as in step 3.
B. D ATASET S ETTING
We evaluate all the models on two widely-benchmarked datasets:
NYUv2 (Nathan Silberman &
Fergus, 2012; Eigen & Fergus, 2015) containing 1,449 images with 40 types of objects (Gupta et al., 2013); Tiny-Taskonomy which is the standard tiny split of the Taskonomy dataset (Zamir et al., 2018).
For Tiny-Taskonomy dataset, since a certain amount of images for each category is required to train a generative network, we keep the images of the top 35 scene categories on Tiny-Taskonomy, with each one consisting of more than 1,000 images.
This resulting dataset contains 358,426 images in total.
For NYUv2, we randomly select 1,049 images as the full training set and 200 images each as the validation/test set.
For Tiny-Taskonomy, we randomly pick 80% of the whole set as the full training set and 10% each as the validation/test set.
C A DDITIONAL C OMPARISONS C. C OMPARISON WITH O THER M ULTI - TASK M ODELS
In the main paper, for a fair comparison we focused on comparing our MGM model with internal models ( e.g ., the multi-task model upon which MGM builds).
To have a more comprehensive understanding of the performance of MGM, we also compare our method with some state-of-the-art multi-task models.
We focus on the 25% data setting for the NYUv2 dataset, where collaboration between different tasks and the utilization of data is vitally important.
We include six models in this experiments.
ST is the single-task model, where the encoder and the decoder are adopted from Zamir et al.
MT is the multi-task model that uses a shared encoder as
ST and separate decoders.
ST and MT are the baselines compared in the main paper.
TaskGrouping uses the optimal network for the three tasks concluded from Standley et al.
Another two wellperforming multi-task models are Cross-stitch (Ren & Jae Lee, 2018) 4 and AdaShare (Sun et al., 2019).
MGM is our proposed model.
Table 6 summarizes the results.
Notably, with a shared
[Table] [Table]
encoder architecture, our MGM model outperforms other state-of-the-art multi-task networks with more sophisticated architectures, which indicates the beneﬁt of introducing generative modeling for multi-task learning .
In addition, our MGM is a model-agnostic framework and could be incorporated with these different multi-task models for further improvement.
C. E XPERIMENTS WITH F EW -S HOT S ETTING
Since the learned generative model facilitates ﬂow of knowledge across tasks and provides meaningful variations in existing images, it is especially beneﬁcial in the low-data regime.
So we designed a 10% data setting for NYUv2 dataset, where around 3 images for each scene is used as the training set.
We also compare our MGM model with ST and MT.
From Table 7, we can see that MGM outperforms the other compared models signiﬁcantly, indicating the gain of generative models in the extremely low-data regime .
D A DDITIONAL A BLATION S TUDY D. I MPACT OF P ARAMETERS
Introducing the reﬁnement, self-supervision, and image generation networks also leads to more parameters.
To validate that the performance improvements come from the novel design of our architecture rather than merely increasing the number of parameters, we provide two model variants as additional baselines: ST l and MT l use ResNet-34 as the encoder network and the corresponding decoder networks.
These two networks have a similar amount of parameters as MGM.
The result in
Table 8 show that simply increasing the number of parameters cannot signiﬁcantly boost performance.
D. T RAINING S TRATEGIES FOR R EFINEMENT N ETWORK
In the main paper, we proposed an Expectation-Maximum (EM) like algorithm to coordinate the training between the reﬁnement network and the main network.
Here we compare our EM-Like
Training (EML) with two alternative ways of the training procedure: Plain End-to-End Training (PEoE) backwards the reﬁnement loss to update the entire network directly; Loosely Separate
Training (LSeT) trains the reﬁnement network and the main network separately, and only backwards the error to the encoder network when dealing with weakly labeled images.
Table 9 shows the comparison results on the NYUv2 dataset in the 50% data setting.
From this table, we could ﬁnd: (1)
A na¨ıve end-to-end training strategy is not able to facilitate the cooperation between different networks and thus hurts the overall performance; (2) Loosely separate training enables the communication
[Table] [Table]
between the reﬁnement network and the main network when needed and thus outperforms the baseline
Our further
D. E XPERIMENTS ON H IGHER I MAGE R ESOLUTION
In principle, the proposed framework is agnostic to the speciﬁc types of multi-task networks and image generation networks, thus ﬂexible with image resolutions.
The practical constraint lies in that it is still challenging and resource-consuming for modern generative models to synthesize very high-resolution images (Zhang et al., 2019;
Brock et al., 2018), although the deep multi-task models normally work better with high-resolution images.
In the main experiments, consistent with exiting image synthesis work (Zhang et al., 2019), we focused on the resolution of 128 × 128 .
Here we further made an attempt to run our experiments with a higher resolution, 256 × 256 on NYUv2.
The results of all the compared models in the 50% data setting are shown in Table 10.
We could ﬁnd that
MGM still consistently outperforms the baselines, indicating the great robustness and ﬂexibility of our proposed framework.
D. G ENERALIZATION OF THE S HARED F EATURE R EPRESENTATION
Intuitively, our MGM achieves state-of-the-art performance by effectively learning a shared feature representation.
We further show the generalization capability of this representation by designing the following experiment: for the multi-task model and our MGM model, we ﬁrst learn the shared feature space with the SS and DE tasks, and we then use that learned feature space to train a new decoder for the SN task.
We report the results on NYUv2 in Table 11.
Our MGM outperforms the multi-task model in all the three data settings, which means that MGM indeed learns a better and robust shared feature space.
D. E XPERIMENTAL R ESULTS WITH DCGAN
The proposed MGM model is a general framework and is ﬂexible with different choices of model components.
Here we show its ﬂexibility for the image generation network: We replace SAGAN with DCGAN (Radford et al., 2015), and Table 12 shows the result on the NYUv2 dataset in the 50% data setting.
Similar to the results in the main paper, MGM equipped with DCGAN (‘MGMDCGAN’) consistently improves the performance on all the tasks, which indicates the robustness of the proposed MGM framework.
In addition, we ﬁnd that ‘MGM-SAGAN’ outperforms ‘MGMDCGAN’, suggesting that a more powerful image generation network leads to better performance. [Table] [Table]
E M ORE V ISUALIZATIONS
In Figure 4 of the main paper we visualized the prediction results.
Here we provide more visualizations of the multi-task predictions for MGM and the compared baselines in Figure 6.
Our MGM model signiﬁcantly outperforms both ST and MT baselines.
F D ISCUSSION AND F UTURE W ORK
In the main paper, we focused on applying generative networks to general multi-task learning.
On the other hand, generative models can also be exploited to address multiple tasks in speciﬁc domains, such as human face modeling (Li et al., 2019; 2016; Han et al., 2017;
Chen et al., 2018a), autonomous driving (Liu et al., 2020; Wang et al., 2019; Chennupati et al., 2019) and bio-medical processing (Zhou et al., 2020; Panagopoulos, 2017).
We leave such exploration of leveraging our
MGM model to these domains as future work.
G A DDITIONAL E XPERIMENTAL E VALUATIONS G. E XPERIMENTAL E VALUATION ON C ITY S CAPE S UBSET
In this section, we demonstrate that MGM can work with datasets when no image-level labels are available.
In an alternative may, the proposed MGM frame model can work with object labels as well since the generative network and reﬁnement network can naturally work with multi-hot labels — the reﬁnement network can work with a multi-label classiﬁer, and the generative network can be a multi-label-conditional GAN.
We conducted semantic segmentation on a subset of CityScape (Cordts et al., 2016) dataset, the Zurich street scene. We focused on the semantic segmentation task, which is a representative task on CityScape. We use the 30 standard multi-hot CityScape semantic object labels for the generative model and also the reﬁnement networks.
We use 80% of the data for training and 20 % for testing and compare the performance of ST and MGM for this experiment.
We generate the same amount of data with random multi-hot labels using MGM.
The results are shown in Table 13.
MGM still outperforms the baseline ST model, indicating the robustness and generalizability of the model with multi-hot object labels.
G. E XPERIMENTS WITH E XTREME L OW DATA AT T ASKONOMY
We noticed that for Tiny-Taskonomy dataset, 25% data setting is still far from low-data regime.
To further explore the effectiveness with our model with low-data, we further conduct an experiment with a subset of Tiny-Taskonomy dataset.
We randomly select 3 nodes (allensville, benevolence, and coffeen) from Tiny-Taskonomy dataset to build a dataset with 17,404 images—around 5% data setting compared with the full Tiny-Taskonomy dataset.
We then conduct experiments with ST,
[Table]
networks .
‘MGM-SAGAN’:
MGM equipped with SAGAN (presented in the main paper); ‘MGMDCGAN’:
MGM equipped with DCGAN.
Both ‘MGM-SAGAN’ and ‘MGM-DCGAN’ consistently improve the performance on all the tasks and outperform single-task (ST) and multi-task (MT) baselines.
This shows the robustness and ﬂexibility of the proposed MGM framework.
In addition, ‘MGM-SAGAN’ outperforms ‘MGM-DCGAN’, suggesting that a more powerful image generation network leads to better performance.
[Table]
MT and MGM for this subset.
All the other experimental settings keep the same as the main paper.
Table 14 shows the comparable results.
Combining the results in Table 2, we can ﬁnd that MGM consistently outperforms ST and MT and is robust and especially helpful in low-data regime.
G. A BLATION WITH S INGLE T ASKS
MGM is a general framework that can be applied to both single tasks and multiple tasks.
In the main paper, we mainly focused on the more challenging multi-task scenario.
In this subsection, we conduct experiments with single tasks.
Here we add three baselines applying MGM to the single tasks (SS,
DE, SN) named MGM-SS , MGM-DE and MGM-SN in the NYUv2 50% data setting.
We further add an additional strong baseline by using the equivalently sampled data from the above three model as the augmented data, but not jointly train the generative network, named MGM-Combine .
The results of these models are shown in Table 15.
We have the following observations: (1)
The proposed
MGM framework can consistently beneﬁt each single tasks though without leveraging shared features from multiple tasks.
Compared with the full MGM model, the performance drops when only using single tasks to jointly train with the generative model.
(3) When using the individually optimized generative models, the performance is slight better than MGM / j but still could not reach MGM, indicating the importance of jointly training mechanism proposed in the main paper.
H V ISUALIZATIONS OF THE G ENERATED I MAGES
We visualize the generated images for Tiny-Taskonomy dataset of SAGAN (Zhang et al., 2019) and MGM in Figure 7.
From Figure 7, We observe that (1) the conventional SAGAN, trained with the realistic objective and without the guidance of the downstream tasks, produces photo-realistic images; (2) the visual quality of the generated images by MGM becomes degraded, where SAGAN is jointly trained with the multi-task learning objective and under the guidance of the downstream tasks.
Interestingly, a similar phenomenon has been observed in (Souly et al., 2017), where a generative model is used to facilitate the semantic segmentation task.
We think it might be related to the difference between machine vision and human vision.
I V ISUALIZATIONS OF THE S EMANTIC A NNOTATIONS
We also visualize two samples and the corresponding semantic segmentation annotations we used for the pilot study in Figure 8.
The visualization shows that the annotation is accurate and using
Taskonomy is sufﬁcient for training.
[Figure]
[Table]
[Table]
MGM-Combine: MGM variant trained with augmented images generated by the above three models.
The proposed MGM framework can consistently beneﬁt each single cannot reach of
[Figure] [Figure]