A UXILIARY L EARNING BY I MPLICIT D IFFERENTIATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Training neural networks with auxiliary tasks is a common practice for improving the performance on a main task of interest.
Two main challenges arise in this multi-task learning setting: (i) designing useful auxiliary tasks; and (ii) combining auxiliary tasks into a single coherent loss.
Here, we propose a novel framework,
AuxiLearn , that targets both challenges based on implicit differentiation.
First, when useful auxiliaries are known, we propose learning a network that combines all losses into a single coherent objective function.
This network can learn non-linear interactions between auxiliary tasks.
Second, when no useful auxiliary task is known, we describe how to learn a network that generates a meaningful, novel auxiliary task.
Evaluation of AuxiLearn in a series of tasks and domains, including image segmentation and learning with attributes in the low data regime, shows consistent improvement in accuracy compared to competing methods.
I NTRODUCTION
The performance of deep neural networks can signiﬁcantly improve by training the main task of interest with additional auxiliary tasks (Goyal et al., 2019; Jaderberg et al., 2016; Mirowski, 2019).
For example, learning to segment an image into objects can be more accurate when the model is simultaneously trained to predict other properties of the image like pixel depth or 3D structure (Standley et al., 2019).
In the low data regime, models trained with the main task only are prone to overﬁt and generalize poorly to unseen data (Vinyals et al., 2016).
In this case, the beneﬁts of learning with multiple tasks are ampliﬁed (Zhang and Yang, 2017).
Training with auxiliary tasks adds an inductive bias that pushes learned models to capture meaningful representations and avoid overﬁtting to spurious correlations.
Unfortunately, training with auxiliaries in practice is very challenging because of two main reasons.
First, combining and selecting auxiliaries .
In some domains, it may be easy to design beneﬁcial auxiliary tasks and collect supervised data.
For example, numerous tasks were proposed for selfsupervised learning in image classiﬁcation, including masking (Doersch et al., 2015), rotation (Gidaris et al., 2018) and patch shufﬂing (Doersch and Zisserman, 2017; Noroozi and Favaro, 2016).
In these cases, combining all auxiliary tasks into a single loss can be challenging (Doersch and Zisserman, 2017).
The common practice is to compute a weighted combination of pretext losses by tuning the weights of individual losses using hyperparameter grid search.
This approach limits the potential of learning with auxiliary tasks because the run time of grid search grows exponentially with the number of tasks.
The second major challenge is obtaining good auxiliaries in the ﬁrst place.
In many domains, it is not clear which auxiliary tasks could be beneﬁcial.
For example, for point cloud classiﬁcation, few self-supervised tasks have been proposed, and their beneﬁts are limited so far (Achituve et al., 2020;
Hassani and Haley, 2019; Sauder and Sievers, 2019; Tang et al., 2020).
For some domains specialized expert knowledge may be needed for designing useful auxiliary tasks.
For these cases, it would be beneﬁcial to automate the process of generating auxiliary tasks without domain expertise.
Our work takes a step forward in automating the use and design of auxiliary learning.
We present an approach to guide the learning of the main task with auxiliary learning, which we term AuxiLearn .
AuxiLearn leverages recent progress made in implicit differentiation for optimizing hyperparameters (Liao et al., 2018; Lorraine et al., 2020).
We show the effectiveness of AuxiLearn in two types of problems.
First, in combining auxiliaries , for cases where auxiliary tasks are predeﬁned, we propose to train a deep neural network (NN) on top of auxiliary losses and combine them non-linearly into a uniﬁed loss.
For instance, we combine per-pixel losses in image segmentation tasks using a convolutional NN (CNN).
Second, designing auxiliaries , for cases where predeﬁned auxiliary tasks are not available, we present an approach for learning such tasks without domain knowledge and from input data alone.
It is achieved by training an auxiliary network to output auxiliary labels while training another, primary, network to predict both the original task and the generated task.
One important distinction from previous works, such as (Kendall et al., 2018; Liu et al., 2019a), is that we do not optimize the auxiliary parameters using the training loss but on a separate (small) auxiliary set , allocated from the training data.
This is a crucial point since the goal of auxiliary learning is improving generalization and not helping the optimization on the training data.
To validate our proposed solution, we perform an extensive experimental study on various tasks in the low data regime.
In this case the models suffer from severe overﬁtting and auxilary learning can be of most use.
Our results indicate that using AuxiLearn leads to improved loss functions and auxiliary tasks, in terms of the performance of the resulting model on the main task.
We complement our experimental section with two interesting theoretical insights regarding our model.
The ﬁrst shows that a relatively simple auxiliary hypothesis class may overﬁt.
The second tries to understand which auxiliaries could help with the main task.
To summarize, we propose a novel general approach for learning with auxiliaries by utilizing implicit differentiation.
We make the following novel contributions: (a) A uniﬁed approach for combining multiple loss terms in a ﬂexible manner and for learning novel auxiliary tasks from the data alone; (b) A theoretical observation on auxiliary learning capacity; (c) We show that the key quantity for determining beneﬁcial auxiliaries is the Newton update; (d) New results on a variety of auxiliary learning tasks with a focus on setups in which training data is limited.
We conclude that implicit differentiation can play a signiﬁcant role in automating the design of auxiliary learning setups.
R ELATED WORK
Learning with multiple tasks.
Multitask Learning (MTL) aims at simultaneously solving multiple learning problems while sharing information and knowledge across tasks.
In some cases, MTL beneﬁts the optimization process and improve task-speciﬁc generalization performance compared to single-task learning (Standley et al., 2019).
In contrast to MTL, auxiliary learning aims at solving a single, main task, and the purpose of all other tasks is to facilitate the learning of the primary task.
At test time, only the main task is considered.
This approach has been successfully applied in multiple domains, including computer vision (Zhang et al., 2014), natural language processing (Fan et al., 2017; Trinh et al., 2018), and reinforcement learning (Jaderberg et al., 2016; Lin et al., 2019).
Dynamic task weighting.
When learning a set of tasks, we assemble the overall loss using a combination of task-speciﬁc losses.
The choice of proper loss blend is crucial, as MTL based models are susceptible to the relative task weights (Kendall et al., 2018).
The most common approach for combining task losses is to use a linear combination.
When the number of tasks is small, task weights are commonly tuned through simple grid search.
This approach naturally cannot extend to a large number of tasks, or a more complex weighting scheme.
Several recent studies proposed scaling task weights using gradient magnitude (Chen et al., 2018), task uncertainty (Kendall et al., 2018), or the rate of change in losses (Liu et al., 2019b).
(Sener and Koltun, 2018) propose casting the multitask learning problem as Multi-Objective Optimization.
These methods assume all tasks are equally important, hence they may not be suited for auxiliary learning.
Both (Du et al., 2018; Lin et al., 2019) proposed to weight auxiliaries using gradient similarity.
However, these methods do not scale well to multiple auxiliaries and do not take into account interactions between auxiliaries.
In contrast, we propose to learn from data how to combine auxiliaries, possibly in a non-linear manner.
Devising auxiliaries.
Designing an auxiliary task for a given main task is challenging because it may require domain expertise and additional labeling effort.
For self-supervised learning (SSL), many approaches have been proposed (see (Jing and Tian, 2020) for a recent survey), but the joint representation learned through SSL may suffer from negative transfer and hurt the main task (Standley et al., 2019).
(Liu et al., 2019a) proposed learning a helpful auxiliary in a meta-learning fashion, removing the need for handcrafted auxiliaries.
However, their system is optimized on the training data, which leads to degenerate auxiliaries.
To address this issue, an entropy term is introduced which forces the auxiliary network to spread the probability mass between the classes.
[Figure]
Implicit differentiation based optimization.
Our formulation gives rise to a bi-level optimization problem.
Such problems naturally arise in the context of meta-learning (Finn et al., 2017; Rajeswaran et al., 2019) and hyperparameter optimization (Bengio, 2000; Foo et al., 2008; Larsen et al., 1996;
Liao et al., 2018; Lorraine et al., 2020; Pedregosa, 2016).
The Implicit Function Theorem (IFT) is frequently used for computing the gradients of the upper-level function, which requires calculating a vector-inverse Hessian product.
However, for modern neural networks, it is infeasible to calculate it explicitly, and an approximation must be devised.
(Luketina et al., 2016) proposed approximating the
Hessian with the identity matrix, whereas (Foo et al., 2008; Pedregosa, 2016; Rajeswaran et al., 2019) use Conjugate Gradient (CG) to approximate the product.
Following (Liao et al., 2018; Lorraine et al., 2020), we use a truncated Neumann series and efﬁcient vector-Jacobian products, as it was empirically shown to be more stable than CG.
O UR M ETHOD
We now describe the general AuxiLearn framework for jointly optimizing the primary network and the auxiliary parameters of the auxiliary network.
First, we introduce our notations and formulate the general objective.
Then, we detail two instances of this framework: combining auxiliaries and learning new auxiliaries.
Finally, we present our optimization approach for optimizing these systems.
3. P ROBLEM DEFINITION
Let { ( x ti , y ti ) } i be the training set and { ( x ai , y ai ) } i be a distinct independent set which we term auxiliary set .
Let f ( · ; W ) denote the primary network, and let g ( · ; φ ) denote the auxiliary network.
Here, W are the parameters of the model optimized on the training set, and φ are the auxiliary parameters trained on the auxiliary set.
The training objective is deﬁned as:
[Equation]
where (cid:96) main denotes the loss of the main task and h is the overall auxiliary loss, controlled by φ .
We note that h has access to both W and φ .
The loss on the auxiliary set is deﬁned as L A =
(cid:80) i (cid:96) main ( x ai ,
y ai ; W ) , since we are interested in the generalization performance of the main task.
We wish to ﬁnd auxiliary parameters such that the parameters W , trained with the combined objective, generalize well.
More formally, we seek
[Equation]
3. L EARNING TO COMBINE AUXILIARY TASKS
Suppose we are given K auxiliary tasks, usually designed using expert domain knowledge.
We wish to learn how to optimally leverage these auxiliaries by learning to combine their corresponding losses.
Let (cid:96) ( x , y ; W ) =
( (cid:96) main ( x , y main ; W ) , (cid:96) 1 ( x , y 1 ; W ) , ..., (cid:96) K ( x , y K ; W )) denote a loss vector.
We wish to learn an auxiliary network g : R K +
→ R over the losses that will be added to (cid:96) main in order to output the training loss L T = (cid:96) main + g ( (cid:96) ; φ ) .
Here, h from Eq.
(1) is given by h ( · ; φ )
=
g ( (cid:96) ; φ ) .
Commonly, g ( (cid:96) ; φ ) is a linear combination of losses, namely g ( (cid:96) ; φ )
= (cid:80)
j φ j (cid:96) j , with positive weights φ j ≥ 0 that are tuned by grid search.
However, this method can only scale to a few auxiliaries, as the run time of grid search is exponential in the number of tasks.
Our method can handle a large number of auxiliaries and easily extends to a more ﬂexible formulation in which g parametrized by a deep NN.
This general form allows us to capture complex interactions between tasks, and learn non-linear combinations of losses.
See Figure 1a for illustration.
One way to view a non-linear combination of losses is as an adaptive linear weighting, where losses have a different set of weights for each datum.
If the loss at point x is (cid:96) main ( x , y main ) + g ( (cid:96) ( x , y )) then the gradients are ∇ W (cid:96) main ( x , y main )
+ (cid:80)
j ∂g∂(cid:96)
j ∇ W (cid:96) j ( x , y j ) .
This is equivalent to an adaptive loss where the loss of datum x is (cid:96) main + (cid:80)
j α j, x (cid:96) j and, α j, x = ∂g∂(cid:96)
j .
This observation connects our approach to other studies that assign adaptive loss weighs (e.g., Du et al.
; Liu et al.
(2019b)).
Convolutional loss network.
In certain problems there exist a spatial relation among losses.
For example, semantic segmentation and depth estimation for images.
The common approach is to average the losses over all locations.
We, on the other hand, can leverage this spatial relation for creating a loss-image in which each task forms a channel of pixel-losses induced by the task.
We then parametrize g as a CNN that acts on this loss-image.
As a result, we learn a spatial-aware loss function that captures interactions between task losses.
See loss image example in Fig.
Monotonicity.
It is common to parametrize the function g ( (cid:96) ; φ ) as a linear combination with nonnegative weights.
Under this parameterization, g is a monotonic non-decreasing function of the losses.
A natural question that arises is whether we should generalize this behavior and constrained g ( (cid:96) ; φ ) to be non-decreasing w.r.t.
the input losses as well?
Empirically, we found that training with monotonic non-decreasing networks tends to be more stable and has a better or equivalent performance.
We impose monotonicity during training by negative weights clipping.
See Appendix C.2 for a detailed discussion and empirical comparison to non-monotonic networks.
3. L EARNING NEW AUXILIARY TASKS
The previous subsection focused on situations where auxiliary tasks are given.
In many cases, however, no useful auxiliary tasks are known in advance, and we are only presented with the main task.
We now describe how to utilize AuxiLearn in such cases.
The intuition is simple: We wish to learn an auxiliary task that pushes the representation of the primary network to better generalize on the main task, as measured by the auxiliary set.
We do so in a student-teacher manner: an auxiliary "teacher" network produces labels and the primary network tries to predict these labels as an auxiliary task.
The auxiliary “teacher" network is trained to deﬁne helpful auxiliary for the main task.
More speciﬁcally for the case of auxiliary classiﬁcation task, we learn a soft labeling function g ( x ; φ ) over the input samples x .
These labels are then treated as ground truth auxiliary labels provided to the main network f
( x ; W ) .
See Figure 1b for illustration.
During training, the primary network f ( x ; W ) outputs two predictions, ˆ y main for the main task and ˆ y aux for the auxiliary task.
The auxiliary network g ( x ; φ ) produces soft auxiliary labels y aux , which we treat as “adaptive ground truth".
We then compute the full training loss L T =
(cid:96) main (ˆ y main , y main )
+ (cid:96) aux
(ˆ y aux , y aux ) to update W .
As before, we update φ using the auxiliary set with L A = (cid:96) main .
Here, h from Eq.
(1) is given by h ( · ; φ ) =
(cid:96) aux
( f ( x ti ; W ) , g ( x ti ; φ )) .
Intuitively, the teacher auxiliary network g is rewarded when it provides labels to the student that help it succeed in the main task, as measured by L A .
Why would learning auxiliaries in this manner help?
AuxiLearn learning procedure can be viewed as learning a data-dependent regularizer that is optimized for the objective of increasing the generalization performance of the primary network.
Therefore, the auxiliary network has an incentive to generate non-trivial labels.
In Figure 4 we provide an example of the learned auxiliary task.
3. O PTIMIZING AUXILIARY PARAMETERS
We now return to the bi-level optimization problem in Eq.
(2) and present the optimizing method for φ .
Solving Eq.
(2) for φ poses a problem due to the indirect dependence of L A on the auxiliary parameters.
To compute the gradients of φ , we need to differentiate through the optimization process over W , since ∇ φ L A = ∇ W L A · ∇ φ W ∗ .
As in (Liao et al., 2018; Lorraine et al., 2020), we use the implicit function theorem (IFT) to evaluate ∇
[Equation]
Thus, we can leverage the IFT to approximate the gradients of the auxiliary parameters φ :
[Equation]
See Appendix A for detailed derivation.
To compute the vector and Hessian inverse product, we utilize the algorithm proposed in (Lorraine et al., 2020), which uses Neumann approximation and efﬁcient Vector-Jacobian Product.
We note that accurately computing ∇ φ L A by IFT requires ﬁnding a point such that ∇ W L T = 0 .
In practice, we only approximate W ∗ , and simultaneously train both
W and φ by altering between optimizing W on L T , and optimizing φ using L A .
We summarize our method in Alg. 1 and 2.
We provide theoretical considerations about our method in Appendix D.
Algorithm Initialize auxiliary parameters φ and weights W ; while not converged do for k = 1 , ..., N do L
T = (cid:96) main ( x , y ; W ) + h ( x , y, W ; φ ) W ← W − α ∇ W L T (cid:12)(cid:12) φ,W end φ ← φ − Hypergradient
( L A , L T , φ, W ) [Table] W
[Table]
A NALYSIS 4. C OMPLEXITY OF AUXILIARY HYPOTHESIS SPACE
In our learning setup, an additional auxiliary set is used for tuning a large set of auxiliary parameters.
A natural question arises: could the auxiliary parameters overﬁt this auxiliary set?
and what is the complexity of the auxiliary hypothesis space H φ ?
Analysing the complexity of this space is difﬁcult, as it is coupled with the hypothesis space H W of the main model.
One can think of this hypothesis space as a subset of the original model hypothesis space H φ
= { h W : ∃ φ s.t.
W = arg
min W L T ( W, φ ) }
⊂ H W .
Due to the coupling with H W the behaviour can be unintuitive and we will show that even simple auxiliaries can have inﬁnite VC dimension.
Example: Consider the following 1D hypothesis space for binary classiﬁcation H W = {(cid:100) cos( Wx ) (cid:101) , W ∈ R } , which has inﬁnite VC-dimension.
Let the main loss be the zero-one loss and the auxiliary loss be h ( φ, W )
=
( φ − W ) 2 , namely, an L 2 regularization with a learned center.
As the model hypothesis space H W has inﬁnite VC-dimension there exists training and auxiliary sets of any size that are shattered by H W .
Therefore, for any labeling on the auxiliary and training sets we can let φ = ˆ φ , the parameter that perfectly classiﬁes both sets.
We then have that ˆ φ is the optimum of the training with this auxiliary loss and we get that H φ also has inﬁnite VC-dimension.
This important example shows that even apparently simple looking auxiliary losses can overﬁt due to the interaction with the model hypothesis space.
Thus, it motivates our use of a separate auxiliary set.
4. A NALYSING AN AUXILIARY TASK EFFECT
When designing or learning auxiliary tasks one important question we can try to investigate is what makes an auxiliary task useful?
Consider the following loss with a single auxiliary L T ( W, φ )
=
(cid:80) i (cid:96) main ( x ti , y ti , W ) + φ · (cid:96) aux
( x ti , y ti , W ) .
Here h = φ · (cid:96) aux .
Assume φ =
so we optimize W only on the standard main task loss.
We could check if d L A dφ | φ =0 > 0 , namely would it help to add this auxiliary task?
Proposition 1.
Let L T ( W, φ )
=
(cid:80) i (cid:96) main ( x ti , y ti , W ) + φ · (cid:96) aux
( x ti , y ti , W ) .
Suppose that φ =and that the main task was trained until convergence.
We have
[Equation]
i.e. the gradient with respect to the auxiliary weight is the inner product between the Newton methods update and the gradient of the loss on the auxiliary set.
Proof.
In the general case, the following holds d L A dφ = −∇ W L A ( ∇ 2 W L T ) − 1 ∇ φ ∇ W L T .
For linear combination, we have ∇ φ ∇ W L T =
(cid:80) i ∇ W (cid:96) aux
( x ti , y ti ) .
Since W is optimized till convergence of the main task we obtain ∇ φ ∇ W L T = ∇ W L T .
This simple result shows that the key quantity to observe is the Newton update, rather than the gradient which is often used (Lin et al., 2019; Du et al., 2018).
Intuitively, the Newton update is the important quantity because if ∆ φ is small then we are almost at the optimum and due to quadratic convergence a single Newton step is sufﬁcient for approximate converging to the new optimum.
E XPERIMENTS
We evaluate the AuxiLearn framework in a series of tasks of two types: combining given auxiliary tasks into a uniﬁed loss (Sections 5.1 - 5.3), and generating a new auxiliary task (Section 5.4).
Further experiments and analysis of both modules are given in Appendix C. Throughout all experiments, we use an extra data split for the auxiliary set.
Hence, we use four data sets: training set, validation set, test set, and auxiliary set.
The samples for the auxiliary set are pre-allocated from the training set.
To ensure a fair comparison, these samples are used as part of the training set by all competing methods.
Effectively, this means we have a slightly smaller training set for optimizing the parameters W of the primary network.
In all experiments, we report the mean performance (e.g., accuracy) along with the Standard Error of the Mean (SEM).
Full implementation details of all experiments are given in
Appendix B. We also provide our code to support the reproducibility of our experiments.
Model variants.
For learning to combine losses, we evaluated the following variants of auxiliary networks:
(1) Linear : a convex linear combination between the loss terms, (2) Linear neural network (Deep linear) :
A deep fully-connected NN with linear activations.
(3) Nonlinear : A standard feed forward NN over the loss terms.
For the segmentation task only, (4) ConvNet :
A CNN over the loss-images.
The expressive power of the deep linear network is equivalent to that of a 1-layer linear network.
However, from an optimization perspective, it was shown that over-parameterization introduced by the network depth could stabilize and accelerate convergence (Arora et al., 2018; Saxe et al., 2014).
All variants are constrained to represent only monotone non-decreasing functions.
5. A N ILLUSTRATIVE EXAMPLE
[Figure]
We ﬁrst present an illustrative example of how AuxiLearn changes the loss landscape and helps generalization in the presence of label noise and harmful tasks.
Consider a regression problem with y main = w (cid:63)T x + (cid:15)and two auxiliary tasks.
The ﬁrst auxiliary is helpful, y 1 = w (cid:63)T x + (cid:15) 1 , whereas the second auxiliary is harmful y 2 = ˜ w T x + (cid:15) 2 , ( ˜ w (cid:54) =
w (cid:63) ).
We let (cid:15) 0 ∼ N (0 , σ 2 main ) and (cid:15) 1 , (cid:15) 2 ∼ N (0 , σ 2 aux ) , with σ 2 main > σ 2 aux .
We optimize a linear model with weights w ∈ R 2 that are shared across tasks, i.e., no task-speciﬁc parameters.
We set w (cid:63) =
(1 , 1) T and ˜ w = (2 , − 4) T .
We train an auxiliary network to output linear task weights and observe the changes to the loss landscape in Figure 2.
The left plot shows the loss landscape for the main task, with a training set optimal solution w train .
Note that w train (cid:54) =
w ∗ due to the noise in the training data.
The loss landscape of the weighted train loss at the beginning ( t = 0 ) and the end ( t = T ) of
[Figure]
training is shown in the middle and right plots, respectively.
Note how AuxiLearn learns to ignore the harmful auxiliary and use the helpful one to ﬁnd a better solution by changing the loss landscape.
In Appendix C.3 we extend this experiment and show that the auxiliary task weight is inversely proportional to the label noise.
5. F INE - GRAINED CLASSIFICATION WITH MANY AUXILIARY TASKS
In tasks of ﬁne-grain visual classiﬁcation, annotating images requires that annotators are domain experts, and this makes data labeling challenging and expensive (e.g., in the medical domain).
In some cases, however, non-experts can annotate predictive visual attributes.
As an example, consider the case of recognizing bird species, which would require an ornithologist, yet a layman can describe the head color or bill shape of a bird.
These features naturally form auxiliary tasks, which can be leveraged for training concurrently with the main task of bird classiﬁcation.
[Table]
We evaluate AuxiLearn in this setup on the ﬁne-grained classiﬁcation dataset, CaltechUCSD Birds (CUB) 200-2011 (Wah et al., 2011).
The CUB dataset contains 200 bird species in 11,788 images, each is associated with a set of 312 binary visual attributes, which we use as auxiliaries.
Since we are interested in setups where optimization based on the main task alone does not generalize well, we demonstrate our method in a semi-supervised setting: we assume auxiliary labels for all images but only 5 and 10 labels per class of the main task (noted as 5-shot and 10-shot).
We compare AuxiLearn the following MTL and auxiliary learning baselines: (1) Single-task learning (STL): Training only on the main task.
(2) Equal: Standard multitask learning with equal weights to all auxiliary tasks.
(3) GradNorm: (Chen et al., 2018), an MTL method that scales losses based on gradient magnitude.
(4) Uncertainty: (Kendall et al., 2018), an MTL approach that uses task uncertainty to adjust task weights.
(5) Gradient Cosine Similarity (GCS): (Du et al., 2018), an auxiliary-learning approach that uses gradient similarity between the main and auxiliary tasks.
Dynamic weight averaging (DWA): (Liu et al., 2019b), an MTL approach that sets task weights based on the rate of loss change over time.
The primary network in all experiments is ResNet-18 (He et al., 2016) pre-trained on ImageNet.
We use a 5-layer fully connected NN for the auxiliary network.
Sensitivity analysis of the network size and auxiliary set size is presented in Appendix C.
Table 1 shows the test set classiﬁcation accuracy.
Most methods signiﬁcantly improve upon the STL baseline, highlighting the beneﬁts of using additional (weak) labels.
Our Nonlinear and Deep linear auxiliary network variants outperform all previous approaches by a large margin.
As expected, a non-linear auxiliary network is better than its linear counterparts.
This suggests that there are some non-linear interactions between the loss terms that the non-linear network is able to capture.
Also, notice the effect of using deep-linear compared to a (shallow) linear model.
This result is an indication that at least part of the improvement achieved by our method is attributed to the over-parameterization of the auxiliary network.
In the Appendix we further analyze the auxiliary networks.
In Appendix
C.5 we visualize the full optimization path of a linear auxiliary network over a polynomial kernel on the losses
[Table]
and in Appendix C.6 we show that the last state of the auxiliary network is not informative enough.
From both experiments, we conclude that the training dynamics and coupling between auxiliary and primary parameters is crucial.
5. P IXEL - WISE LOSSES
[Table]
We consider the indoor-scene segmentation task provided in (Couprie et al., 2013), that utilize the
NYUv2 dataset (Silberman et al., 2012).
We use the 13-class semantic segmentation as the main task, with depth prediction and surface-normal estimation (Eigen and Fergus, 2015) as auxiliaries.
Here, we use a SegNet (Badrinarayanan et al., 2017) based model with 44M parameters for the primary network, and a 4-layer CNN for the auxiliary network.
In this task, since the losses are given at the pixel level, we can apply the ConvNet variant of the auxiliary network to the loss image, in which each task forms a channel.
Table 2 reports the mean Intersection over Union (mIoU) and pixel accuracy for the main segmentation task.
Here, we also compare with MGDA (Sener and Koltun, 2018) which had extremely long training time in CUB experiments due to the large number of auxiliary tasks, and therefore wasn’t compared against in
Section 5.2.
All weighting methods achieve a performance gain over the STL model.
However, the comparison shows that the ConvNet variant of our auxiliary network outperforms all competitors in terms of test mIoU.
Figure 3 shows an example of the loss-images for the auxiliary (c) and main (d) tasks together with the pixel-wise weights (e).
First, note how the loss-images resemble to the actual input images.
This suggest that a spatial relationship can be leveraged by using a CNN auxiliary network.
Second, the pixel weights are a non-trivial combination of main and aux task losses.
In the top (bottom) row, the plant (couch) is having low segmentation loss and intermediate auxiliary loss.
As a result, a higher weight is allocated to these pixels which increase the error signal.
5. L EARNING AN AUXILIARY CLASSIFIER
[Table]
In many cases, designing helpful auxiliaries is challenging.
Here, we evaluate AuxiLearn in learning a multi-class classiﬁcation auxiliary tasks.
We do so on the following three multi-class classiﬁcation datasets: CIFAR10, CIFAR100 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and three ﬁne-grain classiﬁcation datasets: CUB-200-2011 (Wah et al., 2011), Oxford-IIIT Pet (Parkhi et al., 2012), and Cars (Krause et al., 2013).
Oxford-IIIT Pet contains 7349 images of 37 species of dogs and cats, and Cars contains 16,185 images of 196 cars.
The ofﬁcial splits were used in all datasets.
Following (Liu et al., 2019a), we learn a different auxiliary task for each class in the main task.
Intuitively, this allows us to uncover hierarchies within each main task class.
We set the number of classes to be learned to 5 for all experiments and all learned tasks.
To examine the effect of learnable auxiliary in the low-data regime, we evaluate the performance while training with only 5% of the training set in CIFAR10, CIFAR100 and SVHN datasets, and ∼ 30 samples per class in CUB,
Oxford-IIIT Pet and Cars.
We use VGG-16 (Simonyan and Zisserman, 2014) as the backbone for both CIFAR datasets, a 4-layers ConvNet for the SVHN experiment, and ResNet18 for the ﬁne-grain datasets.
In all experiments, the architectures of the auxiliary and primary networks were set the same, and were trained from scratch without pre-training.
We compared our approach against the following baselines: (1) Single-task learning (STL): Training the main task only.
(2) MAXL: Meta AuXiliary Learning (MAXL) recently proposed by (Liu et al., 2019a) for learning auxiliary tasks.
MAXL optimizes the label generator in a meta learning fashion.
(3) MAXL-F: A frozen MAXL label generator, that is initialized randomly.
It decouples the effect of having a teacher network from the additional effect brought by the training process.
Table 3 shows that AuxiLearn outperforms all baselines in all setups, even-though it sacriﬁces some of the training set for the auxiliary set.
It is also worth noting that our optimization approach signiﬁcantly improves upon MAXL, yielding × 3 improvement in the run-time.
In Appendix C.9 and C.10 we show additional experiments for this setup, including an extension of the method to point-cloud part segmentation and experiments with varying training data sizes.
Figure 4 presents a 2D t-SNE projection of the vector of ﬁve auxiliary (soft) labels that are learned using Auxilearn for samples of the main classes Frog (left) and Deer (right) from the CIFAR dataset.
t-SNE was applied to to each auxiliary task separately.
When considering how images are projected to this space of auxiliary soft labels, several structures emerge.
The auxiliary network learns a ﬁne partition of the Frog class that separates real images from illustrations.
More interesting, the soft labels learned for the class Deer have a middle region that only contains deers with antlers (in various poses and varying backgrounds).
This semantic feature captured by the learned label can help the primary network to discriminate between main task classes.
[Figure]
D ISCUSSION
In this paper, we presented a novel and uniﬁed approach for learning how to combine and how to learn auxiliary tasks.
We theoretically showed which auxiliaries can be beneﬁcial and the importance of using a separate auxiliary set.
We empirically showed that our method achieve signiﬁcant improvement over existing methods on various datasets and tasks.
This work opens interesting directions for future research.
First, when training deep linear auxiliary networks, we observed similar learning dynamics to those of non-linear models.
As a result, they generated better performance compared to their linear counterparts.
This effect was observe in standard training setup, but the optimization path in auxiliary networks is very different.
Second, we ﬁnd that shifting labeled data from the training set to an auxiliary set is consistently helpful.
A broader question remains about the most efﬁcient way to do this allocation.
Appendix: Auxiliary Learning by Implicit Differentiation A G RADIENT DERIVATION
We provide here the derivation of Eq.
(4) in Section 3.
One can look at the function ∇ W L T ( W, φ ) around a certain local-minima point ( ˆ W, ˆ φ ) and assume the Hessian ∇ 2 W L T ( ˆ W, ˆ φ ) is positivedeﬁnite.
At that point, we have ∇ W L T ( ˆ W, ˆ φ ) = 0 .
From the IFT, we have that locally around ( ˆ W, ˆ φ ) , there exists a smooth function W ∗ ( φ ) such that ∇ W L T ( W, φ ) =
if W = W ∗ ( φ ) .
Since the function ∇ W L T ( W ∗ ( φ ) , φ ) is constant and equal to zero, we have that its derivative w.r.t.
φ is also zero.
Taking the total derivative we obtain
[Equation]
Multiplying by ∇ 2 W L T ( W, φ ) − 1 and reordering we obtain
[Equation]
We can use this result to compute the gradients of the auxiliary set loss w.r.t φ
[Equation]
As discussed in the main text, fully optimizing W to convergence is too computationally expensive.
Instead, we update φ once for every several update steps for W , as seen in Alg. 1.
To compute the vector inverse-Hessian product, we use Alg. 2 that was proposed in (Lorraine et al., 2020).
B E XPERIMENTAL DETAILS B. CUB 200-
Data.
To examine the effect of varying training set sizes we use all 5994 predeﬁned images for training according to the ofﬁcial split and, we split the predeﬁned test set to 2897 samples for validation and 2897 for testing.
All images were resized to 256 × 256 and Z-score normalized.
During training, images were randomly cropped to 224 and ﬂipped horizontally.
Test images were centered cropped to 224 .
The same processing was applied in all ﬁne-grain experiments.
Training details for baselines.
We ﬁne-tuned a ResNet-18 (He et al., 2016) pre-trained on ImageNet (Deng et al., 2009) with a classiﬁcation layer on top for all tasks.
Because the scale of auxiliary losses differed from that of the main task, we multiplied each auxiliary loss, on all compared method, by the scaling factor τ = 0 .
It was chosen based on a grid search over { 0 . 1 , 0 . 3 , 0 . 6 , 1 . 0 } using the Equal baseline.
We applied grid search over the learning rates in { 1 e − 3 , 1 e − 4 , 1 e − 5 } and the weight decay in { 5 e − 3 , 5 e − 4 , 5 e − 5 } .
For DWA (Liu et al., 2019b), we searched over the temperature in { 0 . 5 , 2 , 5 } and for GradNorm (Chen et al., 2018), over α in { 0 . 3 , 0 . 8 , 1 . 5 } .
The computational complexity of GSC (Du et al., 2018) grows with the number of tasks.
As a result, we were able to run this baseline only in a setup where there are two loss terms: the main and the sum of all auxiliary tasks.
We ran each conﬁguration with 3 different seeds for 100 epochs with ADAM optimizer (Kingma and
Ba, 2014) and used early stopping based on the validation set.
The auxiliary set and auxiliary network.
In our experiments, we found that allocating as little as 20 samples from the training set for the auxiliary set and using a NN with 5 layers and 10 units in each layer yielded good performance for both deep linear and non-linear models.
We found that our method was not sensitive to these design choices.
We use skip connection between the main loss (cid:96) main and the overall loss term and Softplus activation.
Optimization of the auxiliary network.
In all variants of our method, the auxiliary network was optimized using SGD with 0.9 momentum.
We applied grid search over the auxiliary network learning rate in { 1 e − 2 , 1 e − 3 } and weight decay in { 1 e − 5 , 5 e − 5 } .
The total training time of all methods was 3 hours on a 16GB Nvidia V100 GPU.
B. NYU V
The data consists of 1449 RGB-D images, split into 795 train images and 654 test images.
We further split the train set to allocate 79 images, 10% of training examples, to construct a validation set.
Following (Liu et al., 2019b), we resize images to 288 × 384 pixels for training and evaluation and use SegNet (Badrinarayanan et al., 2017) based architecture as the backbone.
Similar to (Liu et al., 2019b), we train the model for 200 epochs using Adam optimizer (Kingma and Ba, 2014) with learning rate 1 e − 4 , and halve the learning rate after 100 epochs.
We choose the best model with early stopping on a pre-allocated validation set.
For DWA (Liu et al., 2019b) we set the temperature hyperparameter to 2, as in the NYUv2 experiment in (Liu et al., 2019b).
For
GradNorm (Chen et al., 2018)
we set α = 1 . 5 .
This value for α was used in (Chen et al., 2018) for the NYUv2 experiments.
In all variants of our method, the auxiliary networks are optimized using
SGD with 0.9 momentum.
We allocate 2 .
5% of training examples to form an auxiliary set.
We use grid search to tune the learning rate { 1 e − 3 , 5 e − 4 , 1 e − 4 } and weight decay { 1 e − 5 , 1 e − 4 } of the auxiliary networks.
Here as well, we use skip connection between the main loss (cid:96) main and the overall loss term and Softplus activation.
B. L EARNING AUXILIARIES
Multi-class classiﬁcation datasets.
On the CIFAR datasets, we train the model for 200 epochs using
SGD with momentum 0 . 9 , weight decay 5 e − 4 , and initial learning rates 1 e − 1 and 1 e − 2 for
CIFAR10 and CIFAR100, respectively.
For the SVHN experiment, we train for 50 epochs using
SGD with momentum 0 . 9 , weight decay 5 e − 4 , and initial learning rates 1 e − 1 .
The learning rate is modiﬁed using a cosine annealing scheduler.
We use VGG-16 (Simonyan and Zisserman, 2014) based architecture for the CIFAR experiments, and a 4-layer ConvNet for the SVHN experiment.
For
MAXL (Liu et al., 2019a) label generating network, we tune the following hyperparameters: learning rate { 1 e − 3 , 5 e − 4 } , weight decay { 5 e − 4 , 1 e − 4 , 5 e − 5 } , and entropy term weight { . 2 , .
6 } (see (Liu et al., 2019a) for details).
We explore the same learning rate and weight decay for the auxiliary network in our method, and also tune the number of optimization steps between every auxiliary parameter update { 5 , 15 , 25 } , and the size of the auxiliary set { 1 . 5% , 2 . 5% } (of training examples).
We choose the best model on the validation set and allow for early stopping.
Fine-grain classiﬁcation datasets.
In CUB experiments we use the same data and splits as described in Sections 5.2 and B.1.
Oxford-IIIT Pet contains 7349 images of 37 species of dogs and cats.
We use the ofﬁcial train-test split.
We pre-allocate 30% from the training set to validation.
As a results, the total number of train/validation/test images are 2576 / 1104 / 3669 respectively.
Cars (Krause et al., 2013) contains 16 , 185 images of 196 car classes.
We use the ofﬁcial train-test split and pre-allocate 30% from the training set to validation.
As a results, the total number of train/validation/test images are 5700 / 2444 / 8041 respectively.
In all experiments we use ResNet-18 as the backbone network for both the primary and auxiliary networks.
Importantly, the networks are not pre-trained.
The task speciﬁc (classiﬁcation) heads in both the primary and auxiliary networks is implemented using a 2-layer NN with sizes 512 and C .
Where C is number of labels (e.g., 200 for CUB and 37 for
Oxford-IIIT Pet).
In all experiments we use the same learning rate of 1 e − 4 and weight decay of 5 e − 3 which were shown to work best, based on a grid search applied on the STL baseline.
For
MAXL and AuxiLearn we applied a grid search over the auxiliary network learning rate and weight decay as described in the Multi-class classiﬁcation datasets subsection.
We tune the number of optimization steps between every auxiliary parameter update in { 30 , 60 } for Oxford-IIIT Pet and { 40 , 80 } for CUB and Cars.
Also, the auxiliary set size was tuned over { 0 .
33% } with stratiﬁed sampling.
For our method, we leverage the module of AuxiLearn for combining auxiliaries.
We use a Nonlinear network with either two or three hidden layers of sizes 10 (which was selected according to a grid search).
The batch size was set to 64 in CUB and Cars experiments and to 16 in Oxford-IIIT Pet experiments.
We ran each conﬁguration with 3 different seeds forepochs with ADAM optimizer and used early stopping based on the validation set.
C A DDITIONAL EXPERIMENTS C. I MPORTANCE OF A UXILIARY S ET
In this section we illustrate the importance of the auxiliary set to complement our theoretical observation in Section 4.
We repeat the experiment in Section 5.1, but this time we optimize the auxiliary parameters φ using the training data.
Figure 5 shows how the tasks’ weights change during training.
The optimization procedure is reduced to single-task learning, which badly hurts generalization (see Figure 2).
These results are consistent with (Liu et al., 2019a) that added an entropy loss term to avoid the diminishing auxiliary task.
C. M ONOTONOCITY
As discussed in the main text, it is a common practice to combine auxiliary losses as a convex combination.
This is equivalent to parametrize the function g ( (cid:96) ; φ ) as a linear combination over losses g
( (cid:96) ; φ ) = (cid:80) Kj =1 φ j (cid:96) j , with non-negative weights, φ j ≥ 0 .
Under this parameterization, g is a monotonic non-decreasing function of the losses, since ∂ L T /∂(cid:96)
j ≥ 0 .
The non-decreasing property means that the overall loss grows (or is left unchanged) with any increase to the auxiliary losses.
As a result, an optimization procedure that operates to minimize the combined loss also operates in the direction of reducing individual losses (or not changing them).
A natural question that arises is whether the function g should generalize this behavior, and be constrained to be non-decreasing w.r.t.
the losses as well?
Non-decreasing networks can "ignore" an auxiliary task by zeroing its corresponding loss, but cannot reverse the gradient of a task by negating its weight.
While monotonicity is a very natural requirement, in some cases, negative task weights (i.e., non-monotonicity) seem desirable if one wishes to "delete" input information not directly related to the task at hand (Alemi et al., 2017; Ganin and Lempitsky, 2015).
For example, in domain adaptation, one might want to remove information that allows a discriminator to recognize the domain of a given sample (Ganin and Lempitsky, 2015).
Empirically, we found that training with monotonic non-decreasing networks to be more stable and has better or equivalent performance, see
Table 4 for comparison.
Table 4 compares monotonic and non-monotonic auxiliary networks in both the semi-supervised and the fully-supervised setting.
Monotonic networks show a small but consistent improvement over non-monotonic ones.
It is also worth mentioning that the non-monotonic networks were harder to stabilize.
[Table]
C. N OISY AUXILIARIES
We demonstrate the effectiveness of AuxiLearn in identifying helpful auxiliaries and ignoring harmful ones.
Consider a regression problem with main task y =
w T x + (cid:15) , where (cid:15) ∼ N (0 , σ 2 ) .
We learn this task jointly with K = 100 auxiliaries of the form y j = w T x + | (cid:15) j | , where (cid:15) j ∼ N (0 , j · σ 2 aux ) for j = 1 , ..., 100 .
We use the absolute value on the noise so that noisy estimations are no longer unbiased, making the noisy labels even less helpful as the noise increases.
We use a linear auxiliary network to weigh the loss terms.
Figure 6 shows the learned weight for each task.
We can see that the auxiliary network captures the noise patterns, and assign weights based on the noise level.
C. CUB SENSITIVITY ANALYSIS
In this section, we provide further analysis for the experiments conducted on the CUB 200dataset in the 5-shot setup.
We examine the sensitivity of a non-linear auxiliary network to the size of the auxiliary set , and the depth of the auxiliary network .
In Figure 7a we test the effect of allocating (labeled) samples from the training set to the auxiliary set.
As seen, allocating between 10 − 50 samples results in similar performance picking at 20.
The ﬁgure shows that removing too many samples from the training set can be damaging.
Nevertheless, we notice that even when allocating 200 labeled samples (out of 1000), our nonlinear method is still better than the best competitor GSC (Du et al., 2018) (which reached an accuracy of 42 . 57 ).
Figure 7b shows how accuracy changes with the number of hidden layers.
As expected, there is a positive trend.
As we increase the number of layers, the network expressivity increases, and the performance improves.
Clearly, making the auxiliary network too large may cause the network to overﬁt the auxiliary set as was shown in Section 4, and empirically in (Lorraine et al., 2020).
C. L
[Table]
WEIGHTED NON - LINEAR TERMS
To further motivate the use of non-linear interactions between tasks, we train a linear auxiliary network over a polynomial kernel on the tasks segmentation, depth estimation and normal prediction from the NYUv2 dataset.
Figure 8 shows the learned loss weights.
From the ﬁgure, we learn that two of the three largest weights at the end of training belong to non-linear terms, speciﬁcally, Seg 2 and
Seg · Depth .
Also, we observe a scheduling effect, in which at the start of training, the auxiliary network focuses on the auxiliary tasks (ﬁrst ∼ 50 steps), and afterwards it draws most of the attention of the primary network towards the main task.
C. F IXED AUXILIARY
As a result of alternating between optimizing the primary network parameters and the auxiliary parameters, the weighting of the loss terms are updated during the training process.
This means that the loss landscape is changed during training.
This effect is observed in the illustrative examples
[Figure] [Figure]
described in Section 5.1 and Section C.5, where the auxiliary network focuses on different tasks during different learning stages.
Since the optimization is non-convex, the end result may depend not only on the ﬁnal parameters but also on the loss landscape during the entire process.
We examined this effect with the following setup on the 5-shot setting on CUB 200-2011 dataset: we trained a non-linear auxiliary network and saved the best model.
Then we retrain with the same conﬁguration, only this time, the auxiliary network is initialized using the best model, and is kept ﬁxed.
We repeat this using ten different random seeds, affecting the primary network initialization and data shufﬂing.
As a result, we observed a drop of 6.7% on average in the model performance with an std of 1.2% (46.7% compared to 40%).
C. F ULL CUB DATASET
In Section 5.2 we evaluated AuxiLearn and the baseline models performance under a semi-supervised scenario in which we have 5 or 10 labeled samples per class.
For completeness sake, we show in
Table 5 the test accuracy results in the standard fully-supervised scenario.
As can be seen, in this case the STL baseline achieves the highest top-1 test accuracy while our nonlinear method is second on the top-1 and ﬁrst on the top-3.
Most baselines suffer from severe negative transfer due to the large number of auxiliary tasks (which are not needed in this case) while our method cause minimal performance degradation.
[Table]
C
[Table]
C ITYSCAPES
Cityscapes (Cordts et al., 2016) is a high-quality urban-scene dataset.
We use the data provided in (Liu et al., 2019b) with 2975 training and 500 test images.
The data comprises of four learning tasks: 19-classes, 7-classes and 2-classes semantic segmentation
[Table]
and depth estimation.
We use the 19-classes semantic segmentation as the main task
[Table]
and all other tasks as auxiliaries.
We allocate 10% of the training data for validation set
[Table]
to allow for hyperparameter tuning and early stopping.
We further allocate 2 .
5% of the remaining training examples to construct the auxiliary set.
All images are resized to 128 × 256 to speed up computation.
We train a SegNet (Badrinarayanan et al., 2017) based model for 150 epochs using Adam optimizer (Kingma and Ba, 2014) with learning rate 1 e −
[Table]
and halve the learning rate after 100 epochs.
We search over weight decay in { 1 e − 4 , 1 e − 5 } .
We compare AuxiLearn to the same baselines used in Section 5.2 and search over the same hyperparameters as in the NYUv2 experiment.
We set the DWA temperature to 2 similar to (Liu et al., 2019b), and the GradNorm hyperparameter α to 1 . 5 , as used in (Chen et al., 2018) for the NYUv2 experiments.
We present the results in Table 6.
The ConvNet variant of the auxiliary network achieves best performance in terms of mIoU and pixel accuracy.
[Table]
C. L EARNING SEGMENTATION AUXILIARY FOR 3D POINT CLOUDS
Recently, several methods were offered for learning auxiliary tasks in point clouds (Achituve et al., 2020; Hassani and Haley, 2019; Sauder and Sievers, 2019); however, this domain is still largely unexplored and it is not yet clear which auxiliary tasks could be beneﬁcial beforehand.
Therefore, it is desirable to automate this process, even at the cost of performance degradation to some extent compared to human designed methods.
We further evaluate our method in the task of generating helpful auxiliary tasks for 3D point-cloud data.
We propose to extend the use of AuxiLearn for segmentation tasks.
In Section 5.4 we trained an auxiliary network to output soft auxiliary labels for classiﬁcation task.
Here, we use a similar
[Table]
approach, assigning a soft label vector to each point.
We then train the primary network on the main task and the auxiliary task of segmenting each point based on the learned labels
[Table]
We evaluated the above approach in a part-segmentation task using the ShapeNet part dataset (Yi et al., 2016).
This dataset contains 16,881 3D shapes from 16 object categories (including Airplane,
Bag, Lamp), annotated with a total of 50 parts (at most 6 parts per object).
The main task is to predict a part label for each point.
We follow the ofﬁcial train/val/test split scheme in (Chang et al., 2015).
We also follow the standard experimental setup in the literature, which assumes known object category labels during segmentation of a shape (see e.g., (Qi et al., 2017; Wang et al., 2019)).
During training we uniformly sample 1024 points from each shape and we ignore points normal.
During evaluation we use all points of a shape.
For all methods (ours and baselines)
we used the DGCNN architecture (Wang et al., 2019) as the backbone feature extractor and for part segmentation.
We evaluated performance using point-Intersection over Union (IoU) following (Qi et al., 2017).
We compared AuxiLearn with the following baselines: (1) Single Task Learning (STL): Training with the main task only.
(2) DefRec: An auxiliary task of reconstructing a shape with a deformed region (Achituve et al., 2020).
(3) Reconstructing Spaces (RS): An auxiliary task of reconstructing a shape from a shufﬂed version of it (Sauder and Sievers, 2019).
and (4) Denoising Auto-encoder (DAE): An auxiliary task of reconstructing a point-cloud perturbed with an iid noise from N (0 , 0 . 01) .
We performed hyper-parameter search over the primary network learning rate in { 1 e − 3 , 1 e − 4 } , weight decay in { 5 e − 5 , 1 e − 5 } and weight ratio between the main and auxiliary task of { 1 : 1 , 1 : 0 . 5 , 1 : 0 . 25 } .
We trained each method for 150 epochs, used the Adam optimizer with cosine scheduler.
We applied early stopping based on the mean IoU of the validation set.
We ran each conﬁguration with 3 different seeds and report the average mean IOU along with the SEM.
We used the segmentation network proposed in (Wang et al., 2019) with an exception that the network wasn’t supplied with the object label as input.
For AuxiLearn, we used a smaller version of PointNet (Qi et al., 2017) as the auxiliary network without input and feature transform layers.
We selected PointNet because its model complexity is light and therefore is a good ﬁt in our case.
We learned a different auxiliary task per each object category (with 6 classes per category) since it showed better results.
We performed hyper-parameter search over the auxiliary network learning rate in { 1 e − 2 , 1 e − 3 } , weight decay in { 5 e − 3 , 5 e − 4 } .
Two training samples from each class were allocated for the auxiliary set.
Table 7 shows the mean IOU per category when training with only 30 segmented point-clouds per object category (total of 480).
As can be seen, AuxiLearn performance is close to RS (Sauder and
Sievers, 2019) and improve upon other baselines.
This shows that in this case, our method generates useful auxiliary tasks that has shown similar or better gain than those designed by humans.
C. L EARNING AN AUXILIARY CLASSIFIER
In Section 5.4 we show how AuxiLearn learns a novel auxiliary to improve upon baseline methods.
For the ﬁne-grained classiﬁcation experiments, we use only 30 samples per class.
Here we also compare AuxiLearn with the baseline methods when there are only 15 images per class.
Tableshows that AuxiLearn is superior to baseline methods in this setup as well, even though it requires to allocate some samples from the training data to the auxiliary set.
To further examine the effect of learning novel auxiliary task with varying train set size, we provide here additional experiments on the CIFAR10 dataset.
We evaluate the methods with of 10%, 15% and 100% of training examples.
The results are presented in Table 9.
As expected, learning with
[Table] [Table]
auxiliaries is mostly helpful in the low data regime.
Nonetheless, AuxiLearn improves over single task learning and MAXL for all training set sizes.
D T HEORETICAL C ONSIDERATIONS
In this section, we discuss the theoretical limitations of AuxiLearn.
First, we discuss the smoothness of our loss criterion while learning to combine losses using DNNs. Next, we present limitations that may arise from utilizing the IFT and their resolution. Finally, we discuss the approximations made for achieving an efﬁcient optimization procedure. Smoothness of the loss criterion.
When learning to combine losses as described in Section 3.2, one must take into consideration the smoothness of the learn loss criterion as a function of W .
This limits, at least in theory, the design choice of the auxiliary network.
In our experiments we use smooth activation functions, namely Softplus, to ensure the existence of ∂ L T /∂W .
Nonetheless, using non-smooth activation (e.g. ReLU) results with a piecewise smooth loss function hence might work well in practice.
Assumptions for IFT.
One assumption for applying the IFT as described in Section 3.4, is that
L T is continuously differentiable w.r.t to the auxiliary and primary parameters.
This assumption limits the design choice of both the auxiliary, and the primary networks.
For instance, one must utilize only smooth activation functions.
However, many non-smooth components can be replaced with smooth counterparts.
For example, ReLU can be replaced with Softplus,
ReLU ( x ) =
lim α →∞ ln (1 + exp( αx )) /α , and the beneﬁcial effects of Batch-Normalization can be captured with Weight-Normalization as argued in (Salimans and Kingma, 2016).
For the setup of learning to combine losses , we use the above substitutes, namely Softplus and Weight
Normalization, however for the learning a novel auxiliary setup, we share architecture between primary and auxiliary network (e.g. ResNet18).
While using non-smooth components may, in theory, cause issues, we show empirically through extensive experiment that AuxiLean performs well in practice, and its optimization is stable.
Furthermore, we note that while ReLUs are non-smooth, they are piecewise smooth, hence the set of non-smoothness points is a zero-measure set.
Approximations.
Our optimization procedure relies on several approximations to efﬁciently solve complex bi-level optimization.
This trade-off between computation efﬁciency and accurate approximation can be controlled by (i)
The number of Neumann series components, and; (ii)
The number of optimization steps between auxiliary parameters update.
While we cannot guarantee that the bi-level optimization process converges, empirically we observe a stable optimization process.
Our work builds on previous studies in the ﬁeld of hyperparameter optimization (Lorraine et al., 2020; Pedregosa, 2016).
Lorraine et al.
(2020) provide an error analysis for both approximations, in a setup for which the exact Hessian can be evaluated in closed form.
We refer the readers to Pedregosa (2016) for theoretical analysis and results regarding the second approximation (i.e. sub-optimally of the inner optimization problem in Eq. 2).