N ESTED L EARNING FOR MULTI - GRANULAR TASKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Standard deep neural networks (DNNs) used for classiﬁcation are trained in an end-to-end fashion for very speciﬁc tasks - object recognition, face identiﬁcation, character recognition, etc.
This speciﬁcity often leads to overconﬁdent models that generalize poorly to samples that are not from the original training distribution.
Moreover, they do not allow to leverage information from heterogeneously annotated data, where for example, labels may be provided with different levels of granularity.
Finally, standard DNNs do not produce results with simultaneous different levels of conﬁdence for different levels of detail, they are most commonly an all or nothing approach.
To address these challenges, we introduce the problem of nested learning : how to obtain a hierarchical representation of the input such that a coarse label can be extracted ﬁrst, and sequentially reﬁne this representation to obtain successively reﬁned predictions, all of them with the corresponding conﬁdence.
We explicitly enforce this behaviour by creating a sequence of nested information bottlenecks.
Looking at the problem of nested learning from an information theory perspective, we design a network topology with two important properties.
First, a sequence of low dimensional (nested) feature embeddings are enforced.
Then we show how the explicit combination of nested outputs can improve both robustness and ﬁner predictions.
Experimental results on CIFAR-10,
MNIST, and FASHION-MNIST demonstrate that nested learning outperforms the same network trained in the standard end-to-end fashion.
Since the network can be naturally trained with mixed data labeled at different levels of nested details, we also study what is the most efﬁcient way of annotating data, when a ﬁxed training budget is given and the cost of labels increases with the levels in the nested hierarchy.
I NTRODUCTION
Deep learning is providing remarkable computational tools for the automatic analysis and understanding of complex high-dimensional problems [Esteva et al.
; Spanhol et al.
; Parkhi et al.
Despite its tremendous value and versatility, methods based on Deep Neural Networks (DNNs) tend to be overconﬁdent about their predictions and limited to the task and data they have been trained on [Hein et al.
; Guo et al.
; Nguyen et al.
This happens, among other reasons, because the standard approach to train DNN models consists in optimizing its performance over a speciﬁc dataset and for a speciﬁc task in an end-to-end fashion Szegedy et al.
Standard DNNs are not designed to be trained with data of different quality and to simultaneously provide results at multiple granularities.
Take as an example the case illustrated in Figure 1 (left); for high quality facial images, we may be able to infer the person’s age group and identity; whereas for poor resolution or occluded examples, only a sub-set of these nested predictions may be achievable.
We expect the network to automatically understand what can and cannot predict, and this is obtained with the framework proposed in this paper (Figure 1, right).
Moreover, nested learning allows us to leverage training information from diverse datasets, with varying granularity and quality of labels, and combine this information into a single model.
On the other hand, when heterogeneous data with different quality and granularity of annotations (as in the example illustrated in Figure 1 (left)) is provided for training, low quality samples with coarse labels can help us to understand the structure of the coarser distributions (person, under 50) while simultaneously data with ﬁner labels can contribute to the coarse and ﬁne tasks.
This will be formalized later in the paper with tools from information theory.
Recently
Alsallakh et al.
(2017) showed that convolutional neural networks (CNNs) naturally tend to learn hierarchical high-level features that discriminate groups of classes in the early layers, while the deeper layers develop more specialized feature detectors.
We explicitly enforce this behaviour by creating a sequence of nested information bottlenecks.
Looking at the problem of nested learning from an information theory perspective we design a network topology with two important properties.
First, a sequence of low dimensional (nested) feature embeddings are enforced for each level in the taxonomy of the labels.
This encourages generalization by forcing information bottlenecks [Tishby et al.
Shwartz-Ziv & Tishby (2017)].
Second, skipped connections allow ﬁner embeddings to access information of the input that may be useful for ﬁner classiﬁcation but not informative on coarser categories Ronneberger et al.
Additionally, we show how the explicit calibration and combination of nested outputs can improve the ﬁner predictions and improve robustness.
Finally, having the ﬂexibility of merging data with different levels of granularity inspired us to study which is the most efﬁcient way of annotating data given a ﬁxed budget that takes into account that more detailed training data is more expensive to annotate.
The source code associated to this work is open source.
The main contributions of this paper are: (1) We introduce the concept of explicit nested learning, where a given level in the hierarchy strictly contains the previous one and strictly adds information; (2) We provide a deep learning architecture that can be trained with data from all levels of the nested structure (all levels of labels and data quality), each one affecting the corresponding component of the network; (3) We provide a model with multiple outputs, one per level of the nested hierarchy, each one with its own conﬁdence; the user does not need to know the “quality” of the data beforehand, the output conﬁdences provide that information.
R ELATED W ORK
The problem of adapting DNNs models and training protocols to encourage nested learning shares similarities with other popular problems in machine learning such as Multi-Task Learning (MTL).
Though our work is related to MTL because of the similar training challenges, most of the MTL methods tackle the task in a parallel way, without imposing a task hierarchy in the architecture [Ranjan et al.
, Kokkinos (2017), Bilen & Vedaldi (2016)].
The idea of learning hierarchical representations to improve classiﬁcation performance has been exploited prior the proliferation of
DNNs, e.g., [Zweig & Weinshall (2007), Fergus et al. (2010), Zhao et al.
(2011), Liu et al.
Some of these ideas have been incorporated into deep learning methods [Alsallakh et al.
Wehrmann et al.
, Deng et al.
(2014), Srivastava & Salakhutdinov (2013)], and exploited in speciﬁc applications [Clark et al.
(2017), Xuehong Mao et al.
, Seo & shik Shin (2019)].
Kim et al.
(2018) proposed a nested sparse network architecture with the emphasis on having a resource-aware versatile architecture to meet (simultaneously) diverse resource requirements.
Wehrmann et al.
(2018) proposed a neural network architecture capable of simultaneously optimizing local and global loss functions to exploit local and global information while penalizing hierarchical violations.
Triguero & Vens (2016) investigated different alternatives to label hierarchical multi-label problems by selecting one or multiple thresholds to map output scores to hierarchical predictions, focusing on performance measures such as the H-loss, HMC-loss and the micro-averaged
F-measure.
Yan et al.
(2015) introduced hierarchical deep CNNs (HD-CNNs) which consists of embedding CNNs into a two-level category hierarchy.
They propose to distinguish a coarse class using an initial classiﬁer and then reﬁne the classiﬁcation into a second level for each individual coarse category.
Although the works listed above are important, relevant, and related to the work presented here, there are notable differences between them and what we propose.
For example, while Kim et al.
(2018) propose a nested architecture providing different (potentially nested) outputs, they do not study how to combine these outputs into a reﬁned single prediction, nor provide a reliable conﬁdence measure associate to them.
Furthermore, the architecture they propose has key differences with ours, while they propose a nested hierarchy in a end-to-end fashion (i.e., features associated to the coarse and ﬁne levels are shared from the top to the bottom of the network), we enforce sequential information bottleneck.
As we show in the following sections this sequence of coarse to ﬁne low dimensional representations facilitate a robust calibration and combination of nested outputs.
Yan et al.
(2015) study the problem of nested learning for two nested levels of granularity, and optimize for a ﬁnal ﬁne prediction.
Their design is speciﬁc for a two-level category hierarchy while our work generalize to any number of nested levels, and we simultaneously can train and test up to an arbitrary level of granularity.
Another important difference is that the focus of their work is on the implementation details and performance of their two-hierarchy network versus traditional end-to-end learning.
As they mention in the conclusion of their work, future work should aim to extend their ideas to more than two hierarchical levels and to contextualize their empirical results into a theoretical framework.
Our work takes steps in these two speciﬁc directions.
This is also a fundamental difference with the works developed by Triguero & Vens (2016) and Wehrmann et al.
Moreover, all these approaches suppose that every sample is annotated for all the granularity levels.
In contrast with them, we can efﬁciently train (and predict) our model on dataset that provide only coarse labels, intermediate levels, or ﬁne labels.
Our approach is therefore more general in that regard.
Also, we show that if testing conditions shift from the ones on training, we can still provide relatively conﬁdent coarser labels while avoiding overconﬁdent (erroneous) ﬁne predictions (see, e.g., Figure 1, right; and Figure 5 in the supplementary material).
Also, the questions of neural network overconﬁdence and output readability are never addressed in those works, which is crucial for both results interpretation, and output combination.
Finally, because our solution can leverage information from datasets with different granularity, we are able to analyze how different proportions of training coarse and ﬁne data affects models cost, performance and robustness.
N ESTED LEARNING
Let us assume we want to classify the popular hand written digits of MNIST LeCun & Cortes (2010).
An input image can be represented as a realization x of the random variable X .
We denote as X the alphabet of X .
Associated to x , there is a label y that corresponds to the actual number the person writing the character wanted to represent.
The label y is a realization of the random variable Y .
In this illustrative case, Y can take 10 different values: Y = { 0 , 1 , ..., 9 } .
Of course Y and X are not independent random variables.
Generally, Y precedes X , and the problem of classiﬁcation can be stated as the problem of inferring y from an observed sample x , i.e., Y → X → ˆ Y .
ˆ Y denotes a new random variable (estimated from X ) which approximate Y .
More precisely, a common practice is to ﬁnd a mapping X
→ ˆ Y such that the probability P ( ˆ Y (cid:54)
=
Y | X ) is minimized.
Nested classiﬁcation.
In this work we focus on the inherent hierarchical structure most classiﬁcation problems have.
For example, imagine now that we have hand written characters including numbers, lower case letters, and capital letters.
It would be intuitive to ﬁrst attempt to classify these characters into three categories: numbers, lower case letters, and capital letters.
Then, depending on this coarse classiﬁcation we can perform a ﬁner classiﬁcation, i.e., classifying the numbers into 0 − 9 classes, the letters into a − z , and so forth.
Of course, we could have an arbitrary number of nested random variables associated to different levels of labels granularity.
Here, subscripts indicate the granularity of the label, for example, Y i −is the closest coarse level of Y i .
Y ki represents the random variable associated to each k value in the closest coarse node, i.e., Y ki represents Y i given that y i − 1 = k , k ∈ Y i − 1 .
Deﬁnition 3.1.
We deﬁne Y 1 , ..., Y n as a discrete sequence of nested labels if H ( Y i | Y i +1 )
=∀ i ∈ [1 , n − 1] .
H denotes the standard deﬁnition of entropy for discrete random variables.
Deﬁnition 3.2.
A discrete sequence of nested labels Y 1 , ..., Y n is strictly nested if H ( Y i | Y i − 1 ) <
H ( Y i ) ∀ i ∈ [2 , n ] .
The core of this work is to formulate classiﬁcation problems such that the information of the input is extracted in a hierarchical way.
To this end, intermediate (coarse) predictions ˆ Y i are jointly learned.
The two key components of the proposed approach are: (i) nested information bottlenecks and (ii) a combination of the predicted coarse and ﬁne labels.
The analysis provided in this section is agnostic to most of the implementation details (which are addressed in coming sections).
Hierarchical information bottlenecks and the role of skipped connections.
We assume the random variable X contains information about the sequence of strictly nested labels Y i , i.e.,
H ( Y i | X ) <
H ( Y i ) , as illustrated in Figure 2.
More precisely, H ( Y i | X ) >
H ( Y i − 1 | X ) and
H ( Y i | X ) <
H ( Y i
| Y i − 1 , X ) .
To exploit these properties, we use standard DNN layers (convolutional, pooling, normalization, and activation layers).
As illustrated in Figure 3, we begin by guiding the network to ﬁnd a low dimensional feature representation f 1 such that H ( f 1 ( X ))
(cid:28) H ( X ) while, I ( f 1 ( X ) , Y 1 ) is close to I ( X, Y 1 ) where I ( · , · ) stands for the standard mutual information.
(DNNs are remarkably efﬁcient at compressing and extracting the mutual information between high dimensional inputs and target labels [Shwartz-Ziv & Tishby (2017), Moshkovitz & Tishby (2017)].)
f 1 is then jointly exploited in addition with complementary
This leads to a second representation f classiﬁcation obtained.
repeated classiﬁcation achieved.
It is important to highlight that this high level description of the proposed model can be implemented in multiple ways.
In the following sections we present our own implementation and provide details of the speciﬁc architecture we chose for experimental validation.
The second step consists in learning the complementary information such that when combined with the representation f 1 , allows us to achieve a second representation f 2 from which the second hierarchical label Y 2 can be inferred.
To this end, skipped connections play a critical role as will be further detailed next.
As we discussed before,
[Equation]
On the other hand, we want each feature embedding f i to compress the information of X while
I ( f i ( X ) , Y i ) ≈
I ( X, Y i ) .
Equation 1 means that the ﬁner the classiﬁcation the more information from X we need.
Notice that while in many DNNs, skipped connections have proved to help on the networks compactness and to mitigate vanishing gradients Szegedy et al.
(2017), in the present work they are included for a more fundamental reason.
(See the experiments presented in the supplementary material Section F.2 for complementary empirical evidence and validation.)
If we do not consider skipped connections, X → f i ( X ) → f i +1 ( X ) forms a Markov chain where
I ( X, f i +1 ( X )) ≤ I ( X, f i ( X ))
(data-processing inequality) which contradicts Equation 1.
Section 4 presents experiments illustrating the impact of skipped connections on nested learning, and complements the discussion started here.
Combination of nested outputs.
We will present in the following sections experimental evidence showing that nested learning leads to an improvement in performance and robustness.
In addition, the explicit combination of nested predictions, since the network simultaneously produces all the outputs
Y i (with corresponding conﬁdence), can improve the accuracy and robustness even further.
Our motivation is to explicitly reﬁne the ﬁne prediction leveraging the information of all the coarser outputs, i.e., { ˆ Y 1 , ..., ˆ Y i } → ˜ Y i .
Let us deﬁne s i ( q ) the network output score associated to the event Y i = q .
In general, if s i ( q ) >
s i ( w ) most likely P ( Y i = q ) >
P ( Y i = w ) , but P ( Y i = q ) (cid:54) =
s i ( q ) .
In other words, a score value of 0 .
3 does not mean the sample belongs to this class with 30% probability.
This can be addressed by calibrating the outputs, which consists of mapping output scores into an estimation of the class probability
s i ( q ) →
P ˆ Y
i ( q ) , as deﬁned and thoroughly explained in Zadrozny & Elkan (2002).
P
ˆ Y
i ( q ) denotes the calibrated output of the network which approximates P ( Y i = q ) .
(We will precise how calibration is performed in the following section.)
Then, we can use the estimated probability associated to a ﬁne label P ˆ Y i to compute the conditional probability P ( Y i =
y i
| Y i − 1 = k ) .
This is achieved by re-normalizing the ﬁner labels associated to the same coarse label, i.e.,
[Equation]
where Y k q i denotes the set of labels at granularity level i that share with q the same coarser label k q .
Finally, the estimated conditional probability is combined with the prior of the coarser prediction to recompute the ﬁne prediction P (cid:48)
ˆ Y i ( q ) =
P
ˆ Y i | ˆ Y i − 1 ( q
) P ˆ Y i − 1 ( k q ) .
3. I MPLEMENTATION C HALLENGES AND D ETAILS
Training.
Let G θ,η ( x ) =
( f i ( x, ( θ j ) j =1 ,..,i ) , g i ( f i , η i ))
i =1 ,..,m be the function coded by our net- work, where m denotes the number of granularity levels and as before x an input sample.
Each sub-function g i corresponds to the output of granularity i (computed from the feature bottleneck f i ).
G depends on parameters ( θ j ) j =1 ,..,i which are common to the sub-functions of coarser granularities, and some granularity-speciﬁc parameters η i .
The general framework of the architecture follows
Figure 3, meaning a trunk of convolutionnal ﬁlters with parameters θ and fully connected layers for each intermediate outputs with parameters η .
Training this kind of model with a disparity of samples per granularity is hard, and naively sampling random batches of training data leads to a noisy gradient computation Kokkinos (2017).
(See the supplementary material Section F.3 for complementary experimental validation and analysis.)
In order to overcome this issue, we organize the training samples and train the network in a cascaded manner.
The dataset D is organized in subsets of samples labeled up to granularity i for i = 1 , .., m .
Formally we can write D =
( x , y ) with x the set of inputs and y the set of labels.
We consider that x = ( x i ) i
= 1 ,.., m and y =
( y i )
i = 1 ,.., m where D i =
( x i , y i ) represents the subset of data for which the label is known up to the granularity level i .
Having the dataset partitioned in this fashion naturally leads to a cascaded training of the network.
We train the model to solve a sequence of optimization problems using ( x i , y i ) as the training examples at each step.
We can write this sequence ( P i ) as: ( P i ) : min ( θ j ,η j ) j =1 ,..,i (cid:80) ij =1 α j L n j ( ˆ Y j , Y j ) , where L n is the n -categorical cross-entropy and α are the weights for each individual loss.
We ﬁrst start to train the network on coarse labels and we gradually add ﬁner labels and start optimizing deeper parameters in an iterative way.
We support our cascaded training protocol against a more standard one with experimental results presented in the supplementary (Section F.3).
(Additional details are presented as supplementary material Section B.)
Scores calibration.
As shown by Hein et al.
, deep neural networks with ReLU activations tend to produce overconﬁdent results for samples which are out of the training distribution.
In order to mitigate this and have meaningful outputs that can be combined, we consider a two step calibration method.
The ﬁrst step consists in adding a “rejection” class for each level of granularity.
Synthetic samples associated to this class are generated from an uniform distribution.
By training the network with this supplementary class we mitigate the problem of overconﬁdence for sample that are far from the training distribution.
However, keeping a ﬁxed coverage on the input space would require a number of samples that grows exponentially with the dimension of the input space.
To overcome this problem, we think of the network as a sequence of encoders in the latent spaces g i ( f i ( x ))
(deﬁned as the penultimate layer prior each prediction), and ﬁne-tune the last layer adding synthetic out-of-distribution samples from an uniform distribution in the latent space.
The latent space has a much smaller dimension than the original input space, and therefore is tractable to synthesise samples with a uniform coverage of the latent space.
(Additional details are provided in the supplementary material Section B.)
The second calibration step is a classic temperature scaling introduced in Guo et al.
This technique consists in scaling the output of the fully-connected layer before the softmax activation by an optimal temperature parameter.
Given x the input data and g the function coded by a neural network before the softmax activation σ , the new calibrated output is given by ¯ g = σ ( gT ) .
The temperature parameter T is tuned so that the mean conﬁdence of the predictions matches the empirical accuracy, more precisely, we want to minimize, E (cid:104)
| P ( ˆ Y
=
Y | ˆ p = p ) − p | (cid:105) , whereas before, ˆ Y denotes the network prediction of Y , and ˆ p is the empirical conﬁdence associated to it.
The previous expression can be approximated, using a validation partition of the data, by computing the Expected
Calibration Error (ECE) ECE = (cid:80) Nj =1 | B j | n | acc ( B j ) − conf
( B j ) | , Naeini et al. (2015).
This measure takes the weighted average between the accuracy and conﬁdence on N bins B j , j = 1 ...N .
n denotes the total number of samples, and | B j | the number of samples on the bin B j .
E XPERIMENTS AND DISCUSSION
We consider three sets of publicly available datasets for experimental evaluation: the handwritten digits from MNIST (LeCun & Cortes (2010)), the small clothes images from Fashion-MNIST (Xiao et al.
and CIFAR10 (Krizhevsky et al.).
First we study how the actual nested nature of the labels affects nested learning.
Then we compare end-to-end training versus the proposed nested approach.
In a third group of experiments we evaluate different combination methods to leverage all the predictions into a reﬁned ﬁne prediction.
Then we evaluate the impact of skipped connections.
Finally, we study how the ratio of coarse and ﬁne labels affects the network’s performance.
Additional experiments are presented in the supplementary material, section A, F
Hierarchical structure versus random grouping.
We group the MNIST samples into two groups of nested labels: “VG” and “RG,” both with three labels of granularity.
VG corresponds to a hierarchy that groups visual similarities; the coarse class groups digits into {{ 3 , 8 , 5 , 0 , 6 } , { 9 , 4 , 7 , 1 , 2 }} , the intermediate class into {{ 3 , 8 , 5 } , { 0 , 6 } , { 9 , 4 , 7 } , { 1 , 2 }} , and of course the ﬁne class into { 0 } − { 9 } .
Additional information on how we grouped the labels for the three datasets is presented in Appendix C. RG consists of a 3 level (coarse/middle/ﬁne) grouping based on the order of N .
Empirical results shows that grouping the labels based on visual similarities leads to better results in terms of accuracy, intuitively supporting the idea of nested learning.
(See the results presented in
Table 4 provided in the supplementary materials.)
Standard end-to-end training versus nested learning.
Let us deﬁne |D
i | as the number of training samples that are annotated up to the level of granularity i .
To understand if adding more samples with coarse annotation helps improving the performance on the ﬁne task, we compared models trained exclusively with ﬁne data D A
= D 3 and models trained ﬁne data plus coarsely annotated data D B =
D 3 + D 2 + D 1 .
Table 1 shows the accuracy for the ﬁne, middle, and coarse outputs for the standard end-to-end network versus the same architecture trained using nested learning and additional coarse and middle data.
If we compare the lines for which D 3 = 20% , we observe that adding coarse and middle granularity samples improves the accuracy even for the ﬁne task.
Moreover, it also improves the robustness of the model.
We tested this models when test data is distorted and shifted from the conditions at training.
Distortion 1 to 4 correspond to four levels (increasing the severity) of “turbulence-like” image distortion, the implementation of this distortion is inspired on the work of
Meinhardt-Llopis & Micheli (2014) (details are provided in the supplementary material, Section D).
Again looking at Table 1 we see that the model trained with additional coarse and middle samples is more robust and less overconﬁdent.
A similar pattern is observed for MNIST and fashion-MNIST datasets (see for example, tables 5, and 6 in the supplementary material).
Complementing these results, Section F.1 in the supplementary material compared the proposed nested architecture with a
Multi-Task learning approach.
[Table]
The previous discussion is interesting as it shows that including additional coarse data tends to help also the discrimination of the ﬁne task.
However one may argue that the comparison is unfair, as one model sees more data than the other.
That is a very interesting point and we address it in following experiments where we study how the proportion of ﬁne and coarse granularity data affect performance (for a ﬁxed budget and different cost models).
For now, let us observe what happens if we increase the amount of ﬁne data from D 3 = 20% to 32% (which assuming a linear cost model equals the budget of training with D
= D
= D 3 = 20% ).
As expected, (see Table 1) the performance on clean test data improves for the end-to-end model.
However, (see columns
Distortion 1-4) it generalizes less to unseen (distorted) test data and also becomes signiﬁcantly more overconﬁdent.
Output combination.
Kuncheva (2004) presented many useful combination methods for both onehot encoding classiﬁers and continuous scores.
We compare some of those combination methods (e.g., Mean, Product, and Majority Vote) to the one that we designed speciﬁcally for multiple nested outputs.
(Details and numerical results are provided in the supplementary material, Section E and
Table 3.)
We observed that different combination methods perform similar on test data that matches the train data, while the proposed method outperforms the others when test samples are distorted.
The role of skipped connections.
Skipped connections are included in order to allow information ﬂow from the input to the ﬁner feature bottlenecks.
Table 2 shows the accuracy for two networks with the same structure, trained on the same data, one including skipped connections (SC) and the other-one not.
Section F.2 in the supplementary material complements these results, and provides empirical measurements of the mutual information between different components of the model with and without skipped connections.
[Table]
Working on a budget.
As we discussed before, establishing a fair comparison between models trained using more ﬁne or coarse data is not trivial.
To address this problem, we investigated three cost models: linear, convex, and concave.
For the linear model, we assume the cost associated to annotate one sample ( x, y i ) is proportional to |Y i | .
Analogously, for the convex/concave cost model, we assume the cost of annotating a sample ( x, y i ) is proportional to g ( |Y i | ) with g ( · ) being strictly convex/concave.
For each budget and cost model, we created hundreds of train sets with different amounts of coarse, middle, and ﬁne samples.
Figure 4 shows the accuracy on the MNIST test set for different budgets and cost models.
It is interesting to observe that for a convex cost model, increasing the number of coarse annotations produces better results for the same budget; while as expected, the opposite is observed for a concave cost model.
We present additional experiments in the Figure 10 in the supplementary material.
C ONCLUSION
In this work we introduced the concept of nested learning , which improves classiﬁcation accuracy and robustness.
Moreover, it allows to leverage information from datasets annotated with different levels of granularity.
Additionally, experiments suggest that nested models have a very desired behaviour, e g., they gradually break as the quality of the test data deteriorates.
We showed that implementing nested learning using a hierarchy of information bottlenecks provides a natural framework to also enforce calibrated outputs, where each level comes with its conﬁdence value.
Given a ﬁxed budget and model we studied what is the economically most efﬁcient strategy for labeling data.
Furthermore, experimental results show that if the amount of ﬁne training samples is constant, then adding samples with only a coarse annotation increases the performance and robustness for the ﬁne task.
To recap, the introduced nested learning framework performs as expected from our own human experience, where for good data we can provide high level inference with high conﬁdence; and when the data is not so good, we can still provide with high conﬁdence some level of inference on it.
Supplementary Material A A DDITIONAL OUTPUT EXAMPLES
D shown in Table 1 the performance of both networks similar distribution
, but the data is
B I MPLEMENTATION D ETAILS B. A RCHITECTURE
The architecture of our model is presented in Figure 6.
We evaluate the same architecture for MNIST and Fashion-MNIST as the images have the same size and number of channels.
For CIFAR10, the architecture of our model is very similar but with an increased depth of the convolutional ﬁlters.
Classifying images from CIFAR10 is indeed a harder problem than classifying images from MNIST or Fashion-MNIST, and therefore, it requires a model with more parameters.
The architecture we use is an adaptation of the U-Net, designed to ﬁt our proposed framework of nested information bottlenecks.
The U-Net is indeed a good starting point, as it meets most of the criteria that we presented in Section 3.
First, it consists of a convolutional network that enforces a bottleneck representation.
Second, it presents skip connections that allow information of the input to ﬂow to deeper components of the network.
We also added Batch-Normalization (BN) layers after every convolutional ﬁlters in order to achieve a faster convergence.
BN is very helpful to mitigate the vanishing gradient phenomenon.
Also, since it introduces randomness during training, BN acts as a regularization.
Classic L1 regularization was shown to be similar to BN in Luo et al.
(2018), while including BN layers also improves training speed and stability as explained by Santurkar et al.
B. T RAINING
We train the models in an iterative way.
First we optimize the weights for the coarse prediction with the samples that are only coarsely annotated and freeze the remaining weights.
Then, we optimize the weights up to the intermediate output with the samples that are coarsely and intermediately annotated.
Finally we train the whole network, with the samples that are coarsely, intermediately, and ﬁnely annotated.
(In general, this process has as many steps as levels of granularity.)
Each training step is performed using ADAM optimizer with different learning rates: 2 × 10 − 3 for the ﬁrst step, 1 × 10 − 3 for the second, and 5 × 10 − 4 for the last.
We stop the training of each step when stagnation of the validation loss is observed.
B. C ALIBRATION
The calibration consists of two main steps.
First a “rejection” class is modeled using an uniform distribution on the latent space.
This models out-of-distribution samples and mitigates overconﬁdent predictions on portions of the feature space where no training data is observed.
The second step consists of temperature scaling to convert output scores into approximations of class probabilities.
The “rejection” class.
For every level of granularity i and for every sample of the training dataset, we store the normalized outputs of the global averaging layer (GAP) in a dataset D i .
The samples have size s i and are normalized with respect to the L2 norm, therefore, they live in B s i (0 , 1) , the unitary sphere in R s i centered in zero.
We randomly sample n i new instances from an uniform distribution in B s i (0 , 1) .
These samples (associated to a new “rejection” class) are aggregated to D i and the fully connected layers ﬁne tuned.
3 Naturally, the larger | D i | and s i , the larger n i should be.
We set n i ∝ | D i |×S ( s i ) where S ( s i ) is the area of the hypersphere of unitary radius in the s idimensional space.
Figure 7 illustrates for a one dimensional toy example, how the proposed ideas provide and efﬁcient solution to reduce outputs overconﬁdence on out-of-distribution input samples.
[Figure]
Temperature scaling.
Temperature scaling improves calibration by converting arbitrary score values into an approximation of class probabilities.
As explained in Section 3.1, we minimize the empirical ECE metric over the validation set, composed of 5000 unseen samples for all the dataset we tested our method on.
To ﬁnd the optimal temperature T , we compute the ECE over 50 values between 1 and 3 and select the one for which the ECE is minimized.
If T = 1 the model is already well calibrated.
On all our experiments, the minimal ECE value was always reached for T values strictly lower than 3.
C L ABEL G ROUPING
In Section 4, we showed that nested learning performs better if the taxonomy of the labels has a real nested structure (e.g., one based on visual similarity).
To group ﬁne labels into a meaningful nested structure, we ﬁrst train a shallow neural network and classify images into the ﬁne classes.
Then we used the confusion matrix M associated to this auxiliary classiﬁer to establish which classes are closer to each other.
For MNIST and Fashion-MNIST for example, we wanted to group the labels in 2 coarse categories which also contained 2 intermediate categories.
To this end, we ﬁnd the 10-permutation l applied to both the rows and columns of M , such that the non-diagonal 5 × 5 matrices of M had the lowest possible L1-norm.
We iterate this process to ﬁnd the intermediate categories.
It is computationally hard to go through all the permutations, therefore, we follow the ideas proposed by Behrisch et al.
(2016) to perform matrix reordering with a reduced complexity.
Figure 8 presents the groups of labels we obtained for MNIST, Fashion-MNIST, and CIFAR10.
Our results show natural and intuitive taxonomies, see, e.g., how MNIST digits are grouped according to a natural shape-oriented similarity, with 3 and 8 in the same intermediate class for example.
[Figure]
D P ERTURBATIONS
Selecting realistic and meaningful perturbations to test DNN models is a non trivial problem.
For example, adding Gaussian noise mainly affects the high frequency components of the input images and we observed that both standard (end-to-end) and nested networks were not severely affected by this type of perturbation.
In this work we focus on structural deformations inspired by a model of turbulence.
The pseudo-code of this perturbation is presented in Algorithm 1 and was inspired in the work of Meinhardt-Llopis & Micheli (2014).
Figure 9 illustrates the distortion of an example image from MNIST dataset for different levels of turbulence intensity .
Algorithm
Data: the input: I ∈ R w,h , the parameters ( S, T )
∈ R
Result: the distorted image
I ( S,T ) dist
Creating a vector ﬁeld (u,v) for the distortion: u, v = normal noise (( w, h )) , normal noise (( w, h ))
u, v = gaussian filter ( u, S ) , gaussian filter ( v, S ) u, v = u
×
T std ( u ) , v ×
[Equation]
( v )
Interpolate the image with the obtained vector ﬁeld:
I ( S,T ) dist = bilinear interpolate ( I, u, v )
E C OMBINATION METHODS
Combining multiple classiﬁers is a standard approach in machine learning.
However, most approaches combine similar outputs and are not designed for the speciﬁc problem of nested learning.
We compared some standard combination methods and our calibration-based combination strategy.
Combination methods can be classiﬁed into two categories.
First those that combine or vote discrete classiﬁcation results, where classiﬁer outputs are considered as one-hot encoding vectors.
For example, the Majority Vote (MA) which consists in aggregating the decision of multiple classiﬁers and selecting the candidate that receives more votes.
The second category of methods combine classiﬁers continuous outputs.
As we show in the following, the second class of methods are more suitable for the combination of nested outputs.
For example, suppose that given an input sample x , the classiﬁer outputs an estimations of the probability associated to each coarse, middle, and ﬁne label: P ˆ Y 1 , P ˆ Y 2 , and P ˆ Y 3 .
In order to apply standard combination methods, we need to transform the coarse and intermediate predictions into a ﬁne prediction and vice-versa.
To transform the probability associated to a coarse label into a ﬁner granularity, we can assume the ﬁner classes associated to each coarse class are equally probable, i.e., the probability associated to a coarse node, is divided equally into the ﬁner nodes associated to this label.
On the other hand, to aggregate probabilities associates to a ﬁner level into a coarser level, we can simply add those probabilities associated to the same coarse node.
Once probabilities associated to coarse levels are propagated to the ﬁne levels and vice-versa, we can combine them using the mean or the product rule as described in Kuncheva (2004).
Tableshows the result of combining nested outputs with standard combination methods and our strategy described in Section 3.1.
F A DDITIONAL EXPERIMENTS
[Table]
These results the results presented in Figure 4.
Figure 4 illustrates the results on clean data grouping ﬁve levels of budgets and for three cost
, and
In this
, we show the performance of we display the well as
[Table]
[Table] [Table]
[Table]
F. C OMPARING WITH A COMMON MTL ARCHITECTURE
Classic Multi-Task Learning (MTL) DNN approaches extract common features that are on the top the architecture branched for the prediction of multiple tasks.
To this end, a common MTL architecture consists of shared convolutionnal blocks followed by task-speciﬁc classiﬁcation (fully connected) layers.
Figure 11 illustrates an MTL architecture designed to obtain a nested classiﬁcation of MNIST characters.
The network enforces an information bottleneck that encodes the input in a (64,1) feature vector.
Then, it is connected to the three classiﬁcation branches implemented by a sequence of fully-connected layers.
This MTL model and our nested model have approximately the same number of parameters and are trained in an identical way on MNIST dataset.
Table 7 shows the classiﬁcation accuracy for our nested architecture presented in Section B and the multi-task learning architecture presented above.
On the original distribution both architectures perform roughly equivalently, but on perturbed test data the proposed nested model is considerably superior.
Nested learning outperforms its MTL counterpart with a gap of 18% for the ﬁne task and a gap of more than 10% for the coarse task for distortion levels 3 and 4.
These results provide additional evidence that having a hierarchical nested architecture promotes robustness and leads to more consistent models.
[Table]
F. S KIPPED CONNECTIONS AND THEIR IMPACT ON THE FLOW OF INFORMATION
Obtaining an empirical measure of the mutual information (MI) between two high dimensional random variables is a very hard numerical problem (Paninski (2003)).
However, recent progresses in deep learning made a numerical approximation (MINE) tractable by exploiting the ﬂexibility of neural networks and properties of the mutual information.
Belghazi et al.
(2018a) proved that a MI estimation can be seen as an optimization problem.
They rely on the following characterization of the mutual information as the Kullback-Leibler (KL- [Equation] divergence,
[Equation]
Based on this, the authors use the Donsker-Varadhan representation of the KL divergence, which introduces a dual optimization problem,
[Equation]
where the supremum is taken over all the functions T such that the two expectations are ﬁnite.
Since the parameters of a neural network can be used to encode a large space of functions, the authors propose to solve Equation equation 4 for T θ ∈ F , where F denotes the space of functions encoded by a pre-deﬁned network architecture with parameters θ ∈ Θ .
An approximation of the MI can be obtained solving the problem
[Equation]
Equation 5 can be numerically solved using standard optimization tools of deep learning (see Belghazi et al.
(2018b)
for details).
In order to measure the impact of skipped-connections from an MI perspective, we measure the empirical approximation of the mutual information between different sections of the proposed nested architecture.
We compared these results for the same network with and without skipped connections.
Let us refer to the network with skipped connections with the subscript 1 and the network without skipped connections with the subscript 2.
We deﬁne different variables of interest at particular stages of the network and estimate the mutual information between them.
We will call F 1 ( X ) and
F 2 ( X ) the variable corresponding to the feature map created after the 2nd Maxpooling layer (see
Figure 11) with or without skipped connections.
Similarly, let G 1 ( X ) and G 2 ( X ) be the feature maps obtained before the second GAP layer.
We will also consider H 1 ( X ) and H 2 ( X ) the coarsest feature maps obtained before the ﬁrst GAP.
(Before feeding these features to MINE algorithm we performed average pooling to reduce the dimension of the input variables.)
We estimated the MI between I ( F i ( X ) , G i ( X )) and I ( F i ( X ) , H i ( X )) , i = 1 meaning “with skipped connections” and i = 2 “without skipped connections” (the network without skipped connections is re-trained to allow the model to adapt to this new conﬁguration).
Figure 12 (left side) sketches in which sections of the model we are measuring the mutual information, and (right side) the evolution of the MI estimation (MINE algorithm) for 3 k steps.
After convergence of MINE algorithm we compared the mutual information between the random variables F i , H i and G i with and without skipped connections (illustrated in green in the sketch of the left side of ﬁgure 12),
[Equation] [Equation]
As shown in Equation 6 skipped connections play an important role to allow information to ﬂow to the feature representation of the ﬁner classes.
This provides additional numerical evidence to the discussion presented in Section 3 supporting the importance of including skipped connections and avoiding information bottleneck at the ﬁner classiﬁcation.
F. C OMPARISON OF TRAINING PROTOCOLS
Our training protocol is organized in a cascaded way.
As explained in Section 3.1, the reason for this is two-fold.
First, this would eliminate the problem of the noisy stochastic gradient computations which happens when we there is not enough samples in the mini-batch for a speciﬁc task/granularity.
Second, it provides a good initialisation of the weights associated to the coarser levels, and we observed empirically that this leads to a faster convergence and better accuracy.
In order to illustrate the impact of the training protocol, we compare the results obtained by training the exact same model, with the same data, but using two different training strategies.
We compared the described cascade methodology with the standard methodology (i.e., selecting batches of random samples out of the training set).
As samples are annotated heterogeneously (with all or some ﬁne to coarse annotations) we deﬁned the uniﬁed loss in each mini-batch as
[Equation]
M denotes the number of samples in the batch, N the number of nested level, where ω ij is a masking vector which indicates if the sample j is annotated or not for the level of granularity i , α i are global weight that help to balance the loss associated to each level L i .
We observed that cascaded training achieves substantially better performances than the traditional training as we can see in Table 8.
The gaps are very signiﬁcant also on the training distribution as we can observe in the ﬁrst column.
Additionally, if we studied the behaviour of the network while, e.g., see the results presented in Figure 13.
As we can see, the proposed protocol is more suitable for the proposed multi-level problem leading to faster convergence and better models.
After 26 epochs of coarse learning, we add the intermediate data for 30 epochs before adding the ﬁne data formore epochs.
We can observe that very few epochs after the introduction of the intermediate data, the accuracy of the intermediate classiﬁer is already superior to a standard training procedure.
We observe this phenomenon again for the ﬁne classiﬁcation, see for example, the training evolution over the green area illustrated in Figure 13.
In addition we can observe that the accuracy of the cascaded training classiﬁers oscillates far less than the accuracy of the traditionally trained classiﬁers.
This provides additional evidence that our method helps to deal with the noisiness of the stochastic gradient estimation.
[Table]