Do deep neural networks learn shallow learnable examples ﬁrst?
Abstract
In this paper, we empirically investigate the training journey of deep neural networks relative to fully trained shallow machine learning models.
We observe that the deep neural networks (DNNs) train by learning to correctly classify shallowlearnable examples in the early epochs before learning the harder examples.
We build on this observation this to suggest a way for partitioning the dataset into hard and easy subsets that can be used for improving the overall training process.
Incidentally, we also found evidence of a subset of intriguing examples across all the datasets we considered, that were shallow learnable but not deep-learnable.
In order to aid reproducibility, we also duly release our code for this work at https : // github . com / karttikeya / Shallow to Deep /
1. Introduction
Analyzing the temporal journey taken by deep neural networks (DNNs) during training has elicited a lot of attention recently.
The authors in (Arpit et al., 2017) suggested that
DNNs learn simple patterns ﬁrst, before memorizing .
More speciﬁcally, they posit that real world datasets are littered with easy examples characterized by simple patterns that are learned in the initial epoch(s) before the conquest of hard examples in the training dataset.
Tishby et al
(Tishby &
Zaslavsky, 2015) conjectured that DNN training was characterized by two distinct phases consisting of an initial ﬁtting phase (memorization) and a subsequent compression phase.
While this claim was questioned in (Saxe et al., 2018) the authors do remark that when an input domain consists of a subset of task-relevant and task-irrelevant information hidden representations do compress the task-irrelevant information .
These works do suggest that the easy-vs-hard dichotomy in real-world datasets does inﬂuence the learning in DNNs and goad a data-dependent approach towards understanding the capacity of DNNs.
Taking cue from this, we strive to contribute to this growing body of literature by bringing in another viewpoint: The dichotomy between shallow learnable examples and deep learnable examples in the dataset.
More speciﬁcally we try to address the questions:
Is the notion of easiness same for models with as different parameterizations and architectures as shallow machine learning models and deep networks the same?
And hence is attached to the example independently of a model?
If we are to investigate the examples that a DNN learns to correctly classify over the training batches, do we observe a shallow learnable to deep learnable regime change ?
Are there examples that are shallow learnable but somehow a DNN with a far better overall accuracy fails to classify?
At the heart of this quest is to understand if shallow learnability is a good proxy for the easiness of an example.
We’d like to reiterate that the motivation behind this work is to obtain insights into the changing scenery of the conquest of the training dataset experienced by deep neural networks and not to delineate the nature of compositional functions that DNNs can learn and shallow algorithms cannot or comment on the amount of training data required to do so.
In (Mhaskar et al., 2017), the authors have already shown how
DNNs can approximate the class of compositional functions as well as shallow networks but with exponentially lower number of training parameters and sample complexity .
The rest of the paper is organized as follows.
In section 2, we present the quantitative methodology we used to answer these questions raised above.
In section 3, we showcase our empirical experiments with the results covered in section 4.
We conclude the paper in section 5.
2. Proposed Methodology
In this section we delineate our proposed method to study the learning process of a deep learning model D relative to a shallow machine learning model M . [Table]
2.1. Tracking the learning process
We propose to measure the generalization capability of a deep learning model D on unknown data using a custom contingency matrix T .
In other words, instead of just calculating the accuracy of the deep learning model after i steps of back propagation, D ( i ) on the test set, we propose to calculate the contingency matrix T ( i ) .
Furthermore, to study the learning process of model D relative to M , we look at how the matrix T ( i ) evolves as the model trains.
2.1.1. P REPARING THE C ONTINGENCY M ATRIX
The contingency matrix T is deﬁned using the pattern of classiﬁcation of the test set examples by the models D and
M .
Referring to Table 1, T ( i ) 00 denotes the number of examples on the test set that both the models D ( i ) and M make an error on.
Similarly, T ( i ) 10 denotes the number of test set examples that D ( i ) classiﬁes correctly but M classiﬁes wrongly and so on and so forth for T ( i ) 01 and T ( i ) 11 .
2.1.2. A NALYZING L EARNING D YNAMICS
Several useful metrics can be derived from T that analyze different aspects of the learning process.
Naively, accuracies of models D and M can be recovered simply by
[Equation]
Note that since we’re tracking the learning dynamics for
D ( i ) and model M is kept the same, Accuracy ( M ) actually remains constant throughout.
More interesting metrics can be derived such as the accuracy of D ( i ) on subsets of the training data that M classiﬁes correctly ( R i + ) and those that
M classiﬁes wrongly ( R i − ).
[Equation]
Also, to measure how many times more accurate D ( i ) is on one subset versus the other, we study the ratio of the above two accuracies R i
= R i + / R i − .
3. Experiments
We perform our experiments under three different regimes divided on the basis of relative performance of shallow
Machine Learning models to Deep Networks to check the robustness of our observations.
We experiment with (a)
The MNIST dataset, where the ML models tend to perform competitively with the ConvNets (b) The CIFAR10 dataset where ML models perform worse than neural networks but not are still quite good and lastly (c) the CIFAR100 dataset, where the deep networks far outperform the shallow Machine Learning models.
We preprocess all the datasets to center and normalize the images before training models but do not augment data to avoid other potential confounding factors in our observations.
3.1. Datasets
MNIST : MNIST (Mixed National Institute of Standards and
Technology) (LeCun, 1998) is a very popular toy dataset consisting of images of handwritten digits and their numeric value as their labels.
It consists of a total of 70 , 000 grayscale 28 ×
28 labelled images divided in training set of 60 , 000 and another test set of 10 , 000 images.
It is widely used to examine image recognition techniques and considered a relatively simple starter dataset.
CIFAR :
The CIFAR database (Krizhevsky, 2009) refers to two related but different datasets namely, the CIFAR and CIFAR100 datasets.
CIFAR10 consists of 60 ,color 32 × 32 images divided into 10 object categories like airplane , bird , cat etc.
It consists of a training set of 50 , 000 and a test set of 10 , 000 images.
CIFAR100 also has the same characterstics and size except it consists of 100 image classes rather than just 10 .
Thus, it consists of 600 examples of each class which are grouped proportionally into 500 in the training set and 100 in the test set.
The dataset also contains the 100 classes grouped into 20 super classes but in this work we use ﬁne grained classes for comparing models on CIFAR100.
3.2. Training shallow ML Models
We train two types of machine learning models to study the learning process – Support Vector Machines (Cortes &
Vapnik, 1995) and Random Forests (Breiman, 2001).
Random Forest : Random Forest are another family of successful Machine Learning models which have been applied to a large number of classiﬁcation and regression problems (Liaw et al., 2002) like real time face detection (Cootes et al., 2012), Gene Selection (D ´ ıaz-Uriarte & De Andres, 2006),
Remote sensing (Pal, 2005) and several other applications.
In this paper, we train the Random Forests using 20 estimators and the gini index criterion (Pedregosa et al., 2011).
Ratio of Accuracies R i plotted against i with M being a Support Vector Machine.
Accuracies of D on the SVM-correct subset ( R i + ), SVM-incorrect subset ( R i − ) and the overall testset (Accuracy( D ( i ) ) ).
Values of the contingency matrix T i plotted against i (smoothed and scaled for plotting).
Referring to Table 1, the entries T 00 , T 01 , T 10 and T 11 are denoted by red, orange, green and blue respectively.
We ﬂatten the preprocessed images into a 784 dimensional vector in case of MNIST and into a 1024 dimensional vector for CIFAR to facilitate training the random forest.
Support Vector Machines : Support Vector Machines (SVM) are extremely popular ML models shown to achieve good results in several limited data domains like hand written character recognition (Decoste & Sch¨olkopf, 2002), face recognition (Heisele et al., 2001), hypertext classiﬁcation (Pradhan et al., 2004) and several biological applications (Cuingnet et al., 2011; Yang, 2004).
We train a SVM with slacks under the Radial Basis Kernel (Pedregosa et al., 2011) with hyperparameters C = 1 . 0 and γ = 0 .
Similar to Random Forests, we ﬂatten out the images for training an SVM.
3.3. Training Deep Neural Networks
We train a different deep network on each dataset because of two reasons.
First, some networks we train have a much larger capacity than others and hence are not suited for a small dataset like MNIST.
Likewise, the architecture suited for MNIST is too small for a more complicated dataset like CIFAR and would not be a optimal choice for good performance.
Second, we want to check robustness of our observations across different model sizes and architectures and hence experiment with very popular but very different ontologies like DenseNet and ResNet.
Convolutional Neural Network : We train a small Convolution Network with the architecture :
[ Conv (cid:46) ReLu (cid:46)
MaxPool ] (cid:55)→ [ Conv (cid:46) ReLu (cid:46) MaxPool ] (cid:55)→ [ FC (cid:46) ReLu ] (cid:55)→ FC (cid:55)→ SoftMax .
The network is trained on the MNIST dataset with SGD with a learning rate 0 .
01 , momentum set to 0 .
5 and a batch size of 64 .
DenseNet121 :
DenseNet 121 (Huang et al., 2017) is a deep network that has been demonstrated to achieve a very good performance on that task of Image recognition on a variety of benchmarks.
In a nutshell, DensetNet contains several
DenseBlocks which inside themselves are a constant feature depth stack of ConvNets; connected in a fully connected fashion.
We train DenseNet 121 on the CIFAR10 dataset with SGD with learning rate 0 .
1 , momentum 0 .
5 and a batch size of 128 .
Further training details are in the appendix.
Resnet 101 :
ResNet 101 belongs to the very popular family of Residual Networks (He et al., 2016) and is widely used as a backbone in a variety of computer vision tasks like Image
Recognition (He et al., 2016), Action Classiﬁcation (Feichtenhofer et al., 2016), Image to Image Translation (Zhu et al., 2017) etc.
We use ResNet 101 for image recognition on the CIFAR100 dataset where it is trained with SGD under same parameters as mentioned for DenseNet121 but with an additional weight decay of 5 × 10 − 4 .
4. Results
Here we discuss our observations while studying the training process under the lens of machine learning models.
Table 2 reports the maximum accuracy achieved with the models discussed in Section 3.2 and 3.3.
While the accuracies are reported for completeness, this work studies the learning process in the early stages and hence is not concerned with the maximum accuracy.
Moreover, in cases of CIFAR10 and
CIFAR100 datasets the operating regime for DenseNet
[Table]
and ResNet101 are far from their optimal performance.
4.1. Ratio of Accuracies
Referring to the top row in Fig 1 and 2, we observe how the ratio of accuracies ( R + ) changes during training.
Note that if the two subsets, M -correct and M -incorrect were completely irrelevant and similar for training process of
D , R + would remain identically 1 .
However, we observe that across datasets and ( M , D ) pairs, the curve has a right skewed unimodal shape with a sharp hump.
We also note that this happens in very early stages of training, sometimes as early as just after 1 / 20 th epoch over the training set.
Also, the accuracies can be very different on the two subsets with
D being upto 8 times more accurate on M -correct subsets than on M -correct subsets in some cases.
Furthermore, we observe that the curve shows a long tail as the ratio R i returns back to 1 .
This observation is the cornerstone in conﬁrming out hypothesis that deep networks training starts from quickly learning shallow classiﬁable easy examples and then slowly extends to the hard ones.
4.2. Accuracy Plots
The second row of Fig.
1 and Fig.
2 depicts test accuracies as training progresses.
Note that the overall trend is increasing as expected from a network in early stages of training, however there are huge gaps in accuracy on the two different subsets.
For example, in the case of CIFAR when the M -incorrect subset is ∼ 60% of the total set, it weighs down the overall accuracy by over as much as 20% at times during training.
Thus, identifying the hard examples with M incorrectness can help in training procedures like curriculum learning (Bengio et al., 2009), teacher forcing (Jordan, 1986; Pineda, 1988) and professor forcing (Lamb et al., 2016).
Furthermore, weighing the training set examples on basis of whether they are correctly classiﬁed by M can provide a more balanced dataset for training.
4.3. Contigency Matrix Values
Observe the long tail decay of T 00 and T 11 and the very fast rise of T 11 and T 10 (bottom row Figure 1).
This shows that the the slow learning of M -incorrect examples is the major factor of slump in accuracy growth for deep networks after the often observed initial fast ascent.
This observation can be used for iterating more on M -incorrect harder data points after the initial phase and achieve faster convergence.
5. Conclusion and Future work
In this work, we track the training of DNNs relative to shallow machine learning models. We showcase some results on analyzing the training trajectory of the DNNs relative to SVM and RF on three different datasets.
Empirically, we observe that the during training the Deep Network quickly learns shallow classiﬁable easy examples ﬁrst and then learns the hard examples in the later epochs .
Furthermore, we ﬁnd that the notion of hardness of an example is largely independent of the model being used and can be evaluated reliably using a shallow learning model.
This observation allows for a procedural slicing of the training set into easy and hard categories that can improve network training.
We also report a slightly surprising ﬁnding pertaining to the existence of a subset of examples in all the datasets considered that were shallow-classiﬁable but not deep-classiﬁable . 5.1. Future work We are currently extending this work along the following two paths.
The ﬁrst entails using the inﬂuence functions framework (Koh & Liang, 2017) to understand the distribution of the inﬂuence of the training examples and juxtaposing this with respect to their shallow/deep learnability.
The second path entails understanding the nature of this conquest of the training space of the deep and shallow classiﬁers from the viewpoint of complexity and interestingness of images (Gygli et al., 2013). We conclude with a conjecture
that complexity of images as measured by, say, it’s
JPEG compressibility will have strong correlations with it’s shallow learnability.
6. Appendix 6.1. Training Details
The reported maximum accuracy (Table 2) results have been achieved with decaying the learning rate by
(a) a factor of 20 after 60 , 120 , 160 and 200 epochs for DenseNet121 on the CIFAR10 dataset (b) a factor of 10 after 150 and 250 epochs for ResNet on the CIFAR100 dataset.