M ULTI - TASK L EARNING ON MNIST I MAGE D ATASETS
Anonymous authors
Paper under double-blind review
A BSTRACT
We apply multi-task learning to image classiﬁcation tasks on MNIST-like datasets.
MNIST dataset has been referred to as the drosophila of machine learning and has been the testbed of many learning theories.
The NotMNIST dataset and the FashionMNIST dataset have been created with the MNIST dataset as reference.
In this work, we exploit these MNIST-like datasets for multi-task learning.
The datasets are pooled together for learning the parameters of joint classiﬁcation networks.
Then the learned parameters are used as the initial parameters to retrain disjoint classiﬁcation networks.
The baseline recognition model are all-convolution neural networks.
Without multi-task learning, the recognition accuracies for MNIST,
NotMNIST and FashionMNIST are 99.56%, 97.22% and 94.32% respectively.
With multi-task learning to pre-train the networks, the recognition accuracies are respectively 99.70%, 97.46% and 95.25%.
The results re-afﬁrm that multi-task learning framework, even with data with different genres, does lead to signiﬁcant improvement.
I NTRODUCTION
Multi-task learning (Caruana (1998)) enjoys the idea of pooling information that can be learned from data collected for multiple related tasks.
Multiple sources of information can stem from multiple datasets, or even a single dataset, for multiple tasks.
In this work, we focus on the case of using multiple datasets for multiple tasks.
Namely, we use MNIST, FashionMNIST, and NotMNIST image datasets collected for digit recognition, fashion item recognition, and letter recognition, respectively.
Information sharing in multi-task training can be achieved in various formality. For neural-network based deep learning, the sharing can happen at the input layer, the hidden layers, or the output layer.
Input-layer multi-tasking combines heterogeneous input data, hidden-layer multi-tasking shares multiple groups of hidden layer units, and output-layer multi-tasking pools multiple output groups of categories.
The implementation of a multi-task learning system depends on the data and the tasks at hand.
Multi-task learning has been successfully applied to many applications of machine learning, from natural language processing (Collobert & Weston (2008)) and speech recognition (Deng et al. to computer vision (Ren et al.
and drug discovery (Ramsundar et al. A recent review of multi-task learning in deep learning can be found in (Ruder (2017)).  MNIST D ATASETS 2. MNIST The MNIST dataset (LeCun et al.
(1998)) consists of a training set of 60,000 images, and a test set of 10,000 images. MNIST is often referred to as the drosophila of machine learning, as it is an ideal testbed for new machine learning theories or methods on real-world data.
Table 1 lists the state-of-the-art performance on MNIST dataset. A few examples of MNIST dataset are shown in
Table 2, alongside with examples of the other MNIST-like datasets. [Table] [Table]
F ASHION MNIST
Xiao et al.
(2017) presents the FashionMNIST dataset.
It consists of images from the assortment on Zalandos website.
As given out by name, the conﬁguration of the FashionMNIST dataset completely parallels the conﬁguration of the MNIST dataset. FashionMNIST consists of a training set of 60,000 images and a test set of 10,000 images.
Each image is a 28 (cid:2) 28 grayscale image associated with a label from 10 classes. FashionMNIST poses a more challenging classiﬁcation task than the MNIST digits data.
A leaderboard for FashionMNIST has been created and maintained at https://github.com/zalandoresearch/fashion-mnist .
[Table]
N OT MNIST
Bulatov (2011) presents the NotMIST dataset.
NotMNIST dataset consists of more than 500k 28 (cid:2)greyscale images of English letters from A to J , including a small hand-cleaned subset and a large uncleaned subset. From the uncleaned subset, we randomly select 60,000 examples as train set and 10,000 examples as test set.
M ETHOD
In short, we apply multi-task learning to learn from the MNIST-like datasets to pre-train the parameters of the all-convolution neural networks for individual image recognition tasks. The overall framework is depicted in Figure 1.
3. A LL -C ONVOLUTION N EURAL N ETWORKS
A convolutional neural networks (CNN) (LeCun et al. (1998)) usually consists of alternating convolution layers and pooling layers. In contrast, an all-convolution neural network ( Springenberg et al.
(2014)) replace pooling layers by convolution layers. A feature map f created at a convolution layer can be represented by a three-dimensional tensor of order W (cid:2) H (cid:2) N , where W and H are the width and height, N is the number of channels.
A pooling operation with a size of k (cid:2) k and a stride of r applied to a three-dimensional tensor results in another tensor s with
[Equation]
where the ﬁrst 2 subscripts index position in the map, and the third subscript indexed the channel. If a convolution operation with the same stride were applied to f , we would have a tensor c with
[Equation]
[Equation]
[Equation]
where t is the convolution kernel tensor, and (cid:27) ( (cid:1) ) is an activation function. Thus, a pooling operation can be seen as a convolution operation with uniform kernel tensor and with L p -norm as the activation function.
E XPERIMENT 4. C LASSIFIER A RCHITECTURE AND M ULTI - TASK L EARNING
The architecture of an all-convolution neural network for a single task is shown in Figure 2. The multi-task learning classiﬁer has the same architecture as a single-task classiﬁer except that the width of the output layer is proportional to the number of tasks. The target label is enhanced accordingly by zero-padding.
As mentioned before, we combine the training data together, so that the training label dimension is increased from [ 1 (cid:2) 10 ] to [ 1 (cid:2) 20 ] by zero-padding. This is shown in Figure 3. Multi-task learning with different combination of datasets are evaluated, namely 3 bi-task learning and 1 tri-task learning.
For example, we combine the MNIST train dataset and FashionMNIST train dataset together. Thus, there are 120,000 examples to the bi-task learning network for MNIST and
FashionMNIST. Each training example has an input image size of 28 (cid:2) 28 and an output 1-hot vector size of 20 . Each network is trained with 50 epochs. A two-stage learning rate decay scheme is implemented.
The initial learning rate is 10 (cid:0) 3 for the ﬁrst stage of 25 epochs, and 10 (cid:0) 5 for the second stage ofepochs. When training a model, it is often recommended to lower the learning rate as the training progresses. So, we let learning rate decay in each epochs. The decay rate is 1 1+ d (cid:2) n where d is the learning rate set in the begin divided by 25.
n is the number of current epochs. Figure 4 shows the learning rate scheme.
The size of a mini-batch is set to 100, and the Adam optimizer is used. When multi-task learning is complete, the parameters in the network is used to initialize single-task classiﬁers. Figure 5 illustrates how parameters are handed over. The single-task classiﬁers are then re-trained to perform their respective classiﬁcation tasks.
The experimental results are summarized in Table 3. We are happy to see that multi-task learning works, even though the MNIST, NotMNIST, and FashionMNIST datasets are images from totally
[Table] different classes. The bi-task learning systems are always better than the single-task systems.
Furthermore, the tri-task learning systems are the best, except for the MNIST (0.01% difference). The relative reduction in error rates by tri-task learning are respectively 31.8% for MNIST (99.56% to 99.70%), 16.4% for FashionMNIST (94.32% to 95.25%), and 8.6% for NotMNIST (97.22% to 97.46%). The results conﬁrm that multi-task learning is able to learn representation which is universal and robust to different tasks. To better understand the effect of multi-task learning, we plot the distribution of the high dimensional-data of the classes by the t -distributed stochastic neighbor embedding (t-SNE) (van der
Maaten & Hinton (2008)). Figure 6 shows the t-SNE of the output values of several hidden layers of the classiﬁers. The separation between the learned class manifolds appears to increase with the integration of multi-task learning. The representation learned with multi-task learning in the loop looks better indeed.
C ONCLUSION
In this paper, we use multi-task learning in pre-training an all-convolution neural network model. We pass the parameters of trained multi-task models to single-task models. Evaluation on MNISTlike datasets show that using multi-task learning can improve image recognition accuracy.
The more data we use, the better results we get. This agrees with statistical learning theory that using more data reduces the generalization gap, thus improving test set performance, even if the data comes from a different domain. The classiﬁcation tasks of the images of digits, letters, and fashion items share parts of their hierarchical representations. By multi-task learning, it is possible to make such common representation robust to help individual classiﬁcation tasks.