We combine supervised learning with unsupervised learning in deep neural networks. The proposed model is trained to simultaneously minimize the sum of supervised and unsupervised cost functions by backpropagation, avoiding the need for layer-wise pre-training. Our work builds on the Ladder network proposed by Valpola (2015), which we extend by combining the model with supervision. We show that the resulting model reaches state-of-the-art performance in semi-supervised MNIST and CIFAR-10 classification, in addition to permutationinvariant MNIST classification with all labels. arXiv:1507.02672v2 [cs.NE] 24 Nov 2015 decoder can recover any details discarded by the encoder. Previously, the Ladder network has only been demonstrated in unsupervised learning (Valpola, 2015; Rasmus et al., 2015a) but we now combine it with supervised learning. The key aspects of the approach are as follows: Compatibility with supervised methods. The unsupervised part focuses on relevant details found by supervised learning. Furthermore, it can be added to existing feedforward neural networks, for example multi-layer perceptrons (MLPs) or convolutional neural networks (CNNs) (Section 3). We show that we can take a state-of-the-art supervised learning method as a starting point and improve the network further by adding simultaneous unsupervised learning (Section 4). Scalability resulting from local learning. In addition to a supervised learning target on the top layer, the model has local unsupervised learning targets on every layer, making it suitable for very deep neural networks. We demonstrate this with two deep supervised network architectures. Computational efficiency. The encoder part of the model corresponds to normal supervised learning. Adding a decoder, as proposed in this paper, approximately triples the computation during training but not necessarily the training time since the same result can be achieved faster through the better utilization of the available information. Overall, computation per update scales similarly to whichever supervised learning approach is used, with a small multiplicative factor. As explained in Section 2, the skip connections and layer-wise unsupervised targets effectively turn autoencoders into hierarchical latent variable models which are known to be well suited for semisupervised learning. Indeed, we obtain state-of-the-art results in semi-supervised learning in the MNIST, permutation invariant MNIST and CIFAR-10 classification tasks (Section 4). However, the improvements are not limited to semi-supervised settings: for the permutation invariant MNIST task, we also achieve a new record with the normal full-labeled setting. 1 2 Derivation and justification Latent variable models are an attractive approach to semi-supervised learning because they can combine supervised and unsupervised learning in a principled way. The only difference is whether the class labels are observed or not. This approach was taken, for instance, by Goodfellow et al. (2013a) with their multi-prediction deep Boltzmann machine. A particularly attractive property of hierarchical latent variable models is that they can, in general, leave the details for the lower levels to represent, allowing higher levels to focus on more invariant, abstract features that turn out to be relevant for the task at hand.
Π-model can also be viewed as a simplification of the Γ-model of the Ladder Network REF .
Semi-supervised learning with Ladder networks
NIPS
Computer Science