We explore the effect of introducing prior information into the intermediate level of deep supervised neural networks for a learning task on which all the black-box state-of-the-art machine learning algorithms tested have failed to learn. We motivate our work from the hypothesis that there is an optimization obstacle involved in the nature of such tasks, and that humans learn useful intermediate concepts from other individuals via a form of supervision or guidance using a curriculum. The experiments we have conducted provide positive evidence in favor of this hypothesis. In our experiments, a two-tiered MLP architecture is trained on a dataset for which each image input contains three sprites, and the binary target class is 1 if all three have the same shape. Black-box machine learning algorithms only got chance on this task. Standard deep supervised neural networks also failed. However, using a particular structure and guiding the learner by providing intermediate targets in the form of intermediate concepts (the presence of each object) allows to nail the task. Much better than chance but imperfect results are also obtained by exploring architecture and optimization variants, pointing towards a difficult optimization task. We hypothesize that the learning difficulty is due to the composition of two highly non-linear tasks. Our findings are also consistent with hypotheses on cultural learning inspired by the observations of effective local minima (possibly due to ill-conditioning and the training procedure not being able to escape what appears like a local minimum). Keywords: Deep Learning, Neural Networks, Optimization, Evolution of Culture, Curriculum Learning, Training with Hints There is a recent emerging interest in different fields of science for cultural learning (Henrich and McElreath, 2003) and how groups of individuals exchanging information can learn in ways superior to individual learning. This is also witnessed by the emergence of new research fields such as "Social Neuroscience". Learning from other agents in an environment by the means of cultural transmission of knowledge with a peer-to-peer communication is an efficient and natural way of acquiring or propagating common knowledge. The most popular belief on how the information is transmitted between individuals is that bits of information are transmitted by small units, called memes, which share some characteristics of genes, such as self-replication, mutation and response to selective pressures (Dawkins, 1976 This paper is based on the hypothesis (which is further elaborated in Bengio (2013a)) that human culture and the evolution of ideas have been crucial to counter an optimization issue: this difficulty would otherwise make it difficult for human brains to capture high level knowledge of the world without the help of other educated humans. In this paper machine learning experiments are used to investigate some elements of this hypothesis by seeking answers for the following questions: are there machine learning tasks which are intrinsically hard for a lone learning agent but that may become very easy when intermediate concepts are provided by another agent as additional intermediate learning cues, in the spirit of Curriculum Learning (Bengio et al., 2009b) ? What makes such learning tasks more difficult? Can specific initial values of the neural network parameters yield success when random initialization yield complete failure? Is it possible to verify that the problem being faced is an optimization problem or with a regularization problem? These are the questions discussed (if not completely addressed) here, which relate to the following broader question: how can humans (and potentially one day, machines) learn complex concepts? In this paper, results of different machine learning algorithms on an artificial learning task involving binary 64×64 images are presented. In that task, each image in the dataset contains 3 Pentomino tetris sprites (simple shapes). The task is to figure out if all the sprites in the image are the same or if there are different sprite shapes in the image. Several state-of-the-art machine learning algorithms have been tested and none of them could perform better than a random predictor on the test set. Nevertheless by providing hints about the intermediate concepts (the presence and location of particular sprite classes), the problem can easily be solved where the same-architecture neural network without the intermediate concepts guidance fails. Surprisingly, our attempts at solving this problem with unsupervised pre-training algorithms failed solve this problem. However, with specific variations in the network architecture or training procedure, it is found that one can make a big dent in the problem. For showing the impact of intermediate level guidance, we experimented with a two-tiered neural network, with supervised pre-training of the first part to recognize the category of sprites independently of their orientation and scale, at different locations, while the second part learns from the output of the first part and predicts the binary task of interest. The objective of this paper is not to propose a novel learning algorithm or architecture, but rather to refine our understanding of the learning difficulties involved with composed tasks (here a logical formula composed with the detection of object classes), in particular the training difficulties involved for deep neural networks. The results also bring empirical evidence in favor of some of the hypotheses from Bengio (2013a), discussed below, as well as introducing a particular form of curriculum learning (Bengio et al., 2009b) . Building difficult AI problems has a long history in computer science. Specifically hard AI problems have been studied to create CAPTCHA's that are easy to solve for humans, but hard to solve for machines (Von Ahn et al., 2003) . In this paper we are investigating a difficult problem for the off-the-shelf black-box machine learning algorithms. 1
REF explore the architecture by providing intermediate targets.
Knowledge Matters: Importance of Prior Information for Optimization
ICLR
CoRR
Computer Science Mathematics