Deep learning development

In recent years, the rapid development of deep learning has been sensational due to Google's AlphaGo some time ago, and China has also begun to usher in the research boom of this technology, so what kind of development process has such a hot deep learning experienced? Let's take a look at the history of deep learning.

1943

Neuroscientist W.S. McCilloch and mathematician W. Pitts published a paper in the Bulletin of Mathematical Biophysics, "A Logical Calculus of the Ideas Immanent in Nervous Activity." Neural networks and mathematical models are built, called MCP models. The so-called MCP model is actually an abstract and simplified model constructed according to the structure and working principle of biological neurons, which gives birth to the so-called "simulated brain", and the door of artificial neural networks is opened.

W.S. McCilloch

W. Pitts

MCP wanted to be able to use computers to simulate the process of human neuronal responses, and the model simplified neurons to three processes: linear weighting of input signals, summation, and nonlinear activation (threshold method). Figure below

1958

Computer scientist Rosenblatt has proposed a neural network of two layers of neurons, called Perceptrons. The first use of MCP for machine learning and classification. The "perceptron" algorithm algorithm uses the MCP model to binary classify the input multidimensional data, and can use gradient descent to automatically learn to update weights from training samples. In 1962, the method proved to converge, and the theoretical and practical effects caused the first wave of neural networks.

1969

Throughout the history of scientific development, it is undoubtedly full of twists and turns, and deep learning is no exception. In 1969, American mathematician and artificial intelligence pioneer Marvin Minsky proved in his book that the perceptron is essentially a linear model that can only handle linear classification problems, and even the simplest XOR (or) problems cannot be classified correctly. This is equivalent to directly pronouncing the death sentence of the perceptron, and the research of neural networks has been stagnant for nearly 20 years.

1986

Geoffrey Hinton, the father of neural networks, invented the BP (Backpropagation) algorithm for multilayer perceptrons (MLPs) in 1986, and used Sigmoid for nonlinear mapping, effectively solving the problem of nonlinear classification and learning. This method caused a second craze in neural networks.

Note: The Sigmoid function is an S-type function commonly found in biology, also known as the S-type growth curve. In information science, due to its properties such as single increase and inverse function single increase, the Sigmoid function is often used as a threshold function for neural networks, mapping variables between 0,1.

90s period

In 1991, the BP algorithm was pointed out that there is a gradient vanishing problem, that is, in the process of error gradient postterm transfer, the post-layer gradient is multiplied to the front layer, due to the saturation characteristics of the Sigmoid function, the post-layer gradient is originally small, and the error gradient is almost 0 when transmitted to the front layer, so it cannot effectively learn the front layer, which directly hinders the further development of deep learning.

In addition, in the mid-90s, various shallow machine learning models such as the birth of support vector machine algorithms (SVM algorithms) were proposed, SVM is also a supervised learning model, which is applied to pattern recognition, classification, and regression analysis. Support vector machines are based on statistics and have obvious differences from neural networks, and the proposal of algorithms such as support vector machines once again hinders the development of deep learning.

Development period 2006 - 2012

In 2006, Geoffrey Hinton, a professor at the University of Toronto, a leader in the field of machine learning, and the father of neural networks, and his student Ruslan Salakhutdinov published an article in the top academic journal Science, which proposed a solution to the gradient vanishing problem in deep network training: unsupervised pre-training initializes weights and supervised training fine-tuning. Stanford University, New York University, University of Montreal, Canada, etc. have become important cities for the study of deep learning, thus opening the wave of deep learning in academia and industry.

Geoffrey Hinton

In 2011, the ReLU activation function was proposed, which can effectively suppress the gradient vanishing problem. Since 2011, Microsoft has made a major breakthrough in applying DL to speech recognition for the first time. Microsoft Research and Google's speech recognition researchers have successively used DNN technology to reduce the speech recognition error rate by 20%~30%, which is the biggest breakthrough in the field of speech recognition for more than ten years. In 2012, DNN technology achieved amazing results in the field of image recognition, reducing the error rate from 26% to 15% on ImageNet reviews. During the year, DNN was also applied to the DrugeActivity prediction problem for pharmaceutical companies and received the world's best results.

Outbreak period 2012 - 2017

In 2012, in order to prove the potential of deep learning, Hinton's group participated in the ImageNet image recognition competition for the first time, and won the championship by building the CNN network AlexNet, and crushed the classification performance of the second place (SVM method). It is precisely because of the competition that CNN has attracted the attention of many researchers.

AlexNet's innovation lies in:

(1) The ReLU activation function is used for the first time, which greatly increases the convergence speed and fundamentally solves the problem of gradient disappearance.

(2) Since the ReLU method can well suppress the gradient vanishing problem, AlexNet abandons the "pre-training fine-tuning" method and completely adopts supervised training. Because of this, DL's mainstream learning method has become pure supervised learning.

(3) The LeNet5 structure is extended, the Dropout layer is added to reduce overfitting, and the LRN layer enhances the generalization ability/reduces overfitting.

(4) The first time to use the GPU-accelerated model calculation.

In 2013, 2014, 2015 and 2016, through the ImageNet image recognition competition, DL network structure, training methods, and GPU hardware continued to improve, prompting it to continue to conquer the battlefield in other fields.

In March 2016, AlphaGo (based on deep learning), developed by Google's DeepMind, played a Go man-machine battle with Go world champion and professional nine-dan player Lee Sedol, winning 4-1 on aggregate; In late 2016 and early 2017, the program used "Master" as a registered account on the Chinese chess website to play fast chess duels with dozens of Go masters in China, Japan and South Korea, and did not lose a single game for 60 consecutive games; In May 2017, at the Go Summit in Wuzhen, China, it played against world number one Go champion Ke Jie, winning 3-0 on aggregate. The Go community recognizes that the chess power of Alpha Go has exceeded the top level of human professional Go.

Summary

Deep learning is still in the development stage, whether it is theoretical or practical, there are still many problems to be solved, but because we are in a "big data" era, and the greatly improved computing resources, the verification cycle of new models and new theories will be greatly shortened. The opening of the era of artificial intelligence will inevitably change the world to a great extent, whether it is from transportation, medical care, shopping, military and other aspects, perhaps we are in the best era, or maybe we are in the worst era, the future is unpredictable, then with an optimistic attitude to meet this third industrial revolution.