Development

## Overview

From the beginning to now, artificial intelligence has been exploring on a road full of unknowns, with ups and downs. We can roughly divide this development process into five stages:

Initial development period: 1943-1960s

Reflective Development Period: 1970s

Application development period: 1980s

Steady development period: 1990s-2010

Booming period: 2011 to present.

## Development

### Initial development period: 1943-1960s

After the concept of artificial intelligence was proposed, symbolism and connectionism (neural network) were developed, and a number of remarkable research results have been obtained one after another, such as machine theorem proving, checkers program, man-machine dialogue, etc., which set off the development of artificial intelligence. First orgasm.

* In 1943, American neuroscientist Warren McCulloch and logician Water Pitts proposed a mathematical model of neurons, which is one of the cornerstones of modern artificial intelligence.
* In 1950, Alan Mathison Turing (Alan Mathison Turing) proposed the "Turing Test" (testing whether a machine can exhibit intelligence that is indistinguishable from humans), and the idea of making machines intelligent began to enter people's vision.
* In 1950, Claude Shannon proposed computer games.
* In 1956, the term artificial intelligence (AI) was officially used at the Summer Symposium on Artificial Intelligence at Dartmouth College. This is the first artificial intelligence seminar in human history, marking the birth of the discipline of artificial intelligence.
* In 1957, Frank Rosenblatt (Frank Rosenblatt) simulated on an IBM-704 computer a neural network model he invented called "Perceptron".
* In 1958, David Cox proposed logistic regression.
* In 1959, Arthur Samuel gave machine learning a clear concept: Field of study that gives computers the ability to learn without being explicitly programmed. (Machine learning is the study of how to enable computers to learn without explicit programs).
* In 1961, Leonard Merrick Uhr and Charles M Vossler published a pattern recognition paper titled A Pattern Recognition Program That Generates, Evaluates and Adjusts its Own Operators, which described a pattern recognition program designed using machine learning or self-organizing processes try.
* In 1965, I. J. Good (I. J. Good) published an article on the threat that artificial intelligence may pose to human beings in the future, which can be regarded as the pioneer of "AI threat theory". He believes that the superintelligence of machines and the inevitable explosion of intelligence will eventually be beyond the control of human beings. Later, famous scientist Stephen Hawking, inventor Musk and others' horror predictions about artificial intelligence echoed Goode's warning half a world ago.
* In 1966, Joseph Weizenbaum, a scientist at the Massachusetts Institute of Technology, published an article entitled "ELIZA-a computer program for the study of natural language communication between man and machine" on ACM, describing how the ELIZA program makes humans and computers communicate to a certain extent. It is possible to conduct natural language dialogues. The implementation technology of ELIZA is to decompose the input through keyword matching rules, and then generate replies according to the reorganization rules corresponding to the decomposition rules.
* In 1967, Thomas et al. proposed the K nearest neighbor algorithm (The nearest neighbor algorithm).
* In 1959, Arthur Samuel gave machine learning a clear concept: Field of study that gives computers the ability to learn without being explicitly programmed. (Machine learning is the study of how to enable computers to learn without explicit programs).
* In 1961, Leonard Merrick Uhr and Charles M Vossler published a pattern recognition paper titled A Pattern Recognition Program That Generates, Evaluates and Adjusts its Own Operators, which described a pattern recognition program designed using machine learning or self-organizing processes try.
* In 1965, I. J. Good (I. J. Good) published an article on the threat that artificial intelligence may pose to human beings in the future, which can be regarded as the pioneer of "AI threat theory". He believes that the superintelligence of machines and the inevitable explosion of intelligence will eventually be beyond the control of human beings. Later, famous scientist Stephen Hawking, inventor Musk and others' horror predictions about artificial intelligence echoed Goode's warning half a world ago.
* In 1966, Joseph Weizenbaum, a scientist at the Massachusetts Institute of Technology, published an article entitled "ELIZA-a computer program for the study of natural language communication between man and machine" on ACM, describing how the ELIZA program makes humans and computers communicate to a certain extent. It is possible to conduct natural language dialogues. The implementation technology of ELIZA is to decompose the input through keyword matching rules, and then generate replies according to the reorganization rules corresponding to the decomposition rules.
* In 1967, Thomas et al. proposed the K nearest neighbor algorithm (The nearest neighbor algorithm).
* In 1969, the book "Perceptron" by Marvin Minsky, a representative of "symbolicism", raised the problem of linear inseparability of XOR: a single-layer perceptron cannot divide XOR original data, and to solve this problem requires the introduction of higher-dimensional non-linear Linear network (MLP, requires at least two layers), but there is no effective training algorithm for multi-layer networks. These arguments have dealt a heavy blow to neural network research, and the research of neural network has reached a 10-year low ebb.

### Reflective Development Period: 1970s

The breakthroughs in the early stages of artificial intelligence development greatly raised people's expectations for artificial intelligence, and people began to try more challenging tasks. However, the lack of computing power and theory made unrealistic goals come to nothing. The development of artificial intelligence entered trough.

* In 1974, in the doctoral dissertation of Paul Werbos of Harvard University, it was first proposed to train the artificial neural network through error backpropagation (BP), but it did not attract attention during this period.
* In 1975, Marvin Minsky proposed a knowledge representation learning framework theory for artificial intelligence in his paper "A Framework for Representing Knowledge".
* In 1976, Randall Davis (Randall Davis) built and maintained a large-scale knowledge base, and proposed that the use of an integrated object-oriented model can improve the integrity of knowledge base (KB) development, maintenance and use.
* In 1976, Edward H. Shortliffe of Stanford University and others completed the first medical expert system MYCIN for the diagnosis, treatment and consultation of blood infectious diseases.
* In 1976, Dr. Lenart of Stanford University published the paper "Artificial Intelligence Methods Discovered in Mathematics-Heuristic Search", describing a program called "AM", which develops new concepts of mathematics under the guidance of a large number of heuristic rules. , eventually rediscovering hundreds of common concepts and theorems.
* In 1977, the logic-based machine learning system of Hayes. Roth and others made great progress, but it could only learn a single concept and failed to be put into practical application.
* In 1979, a computer program built by Hans Berliner became iconic when it defeated the world champion backgammon. (Subsequently, behavior-based robotics developed rapidly under the impetus of Rodney Brooks and Sutton, etc., and became an important development branch of artificial intelligence. The self-learning backgammon program created by Gerry Tesoro and others It also laid the foundation for the later development of reinforcement learning.)

### Application development period: 1980s

Artificial intelligence has entered a new upsurge in application development. The expert system simulates the knowledge and experience of human experts to solve problems in specific fields and realizes a major breakthrough in artificial intelligence from theoretical research to practical application, from general reasoning strategies to the use of specialized knowledge. And machine learning (especially neural network) explores different learning strategies and various learning methods, and it is slowly recovering in a large number of practical applications.

* In 1980, the first International Symposium on Machine Learning was held at Carnegie Mellon University (CMU) in the United States, marking the rise of machine learning research all over the world.
* In 1980, Drew McDermott and Jon Doyle proposed non-monotonic logic, and later robotic systems.
* In 1980, Carnegie Mellon University developed an expert system called XCON for DEC, which saved the company $40 million a year and achieved great success.
* In 1981, Paul (R.P.Paul) published the first robotics textbook, "Robot Manipulator: Mathematics, Programmings and Control", marking the maturity of the robotics discipline.
* In 1982, David Marr published his masterpiece "Visual Computing Theory" and proposed the concept of computer vision (Computer Vision), and constructed a systematic vision theory, which also had a profound impact on cognitive science (CognitiveScience).
* In 1982, John Hopfield (John Hopfield) invented the Hopfield network, which is the prototype of the earliest RNN. The Hopfield neural network model is a single-layer feedback neural network (the neural network structure can be mainly divided into feedforward neural network, feedback neural network and graph network), and there is a feedback connection from output to input. Its appearance has inspired the field of neural networks, and it has been widely used in machine learning of artificial intelligence, associative memory, pattern recognition, optimization computing, parallel implementation of VLSI and optical devices, etc.
* In 1983, Terrence Sejnowski, Hinton and others invented Boltzmann Machines (Boltzmann Machines), also known as random Hopfield network, which is essentially an unsupervised model for reconstructing input data to extract data features for predictive analysis.
* In 1985, Judea Pearl proposed Bayesian network (Bayesian network). He is famous for advocating the probabilistic method of artificial intelligence and developing Bayesian network. Inference theory is praised.
* In 1986, Rodney Brooks (Brooks) published the paper "Robust Hierarchical Control System for Mobile Robots", which marked the establishment of the subject of behavior-based robotics, and the robotics community began to focus on practical engineering topics.
* In 1986, Geoffrey Hinton and others successively proposed the concept of combining multi-layer perceptron (MLP) and backpropagation (BP) training (this method still had many challenges in terms of computing power at the time, basically It is related to the gradient algorithm of the chain derivation), which also solves the problem that the single-layer perceptron cannot do nonlinear classification, and opens a new round of climax of the neural network.
* In 1989, George Cybenko proved the "universal approximation theorem" (universal approximation theorem). Simply put, a multi-layer feedforward network can approximate any function, and its expressive power is equivalent to that of a Turing machine. This fundamentally eliminates Minsky's doubts about the expressiveness of neural networks.

### Period of steady development: 1990s-2010

Due to the rapid development of Internet technology, the innovative research of artificial intelligence has been accelerated, and artificial intelligence technology has been further moved towards practicality, and various fields related to artificial intelligence have made great progress. In the early 2000s, the focus of AI research shifted from knowledge-based systems to machine learning because expert system projects required encoding too many explicit rules, which reduced efficiency and increased costs.

* In 1995, Cortes and Vapnik proposed the classic connectionist support vector machine (Support Vector Machine), which shows many unique advantages in solving small samples, nonlinear and high-dimensional pattern recognition, and can be extended to function fitting, etc. among other machine learning problems.
* In 1995, Freund and Schapire proposed the AdaBoost (Adaptive Boosting) algorithm. AdaBoost uses the Boosting ensemble learning method - serial combination of weak learners to achieve better generalization performance. Another important integration method is the parallel combination of Bagging represented by random forest. Based on the "bias-variance decomposition" analysis, the Boosting method mainly optimizes the deviation, and the Bagging mainly optimizes the variance.
* In 1995, Freund and Schapire proposed the AdaBoost (Adaptive Boosting) algorithm. AdaBoost uses the Boosting ensemble learning method - serial combination of weak learners to achieve better generalization performance. Another important integration method is the parallel combination of Bagging represented by random forest. Based on the "bias-variance decomposition" analysis, the Boosting method mainly optimizes the deviation, and the Bagging mainly optimizes the variance.
* In 1998, Tim Berners-Lee of the World Wide Web Consortium proposed the concept of the Semantic Web. Its core idea is: by adding semantics (Meta data) that can be understood by computers to documents on the World Wide Web (such as HTML), so as to make the entire Internet a universal information exchange medium based on semantic links. In other words, it is to build an intelligent network that can realize barrier-free communication between humans and computers.
* In 2001, John Lafferty first proposed the conditional random field model (Conditional random field, CRF). CRF is a discriminative probabilistic graphical model based on the Bayesian theoretical framework. Given a conditional random field P ( Y ∣ X ) and an input sequence x, find the output sequence y \* with the highest conditional probability. It performs particularly well in many natural language processing tasks such as word segmentation and named entity recognition.
* In 2001, Dr. Breiman proposed Random Forest. Random forest is an integrated learning method that combines multiple different weak learners (decision trees) Bagging in parallel to optimize generalization performance by establishing multiple better-fitting and different models to combine decision-making. Diverse differences can reduce the dependence on certain feature noise, reduce variance (overfitting), and combined decision-making can eliminate some deviations between learners.
* In 2003, David Blei, Andrew Ng and Michael I. Jordan proposed LDA (Latent Dirichlet Allocation) in 2003.
* In 2005, Boston Dynamics launched a power-balanced four-legged robot dog, which has strong versatility and can adapt to more complex terrain.
* In 2006, Jeffrey Hinton and his student Ruslan Salahudinoff formally proposed the concept of deep learning (Deep Learning), which opened the wave of deep learning in academia and industry. 2006 is also known as the first year of deep learning, and Jeffrey Hinton is therefore known as the father of deep learning.
* In 2010, Sinno Jialin Pan and Qiang Yang published the article "A Survey of Transfer Learning".

### Booming period: 2011 to present

With the development of information technologies such as big data, cloud computing, the Internet, and the Internet of Things, computing platforms such as ubiquitous perception data and graphics processors have promoted the rapid development of artificial intelligence technology represented by deep neural networks, which has greatly surpassed the gap between science and application. Artificial intelligence technologies such as image classification, speech recognition, knowledge quizzes, man-machine games, and unmanned driving have achieved major technological breakthroughs and ushered in a new climax of explosive growth.

* In 2011, the IBM Watson question-answering robot participated in the Jeopardy answering quiz competition and won the championship. Watson is a computer question answering (Q&A) system that integrates technologies such as natural language processing, knowledge representation, automatic reasoning, and machine learning.
* In 2012, the AlexNet neural network model designed by Hinton and his student Alex Krizhevsky won the ImageNet competition. This is the first time in history that a model has performed so well in the ImageNet dataset and ignited the enthusiasm for neural network research.
* In 2012, Google officially released the Google Knowledge Graph (Google Knowledge Graph), which is a knowledge base collected from various information sources by Google. It uses Knowledge Graph to superimpose a layer of mutual relationships on ordinary string searches, assisting While users find the information they need faster, they can also take a step forward in knowledge-based search to improve the quality of Google search.
* In 2013, Durk Kingma and Max Welling proposed the Variational Auto-Encoder (VAE) in the article "Auto-Encoding Variational Bayes" on ICLR.
* In 2013, Durk Kingma and Max Welling proposed the Variational Auto-Encoder (VAE) in the article "Auto-Encoding Variational Bayes" on ICLR.
* In 2014, the chat program "Eugene Goostman" (Eugene Goostman) "passed" the Turing test for the first time at the "2014 Turing Test" conference held by the Royal Society of England.
* In 2014, Goodfellow and Bengio et al. proposed Generative Adversarial Network (GAN), known as the coolest neural network in recent years.
* In 2015, to commemorate the 60th anniversary of the concept of artificial intelligence, the three deep learning giants LeCun, Bengio and Hinton (who jointly won the Turing Award in 2018) launched a joint review of deep learning "Deep learning".
* In 2015, Google open sourced the TensorFlow framework. It is a symbolic mathematical system based on dataflow programming, which is widely used in the programming of various machine learning algorithms. Its predecessor is Google's neural network algorithm library DistBelief.
* In 2015, Musk and others co-founded OpenAI. It is a not-for-profit research organization whose mission is to ensure that artificial general intelligence (a system that is highly autonomous and outperforms humans in most tasks of economic value) will benefit all human beings. It releases popular products such as: OpenAI Gym, GPT, etc.
* In 2016, Google proposed the federated learning method, which trains algorithms on multiple distributed edge devices or servers that hold local data samples without exchanging their data samples.
* In 2017, Sophia, a humanoid robot developed by Hanson Robotics in Hong Kong, China, was the first robot in history to be granted citizenship. Sophia looks like a human female, with rubber skin and the ability to display more than 62 natural facial expressions. Algorithms in its "brain" can understand language, recognize faces, and interact with people.
* In 2018, Google proposed the paper "Pre-training of Deep Bidirectional Transformers for Language Understanding" and released the Bert (Bidirectional Encoder Representation from Transformers) model, which successfully achieved state of the art results in 11 NLP tasks.
* In 2019, IBM announced the Q System One, the world's first integrated general-purpose approximate quantum computing system designed for scientific and commercial use.
* In 2019, the research team of Hong Kong Insilico Medicine Company and the University of Toronto achieved a major experimental breakthrough, and discovered several drug candidates through deep learning and generative model-related technologies, which proved the effectiveness of the AI discovery molecular strategy and largely solved the problem of Traditional new drug development is difficult and time-consuming in molecular identification.
* In 2020, Google and Facebook respectively proposed SimCLR and MoCo two unsupervised learning algorithms, both of which can learn image data representation on unlabeled data. The framework behind the two algorithms is contrastive learning, and the core training signal of contrastive learning is the "distinguishability" of pictures.
* In 2020, the text generation (text generation) artificial intelligence GPT-3 developed by OpenAI has a natural language deep learning model with 175 billion parameters, which is 100 times higher than the previous version GPT-2. Word pre-training can achieve state-of-the-art performance on multiple NLP task benchmarks (question answering, translation, article writing).
* In 2020, Neuralink, Musk's brain-computer interface (BCI) company, held a live broadcast, showing the brain activity of experimental pigs implanted with Neuralink devices.
* In 2020, the AlphaFold2 artificial intelligence system of Google's DeepMind effectively solved the milestone problem of protein structure prediction. It beat the rest of the participants in the International Competition for Protein Structure Prediction (CASP), predicting the three-dimensional structure of proteins with accuracy comparable to experimental techniques such as cryo-electron microscopy (cryo-EM), nuclear magnetic resonance or X-ray crystallography Comparable.
* In 2020, Pan Jianwei of the University of Science and Technology of China and others successfully built a quantum computing prototype "Nine Chapters" with 76 photons. It only takes 200 seconds to solve the mathematical algorithm "Gaussian Bose Sampling", while the world's fastest supercomputer currently takes 6 billion years.
* In 2021, OpenAI proposes two neural networks connecting text and images: DALL·E and CLIP. DALL·E can directly generate images based on text, and CLIP can complete the matching of images and text categories.
* In 2021, the German Eleuther artificial intelligence company launched an open source text AI model GPT-Neo in late March this year. The difference compared to GPT-3 is that it is open source and free.
* In 2021, researchers at Stanford University in the United States developed a brain-computer interface (brain–computer interface, BCI) for typing. This system can decode the imaginary handwriting actions of paralyzed patients from the neural activity of the motor cortex, and These handwriting actions are converted into text in real-time using a recurrent neural network (RNN) decoding method. The relevant research results were published in the journal Nature on May 13, 2021, with the title of the paper "High-performance brain-to-text communication via handwriting".