The rise and fall of artificial neural networks is well documented in the scientific literature of both computer science and computational chemistry. Yet almost two decades later, we are now seeing a resurgence of interest in deep learning, a machine learning algorithm based on multilayer neural networks. Within the last few years, we have seen the transformative impact of deep learning in many domains, particularly in speech recognition and computer vision, to the extent that the majority of expert practitioners in those field are now regularly eschewing prior established models in favor of deep learning models. In this review, we provide an introductory overview into the theory of deep neural networks and their unique properties that distinguish them from traditional machine learning algorithms used in cheminformatics. By providing an overview of the variety of emerging applications of deep neural networks, we highlight its ubiquity and broad applicability to a wide range of challenges in the field, including QSAR, virtual screening, protein structure prediction, quantum chemistry, materials design and property prediction. In reviewing the performance of deep neural networks, we observed a consistent outperformance against nonneural networks state-of-the-art models across disparate research topics, and deep neural network based models often exceeded the "glass ceiling" expectations of their respective tasks. Coupled with the maturity of GPU-accelerated computing for training deep neural networks and the exponential growth of chemical data on which to train these networks on, we anticipate that deep learning algorithms will be a valuable tool for computational chemistry. 3 Deep Learning is the key algorithm used in the development of AlphaGo, a Deep learning is a machine learning algorithm, not unlike those already in use in various applications in computational chemistry, from computer-aided drug design to materials property prediction. [5] [6] [7] [8] Amongst some of its more high profile achievements include the Merck activity prediction challenge in 2012, where a deep neural network not only won the competition and outperformed Merck's internal baseline model, but did so without having a single chemist or biologist in their team. In a repeated success by a different research team, deep learning models achieved top positions in the Tox21 toxicity prediction challenge issued by NIH in 2014. 9 The unusually stellar performance of deep learning models in both predicting activity and toxicity in these recent examples, originate from the unique characteristics that distinguishes deep learning from traditional machine learning algorithms. For those unfamiliar with the intricacies of machine learning algorithms, we will highlight some of the key differences between traditional (shallow) machine learning and deep learning. The simplest example of a machine learning algorithm would be the ubiquitous least-squares linear regression. In linear regression, the underlying nature of the model is known (linear in this The mapping of features into an output using the function provided is the task of a processing unit, and deep learning algorithms are constructed from a collection of these processing units, arranged in a layered and hierarchical fashion. Therefore, unlike simpler machine learning algorithms, it maps features through a series of non-linear functions that are stitched together in a combinatorial fashion to optimally maximize the accuracy of the model. As a result of this complex architecture, a deep learning algorithm learns multiple levels of representations that correspond to different levels of abstraction, forming a hierarchy of concepts. By constructing these so-called "hierarchical representations," deep learning has an internal state that may be transferred to new problems, partially overcoming the template matching problem. Finally, information can take 5 many paths through the network, and, as a result, expressive power grows exponentially with depth. 10 Ultimately, these unique characteristics of deep learning enables it to utilize raw data directly as opposed to engineered features, and often the resulting models constructed produce a comparable level of predictive accuracy. Deep learning achieves this ability because within the multiple layers of the non-linear functions, the algorithm transforms the raw data and maps it to intermediate "output" that serve as input (features) for the latter layers in the algorithm, in the process gradually transforming raw data into learned features. In short, deep learning algorithms are potentially capable of automatically (i.e. without expert intervention) engineering the necessary features that are relevant to optimally predict the output of interest. The majority of deep learning algorithms currently developed are based off artificial neural networks, and for the purpose of this review we will focus on deep neural networks exclusively. In the first half of this review, we will provide a brief non-technical introduction to deep learning, starting with a basic background on artificial neural networks and highlighting the key technical developments in the last decade that enabled deep neural networks. In addition, we will focus on how deep learning differs from traditional machine learning algorithms that are used in computational chemistry, and how the ongoing resurgence of deep learning differs from artificial neural network models in the 1980s, which may be regarded as its "parent" algorithm. In the next half of the review, we will include a survey of recent developments of deep learning applications across the field of computational chemistry, where we will examine its performance against existing machine learning models, and future prospects for contributing to the field. This review was written primarily to serve as an introductory entry point for computational chemists who wish to explore or integrate deep learning models in their research from an applications standpoint, and 6 additional references to existing literature reviews will be provided to cover the more technical aspects of deep learning neural network architecture and optimization. Artificial neural networks (ANNs), on which most deep learning algorithms are based on, are a class of machine learning algorithm inspired by biological neural networks, used to estimate or approximate functions by translating a large number of inputs into a target output (Figure 1a ). ANNs are constructed from a series of layers, and each layer comprises many "neurons". Each neuron accepts an input value from the previous layer, and maps it onto a non-linear function. The output of this function is used as the input for the next layer in the ANN, until it reaches the last layer, where the output corresponds to the objective that is to be predicted. In addition, a tunable parameter, the "weight" (or coefficient) of each neuron's function is adjusted in the construction of this model to minimize the error of the predicted value, a process known as "training" the neural network. Figuratively, the collection of these neurons in ANNs mimics the way neurons work in biological systems, hence its name, artificial neural network. The power of ANNs, as hinted earlier, lies in their ability to make multiple non-linear transformations through many hidden layers of neurons, where the "hidden" term refers to layers that are not directly adjacent to the input or output. In this process, increasingly complex and abstract features can be constructed, through the addition of more layers and/or increasing the width of layers (i.e. increasing the number of neurons per layer). Correspondingly, the model can learn increasingly complex and abstract representations (i.e. "concepts" if the term is used loosely). However, for one to use more than a single hidden layer, it is necessary to determine how to assign error attribution and make corrections to its weights by working backwards originating from the predicted output, and back through the neural network. This backwards propagation of errors is known formally as "backpropagation". Although the conceptual foundation of backpropagation was discovered in 1963, 11 it was not until 1986 that Hinton and co-workers discovered a way for this algorithm to be applied to neural networks, 12 which was a key historical development that enabled practically usable ANNs. During the process of backpropagation, an algorithm known as gradient descent is used to find the minimum in the error surface caused by each respective neuron when generating a corresponding output. Conceptually, gradient descent is no different from the steepest descent algorithm used in classical molecular dynamics simulation. The major difference is instead of iteratively minimizing an energy function and updating atomic coordinates for each step, an error function of the target output of the ANN is iteratively minimized and the weights of the neurons 8 are updated each step, which are also known as "iteration" in the ANN literature. The data in the training set may be iterated over multiple times, with a complete pass over the data being called an "epoch." A key issue with backpropagation is that the error signals become progressively more diffused as the signal goes back through each hidden layer. This is because, as the signal goes deeper into the model, an increasing number of neurons and weights are associated with a given error. Until recently, this made it difficult to train many layers efficiently; anything more than a few layers that required a long time to converge with a high probability of overfitting, especially for the layers closest to the output. In addition, the nonlinear transformation functions, such as sigmoids, had finite dynamic range, so error signals tends to decay as they passed through multiple layers, which is more commonly known as the "vanishing gradient problem". As these methods enabled the training of deeper and more complex neural network architecture, overfitting also became more of an issue, which led to the development of the dropout algorithm. In dropout, for each epoch of the training process, a fixed proportion of neurons are randomly selected to be temporarily excluded from the model. The net effect of dropout is that it simulates many different architectures during training, which prevents co-dependency among neurons and reduces overfitting. 16 Whilst the architecture of modern DNNs vary widely, a popular configuration is ReLU-based neural networks. When coupled with dropout and early stopping, such ReLU networks have often been enough to regularize the model (i.e. prevent overfitting). Having provided a summary of the key developments in ANNs and its associated algorithms, we note that it is by no means comprehensive. In addition to the traditional feedforward DNN (Figure 1b ) that has been discussed thus far, more recent developments include alternative architectures, notably convolutional neural networks (Figure 3a) , 18, 19 Many decades of chemistry research has led to the development of several thousand molecular descriptors that describe a range of properties of conceivably any compound. Molecular descriptors thus serve as features constructed using chemical knowledge and intuition (i.e. domain expertise) that can be used in traditional machine learning models, which have achieved reasonable success in computational chemistry applications. [26] [27] [28] [29] [30] [31] Traditional machine learning algorithms such as linear regression and decision trees are intuitive and create simple models that humans can understand. Nevertheless, as we progress to the prediction of more complex properties with nonlinear relationship, typically those associated with biological processes and materials engineering, it is often necessary to rely on more sophisticated and less transparent algorithms such as support vector machines (SVM) and random forests (RF) in order to achieve an acceptable level of predictive accuracy. At a first glance, deep learning algorithms falls under the latter category, but it has one major difference. Unlike SVMs and RFs, DNNs transform inputs and reconstruct them 12 into a distributed representation across the neurons of the hidden layers. With appropriate training methods, different features will be learned by the neurons in the hidden layers of the system; this is referred to as automatic feature extraction. As each hidden layer becomes the input for the next layer of the system and non-linear transformations can be applied along the way, it creates a model that progressively "learns" increasingly abstract, hierarchical and deep features. Automatic feature extraction, a process that requires no domain knowledge, is therefore one of the most significant benefits of a deep learning algorithm. This is unlike traditional machine learning algorithms, where a model must be carefully constructed with the "correct" features based off chemical knowledge and intuition for it to perform and generalize well. It is for this reason, that deep learning has become the dominant algorithm used in speech recognition 32 and computer vision 18,33-35 today. ImageNet is an annual assessment and competition of various algorithms for image classification. Prior to deep learning, the state-of-the-art models employed hovered in the 25-30% error rate, which falls short from the ideal goal of matching a trained human error rate of 5.1%. 36 In 2012, deep learning algorithms were first introduced to this community by Hinton and co-workers, 18 and their DNN-based model achieved a 16.4% error rate. That was a significant improvement from established models in computer vision at that time, and the second-best performing model based off traditional machine learning algorithms only achieved a 26.2% error rate. Subsequent improvements in DNN-based models eventually achieved an error rate of under 5.
For example, neural networks have been readily adopted in many chemistry applications such as predicting molecular properties from descriptors, protein contact prediction and computational material design REF .
Deep Learning for Computational Chemistry
Journal of computational chemistry
Journal of computational chemistry
Mathematics Computer Science Physics Medicine