Deep learning is a broad set of techniques that uses multiple layers of representation to automatically learn relevant features directly from structured data. Recently, such techniques have yielded record-breaking results on a diverse set of difficult machine learning tasks in computer vision, speech recognition, and natural language processing. Despite the enormous success of deep learning, relatively little is understood theoretically about why these techniques are so successful at feature learning and compression. Here, we show that deep learning is intimately related to one of the most important and successful techniques in theoretical physics, the renormalization group (RG). RG is an iterative coarse-graining scheme that allows for the extraction of relevant features (i.e. operators) as a physical system is examined at different length scales. We construct an exact mapping from the variational renormalization group, first introduced by Kadanoff, and deep learning architectures based on Restricted Boltzmann Machines (RBMs). We illustrate these ideas using the nearest-neighbor Ising Model in one and two-dimensions. Our results suggests that deep learning algorithms may be employing a generalized RG-like scheme to learn relevant features from data. A central goal of modern machine learning research is to learn and extract important features directly from data. Among the most promising and successful techniques for accomplishing this goal are those associated with the emerging sub-discipline of deep learning. Deep learning uses multiple layers of representation to learn descriptive features directly from training data [1, 2] and has been successfully utilized, often achieving record breaking results, in difficult machine learning tasks including object labeling [3], speech recognition [4] , and natural language processing [5] . In this work, we will focus on a set of deep learning algorithms known as deep neural networks (DNNs) [6] . DNNs are biologically-inspired graphical statistical models that consist of multiple layers of "neurons", with units in one layer receiving inputs from units in the layer below them. Despite their enormous success, it is still unclear what advantages these deep, multi-layer architectures possess over shallower architectures with a similar number of parameters. In particular, it is still not well understood theoretically why DNNs are so successful at uncovering features in structured data. (But see [7] [8] [9] .) One possible explanation for the success of DNN architectures is that they can be viewed as an iterative coarse-graining scheme, where each new high-level layer of the neural network learns increasingly abstract higherlevel features from the data [1, 10] . The initial layers of the the DNN can be thought of as low-level feature detecters which are then fed into higher layers in the DNN which combine these low-level features into more abstract higher-level features, providing a useful, and at times reduced, representation of the data. By successively applying feature extraction, DNNs learn to deemphasize irrelevant features in the data while simultaneously learning relevant ones. (Note that in a supervised setting, such as classification, relevant and irrelevant are ultimately determined by the problem at hand. Here we are concerned solely with the unsupervised aspect of training DNNs, and the use of DNNs for compression [6] .) In what follows, we make this explanation precise. This successive coarse-graining procedure is reminiscent of one of the most successful and important tools in theoretical physics, the renormalization group (RG) [11, 12] . RG is an iterative coarse-graining procedure designed to tackle difficult physics problems involving many length scales. The central goal of RG is to extract relevant features of a physical system for describing phenomena at large length scales by integrating out (i.e. marginalizing over) short distance degrees of freedom. In any RG sequence, fluctuations are integrated out starting at the microscopic scale and then moving iteratively on to fluctuations at larger scales. Under this procedure, certain features, called relevant operators, become increasingly important while other features, dubbed irrelevant operators, have a diminishing effect on the physical properties of the system at large scales. In general, it is impossible to carry out the renormalization procedure exactly. Therefore, a number of approximate RG procedures have been developed in the theoretical physics community [12] [13] [14] [15] . One such approximate method is a class of variational "real-space" renormalization schemes introduced by Kadanoff for performing RG on spin systems [14, 16, 17 ]. Kadanoff's variational RG scheme introduces coarse-grained auxiliary, or "hidden", spins that are coupled to the physical spin systems through some unknown coupling parameters. A parameter-dependent free energy is calculated for the coarse-grained spin system from the coupled system by integrating out the physical spins. The coupling parameters are chosen through a variational procedure that minimizes the difference between the free energies of the physical and hidden spin systems. This ensures that the coarse-grained system preserves the long-distance information present in the physical system. Carrying out this
REF recently showed an intriguing connection between Renormalization group flow 5 and deep-learning.
1 4 O ct 2 01 4 An exact mapping between the Variational Renormalization Group and Deep Learning