Original: C OMPACT

Original: N EURAL N ETWORKS
Revised: N EURAL N ETWORKS

Original: BASED

Original: ON

Original: THE

Original: M ULTISCALE E NTANGLEMENT R ENORMALIZATION A NSATZ
Revised: M ULTISCALE E NTANGLEMENT R ENORMALIZATION A NSATZ

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The goal of this paper is to demonstrate a method for tensorizing neural networks based upon an efﬁcient way of approximating scale invariant quantum states, the 
Revised: This paper demonstrates a method for tensorizing neural networks based upon an efﬁcient way of approximating scale invariant quantum states, the Multi-scale

Original: Multi-scale Entanglement Renormalization Ansatz (MERA).
Revised: Entanglement Renormalization Ansatz (MERA).

Original: We employ MERA as a replacement for linear layers in a  neural network and test this implementation on the CIFAR-10 dataset.
Revised: We employ MERA as a replacement for the fully connected layers in a convolutional neural network and test this implementation on the CIFAR-10 dataset.

Original: The proposed method outperforms factorization using tensor trains, providing greater compression for the same level of accuracy and greater accuracy for the same level of compression.
Revised: The proposed method outperforms factorization using tensor trains, providing greater compression for the same level of accuracy and greater accuracy for the same level of compression.

Original: We demonstrate MERAlayers with 3900 times fewer parameters and a reduction in accuracy of less than 1% compared to the equivalent fully connected layers
Revised: We demonstrate MERA layers with 3900 times fewer parameters and a reduction in accuracy of less than 1% compared to the equivalent fully connected layers, scaling like O ( N log 2 3 ) .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The curse of dimensionality is a major bottleneck in machine learning, stemming from the exponential growth of variables with the number of modes in a data set (Cichocki et al.
Revised: The curse of dimensionality is a major bottleneck in machine learning, stemming from the exponential growth of variables with the number of modes in a data set (Cichocki et al.

Revised: Typically state-of-the-art convolutional neural networks have millions or billions of parameters.

Revised: However, previous work has demonstrated that representations stored in the network parameters can be highly compressed without signiﬁcant reduction in network performance (Novikov et al.

Revised: (2015), Garipov et al.

Revised: (2016), Hinton et al.

Original:  Descriptions of quantum mechanical systems raise a similar challenge; representing n qubits requires a rank- n tensor whose memory cost scales as 2 n .
Revised: Determining the best network architecture for a given task remains an open problem Descriptions of quantum mechanical systems raise a similar challenge; representing n d -dimensional particles requires a rank- n tensor whose memory cost scales as d n .

Original: Indeed, it was the promise of harnessing this that led Richard Feynman (Feynman (1982)) to suggest the possibility of quantum computation.
Revised: Indeed, it was the promise of harnessing this that led Richard Feynman (Feynman (1982)) to suggest the possibility of quantum computation.

Original: In the absence of a quantum computer, however, one must use compressed representations of quantum states.
Revised: In the absence of a quantum computer, however, one must use compressed representations of quantum states.

Original: A level of compression can be achieved by factorizing the tensorial description of the quantum wavefunction.
Revised: A level of compression can be achieved by factorizing the tensorial description of the quantum wavefunction.

Original: Many such factorizations are possible, the optimal structure of the factorization being determined by the structure of correlations in the quantum system being studied.
Revised: Many such factorizations are possible, the optimal structure of the factorization being determined by the structure of correlations in the quantum system being studied.

Original: A revolution in quantum mechanics was made by realizing that the best way to characterize the distribution of correlations and information in a state is by a quantity known as entanglement – loosely the mutual quantum information between partitions of a quantum system (Eisert et al.
Revised: A revolution in quantum mechanics was made by realizing that the best way to characterize the distribution of correlations and information in a state is by a quantity known as entanglement – loosely the mutual quantum information between partitions of a quantum system (Eisert et al.

Original: This has led to many successful applications of tensorial approaches to problems in solid state physics and quantum chemistry over the past 25 years (Or ´ us (2014), Kin-Lic Chan et al.
Revised: This has led to many successful applications of tensorial approaches to problems in solid state physics and quantum chemistry over the past 25 years (Or´us (2014), Kin-Lic Chan et al.

Original: Intriguing ideas have also emerged over the past few years attempting to bridge the successes of neural networks in machine learning with those of tensorial methods in quantum physics, both at a fundamental level (Lin et al.
Revised: Intriguing ideas have also emerged over the past few years attempting to bridge the successes of neural networks in machine learning with those of tensorial methods in quantum physics, both at a fundamental level (Lin et al.

Original: (2017), Mehta & Schwab (2014)), and as a practical tool for network design (
Revised: (2017), Mehta & Schwab (2014)), and as a practical tool for network design (

Original: Levine et al.
Revised: Levine et al.

Original: Recent work has suggested that entanglement itself is a useful quantiﬁer of the performance of neural networks (Levine et al.
Revised: Recent work has suggested that entanglement itself is a useful quantiﬁer of the performance of neural networks (Levine et al.

Original: (2017), Liu et al.
Revised: (2017), Liu et al.

Original: The simplest factorization employed in quantum systems is known as the matrix product state (Or´us (2014)).
Revised: The simplest factorization employed in quantum systems is known as the matrix product state (Or´us (2014)).

Original: In essence, it expresses the locality of information in certain quantum states.
Revised: In essence, it expresses the locality of information in certain quantum states.

Original: It has already been adopted to replace expensive linear layers in neural networks – in which context it has been independently termed tensor trains (Oseledets (2011)).
Revised: It has already been adopted to replace expensive linear layers in neural networks – in which context it has been independently termed tensor trains (Oseledets (2011)).

Original: This led to substantial compression of neural networks with only a small reduction in the accuracy (Novikov et al.
Revised: This led to substantial compression of neural networks with only a small reduction in the accuracy (Novikov et al.

Original: (2015), Garipov et al.
Revised: (2015), Garipov et al.

Original: Here we use a different tensor factorization – known as the Multi-scale Entanglement Renormalization Ansatz (MERA) – that encodes information in a hierarchical manner (Vidal (2008)).
Revised: Here we use a different tensor factorization – known as the Multi-scale Entanglement Renormalization Ansatz (MERA) – that encodes information in a hierarchical manner (Vidal (2008)).

Original: MERA works through a process of coarse graining or renormalization.
Revised: MERA works through a process of coarse graining or renormalization.

Original: There have been a number of papers looking at the relationship between renormalization and deep learning.
Revised: There have been a number of papers looking at the relationship between renormalization and deep learning.

Original: MERA is a concrete realization of such a renormalization procedure (Vidal (2009)) and so possesses a multi-scale structure that one might anticipate in complex data.
Revised: MERA is a concrete realization of such a renormalization procedure (Vidal (2009)) and so possesses a multi-scale structure that one might anticipate in complex data.

Original: A number of works have utilised tree tensor network models that possess a similar hierarchical structure.
Revised: A number of works have utilized tree tensor network models that possess a similar hierarchical structure.

Original: However,while some of these works claim to be MERA-inspired, they do not include the disentangler tensors that are essential if each layer of the MERA is to capture correlations on different length scales (Liu et al.
Revised: However, they do not include the disentangler tensors that are essential if each layer of the MERA is to capture correlations on different length scales (Liu et al.

Original: In this work we employ MERA as a replacement for linear layers in a neural network used to classify the CIFAR-10 dataset.
Revised: In this work we employ MERA as a replacement for linear layers in a neural network used to classify the CIFAR-10 dataset.

Original: Our results show that this performs better than the tensor train decomposition of the same linear layer, and gives better accuracy for the same level of compression and better compression for the same level of accuracy.
Revised: Our results show that this performs better than the tensor train decomposition of the same linear layer, and gives better accuracy for the same level of compression and better compression for the same level of accuracy.

Original: In Section 2 we introduce factorizations of fully connected linear layers, starting with the tensor train factorization followed by a tree-like factorization and ﬁnally the MERA factorization.
Revised: In Section 2 we introduce factorizations of fully connected linear layers, starting with the tensor train factorization followed by a tree-like factorization and ﬁnally the MERA factorization.

Original: In Section 3 we discuss how this is employed as a replacement for a fully connected linear layer in deep learning networks.
Revised: In Section 3 we discuss how this is employed as a replacement for a fully connected linear layer in deep learning networks.

Original: Section 4 gives our main results and we note connections with the existing literature in Section 5.
Revised: Section 4 gives our main results and we note connections with the existing literature in Section 5.

Original: Finally, in Section 6 we discuss some potential developments of the work.
Revised: Finally, in Section 6 we discuss some potential developments of the work.

Original:  T ENSOR F ACTORIZATION OF L INEAR L AYERS
Revised:  T ENSOR F ACTORIZATION OF L INEAR L AYERS

Revised: [Figure]

Original: {+N ααα N jji 2 ij 1 jj 3 jj 5 jj 7 ji 1 ii 3 ii 5 ii 
+}
Revised: {+N ααα N jji 2 ij 1 jj 3 jj 5 jj 7 ji 1 ii 3 ii 5 ii 
+}

Original:  [Figure]
Revised: i [Figure]

Original: Figure 1a gives a graphical representation of a rank 2 n tensor W i 1 ,i 2 ,...,i n j 1 ,j 2 ,...,j n and Figure 1b illustrates its tensor train decomposition.
Revised: j 1 jj 3 jj 5 jj 7 ji 1 ii 3 ii 5 ii 7 i [Figure] In this report we have replaced the linear layers of the standard neural network with tensorial MERA layers. The ﬁrst step in achieving this involves expressing a linear layer as a tensor.

Revised: This can be accomplished by taking a matrix W and reshaping it to be a higher dimensional array.

Revised: For example, suppose W is d n by d n dimensional with components W AB .

Revised: It can be transformed into a rank 2 n tensor by mapping A to n elements A → i 1 , i 2 , ..., i n and B to another n elements

Revised: B → j 1 , j 2 , ..., j n .

Revised: In this case each of the elements of the new tensor will be of size d .

Revised: Figure 1a gives a graphical representation of this rank 2 n tensor W i 1 ,i 2 ,...,i n j 1 ,j 2 ,...,j n .

Revised: It is important to note that in this representation, the lines represent the indices of the tensors rather than weights.

Revised: Figure 1b illustrates the tensor train decomposition of W .

Original: This consists of writing the larger tensor as the contraction of a train of smaller tensors:
Revised: This consists of writing the larger tensor as the contraction of a train of smaller tensors:

Original: [Equation]
Revised: [Equation]

Revised: In the tensor graphical notation, closed legs represent indices being summed over and free legs represent indices that aren’t being summed over.

Original:  If each index runs over values from 1 to d , this represents an exponential reduction from d 2 n parameters to n ( Dd ) 2 , where the indices α run over values from 1 to D (known as the bond order or Schmidt rank in the quantum context).
Revised: For example, in equation 1 the α i indices are being summed over and in Figure 1b the α i lines are connected to tensors at both ends. If each index runs over values from 1 to d , this represents an exponential reduction from d 2 n parameters to n ( Dd ) 2 , where the indices α run over values from 1 to D (known as the bond order or Schmidt rank in the quantum context).

Original: As noted above, this type of tensor factorization works well  when the information has a local structure (Eisert et al.
Revised: As noted above, this type of tensor factorization works well in physics when the information has a local structure (Eisert et al.

Original: (2010), Verstraete & Cirac (2006))
Revised: (2010), Verstraete & Cirac (2006))

Original: ; tensor trains capture correlations effectively up to length scales of order log D (Schollw¨ock (2011)).
Revised: ; tensor trains capture correlations effectively up to length scales of order log D (Schollw ¨ ock (2011)).

Original: Image data often possesses a more hierarchical structure.
Revised: This means that while useful for many tasks, the learned representations will be highly local. Tensors at either end of a tensor train decomposition of a linear layer will not be strongly correlated with one another. A hierarchically structured tensor network can better represent correlations across the linear layer.

Original: The tree tensor network shown in Figure 1c represents one possible hierarchical factorization.
Revised: The tree tensor network shown in Figure 1c represents one possible hierarchical factorization.

Original:  Reading from left to right this can be interpreted as follows: the tree-like connectivity imbues the network with a causal structure whereby a given linear element and its outputs are inﬂuenced by inputs in a region determined by its height in the tree.
Revised: Each element of this network is a rank 4 tensor tensors Reading from left to right Figure 1c can be interpreted as follows: the tree-like connectivity imbues the network with a causal structure whereby a given linear element and its outputs are inﬂuenced by inputs in a region determined by its height in the tree.

Original: For example, the rightmost element in Figure 1c is inﬂuenced by all of the inputs, whereas the top element in the middle column is inﬂuenced by inputs i 1 to i 4 .
Revised: For example, the rightmost element in Figure 1c is inﬂuenced by all of the inputs, whereas the top element in the middle column is inﬂuenced by inputs i 1 to i 4 .

Original: Elements other than the rightmost tensor have one dashed output (that connects directly to the overall output) and one solid output (that ties it to the branching tree structure).
Revised: Elements other than the rightmost tensor have one dashed output (that connects directly to the overall output) and one solid output (that ties it to the branching tree structure).

Original: These dashed lines are controlled by features occurring on a particular scale in the data.
Revised: These dashed lines are controlled by representations occurring on a particular scale in the data.

Original: Notice that removing these dashed lines, the network has a true tree structure and represents a coarse graining or renormalization of the network.
Revised: Notice that removing these dashed lines, the network has a true tree structure and represents a coarse graining or renormalization of the network.

Original: In this case, the linear elements are the isometries of the original MERA’s deﬁnition (Vidal (2008; 2009)).
Revised: In this case, the linear elements are the isometries of the original MERA’s deﬁnition (Vidal (2008; 2009)).

Original: The simple tree network, which has been studied before in the context of neural networks, has a major deﬁciency.
Revised: The simple tree network, which has been studied before in the context of neural networks, has a major deﬁciency.

Original: At each branching, it partitions the system in two, so that in extremis , the correlations between neighbouring outputs – for example i 4 and j 5 in Figure 1c – are only controlled by the element at the end of the network, thus breaking the  hierarchical representation of correlations.
Revised: At each branching, it partitions the system in two, so that in extremis , the correlations between neighbouring inputs – for example i 4 and i 5 in Figure 1c – are only controlled by the element at the end of the network. Requiring the higher elements in the tree-structure to capture correlations between neighbouring inputs restricts their ability to describe the longer length scale correlations you would hope to capture by using a hierarchical structure.

Original: The 
Revised: The MERA (Vidal (2009)) factorization was introduced in order to solve this problem. As can be seen in Figure 1d it adds an additional set of rank 4 tensors called disentanglers . The MERA is constructed by taking a tree network and placing one of these rank 4 tensors D β 1 ,β 2 γ 1 ,γ 2 such that its right-going legs β 1 and β 2 connect to two adjacent tensors of the tree network.

Original: MERA (Vidal (2009)) factorization shown in Figure 1d ﬁxes this by adding additional disentanglers , whose role is toreduce correlations between these partitions of the system as far as possible before they lose causal connection in the branching structure.
Revised: For example, if we consider the top left-most disentangler in Figure 1d it has elements D β 1 ,β 2 i 2 ,i 3 and connects to the tree elements M (cid:48) j 1 ,α 1 i 1 ,β 1 and N (cid:48) j 2 ,α 2 β 2 ,i 4 with β 1 and β 2 then being summed over.

Original: E XPERIMENTS & NETWORK STRUCTURE
Revised: The role of the disentanglers is to cause all correlations on the same length scale to be treated similarly. For example, correlations between any two neighbouring input indices i n and i n +1 will be captured by either the ﬁrst row of tree elements or the disentanglers. This allows the later elements in the network to work at capturing longer range correlations. In summary, a rank- N MERA layer can be constructed in the following manner:

Revised: Create a tree tensor layer.

Revised: For example, an N = 2 τ tree can be constructed from 2 τ −rank-4 tree tensors M β 1 ,β 2 γ 1 ,γ 2 in the ﬁrst layer, followed by 2 τ − 2 tree tensors in the second layer until after τ layers there is only a single tree tensor.

Revised: A set of disentanglers are introduced.

Revised: These are rank-4 tensors D β 1 ,β 2 γ 1 ,γ 2 which are placed such that every disentangler is contracted with two neighbouring tree tensors in an upcoming layer of the tree tensor.

Original: We have considered the performance of a neural network with the  fully connected layers of the model replaced with MERA layers, similar to the Novikov et al.
Revised: We have considered the performance of a neural network with the two penultimate fully connected layers of the model replaced with MERA layers, similar to the Novikov et al.

Original: (2015) study of compression of fully connected layers using tensor trains.
Revised: (2015) study of compression of fully connected layers using tensor trains.

Original: We have quantiﬁed the performance of the MERA layer through comparisons with two other classes of networks: fully connected layers with varying numbers of nodes and tensor train layers with varying internal dimension.
Revised: We have quantiﬁed the performance of the MERA layer through comparisons with two other classes of networks: fully connected layers with varying numbers of nodes and tensor train layers with varying internal dimension.

Original: The three types of network are otherwise identical.
Revised: The three types of network are otherwise identical.

Original: The networks consist of three sets of two convolutional layers each followed by max pooling layers.
Revised: The networks consisted of three sets of two convolutional layers each followed by max pooling layers with 3 × 3 kernels and stride 2 .

Original: The convolutional kernels were 3 × 3 .
Revised: The convolutional kernels were 3 × 3 .

Original: There were 64 channels in all of the convolutional layers except for the input, which had three channels, and the last convolutional layer, which haschannels.
Revised: There werechannels in all of the convolutional layers except for the input, which had three channels, and the last convolutional layer, which had 256 channels.

Original: The ﬁnal convolutional layer was followed by two more  layers, these were either fully connected, MERA-layers or TT-layers depending upon the network.
Revised: The ﬁnal convolutional layer was followed by two more hidden layers, these were either fully connected, MERA layers or TT-layers depending upon the network.

Original: The ﬁrst of these layers was of size 4096 × x , the second is of size x × 64 .
Revised: The ﬁrst of these layers was of size 4096 × x , the second is of size x × 64 .

Original: For all MERA and TT networks, these layers were 4096 × 4096 and 4096 × 64 .
Revised: For all MERA and TT networks, these layers were 4096 × 4096 and 4096 × 64 .

Original: The ﬁnal layer had 10 nodes corresponding to the 10 image classes in CIFAR-10.
Revised: The ﬁnal layer had 10 nodes corresponding to the 10 image classes in CIFAR-10.

Original: Leaky rectiﬁed linear units (LReLU) were used on all layers except the ﬁnal layer, with leak = 0 .
Revised: Leaky rectiﬁed linear units (LReLU) were used on all layers except the ﬁnal layer, with leak = 0 .

Original: 2 (Maas et al.
Revised: 2 (Maas et al.

Original: During training, nodes in the ﬁnal convolutional layer and the two ﬁrst fully connected layers were dropped with probability 0 .
Revised: During training, nodes in the ﬁnal convolutional layer and the two ﬁrst fully connected layers were dropped with probability 0 .

Original: The penultimate convolutional layer nodes were dropped with probability 0 .
Revised: The penultimate convolutional layer nodes were dropped with probability 0 .

Original: 2 (Srivastava et al.
Revised: 2 (Srivastava et al.

Original: Batch-normalization was used on all layers after dropout and max pooling (Ioffe & Szegedy (2015)).
Revised: Batch-normalization was used on all layers after dropout and max pooling (Ioffe & Szegedy (2015)).

Original: We did not use bias units.
Revised: We did not use bias units.

Original: Gaussian weight initialization was employed in the fully connected models with standard deviation equal to(cid:112) 1 /n in , where n in is the number of input channels to the layer (He et al.
Revised: Gaussian weight initialization was employed in the fully connected models with standard deviation equal to 1 √ n in , where n in was the number of inputs (He et al.

Original: In this report we considered networks with two varieties of fully-connected layers.
Revised: In this report we considered networks with two varieties of fully-connected layers.

Original: The ﬁrst of these networks has a 4096 × 4096 fully connected layer followed by one which is 4096 × 64 ; this network was used as a benchmark against which the other models could be compared.
Revised: The ﬁrst of these networks had a 4096 × 4096 fully connected layer followed by one which was 4096 × 64 ; this network was used as a benchmark against which the other models could be compared.

Original: The second network instead has a 4096 × 10 fully connected layer followed by a 10 × 64 layer.
Revised: The second network instead had a 4096 × 10 fully connected layer followed by a 10 × 64 layer.

Original: We trained this network to compare the MERA and tensor train layers to a fully connected model with a comparable number of parameters, in order to evaluate how detrimental naive compression is to accuracy.
Revised: We trained this network to compare the MERA and tensor train layers to a fully connected model with a comparable number of parameters, in order to evaluate how detrimental naive compression is to accuracy.

Original: A schematic of the two MERA-layers can be found in Figure 2.
Revised: A schematic of the two MERA layers can be found in Figure 2.

Original: The input to the MERA-layer was reshaped in to a rank-12 tensor with each index being dimension two.
Revised: The input to the ﬁrst MERA layer was reshaped in to a rank-12 tensor with each index being dimension 2 , as described in Section 2.

Original: The MERA-layer was then constructed from a set of rank- 4 tensorsB β k β l α i α j , using the method described in Section 2. 
Revised: The MERA layer was then constructed from a set of rank- 4 tensors using the method described in Section 2. The ﬁrst MERA layer works as follows:

Original: [Figure]
Revised: It contains a column of 6 rank- 4 tree elements, followed by 3 tree elements and ﬁnally a single tree element. 5 disentanglers are placed before the ﬁrst column of tree elements and 2 more disentanglers are placed before the second column of tree elements.

Original: In the second MERA-layer, the output index for the ﬁrst set of isometry tensors was ﬁxed to produce a  64 node output.
Revised: The second MERA layer has an identical structure to the ﬁrst MERA layer, one of the outputs of the ﬁrst set of tree elements is ﬁxed. As a result the output of the second MERA layer is 64 nodes.

Original: The dimensions of the internal indices in the MERA layers were allowed to vary, with a MERA- D being a MERA layer with internal dimension D .
Revised: The dimensions of the internal indices in the MERA layers were allowed to vary, with a MERA- D being a MERA layer with internal dimension D .

Original: MERA weights were initialized using elements of randomized orthogonal matrices (Saxe et al. 
Revised: MERA weights were initialized using elements of randomized orthogonal matrices (Saxe et al. The tensors themselves were constructed by reshaping these matrices, as described in

Revised: Section 2.

Original: – randomization was done with respect to the Haar measure.
Revised: The random orthogonal matrix was constructed using the method of Stewart (Stewart [Figure] (1980), Mezzadri (2007)). Starting from a random n − 1 × n − 1 dimensional orthogonal matrix, a random n × n dimensional orthogonal matrix can be constructed by taking a randomly distributed n -dimensional vector, constructing its Householder transformation, and then applying the n −dimensional matrix to this vector.

Original: Finally, a network with its fully connected layers replaced with a tensor train decomposition was trained in order to provide a comparison with the MERA-layers.
Revised: Finally, a network with its fully connected layers replaced with a tensor train decomposition was trained in order to provide a comparison with the MERA layers.

Original: The tensor train layers were constructed as described in Section 2 with the internal dimension being allowed to vary from D = 3 to
Revised: The tensor train layers were constructed as described in Section 2 with the internal dimension being allowed to vary from D = 3 to

Original: D
Revised: D

Original: = 5 .
Revised: = 5 .

Original: In the second tensor train layer, half of the output indices were ﬁxed to match the second
Revised: In the second tensor train layer, half of the output indices were ﬁxed to match the second

Original: MERA-layer.
Revised: MERA layer.

Original: We tested performance on the CIFAR-10 dataset.
Revised: We tested performance on the CIFAR-10 dataset.

Original: We used 50 , 000 images for training and 10 ,for testing.
Revised: We used 50 , 000 images for training and 10 ,for testing.

Original: We used 100 , 000 training iterations, each consisting of batches of 50 images.
Revised: Each training batch consisted of 50 images.

Original: Training data was augmentedonline by randomly ﬂipping and translating the input images by up to 4 pixels.
Revised: Training data was augmented by randomly ﬂipping and translating the input images by up to 4 pixels.

Original: Translated images were padded with zeros.
Revised: Translated images were padded with zeros.

Original: All images were normalized by dividing by 255 and subtracting the mean pixels value from the training set.
Revised: All images were normalized by dividing by 255 and subtracting the mean pixels value from the training set.

Original: Test accuracy was recorded every 500 iterations and training was stopped when test accuracy did not improve for 10 successive tests.
Revised: Test accuracy was recorded every 500 iterations and training was stopped when test accuracy did not improve for 10 successive tests.

Original: The network was trained using backpropagation and the Adam optimizer, with initial learning rate 0 .
Revised: The network was trained using backpropagation and the Adam optimizer, with initial learning rate 0 .

Original: 001 (Kingma & Ba (2014)) and a softmax-cross-entropy objective.
Revised: 001 (Kingma & Ba (2014)) and a softmax-cross-entropy objective.

Original: The networks were implemented in Tensorﬂow r1.3 and trained on NVIDIA Titan Xp and 1080ti
Revised: The networks were implemented in Tensorﬂow r1.3 and trained on NVIDIA Titan Xp and 1080ti

Original: GPUs.
Revised: GPUs.

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: In Table 1 we compare the number of parameters used, the compression rate and the accuracy obtained for each of the models described in Section 3.
Revised: In Table 1 we compare the number of parameters used, the compression rate and the accuracy obtained for each of the models described in Section 3.

Original: The compression rate stated is with respect to the number of parameters used in the fully-connected benchmark model, FC-1.
Revised: The compression rate stated is with respect to the number of parameters used in the fully-connected benchmark model, FC-1.

Original: When comparing the MERA networks to the fully connected model we can see a considerable drop in the number of parameters required with only a modest drop in the accuracy.
Revised: When comparing the MERA networks to the fully connected model we can see a considerable drop in the number of parameters required with only a modest drop in the accuracy.

Original: MERA-2 compressed the fully connected layers by a factor of nearly 5500 with an accuracy drop of 1 . 48% .
Revised: MERA-2 compressed the fully connected layers by a factor of nearly 5500 with an accuracy drop of 1 . 48% .

Original: Increasing the internal dimension of the MERA to 3 we found the compression rate decreased to approximately 3900 but the accuracy was 86 .
Revised: Increasing the internal dimension of the MERA to 3 we found the compression rate decreased to approximately 3900 but the accuracy was 86 .

Original: 2% , a drop of less than 1% compared to the fully connected model.
Revised: 2% , a drop of less than 1% compared to the fully connected model.

Original: The compression rate of an entire MERA network compared to the fully connected network was approximately 57 .
Revised: The compression rate of an entire MERA network compared to the fully connected network was approximately 57 .

Original: We did not attempt to compress the convolutional layers in this work, and therefore the vast majority of parameters were focused in these layers for the MERA networks.
Revised: We did not attempt to compress the convolutional layers in this work, and therefore the vast majority of parameters were focused in these layers for the MERA networks.

Original: How signiﬁcant is the MERA network structure we have chosen to the results obtained?
Revised: How signiﬁcant is the MERA network structure we have chosen to the results obtained?

Original: The results of the MERA networks can be compared against the fully connected layer with only 10 nodes, see
Revised: The results of the MERA networks can be compared against the fully connected layer with only 10 nodes, see

Original: FC-2 in Table 1.
Revised: FC-2 in Table 1.

Original: This network possesses 100 times more parameters than MERA-2 but the drop in the accuracy is 6 .
Revised: This network possesses 100 times more parameters than MERA-2 but the drop in the accuracy is 6 .

Original: 04% , signiﬁcantly more than any MERA network.
Revised: 04% , signiﬁcantly more than any MERA network.

Original: The MERA network also compares favourably to tensor train methods.
Revised: The MERA network also compares favourably to tensor train methods.

Original: TT-3 had a comparable compression rate to the MERA methods but a more signiﬁcant drop in accuracy, 2 . 22% .
Revised: TT-3 had a comparable compression rate to the MERA methods but a more signiﬁcant drop in accuracy, 2 . 22% .

Original: For a tensor train network to achieve an accuracy drop of only 1% the internal dimension had to be increased to 7, resulting in an inferior compression rate compared to MERA- 2 .
Revised: For a tensor train network to achieve an accuracy drop of only 1% the internal dimension had to be increased to 7, resulting in an inferior compression rate compared to MERA- 2 .

Original: In addition to the degree of compression achieved by these networks, we also address the time to optimize.
Revised: In addition to the degree of compression achieved by these networks, we also address the time to optimize.

Original: There is evidently a degree of compromise required here: the number of multiplications required to apply a MERA layer scales with the input size N and bond order D as N log 2 D .
Revised: There is evidently a degree of compromise required here: the number of multiplications required to apply a MERA layer scales with the input size N and bond order D as N log 2 D .

Original: The equivalent scaling for a tensor train and fully connected layer are ND 2 and N 2 , respectively.
Revised: The equivalent scaling for a tensor train and fully connected layer are ND 2 and N 2 , respectively.

Original: This is reﬂected in the times taken to optimize these networks.
Revised: This is reﬂected in the times taken to optimize these networks.

Original: Note however, that MERA can accommodate correlations at all scales of its input even at low bond order, whereas tensor trains require a bond order that scales exponentially with the length scale of correlation (Or´us (2014)).
Revised: Note however, that MERA can accommodate correlations at all scales of its input even at low bond order, whereas tensor trains require a bond order that scales exponentially with the length scale of correlation (Or´us (2014)).

Original: MERA is, therefore, expected to scale better for very large data sets than either tensor trains or fully connected layers. 
Revised: MERA is, therefore, expected to scale better for very large data sets than either tensor trains or fully connected layers. [Table]

Original: [Table]
Revised: [Table]

Revised: S TUDY OF ABLATED NETWORKS

Revised: As can be seen in Table 1, the number of parameters in the fully convolutional layers far outweigh the parameters in the fully connected, MERA or tensor train layers of the network.

Revised: In order to study the behaviour of the MERA and tensor train layers without the convolutional layers distorting the results we have created an ablated network with the number of parameters in the convolutional layers minimized.

Original: R ELATED W ORK
Revised: In this case the only convolutional layer was had a 1 × 1 kernel with 3 channels in and 4 channels out. This was followed by 4096 × 64 MERA or tensor train layer. The ﬁnal layer was a 64 ×layer as in the previous network structure. A MERA- 2 layer was used, with 232 parameters.

Revised: The tensor-train layer used a mixture of TT- 2 and TT- 3 so it would contain 236 parameters, as close as possible to the MERA- 2 layer.

Revised: For this simpliﬁed network we found that the MERA- 2 network achieved an accuracy of 47 .

Revised: 36% ( σ = 0 .

Revised: 48% ) while the tensor train achieved an accuracy of 46 .

Revised: 65% ( σ = 0 .

Original:  Given how memory intensive deep neural networks typically are, substantial effort has been made to reduce number of parameters these networks require without signiﬁcantly reducing their accuracy.
Revised: Evidence of a MERA layer once again outperforming a tensor train layer with a comparable number of parameters.  R ELATED W ORK Given how memory intensive deep neural networks typically are, substantial effort has been made to reduce number of parameters these networks require without signiﬁcantly reducing their accuracy.

Original: Some of these have taken a similar approach to the MERA network described above, using tensor decompositions of the fully connected layers.
Revised: Some of these have taken a similar approach to the MERA network described above, using tensor decompositions of the fully connected layers.

Original: These include the tensor train models of Novikov et al.
Revised: These include the tensor train models of Novikov et al.

Original: (2015) and Garipov et al.
Revised: (2015) and Garipov et al.

Original: Here we have found replacing a fully connected linear layer with a MERA factorization resulted in superior accuracy for a comparable number of parameters.
Revised: Here we have found replacing a fully connected linear layer with a MERA factorization resulted in superior accuracy for a comparable number of parameters.

Original: More directly related to this MERA model are a number of tree tensor network models (Liu et al.
Revised: More directly related to this MERA model are a number of tree tensor network models (Liu et al.

Original: (2017), Levine et al.
Revised: (2017), Levine et al.

Original: As section 2 explained, the disentangler tensors are essential if each layer of a MERA is to capture correlations on different length scales.
Revised: As Section 2 explained, tree tensor networks inconsistently capture correlations on the same length scale, this is the reason for the introduction of disentanglers.

Original: Tree tensors do not possess these and we expect them to struggle to capture long range correlations as effectively as MERA.
Revised: Tree tensors do not possess these and we expect them to struggle to capture long range correlations as effectively as MERA.

Original: A MERA works through a process of coarse graining or renormalization.
Revised: A MERA works through a process of coarse graining or renormalization.

Original: There have been a number of other papers looking at the relationship between renormalization and deep learning.
Revised: There have been a number of other papers looking at the relationship between renormalization and deep learning.

Original: Lin et al.
Revised: Lin et al.

Original: (2017) argue that the effectiveness of deep neural networks should be thought of in terms of renormalization and Mehta & Schwab (2014) demonstrate an exact mapping between the variational renormalization group and restricted Boltzmann machines.
Revised: (2017) argue that the effectiveness of deep neural networks should be thought of in terms of renormalization and Mehta & Schwab (2014) demonstrate an exact mapping between the variational renormalization group and restricted Boltzmann machines.

Original: In this report we have taken a different approach: only one layer of the network was replaced with a MERA.
Revised: In this report we have taken a different approach: only the fully connected layers of the network were replaced with MERA layers.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have shown that replacing the fully connected layers of a deep neural network with layers based upon the multi-scale entanglement renormalization ansatz can generate signiﬁcant efﬁciency gains with only small reduction in accuracy.
Revised: We have shown that replacing the fully connected layers of a deep neural network with layers based upon the multi-scale entanglement renormalization ansatz can generate signiﬁcant efﬁciency gains with only small reduction in accuracy.

Original: When applied to the CIFAR-10 data we found the fully connected layers can be replaced with MERA layers with 3900 times less parameters with a reduction in the accuracy of less than 1% .
Revised: When applied to the CIFAR-10 data we found the fully connected layers can be replaced with MERA layers with 3900 times less parameters with a reduction in the accuracy of less than 1% .

Original: The model signiﬁcantly outperformed compact fully connected layers with 70 − 100 times as many parameters.
Revised: The model signiﬁcantly outperformed compact fully connected layers with 70 − 100 times as many parameters.

Original: Moreover, it outperformed a similar replacement of the fully connected layers with tensor trains, both in terms of accuracy for a given compression and compression for a given accuracy.
Revised: Moreover, it outperformed a similar replacement of the fully connected layers with tensor trains, both in terms of accuracy for a given compression and compression for a given accuracy.

Original: An added advantage — not explored here — is that a factorized layer can potentially handle much larger input data sets, thus enabling entirely new types of computation.
Revised: An added advantage — not explored here — is that a factorized layer can potentially handle much larger input data sets, thus enabling entirely new types of computation.

Original: Correlations across these large inputs can be handled much more efﬁciently by MERA than by tensor trains.
Revised: Correlations across these large inputs can be handled much more efﬁciently by MERA than by tensor trains.

Original: Moreover, a compressed network may provide a convenient way to avoid over-ﬁtting of large data sets.
Revised: Moreover, a compressed network may provide a convenient way to avoid over-ﬁtting of large data sets.

Original: The compression achieved by networks with these factorized layers comes at a cost.
Revised: The compression achieved by networks with these factorized layers comes at a cost.

Original: They can take longer to train than networks containing the large fully connected layers due to the number of tensor contractions required to apply the factorized layer.
Revised: They can take longer to train than networks containing the large fully connected layers due to the number of tensor contractions required to apply the factorized layer.

Original: Our results suggest several immediate directions for future inquiry.
Revised: Our results suggest several immediate directions for future inquiry.

Original: Firstly, there are some questions about how to improve the existing model.
Revised: Firstly, there are some questions about how to improve the existing model.

Original: For example, before the MERA layer is used the input is reshaped into a rank-12 tensor.
Revised: For example, before the MERA layer is used the input is reshaped into a rank-12 tensor.

Original: There isn’t a well deﬁned method for how to perform this reshaping optimally and some experimentation is necessary.
Revised: There isn’t a well deﬁned method for how to perform this reshaping optimally and some experimentation is necessary.

Original: The best way to initialize the MERA layers is also still an open question.
Revised: The best way to initialize the MERA layers is also still an open question.

Original: The results presented here are a promising ﬁrst step for using MERA in a more fundamental way.
Revised: The results presented here are a promising ﬁrst step for using MERA in a more fundamental way.

Original: Since MERA can be viewed as a coarse graining procedure (as explained in section 2), and image data is often well represented in a hierarchical manner, one possibility would be to simply train a two-dimensional MERA directly on an image dataset, with no reference to a neural network.
Revised: Since MERA can be viewed as a coarse graining procedure (as explained in Section 2), and image data is often well represented in a hierarchical manner, one possibility would be to simply train a two-dimensional MERA directly on an image dataset, with no reference to a neural network.

Original: In
Revised: In

Original: Stoudenmire & Schwab (2016) a similar idea was explored with matrix product states being trained directly on MNIST.
Revised: Stoudenmire & Schwab (2016) a similar idea was explored with matrix product states being trained directly on MNIST.

Original: An alternative possibility would be the replacement of just the convolutional layers of the network with a two-dimensional MERA.
Revised: An alternative possibility would be the replacement of just the convolutional layers of the network with a two-dimensional MERA.

Original: Both of these approaches would be closer in spirit to the fundamental ideas about the relationships between quantum physics and machine learning proposed in Lin et al.
Revised: Both of these approaches would be closer in spirit to the fundamental ideas about the relationships between quantum physics and machine learning proposed in Lin et al.

Original: (2017) and Mehta & Schwab (2014).
Revised: (2017) and Mehta & Schwab (2014).

Original: Additionally, there has been some work using entanglement measures to explore how correlations are distributed in deep neural networks, and then utilizing these in order to optimize the design of networks (Liu et al.
Revised: Additionally, there has been some work using entanglement measures to explore how correlations are distributed in deep neural networks, and then utilizing these in order to optimize the design of networks (Liu et al.

Original: (2017), Levine et al.
Revised: (2017), Levine et al.

Original: It would be intriguing to explore such ideas using
Revised: It would be intriguing to explore such ideas using

Original: MERA, for example by using the concrete MERA model explored in this paper, or one of the more ambitious possibilities mentioned above.
Revised: MERA, for example by using the concrete MERA model explored in this paper, or one of the more ambitious possibilities mentioned above.

Original: We end by noting two facts: any variational approximation to a quantum wavefunction can be used to construct a replacement for linear layers of a network.
Revised: We end by noting two facts: any variational approximation to a quantum wavefunction can be used to construct a replacement for linear layers of a network.

Original: There are many examples and each may have its sphere of useful application.
Revised: There are many examples and each may have its sphere of useful application.

Original: Moreover, quantum computers of the type being developed currently by several groups are precisely described by a type of tensor network (a ﬁnite-depth circuit - and one that may very soon be too large to manipulate classically) and could be used as direct replacement for linear layers in a hybrid quantum/classical neural computation scheme.
Revised: Moreover, quantum computers of the type being developed currently by several groups are precisely described by a type of tensor network (a ﬁnite-depth circuit - and one that may very soon be too large to manipulate classically) and could be used as direct replacement for linear layers in a hybrid quantum/classical neural computation scheme A CKNOWLEDGMENTS This work was supported by the Engineering and Physical Sciences Research Council [grant number EP/P510270/1].

Revised: The Titan Xp used for this research was donated by the NVIDIA Corporation.

Revised: We would like to thank Miles Stoudenmire for many enlightening discussions