Original: C OMPACT
Revised: C OMPACT

Original: N EURAL
Revised: N EURAL

Original: N ETWORKS
Revised: N ETWORKS

Original: BASED
Revised: BASED

Original: ON
Revised: ON

Original: THE
Revised: THE

Original: M ULTISCALE
Revised: M ULTISCALE

Original: E NTANGLEMENT
Revised: E NTANGLEMENT

Original: R ENORMALIZATION
Revised: R ENORMALIZATION

Original: A NSATZ
Revised: A NSATZ

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The goal of this paper is to demonstrate a method for tensorizing neural networks based upon an efﬁcient way of approximating scale invariant quantum states, the
Revised: The goal of this paper is to demonstrate a method for tensorizing neural networks based upon an efﬁcient way of approximating scale invariant quantum states, the

Original: Multi-scale Entanglement Renormalization Ansatz (MERA).
Revised: Multi-scale Entanglement Renormalization Ansatz (MERA).

Original: We employ MERA as a replacement for linear layers in a neural network and test this implementation on the CIFAR-10 dataset.
Revised: We employ MERA as a replacement for linear layers in a neural network and test this implementation on the CIFAR-10 dataset.

Original: The proposed method outperforms factorization using tensor trains, providing greater compression for the same level of accuracy and greater accuracy for the same level of compression.
Revised: The proposed method outperforms factorization using tensor trains, providing greater compression for the same level of accuracy and greater accuracy for the same level of compression.

Original: We demonstrate MERAlayers with 3900 times fewer parameters and a reduction in accuracy of less than 1% compared to the equivalent fully connected layers
Revised: We demonstrate MERAlayers with 3900 times fewer parameters and a reduction in accuracy of less than 1% compared to the equivalent fully connected layers

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The curse of dimensionality is a major bottleneck in machine learning, stemming from the exponential growth of variables with the number of modes in a data set (Cichocki et al.
Revised: The curse of dimensionality is a major bottleneck in machine learning, stemming from the exponential growth of variables with the number of modes in a data set (Cichocki et al.

Original: Descriptions of quantum mechanical systems raise a similar challenge; representing n qubits requires a rank- n tensor whose memory cost scales as 2 n .
Revised: Descriptions of quantum mechanical systems raise a similar challenge; representing n qubits requires a rank- n tensor whose memory cost scales as 2 n .

Original: Indeed, it was the promise of harnessing this that led
Revised: Indeed, it was the promise of harnessing this that led

Original: Richard Feynman (Feynman (1982)) to suggest the possibility of quantum computation.
Revised: Richard Feynman (Feynman (1982)) to suggest the possibility of quantum computation.

Original: In the absence of a quantum computer, however, one must use compressed representations of quantum states.
Revised: In the absence of a quantum computer, however, one must use compressed representations of quantum states.

Original: A level of compression can be achieved by factorizing the tensorial description of the quantum wavefunction.
Revised: A level of compression can be achieved by factorizing the tensorial description of the quantum wavefunction.

Original: Many such factorizations are possible, the optimal structure of the factorization being determined by the structure of correlations in the quantum system being studied.
Revised: Many such factorizations are possible, the optimal structure of the factorization being determined by the structure of correlations in the quantum system being studied.

Original: A revolution in quantum mechanics was made by realizing that the best way to characterize the distribution of correlations and information in a state is by a quantity known as entanglement – loosely the mutual quantum information between partitions of a quantum system (Eisert et al.
Revised: A revolution in quantum mechanics was made by realizing that the best way to characterize the distribution of correlations and information in a state is by a quantity known as entanglement – loosely the mutual quantum information between partitions of a quantum system (Eisert et al.

Original: This has led to many successful applications of tensorial approaches to problems in solid state physics and quantum chemistry over the past 25 years (Or ´ us (2014), Kin-Lic Chan et al.
Revised: This has led to many successful applications of tensorial approaches to problems in solid state physics and quantum chemistry over the past 25 years (Or ´ us (2014), Kin-Lic Chan et al.

Original: Intriguing ideas have also emerged over the past few years attempting to bridge the successes of neural networks in machine learning with those of tensorial methods in quantum physics, both at a fundamental level (Lin et al.
Revised: Intriguing ideas have also emerged over the past few years attempting to bridge the successes of neural networks in machine learning with those of tensorial methods in quantum physics, both at a fundamental level (Lin et al.

Original: (2017), Mehta & Schwab (2014)), and as a practical tool for network design (
Revised: (2017), Mehta & Schwab (2014)), and as a practical tool for network design (

Original: Levine et al.
Revised: Levine et al.

Original: Recent work has suggested that entanglement itself is a useful quantiﬁer of the performance of neural networks (Levine et al.
Revised: Recent work has suggested that entanglement itself is a useful quantiﬁer of the performance of neural networks (Levine et al.

Original: (2017), Liu et al.
Revised: (2017), Liu et al.

Original: The simplest factorization employed in quantum systems is known as the matrix product state (Or´us (2014)).
Revised: The simplest factorization employed in quantum systems is known as the matrix product state (Or´us (2014)).

Original: In essence, it expresses the locality of information in certain quantum states.
Revised: In essence, it expresses the locality of information in certain quantum states.

Original: It has already been adopted to replace expensive linear layers in neural networks – in which context it has been independently termed tensor trains (Oseledets (2011)).
Revised: It has already been adopted to replace expensive linear layers in neural networks – in which context it has been independently termed tensor trains (Oseledets (2011)).

Original: This led to substantial compression of neural networks with only a small reduction in the accuracy (Novikov et al.
Revised: This led to substantial compression of neural networks with only a small reduction in the accuracy (Novikov et al.

Original: (2015), Garipov et al.
Revised: (2015), Garipov et al.

Original: Here we use a different tensor factorization – known as the Multi-scale Entanglement Renormalization Ansatz (MERA) – that encodes information in a hierarchical manner (Vidal (2008)).
Revised: Here we use a different tensor factorization – known as the Multi-scale Entanglement Renormalization Ansatz (MERA) – that encodes information in a hierarchical manner (Vidal (2008)).

Original: MERA works through a process of coarse graining or renormalization.
Revised: MERA works through a process of coarse graining or renormalization.

Original: There have been a number of papers looking at the relationship between renormalization and deep learning.
Revised: There have been a number of papers looking at the relationship between renormalization and deep learning.

Original: MERA is a concrete realization of such a renormalization procedure (Vidal (2009)) and so possesses a multi-scale structure that one might anticipate in complex data.
Revised: MERA is a concrete realization of such a renormalization procedure (Vidal (2009)) and so possesses a multi-scale structure that one might anticipate in complex data.

Original: A number of works have utilised tree tensor network models that possess a similar hierarchical structure.
Revised: A number of works have utilised tree tensor network models that possess a similar hierarchical structure.

Original: However, while some of these works claim to be MERA-inspired, they do not include the disentangler tensors that are essential if each layer of the MERA is to capture correlations on different length scales (Liu et al.
Revised: However, while some of these works claim to be MERA-inspired, they do not include the disentangler tensors that are essential if each layer of the MERA is to capture correlations on different length scales (Liu et al.

Original: In this work we employ MERA as a replacement for linear layers in a neural network used to classify the CIFAR-10 dataset.
Revised: In this work we employ MERA as a replacement for linear layers in a neural network used to classify the CIFAR-10 dataset.

Original: Our results show that this performs better than the tensor train decomposition of the same linear layer, and gives better accuracy for the same level of compression and better compression for the same level of accuracy.
Revised: Our results show that this performs better than the tensor train decomposition of the same linear layer, and gives better accuracy for the same level of compression and better compression for the same level of accuracy.

Original: In Section 2 we introduce factorizations of fully connected linear layers, starting with the tensor train factorization followed by a tree-like factorization and ﬁnally the MERA factorization.
Revised: In Section 2 we introduce factorizations of fully connected linear layers, starting with the tensor train factorization followed by a tree-like factorization and ﬁnally the MERA factorization.

Original: In Section 3 we discuss how this is employed as a replacement for a fully connected linear layer in deep learning networks.
Revised: In Section 3 we discuss how this is employed as a replacement for a fully connected linear layer in deep learning networks.

Original: Section 4 gives our main results and we note connections with the existing literature in Section 5.
Revised: Section 4 gives our main results and we note connections with the existing literature in Section 5.

Original: Finally, in Section 6 we discuss some potential developments of the work.
Revised: Finally, in Section 6 we discuss some potential developments of the work.

Original:  T ENSOR F ACTORIZATION OF L INEAR L AYERS
Revised:  T ENSOR F ACTORIZATION OF L INEAR L AYERS

Original: [Figure]
Revised: [Figure]

Original: Figure 1a gives a graphical representation of a rank 2 n tensor W i 1 ,i 2 ,...,i n j 1 ,j 2 ,...,j n and Figure 1b illustrates its tensor train decomposition.
Revised: Figure 1a gives a graphical representation of a rank 2 n tensor W i 1 ,i 2 ,...,i n j 1 ,j 2 ,...,j n and Figure 1b illustrates its tensor train decomposition.

Original: This consists of writing the larger tensor as the contraction of a train of smaller tensors:
Revised: This consists of writing the larger tensor as the contraction of a train of smaller tensors:

Original: [Equation]
Revised: [Equation]

Original: If each index runs over values from 1 to d , this represents an exponential reduction from d 2 n parameters to n ( Dd ) 2 , where the indices α run over values from 1 to D (known as the bond order or
Revised: If each index runs over values from 1 to d , this represents an exponential reduction from d 2 n parameters to n ( Dd ) 2 , where the indices α run over values from 1 to D (known as the bond order or

Original: Schmidt rank in the quantum context).
Revised: Schmidt rank in the quantum context).

Original: As noted above, this type of tensor factorization works well when the information has a local structure (Eisert et al.
Revised: As noted above, this type of tensor factorization works well when the information has a local structure (Eisert et al.

Original: (2010), Verstraete & Cirac (2006))
Revised: (2010), Verstraete & Cirac (2006))

Original: ; tensor trains capture correlations effectively up to length scales of order log D (Schollw¨ock (2011)).
Revised: ; tensor trains capture correlations effectively up to length scales of order log D (Schollw¨ock (2011)).

Original: Image data often possesses a more hierarchical structure.
Revised: Image data often possesses a more hierarchical structure.

Original: The tree tensor network shown in Figure 1c represents one possible hierarchical factorization.
Revised: The tree tensor network shown in Figure 1c represents one possible hierarchical factorization.

Original: Reading from left to right this can be interpreted as follows: the tree-like connectivity imbues the network with a causal structure whereby a given linear element and its outputs are inﬂuenced by inputs in a region determined by its height in the tree.
Revised: Reading from left to right this can be interpreted as follows: the tree-like connectivity imbues the network with a causal structure whereby a given linear element and its outputs are inﬂuenced by inputs in a region determined by its height in the tree.

Original: For example, the rightmost element in Figure 1c is inﬂuenced by all of the inputs, whereas the top element in the middle column is inﬂuenced by inputs i 1 to i 4 .
Revised: For example, the rightmost element in Figure 1c is inﬂuenced by all of the inputs, whereas the top element in the middle column is inﬂuenced by inputs i 1 to i 4 .

Original: Elements other than the rightmost tensor have one dashed output (that connects directly to the overall output) and one solid output (that ties it to the branching tree structure).
Revised: Elements other than the rightmost tensor have one dashed output (that connects directly to the overall output) and one solid output (that ties it to the branching tree structure).

Original: These dashed lines are controlled by features occurring on a particular scale in the data.
Revised: These dashed lines are controlled by features occurring on a particular scale in the data.

Original: Notice that removing these dashed lines, the network has a true tree structure and represents a coarse graining or renormalization of the network.
Revised: Notice that removing these dashed lines, the network has a true tree structure and represents a coarse graining or renormalization of the network.

Original: In this case, the linear elements are the isometries of the original MERA’s deﬁnition (Vidal (2008; 2009)).
Revised: In this case, the linear elements are the isometries of the original MERA’s deﬁnition (Vidal (2008; 2009)).

Original: The simple tree network, which has been studied before in the context of neural networks, has a major deﬁciency.
Revised: The simple tree network, which has been studied before in the context of neural networks, has a major deﬁciency.

Original: At each branching, it partitions the system in two, so that in extremis , the correlations between neighbouring outputs – for example i 4 and j 5 in Figure 1c – are only controlled by the element at the end of the network, thus breaking the hierarchical representation of correlations.
Revised: At each branching, it partitions the system in two, so that in extremis , the correlations between neighbouring outputs – for example i 4 and j 5 in Figure 1c – are only controlled by the element at the end of the network, thus breaking the hierarchical representation of correlations.

Original: The
Revised: The

Original: MERA (Vidal (2009)) factorization shown in Figure 1d ﬁxes this by adding additional disentanglers , whose role is to reduce correlations between these partitions of the system as far as possible before they lose causal connection in the branching structure.
Revised: MERA (Vidal (2009)) factorization shown in Figure 1d ﬁxes this by adding additional disentanglers , whose role is to reduce correlations between these partitions of the system as far as possible before they lose causal connection in the branching structure.

Original:  E XPERIMENTS & NETWORK STRUCTURE
Revised:  E XPERIMENTS & NETWORK STRUCTURE

Original: We have considered the performance of a neural network with the fully connected layers of the model replaced with MERA layers, similar to the Novikov et al.
Revised: We have considered the performance of a neural network with the fully connected layers of the model replaced with MERA layers, similar to the Novikov et al.

Original: (2015) study of compression of fully connected layers using tensor trains.
Revised: (2015) study of compression of fully connected layers using tensor trains.

Original: We have quantiﬁed the performance of the MERA layer through comparisons with two other classes of networks: fully connected layers with varying numbers of nodes and tensor train layers with varying internal dimension.
Revised: We have quantiﬁed the performance of the MERA layer through comparisons with two other classes of networks: fully connected layers with varying numbers of nodes and tensor train layers with varying internal dimension.

Original: The three types of network are otherwise identical.
Revised: The three types of network are otherwise identical.

Original: The networks consist of three sets of two convolutional layers each followed by max pooling layers.
Revised: The networks consist of three sets of two convolutional layers each followed by max pooling layers.

Original: The convolutional kernels were 3 × 3 .
Revised: The convolutional kernels were 3 × 3 .

Original: There were 64 channels in all of the convolutional layers except for the input, which had three channels, and the last convolutional layer, which haschannels.
Revised: There were 64 channels in all of the convolutional layers except for the input, which had three channels, and the last convolutional layer, which haschannels.

Original: The ﬁnal convolutional layer was followed by two more layers, these were either fully connected, MERA-layers or TT-layers depending upon the network.
Revised: The ﬁnal convolutional layer was followed by two more layers, these were either fully connected, MERA-layers or TT-layers depending upon the network.

Original: The ﬁrst of these layers was of size 4096 × x , the second is of size x × 64 .
Revised: The ﬁrst of these layers was of size 4096 × x , the second is of size x × 64 .

Original: For all MERA and TT networks, these layers were 4096 × 4096 and 4096 × 64 .
Revised: For all MERA and TT networks, these layers were 4096 × 4096 and 4096 × 64 .

Original: The ﬁnal layer had 10 nodes corresponding to the 10 image classes in CIFAR-10.
Revised: The ﬁnal layer had 10 nodes corresponding to the 10 image classes in CIFAR-10.

Original: Leaky rectiﬁed linear units (LReLU) were used on all layers except the ﬁnal layer, with leak = 0 .
Revised: Leaky rectiﬁed linear units (LReLU) were used on all layers except the ﬁnal layer, with leak = 0 .

Original: 2 (Maas et al.
Revised: 2 (Maas et al.

Original: During training, nodes in the ﬁnal convolutional layer and the two ﬁrst fully connected layers were dropped with probability 0 .
Revised: During training, nodes in the ﬁnal convolutional layer and the two ﬁrst fully connected layers were dropped with probability 0 .

Original: The penultimate convolutional layer nodes were dropped with probability 0 .
Revised: The penultimate convolutional layer nodes were dropped with probability 0 .

Original: 2 (Srivastava et al.
Revised: 2 (Srivastava et al.

Original: Batch-normalization was used on all layers after dropout and max pooling (Ioffe & Szegedy (2015)).
Revised: Batch-normalization was used on all layers after dropout and max pooling (Ioffe & Szegedy (2015)).

Original: We did not use bias units.
Revised: We did not use bias units.

Original: Gaussian weight initialization was employed in the fully connected models with standard deviation equal to (cid:112) 1 /n in , where n in is the number of input channels to the layer
Revised: Gaussian weight initialization was employed in the fully connected models with standard deviation equal to (cid:112) 1 /n in , where n in is the number of input channels to the layer

Original: (He et al.
Revised: (He et al.

Original: In this report we considered networks with two varieties of fully-connected layers.
Revised: In this report we considered networks with two varieties of fully-connected layers.

Original: The ﬁrst of these networks has a 4096 × 4096 fully connected layer followed by one which is 4096 × 64 ; this network was used as a benchmark against which the other models could be compared.
Revised: The ﬁrst of these networks has a 4096 × 4096 fully connected layer followed by one which is 4096 × 64 ; this network was used as a benchmark against which the other models could be compared.

Original: The second network instead has a 4096 × 10 fully connected layer followed by a 10 × 64 layer.
Revised: The second network instead has a 4096 × 10 fully connected layer followed by a 10 × 64 layer.

Original: We trained this network to compare the MERA and tensor train layers to a fully connected model with a comparable number of parameters, in order to evaluate how detrimental naive compression is to accuracy.
Revised: We trained this network to compare the MERA and tensor train layers to a fully connected model with a comparable number of parameters, in order to evaluate how detrimental naive compression is to accuracy.

Original: A schematic of the two MERA-layers can be found in Figure 2.
Revised: A schematic of the two MERA-layers can be found in Figure 2.

Original: The input to the MERA-layer was reshaped in to a rank-12 tensor with each index being dimension two.
Revised: The input to the MERA-layer was reshaped in to a rank-12 tensor with each index being dimension two.

Original: The MERA-layer was then constructed from a set of rank- 4 tensors B β k β l α i α j , using the method described in Section 2.
Revised: The MERA-layer was then constructed from a set of rank- 4 tensors B β k β l α i α j , using the method described in Section 2.

Original: [Figure]
Revised: [Figure]

Original: In the second MERA-layer, the output index for the ﬁrst set of isometry tensors was ﬁxed to produce a 64 node output.
Revised: In the second MERA-layer, the output index for the ﬁrst set of isometry tensors was ﬁxed to produce a 64 node output.

Original: The dimensions of the internal indices in the MERA layers were allowed to vary, with a MERA- D being a MERA layer with internal dimension D .
Revised: The dimensions of the internal indices in the MERA layers were allowed to vary, with a MERA- D being a MERA layer with internal dimension D .

Original: MERA weights were initialized using elements of randomized orthogonal matrices (Saxe et al.
Revised: MERA weights were initialized using elements of randomized orthogonal matrices (Saxe et al.

Original: – randomization was done with respect to the Haar measure.
Revised: – randomization was done with respect to the Haar measure.

Original: Finally, a network with its fully connected layers replaced with a tensor train decomposition was trained in order to provide a comparison with the MERA-layers.
Revised: Finally, a network with its fully connected layers replaced with a tensor train decomposition was trained in order to provide a comparison with the MERA-layers.

Original: The tensor train layers were constructed as described in Section 2 with the internal dimension being allowed to vary from D = 3 to
Revised: The tensor train layers were constructed as described in Section 2 with the internal dimension being allowed to vary from D = 3 to

Original: D
Revised: D

Original: = 5 .
Revised: = 5 .

Original: In the second tensor train layer, half of the output indices were ﬁxed to match the second
Revised: In the second tensor train layer, half of the output indices were ﬁxed to match the second

Original: MERA-layer.
Revised: MERA-layer.

Original: We tested performance on the CIFAR-10 dataset.
Revised: We tested performance on the CIFAR-10 dataset.

Original: We used 50 , 000 images for training and 10 ,for testing.
Revised: We used 50 , 000 images for training and 10 ,for testing.

Original: We used 100 , 000 training iterations, each consisting of batches of 50 images.
Revised: We used 100 , 000 training iterations, each consisting of batches of 50 images.

Original: Training data was augmented online by randomly ﬂipping and translating the input images by up to 4 pixels.
Revised: Training data was augmented online by randomly ﬂipping and translating the input images by up to 4 pixels.

Original: Translated images were padded with zeros.
Revised: Translated images were padded with zeros.

Original: All images were normalized by dividing by 255 and subtracting the mean pixels value from the training set.
Revised: All images were normalized by dividing by 255 and subtracting the mean pixels value from the training set.

Original: Test accuracy was recorded every 500 iterations and training was stopped when test accuracy did not improve for 10 successive tests.
Revised: Test accuracy was recorded every 500 iterations and training was stopped when test accuracy did not improve for 10 successive tests.

Original: The network was trained using backpropagation and the Adam optimizer, with initial learning rate 0 .
Revised: The network was trained using backpropagation and the Adam optimizer, with initial learning rate 0 .

Original: 001 (Kingma & Ba (2014)) and a softmax-cross-entropy objective.
Revised: 001 (Kingma & Ba (2014)) and a softmax-cross-entropy objective.

Original: The networks were implemented in Tensorﬂow r1.3 and trained on NVIDIA Titan Xp and 1080ti
Revised: The networks were implemented in Tensorﬂow r1.3 and trained on NVIDIA Titan Xp and 1080ti

Original: GPUs.
Revised: GPUs.

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: In Table 1 we compare the number of parameters used, the compression rate and the accuracy obtained for each of the models described in Section 3.
Revised: In Table 1 we compare the number of parameters used, the compression rate and the accuracy obtained for each of the models described in Section 3.

Original: The compression rate stated is with respect to the number of parameters used in the fully-connected benchmark model, FC-1.
Revised: The compression rate stated is with respect to the number of parameters used in the fully-connected benchmark model, FC-1.

Original: When comparing the MERA networks to the fully connected model we can see a considerable drop in the number of parameters required with only a modest drop in the accuracy.
Revised: When comparing the MERA networks to the fully connected model we can see a considerable drop in the number of parameters required with only a modest drop in the accuracy.

Original: MERA-2 compressed the fully connected layers by a factor of nearly 5500 with an accuracy drop of 1 . 48% .
Revised: MERA-2 compressed the fully connected layers by a factor of nearly 5500 with an accuracy drop of 1 . 48% .

Original: Increasing the internal dimension of the MERA to 3 we found the compression rate decreased to approximately 3900 but the accuracy was 86 .
Revised: Increasing the internal dimension of the MERA to 3 we found the compression rate decreased to approximately 3900 but the accuracy was 86 .

Original: 2% , a drop of less than 1% compared to the fully connected model.
Revised: 2% , a drop of less than 1% compared to the fully connected model.

Original: The compression rate of an entire MERA network compared to the fully connected network was approximately 57 .
Revised: The compression rate of an entire MERA network compared to the fully connected network was approximately 57 .

Original: We did not attempt to compress the convolutional layers in this work, and therefore the vast majority of parameters were focused in these layers for the MERA networks.
Revised: We did not attempt to compress the convolutional layers in this work, and therefore the vast majority of parameters were focused in these layers for the MERA networks.

Original: How signiﬁcant is the MERA network structure we have chosen to the results obtained?
Revised: How signiﬁcant is the MERA network structure we have chosen to the results obtained?

Original: The results of the MERA networks can be compared against the fully connected layer with only 10 nodes, see
Revised: The results of the MERA networks can be compared against the fully connected layer with only 10 nodes, see

Original: FC-2 in Table 1.
Revised: FC-2 in Table 1.

Original: This network possesses 100 times more parameters than MERA-2 but the drop in the accuracy is 6 .
Revised: This network possesses 100 times more parameters than MERA-2 but the drop in the accuracy is 6 .

Original: 04% , signiﬁcantly more than any MERA network.
Revised: 04% , signiﬁcantly more than any MERA network.

Original: The MERA network also compares favourably to tensor train methods.
Revised: The MERA network also compares favourably to tensor train methods.

Original: TT-3 had a comparable compression rate to the MERA methods but a more signiﬁcant drop in accuracy, 2 . 22% .
Revised: TT-3 had a comparable compression rate to the MERA methods but a more signiﬁcant drop in accuracy, 2 . 22% .

Original: For a tensor train network to achieve an accuracy drop of only 1% the internal dimension had to be increased to 7, resulting in an inferior compression rate compared to MERA- 2 .
Revised: For a tensor train network to achieve an accuracy drop of only 1% the internal dimension had to be increased to 7, resulting in an inferior compression rate compared to MERA- 2 .

Original: In addition to the degree of compression achieved by these networks, we also address the time to optimize.
Revised: In addition to the degree of compression achieved by these networks, we also address the time to optimize.

Original: There is evidently a degree of compromise required here: the number of multiplications required to apply a MERA layer scales with the input size N and bond order D as N log 2 D .
Revised: There is evidently a degree of compromise required here: the number of multiplications required to apply a MERA layer scales with the input size N and bond order D as N log 2 D .

Original: The equivalent scaling for a tensor train and fully connected layer are ND 2 and N 2 , respectively.
Revised: The equivalent scaling for a tensor train and fully connected layer are ND 2 and N 2 , respectively.

Original: This is reﬂected in the times taken to optimize these networks.
Revised: This is reﬂected in the times taken to optimize these networks.

Original: Note however, that MERA can accommodate correlations at all scales of its input even at low bond order, whereas tensor trains require a bond order that scales exponentially with the length scale of correlation (Or´us (2014)).
Revised: Note however, that MERA can accommodate correlations at all scales of its input even at low bond order, whereas tensor trains require a bond order that scales exponentially with the length scale of correlation (Or´us (2014)).

Original: MERA is, therefore, expected to scale better for very large data sets than either tensor trains or fully connected layers.
Revised: MERA is, therefore, expected to scale better for very large data sets than either tensor trains or fully connected layers.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Given how memory intensive deep neural networks typically are, substantial effort has been made to reduce number of parameters these networks require without signiﬁcantly reducing their accuracy.
Revised: Given how memory intensive deep neural networks typically are, substantial effort has been made to reduce number of parameters these networks require without signiﬁcantly reducing their accuracy.

Original: Some of these have taken a similar approach to the MERA network described above, using tensor decompositions of the fully connected layers.
Revised: Some of these have taken a similar approach to the MERA network described above, using tensor decompositions of the fully connected layers.

Original: These include the tensor train models of Novikov et al.
Revised: These include the tensor train models of Novikov et al.

Original: (2015) and Garipov et al.
Revised: (2015) and Garipov et al.

Original: Here we have found replacing a fully connected linear layer with a MERA factorization resulted in superior accuracy for a comparable number of parameters.
Revised: Here we have found replacing a fully connected linear layer with a MERA factorization resulted in superior accuracy for a comparable number of parameters.

Original: More directly related to this MERA model are a number of tree tensor network models (Liu et al.
Revised: More directly related to this MERA model are a number of tree tensor network models (Liu et al.

Original: (2017), Levine et al.
Revised: (2017), Levine et al.

Original: As section 2 explained, the disentangler tensors are essential if each layer of a MERA is to capture correlations on different length scales.
Revised: As section 2 explained, the disentangler tensors are essential if each layer of a MERA is to capture correlations on different length scales.

Original: Tree tensors do not possess these and we expect them to struggle to capture long range correlations as effectively as MERA.
Revised: Tree tensors do not possess these and we expect them to struggle to capture long range correlations as effectively as MERA.

Original: A MERA works through a process of coarse graining or renormalization.
Revised: A MERA works through a process of coarse graining or renormalization.

Original: There have been a number of other papers looking at the relationship between renormalization and deep learning.
Revised: There have been a number of other papers looking at the relationship between renormalization and deep learning.

Original: Lin et al.
Revised: Lin et al.

Original: (2017) argue that the effectiveness of deep neural networks should be thought of in terms of renormalization and Mehta & Schwab (2014) demonstrate an exact mapping between the variational renormalization group and restricted Boltzmann machines.
Revised: (2017) argue that the effectiveness of deep neural networks should be thought of in terms of renormalization and Mehta & Schwab (2014) demonstrate an exact mapping between the variational renormalization group and restricted Boltzmann machines.

Original: In this report we have taken a different approach: only one layer of the network was replaced with a MERA.
Revised: In this report we have taken a different approach: only one layer of the network was replaced with a MERA.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have shown that replacing the fully connected layers of a deep neural network with layers based upon the multi-scale entanglement renormalization ansatz can generate signiﬁcant efﬁciency gains with only small reduction in accuracy.
Revised: We have shown that replacing the fully connected layers of a deep neural network with layers based upon the multi-scale entanglement renormalization ansatz can generate signiﬁcant efﬁciency gains with only small reduction in accuracy.

Original: When applied to the CIFAR-10 data we found the fully connected layers can be replaced with MERA layers with 3900 times less parameters with a reduction in the accuracy of less than 1% .
Revised: When applied to the CIFAR-10 data we found the fully connected layers can be replaced with MERA layers with 3900 times less parameters with a reduction in the accuracy of less than 1% .

Original: The model signiﬁcantly outperformed compact fully connected layers with 70 − 100 times as many parameters.
Revised: The model signiﬁcantly outperformed compact fully connected layers with 70 − 100 times as many parameters.

Original: Moreover, it outperformed a similar replacement of the fully connected layers with tensor trains, both in terms of accuracy for a given compression and compression for a given accuracy.
Revised: Moreover, it outperformed a similar replacement of the fully connected layers with tensor trains, both in terms of accuracy for a given compression and compression for a given accuracy.

Original: An added advantage — not explored here — is that a factorized layer can potentially handle much larger input data sets, thus enabling entirely new types of computation.
Revised: An added advantage — not explored here — is that a factorized layer can potentially handle much larger input data sets, thus enabling entirely new types of computation.

Original: Correlations across these large inputs can be handled much more efﬁciently by MERA than by tensor trains.
Revised: Correlations across these large inputs can be handled much more efﬁciently by MERA than by tensor trains.

Original: Moreover, a compressed network may provide a convenient way to avoid over-ﬁtting of large data sets.
Revised: Moreover, a compressed network may provide a convenient way to avoid over-ﬁtting of large data sets.

Original: The compression achieved by networks with these factorized layers comes at a cost.
Revised: The compression achieved by networks with these factorized layers comes at a cost.

Original: They can take longer to train than networks containing the large fully connected layers due to the number of tensor contractions required to apply the factorized layer.
Revised: They can take longer to train than networks containing the large fully connected layers due to the number of tensor contractions required to apply the factorized layer.

Original: Our results suggest several immediate directions for future inquiry.
Revised: Our results suggest several immediate directions for future inquiry.

Original: Firstly, there are some questions about how to improve the existing model.
Revised: Firstly, there are some questions about how to improve the existing model.

Original: For example, before the MERA layer is used the input is reshaped into a rank-12 tensor.
Revised: For example, before the MERA layer is used the input is reshaped into a rank-12 tensor.

Original: There isn’t a well deﬁned method for how to perform this reshaping optimally and some experimentation is necessary.
Revised: There isn’t a well deﬁned method for how to perform this reshaping optimally and some experimentation is necessary.

Original: The best way to initialize the MERA layers is also still an open question.
Revised: The best way to initialize the MERA layers is also still an open question.

Original: The results presented here are a promising ﬁrst step for using MERA in a more fundamental way.
Revised: The results presented here are a promising ﬁrst step for using MERA in a more fundamental way.

Original: Since MERA can be viewed as a coarse graining procedure (as explained in section 2), and image data is often well represented in a hierarchical manner, one possibility would be to simply train a two-dimensional MERA directly on an image dataset, with no reference to a neural network.
Revised: Since MERA can be viewed as a coarse graining procedure (as explained in section 2), and image data is often well represented in a hierarchical manner, one possibility would be to simply train a two-dimensional MERA directly on an image dataset, with no reference to a neural network.

Original: In
Revised: In

Original: Stoudenmire & Schwab (2016) a similar idea was explored with matrix product states being trained directly on MNIST.
Revised: Stoudenmire & Schwab (2016) a similar idea was explored with matrix product states being trained directly on MNIST.

Original: An alternative possibility would be the replacement of just the convolutional layers of the network with a two-dimensional MERA.
Revised: An alternative possibility would be the replacement of just the convolutional layers of the network with a two-dimensional MERA.

Original: Both of these approaches would be closer in spirit to the fundamental ideas about the relationships between quantum physics and machine learning proposed in Lin et al.
Revised: Both of these approaches would be closer in spirit to the fundamental ideas about the relationships between quantum physics and machine learning proposed in Lin et al.

Original: (2017) and Mehta & Schwab (2014).
Revised: (2017) and Mehta & Schwab (2014).

Original: Additionally, there has been some work using entanglement measures to explore how correlations are distributed in deep neural networks, and then utilizing these in order to optimize the design of networks (Liu et al.
Revised: Additionally, there has been some work using entanglement measures to explore how correlations are distributed in deep neural networks, and then utilizing these in order to optimize the design of networks (Liu et al.

Original: (2017), Levine et al.
Revised: (2017), Levine et al.

Original: It would be intriguing to explore such ideas using
Revised: It would be intriguing to explore such ideas using

Original: MERA, for example by using the concrete MERA model explored in this paper, or one of the more ambitious possibilities mentioned above.
Revised: MERA, for example by using the concrete MERA model explored in this paper, or one of the more ambitious possibilities mentioned above.

Original: We end by noting two facts: any variational approximation to a quantum wavefunction can be used to construct a replacement for linear layers of a network.
Revised: We end by noting two facts: any variational approximation to a quantum wavefunction can be used to construct a replacement for linear layers of a network.

Original: There are many examples and each may have its sphere of useful application.
Revised: There are many examples and each may have its sphere of useful application.

Original: Moreover, quantum computers of the type being developed currently by several groups are precisely described by a type of tensor network (a ﬁnite-depth circuit - and one that may very soon be too large to manipulate classically) and could be used as direct replacement for linear layers in a hybrid quantum/classical neural computation scheme.A CKNOWLEDGMENTS This work was supported by the Engineering and Physical Sciences Research Council [grant number EP/P510270/1].
Revised: Moreover, quantum computers of the type being developed currently by several groups are precisely described by a type of tensor network (a ﬁnite-depth circuit - and one that may very soon be too large to manipulate classically) and could be used as direct replacement for linear layers in a hybrid quantum/classical neural computation scheme.

Original: The Titan Xp used for this research was donated by the NVIDIA Corporation.