G ENERALIZED T ENSOR M ODELS FOR R ECURRENT N EURAL N ETWORKS
A BSTRACT
Recurrent Neural Networks (RNNs) are very successful at solving challenging problems with sequential data.
However, this observed efﬁciency is not yet entirely explained by theory.
It is known that a certain class of multiplicative RNNs enjoys the property of depth efﬁciency — a shallow network of exponentially large width is necessary to realize the same score function as computed by such an RNN.
Such networks, however, are not very often applied to real life tasks.
In this work, we attempt to reduce the gap between theory and practice by extending the theoretical analysis to RNNs which employ various nonlinearities, such as Rectiﬁed
Linear Unit (ReLU), and show that they also beneﬁt from properties of universality and depth efﬁciency.
Our theoretical results are veriﬁed by a series of extensive computational experiments.
I NTRODUCTION
Recurrent Neural Networks are ﬁrmly established to be one of the best deep learning techniques when the task at hand requires processing sequential data, such as text, audio, or video (Graves et al., 2013; Mikolov et al., 2011; Gers et al., 1999).
The ability of these neural networks to efﬁciently represent a rich class of functions with a relatively small number of parameters is often referred to as depth efﬁciency , and the theory behind this phenomenon is not yet fully understood.
A recent line of work (Cohen & Shashua, 2016; Cohen et al., 2016;
Khrulkov et al., 2018;
Cohen et al., 2018) focuses on comparing various deep learning architectures in terms of their expressive power .
It was shown in (Cohen et al., 2016) that ConvNets with product pooling are exponentially more expressive than shallow networks, that is there exist functions realized by ConvNets which require an exponentially large number of parameters in order to be realized by shallow nets.
A similar result also holds for RNNs with multiplicative recurrent cells (Khrulkov et al., 2018).
We aim to extend this analysis to RNNs with rectiﬁer nonlinearities which are often used in practice.
The main challenge of such analysis is that the tools used for analyzing multiplicative networks, namely, properties of standard tensor decompositions and ideas from algebraic geometry, can not be applied in this case, and thus some other approach is required.
Our objective is to apply the machinery of generalized tensor decompositions , and show universality and existence of depth efﬁciency in such RNNs.
R ELATED WORK
Tensor methods have a rich history of successful application in machine learning.
(Vasilescu &
Terzopoulos, 2002), in their framework of TensorFaces, proposed to treat facial image data as multidimensional arrays and analyze them with tensor decompositions, which led to signiﬁcant boost in face recognition accuracy.
(Bailey & Aeron, 2017) employed higher-order co-occurence data and tensor factorization techniques to improve on word embeddings models.
Tensor methods also allow to produce more accurate and robust recommender systems by taking into account a multifaceted nature of real environments (Frolov & Oseledets, 2017).
In recent years a great deal of work was done in applications of tensor calculus to both theoretical and practical aspects of deep learning algorithms.
(Lebedev et al., 2015) represented ﬁlters in a convolutional network with CP decomposition (Harshman, 1970; Carroll & Chang, 1970) which allowed for much faster inference at the cost of a negligible drop in performance.
(Novikov et al., 2015) proposed to use Tensor Train (TT) decomposition (Oseledets, 2011) to compress fully–connected layers of large neural networks while preserving their expressive power.
Later on, TT was exploited to reduce the number of parameters and improve the performance of recurrent networks in long–term forecasting (Yu et al., 2017) and video classiﬁcation (Yang et al., 2017) problems.
In addition to the practical beneﬁts, tensor decompositions were used to analyze theoretical aspects of deep neural nets.
(Cohen et al., 2016) investigated a connection between various network architectures and tensor decompositions, which made possible to compare their expressive power.
Speciﬁcally, it was shown that CP and Hierarchial Tucker (Grasedyck, 2010) decompositions correspond to shallow networks and convolutional networks respectively.
Recently, this analysis was extended by (Khrulkov et al., 2018) who showed that TT decomposition can be represented as a recurrent network with multiplicative connections.
This speciﬁc form of RNNs was also empirically proved to provide a substantial performance boost over standard RNN models (Wu et al., 2016).
First results on the connection between tensor decompositions and neural networks were obtained for rather simple architectures, however, later on, they were extended in order to analyze more practical deep neural nets.
It was shown that theoretical results can be generalized to a large class of CNNs with ReLU nonlinearities (Cohen & Shashua, 2016) and dilated convolutions (Cohen et al., 2018), providing valuable insights on how they can be improved.
However, there is a missing piece in the whole picture as theoretical properties of more complex nonlinear RNNs have yet to be analyzed.
In this paper, we elaborate on this problem and present new tools for conducting a theoretical analysis of such RNNs, speciﬁcally when rectiﬁer nonlinearities are used.
A RCHITECTURES INSPIRED BY TENSOR DECOMPOSITIONS
Let us now recall the known results about the connection of tensor decompositions and multiplicative architectures, and then show how they are generalized in order to include networks with ReLU nonlinearities.
3. S CORE FUNCTIONS AND FEATURE TENSOR
Suppose that we are given a dataset of objects with a sequential structure, i.e. every object in the dataset can be written as
[Equation]
We also introduce a parametric feature map f θ : R N → R M which essentially preprocesses the data before it is fed into the network.
Assumption 1 holds for many types of data, e.g. in the case of natural images we can cut them into rectangular patches which are then arranged into vectors x ( t ) .
A typical choice for the feature map f θ in this particular case is an afﬁne map followed by a nonlinear activation: f θ ( x ) =
σ ( Ax + b ) .
To draw the connection between tensor decompositions and feature tensors we consider the following score functions (logits 1 ):
[Equation]
where W ∈ R M × M ×···× M is a trainable T –way weight tensor and Φ ( X ) ∈ R M × M ×···× M is a rank 1 feature tensor , deﬁned as
[Equation]
where we have used the operation of outer product ⊗ , which is important in tensor calculus.
For a tensor A of order N and a tensor B of order M their outer product C =
A ⊗ B is a tensor of order
N + M deﬁned as:
[Equation]
It is known that equation 2 possesses the universal approximation property (it can approximate any function with any prescribed precision given sufﬁciently large M ) under mild assumptions on f θ (Cohen et al., 2016; Girosi & Poggio, 1990).
3. T ENSOR D ECOMPOSITIONS
Working the entire weight tensor W in eq.
(2) is impractical for large M and T , since it requires exponential in T number of parameters.
Thus, we compactly represent it using tensor decompositions , which will further lead to different neural network architectures, referred to as tensor networks (Cichocki et al., 2017).
CP-decomposition The most basic decomposition is the so-called Canonical (CP) decomposition (Harshman, 1970; Carroll & Chang, 1970) which is deﬁned as follows
[Equation]
where v ( t ) r ∈ R M and minimal value of R such that decomposition equation 5 exists is called canonical rank of a tensor (CP–rank) .
By substituting eq.
(5) into eq.
(2) we ﬁnd that (cid:96)
[Equation]
In the equation above, outer products ⊗ are taken between scalars and coincide with the ordinary products between two numbers.
However, we would like to keep this notation as it will come in handy later, when we generalize tensor decompositions to include various nonlinearities.
TT-decomposition Another tensor decomposition is Tensor Train (TT) decomposition (Oseledets, 2011) which is deﬁned as follows
[Equation]
where g ( t ) r t − 1 r t ∈ R M and r 0 = r T = 1 by deﬁnition.
If we gather vectors g ( t ) r t − 1 r t for all corresponding indices r t − 1 ∈ { 1 , . . .
, R t − 1 } and r t ∈ { 1 , . . .
, R t } we will obtain three–dimensional tensors G ( t
) ∈ R M × R t − 1 × R t (for t = 1 and t =
T we will get matrices G (
1) ∈ R M × 1 × R 1 and
G ( T ) ∈ R M × R T − 1 × 1 ).
The set of all such tensors { G ( t ) } Tt =1 is called TT–cores and minimal values of { R t } T − 1 t =1 such that decomposition equation 7 exists are called TT–ranks .
In the case of TT decomposition, the score function has the following form:
[Equation]
3. C ONNECTION BETWEEN TT AND RNN
Now we want to show that the score function for Tensor Train decomposition exhibits particular recurrent structure similar to that of RNN.
We deﬁne the following hidden states :
[Equation]
Such deﬁnition of hidden states allows for more compact form of the score function.
Lemma 3.1.
Under the notation introduced in eq.
(9) , the score function can be written as
[Equation]
Proof of Lemma 3.1 as well as the proofs of our main results from Section 5 were moved to Appendix A due to limited space.
Note that with a help of TT–cores we can rewrite eq.
(9) in a more convenient index form:
[Equation]
where the operation of tensor contraction is used.
Combining all weights from G ( t ) and f θ ( · ) into a single variable Θ ( t ) G and denoting the composition of feature map, outer product, and contraction as g : [Equation] ×
[Equation]
t → R R t we arrive at the following vector form:
[Equation]
This equation can be considered as a generalization of hidden state equation for Recurrent Neural
Networks as here all hidden states h ( t ) may in general have different dimensionalities and weight tensors
Θ ( t ) G depend on the time step.
However, if we set R = R 1 = · · · = R T − 1 and G = G (2) = · · · = G ( T − 1) we will get simpliﬁed hidden state equation used in standard recurrent architectures:
[Equation]
Note that this equation is applicable to all hidden states except for the ﬁrst h (1) =
G (1) f θ ( x (1) ) and for the last h ( T ) =
f (cid:62) θ ( x ( T ) )
G ( T ) h
( T − 1) , due to two–dimensional nature of the corresponding
TT–cores.
However, we can always pad the input sequence with two auxiliary vectors x (0) and x ( T +1) to get full compliance with the standard RNN structure.
Figure 1 depicts tensor network induced by TT decomposition with cores { G ( t ) }
Tt =1 .
G ENERALIZED TENSOR NETWORKS 4. G ENERALIZED OUTER PRODUCT
In the previous section we showed that tensor decompositions correspond to neural networks of speciﬁc structure, which are simpliﬁed versions of those used in practice as they contain multiplicative nonlinearities only.
One possible way to introduce more practical nonlinearities is to replace outer product ⊗ in eq.
(6) and eq.
(10) with a generalized operator ⊗ ξ in analogy to kernel methods when scalar product is replaced by nonlinear kernel function.
Let ξ : R × R → R be an associative and commutative binary operator ( ∀ x, y, z ∈ R : ξ ( ξ ( x, y ) , z )
=
ξ ( x, ξ ( y, z )) and ∀ x, y ∈ R : ξ ( x, y ) =
ξ ( y, x ) ).
Note that this operator easily generalizes to the arbitrary number of operands due to associativity.
For a tensor A of order N and a tensor B of order M we deﬁne their generalized outer product C =
A ⊗ ξ B as an ( N + M ) order tensor with entries given by:
[Equation]
Now we can replace ⊗ in eqs.
(6) and (10) with ⊗ ξ and get networks with various nonlinearities.
For example, if we take ξ ( x, y ) =
max( x, y, 0) we will get an RNN with rectiﬁer nonlinearities; if we take ξ ( x, y ) =
ln( e x + e y
we will get an RNN with softplus nonlinearities; if we take ξ ( x, [Equation]
) =
xy we will get a simple RNN deﬁned in the previous section.
Concretely, we will analyze the following networks.
Generalized shallow network with ξ –nonlinearity
• Score function:
[Equation]
• Parameters of the network:
[Equation]
Generalized RNN with ξ –nonlinearity
• Score function:
[Equation]
• Parameters of the network:
[Equation]
[Equation]
Note that in eq.
(16) we have introduced the matrices C ( t ) acting on the input states.
The purpose of this modiﬁcation is to obtain the plausible property of generalized shallow networks being able to be represented as generalized RNNs of width 1 (i.e., with all R i = 1 ) for an arbitrary nonlinearity ξ .
In the case of ξ ( x, y ) =
xy , the matrices C ( t ) were not necessary, since they can be simply absorbed by
G ( t ) via tensor contraction (see Appendix A for further clariﬁcation on these points).
Initial hidden state Note that generalized RNNs require some choice of the initial hidden state h (0) .
We ﬁnd that it is convenient both for theoretical analysis and in practice to initialize h (0) as unit of the operator ξ , i.e. such an element u that ξ ( x, y, u ) =
ξ
( x, y ) ∀ x, y ∈ R .
Henceforth, we will assume that such an element exists (e.g., for ξ ( x, y )
=
max( x, y, 0) we take u = 0 , for ξ ( x, y )
=
xy we take u = 1 ), and set h (0) =
u .
For example, in eq.
(9) it was implicitly assumed that h (0) = 1 .
G RID TENSORS
Introduction of generalized outer product allows us to investigate RNNs with wide class of nonlinear activation functions, especially ReLU.
While this change looks appealing from the practical viewpoint, it complicates following theoretical analysis, as the transition from obtained networks back to tensors is not straightforward.
In the discussion above, every tensor network had corresponding weight tensor W and we could compare expressivity of associated score functions by comparing some properties of this tensors, such as ranks (Khrulkov et al., 2018; Cohen et al., 2016).
This method enabled comprehensive analysis of score functions, as it allows us to calculate and compare their values for all possible input sequences
X = (cid:0) x (1) , . . .
, x ( T ) (cid:1) .
Unfortunately, we can not apply it in case of generalized tensor networks, as the replacement of standard outer product ⊗ with its generalized version ⊗ ξ leads to the loss of conformity between tensor networks and weight tensors.
Speciﬁcally, not for every generalized tensor network with corresponding score function (cid:96) ( X ) now exists a weight tensor W such that (cid:96) ( X ) =
(cid:104) W , Φ ( X ) (cid:105) .
Also, such properties as universality no longer hold automatically and we have to prove them separately.
Indeed as it was noticed in (Cohen & Shashua, 2016) shallow networks with ξ ( x, y ) = max( x, 0)
+ max( y, 0) no longer have the universal approximation property.
In order to conduct proper theoretical analysis, we adopt the apparatus of so-called grid tensors , ﬁrst introduced in (Cohen & Shashua, 2016).
Given a set of ﬁxed vectors X = (cid:8) x (1) , . . .
, x ( M ) (cid:9) referred to as templates , the grid tensor of X is deﬁned to be the tensor of order T and dimension M in each mode, with entries given by:
[Equation]
where each index i t can take values from { 1 , . .
, M } , i.e. we evaluate the score function on every possible input assembled from the template vectors { x ( i ) }
Mi =1 .
To put it simply, we previously considered the equality of score functions represented by tensor decomposition and tensor network on set of all possible input sequences X
= (cid:0) x (1) , . . .
, x ( T ) (cid:1) , x ( t ) ∈ R N , and now we restricted this set to exponentially large but ﬁnite grid of sequences consisting of template vectors only.
Deﬁne the matrix F ∈ R M × M which holds the values taken by the representation function f θ : R N → R M on the selected templates X : [Equation]
[Equation]
Using the matrix F we note that the grid tensor of generalized shallow network has the following form (see Appendix A for derivation):
[Equation]
Construction of the grid tensor for generalized RNN is a bit more involved.
We ﬁnd that its grid tensor Γ (cid:96) ( X ) can be computed recursively, similar to the hidden state in the case of a single input sequence.
The exact formulas turned out to be rather cumbersome and we moved them to Appendix A.
M AIN RESULTS
With grid tensors at hand we are ready to compare the expressive power of generalized RNNs and generalized shallow networks.
In the further analysis, we will assume that ξ ( x, y ) =
max( x, y, 0) , i.e., we analyze RNNs and shallow networks with rectiﬁer nonlinearity .
However, we need to make two additional assumptions.
First of all, similarly to (Cohen & Shashua, 2016)
we ﬁx some templates X such that values of the score function outside of the grid generated by X are irrelevant for classiﬁcation and call them covering templates.
It was argued that for image data values of M of order 100 are sufﬁcient (corresponding covering template vectors may represent Gabor ﬁlters).
Secondly, we assume that the feature matrix F is invertible, which is a reasonable assumption and in the case of f θ ( x ) =
σ ( Ax + b ) for any distinct template vectors X the parameters A and b can be chosen in such a way that the matrix F is invertible.
5. U NIVERSALITY
As was discussed in section 4.2 we can no longer use standard algebraic techniques to verify universality of tensor based networks.
Thus, our ﬁrst result states that generalized RNNs with ξ ( x, y ) =
max( x, y, 0) are universal in a sense that any tensor of order T and size of each mode being m can be realized as a grid tensor of such RNN (and similarly of a generalized shallow network).
Theorem 5.1 (Universality) .
Let H ∈ R M × M ×···× M be an arbitrary tensor of order T .
Then there exist a generalized shallow network and a generalized RNN with rectiﬁer nonlinearity ξ ( x, y )
=
max( x, y, 0) such that grid tensor of each of the networks coincides with H .
Part of Theorem 5.1 which corresponds to generalized shallow networks readily follows from (Cohen & Shashua, 2016, Claim 4).
In order to prove the statement for the RNNs the following two lemmas are used.
Lemma 5.1.
Given two generalized RNNs with grid tensors Γ (cid:96) A ( X ) , Γ (cid:96) B ( X ) , and arbitrary ξ -nonlinearity, there exists a generalized RNN with grid tensor Γ (cid:96) C ( X ) satisfying
[Equation]
This lemma essentially states that the collection of grid tensors of generalized RNNs with any nonlinearity is closed under taking arbitrary linear combinations.
Note that the same result clearly holds for generalized shallow networks because they are linear combinations of rank 1 shallow networks by deﬁnition.
Lemma 5.2.
Let E ( j 1 j 2 ...
j T ) be an arbitrary one–hot tensor, deﬁned as
[Equation]
Then there exists a generalized RNN with rectiﬁer nonlinearities such that its grid tensor satisﬁes
[Equation]
This lemma states
that in the special case of rectiﬁer nonlinearity ξ ( x, y ) =
max( x, y, 0) any basis tensor can be realized by some generalized RNN.
Proof of Theorem 5.1.
By Lemma 5.2 for each one–hot tensor E ( i 1 i 2 ...
i T ) there exists a generalized RNN with rectiﬁer nonlinearities, such that its grid tensor coincides with this tensor.
Thus, by Lemma 5.1 we can construct an RNN with
[Equation]
For generalized shallow networks with rectiﬁer nonlinearities see the proof of (Cohen & Shashua, 2016, Claim 4).
The same result regarding networks with product nonlinearities considered in (Khrulkov et al., 2018) directly follows from the well–known properties of tensor decompositions (see Appendix A).
We see that at least with such nonlinearities as ξ ( x, y )
=
max( x, y, 0) and ξ ( x, y ) =
xy all the networks under consideration are universal and can represent any possible grid tensor.
Now let us head to a discussion of expressivity of these networks.
5. E XPRESSIVITY
As was discussed in the introduction, expressivity refers to the ability of some class of networks to represent the same functions as some other class much more compactly.
In our case the parameters deﬁning size of networks are ranks of the decomposition, i.e. in the case of generalized RNNs ranks determine the size of the hidden state, and in the case of generalized shallow networks rank determines the width of a network.
It was proven in (Cohen et al., 2016; Khrulkov et al., 2018) that ConvNets and RNNs with multiplicative nonlinearities are exponentially more expressive than the equivalent shallow networks: shallow networks of exponentially large width are required to realize the same score functions as computed by these deep architectures.
Similarly to the case of
ConvNets (Cohen & Shashua, 2016), we ﬁnd that expressivity of generalized RNNs with rectiﬁer nonlinearity holds only partially, as discussed in the following two theorems.
For simplicity, we assume that T is even.
Theorem 5.2 (Expressivity 1) .
For every value of R there exists a generalized RNN with ranks ≤ R and rectiﬁer nonlinearity which is exponentially more efﬁcient than shallow networks, i.e., the corresponding grid tensor may be realized only by a shallow network with rectiﬁer nonlinearity of width at least 2 MT min( M, R ) T / 2 .
This result states that at least for some subset of generalized RNNs expressivity holds: exponentially wide shallow networks are required to realize the same grid tensor.
Proof of the theorem is rather straightforward: we explicitly construct an example of such RNN which satisﬁes the following description.
Given an arbitrary input sequence X = (cid:0) x (1) , . . .
x ( T ) (cid:1) assembled from the templates, these networks (if M = R ) produce 0 if X has the property that x (1) =
x (2) , x (3) =
x (4) , . . .
, x ( T − 1) =
x ( T ) , and 1 in every other case, i.e. they measure pairwise similarity of the input vectors.
A precise proof is given in Appendix A.
In the case of multiplicative RNNs (Khrulkov et al., 2018) almost every network possessed this property.
This is not the case, however, for generalized RNNs with rectiﬁer nonlinearities.
Theorem 5.3 (Expressivity 2) .
For every value of R there exists an open set (which thus has positive measure) of generalized RNNs with rectiﬁer nonlinearity ξ ( x, y )
=
max( x, y, 0) , such that for each
RNN in this open set the corresponding grid tensor can be realized by a rank 1 shallow network with rectiﬁer nonlinearity.
In other words, for every rank R we can ﬁnd a set of generalized RNNs of positive measure such that the property of expressivity does not hold.
In the numerical experiments in Section 6 and
Appendix A we validate whether this can be observed in practice, and ﬁnd that the probability of obtaining CP–ranks of polynomial size becomes negligible with large T and R .
Proof of Theorem 5. is provided in Appendix A.
Shared case Note that all the RNNs used in practice have shared weights , which allows them to process sequences of arbitrary length.
So far in the analysis we have not made such assumptions about RNNs (i.e., G (2) = · · · = G ( T − 1) ).
By imposing this constraint, we lose the property of universality; however, we believe that the statements of Theorems 5.2 and 5.3 still hold (without requiring that shallow networks also have shared weights).
Note that the example constructed in the proof of Theorem 5.3 already has this property, and for Theorem 5.2 we provide numerical evidence in Appendix A.
E XPERIMENTS
In this section, we study if our theoretical ﬁndings are supported by experimental data.
In particular, we investigate whether generalized tensor networks can be used in practical settings, especially in problems typically solved by RNNs (such as natural language processing problems).
Secondly, according to Theorem 5.3 for some subset of RNNs the equivalent shallow network may have a low rank.
To get a grasp of how strong this effect might be in practice we numerically compute an estimate for this rank in various settings.
Performance For the ﬁrst experiment, we use two computer vision datasets MNIST (LeCun et al., 1990) and CIFAR–10 (Krizhevsky & Hinton, 2009), and natural language processing dataset for sentiment analysis IMDB (Maas et al., 2011).
For the ﬁrst two datasets, we cut natural images into rectangular patches which are then arranged into vectors x ( t ) (similar to (Khrulkov et al., 2018)) and for IMDB dataset the input data already has the desired sequential structure.
Figure 2 depicts test accuracy on IMDB dataset for generalized shallow networks and RNNs with rectiﬁer nonlinearity.
We see that generalized shallow network of much higher rank is required to get the level of performance close to that achievable by generalized RNN.
Due to limited space, we have moved the results of the experiments on the visual datasets to Appendix B. respect to the total number of parameters ( M = 50 , T = 100 , ξ ( x, y )
=
max( x, y, 0) )
Expressivity For the second experiment we generate a number of generalized RNNs with different values of TT-rank r and calculate a lower bound on the rank of shallow network necessary to realize the same grid tensor (to estimate the rank we use the same technique as in the proof of Theorem 5.2).
Figure 3 shows that for different values of R and generalized RNNs of the corresponding rank there exist shallow networks of rank 1 realizing the same grid tensor, which agrees well with Theorem 5.3.
This result looks discouraging, however, there is also a positive observation.
While increasing rank of generalized RNNs, more and more corresponding shallow networks will necessarily have exponentially higher rank.
In practice we usually deal with RNNs of R = 10 2 − 10 3 (dimension of hidden states), thus we may expect that effectively any function besides negligible set realized by generalized RNNs can be implemented only by exponentially wider shallow networks.
The numerical results for the case of shared cores and other nonlinearities are given in Appendix B.
C ONCLUSION
In this paper, we sought a more complete picture of the connection between Recurrent Neural
Networks and Tensor Train decomposition, one that involves various nonlinearities applied to hidden states.
We showed how these nonlinearities could be incorporated into network architectures and provided complete theoretical analysis on the particular case of rectiﬁer nonlinearity, elaborating on points of generality and expressive power.
We believe our results will be useful to advance theoretical understanding of RNNs.
In future work, we would like to extend the theoretical analysis to most competitive in practice architectures for processing sequential data such as LSTMs and attention mechanisms.
A CKNOWLEDGEMENTS
We would like to thank Andrzej Cichocki for constructive discussions during the preparation of the manuscript and anonymous reviewers for their valuable feedback.
This work was supported by the
Ministry of Education and Science of the Russian Federation (grant 14.756.31.0001).
A P ROOFS
Lemma 3.1.
Under the notation introduced in eq.
[Equation] the score function can be written as
[Equation]
Proof.
[Equation]
Proposition A
[Equation]
If we replace the generalized outer product ⊗ ξ in eq.
(16) with the standard outer product ⊗ [Equation] we can subsume matrices C ( t ) into tensors G ( t ) without loss of generality.
Proof.
Let us rewrite hidden state equation eq.
(16) after transition from ⊗ ξ to ⊗ :
[Equation]
We see that the obtained expression resembles those presented in eq.
(10) with TT-cores G ( t ) replaced by ˜ G ( t ) and thus all the reasoning applied in the absence of matrices C ( t ) holds valid.
Proposition A.2.
Grid tensor of generalized shallow network has the following form (eq.
[Equation]
Proof.
Let X = (cid:0) x
( i 1 ) , x ( i 2 ) , . . .
, x ( i T ) (cid:1) denote an arbitrary sequence of templates .
Corresponding element of the grid tensor deﬁned in eq.
(20) has the following form:
[Equation]
Proposition A.3.
Grid tensor of a generalized RNN has the following form:
[Equation]
Proof.
Proof is similar to that of Proposition A.2 and uses eq.
(16) to compute the elements of the grid tensor.
Lemma 5.1.
Given two generalized RNNs with grid tensors Γ (cid:96) A ( X ) , Γ (cid:96) B ( X ) , and arbitrary ξ -nonlinearity, there exists a generalized RNN with grid tensor Γ (cid:96) C ( X ) satisfying Γ
[Equation]
Proof.
Let these RNNs be deﬁned by the weight parameters
[Equation]
and Θ
[Equation]
Tt
[Equation]
1 ∈
[Equation]
R
[Equation]
We claim that the desired grid tensor is given by the RNN with the following weight settings.
[Equation]

[Equation]
{ L A + 1 ,
[Equation]
[Equation]
A + L B } .
It is straightforward to verify that the network deﬁned by these weights possesses the following property:
[Equation]
and h
[Equation]
concluding the proof.
We also note that these formulas generalize the well–known formulas for addition of two tensors in the Tensor Train format (Oseledets, 2011).
Proposition A.4.
For any associative and commutative binary operator ξ , an arbitrary generalized rank 1 shallow network with ξ –nonlinearity can be represented in a form of generalized RNN with unit ranks ( R 1 = · · · = R T − 1 = 1 ) and ξ –nonlinearity.
Proof.
Let Θ = (cid:0) λ, { v ( t ) }
Tt =1 (cid:1) be the parameters specifying the given generalized shallow network.
Then the following weight settings provide the equivalent generalized RNN (with h (0) being the unity of the operator ξ ).
[Equation]
Indeed, in the notation deﬁned above, hidden states of generalized RNN have the following form:
[Equation]
The score function of generalized RNN is given by eq.
[Equation]
which coincides with the score function of rank 1 shallow network deﬁned by parameters Θ .
Lemma 5.2.
Let E ( j 1 j 2 ...
j T ) be an arbitrary one–hot tensor, deﬁned as
[Equation]
Then there exists a generalized RNN with rectiﬁer nonlinearities such that its grid tensor satisﬁes
[Equation]
Proof.
It is known that the statement of the lemma holds for generalized shallow networks with rectiﬁer nonlinearities (see (Cohen & Shashua, 2016, Claim 4)).
Based on Proposition A.4 and
Lemma 5.1 we can conclude that it also holds for generalized RNNs with rectiﬁer nonlinearities.
Proposition A.5.
Statement of Theorem 5.1 holds with ξ ( x, y )
=
xy .
Proof.
By assumption the matrix F is invertible.
Consider the following tensor (cid:98) H :
[Equation]
and the score function in the form of eq.
[Equation]
Note that by construction for any input assembled from the template vectors we obtain (cid:96) (cid:0) ( x ( i 1 ) , . . .
, x ( i T ) )
(cid:1)
=
H i 1 ...
i T .
By taking the standard TT and CP decompositions of (cid:98) H which always exist (Oseledets, 2011; Kolda & Bader, 2009), and using Lemma 3.1 and eq.
(6) we conclude that universality holds.
Theorem 5.2 (Expressivity 1) .
For every value of R there exists a generalized RNN with ranks ≤ R and rectiﬁer nonlinearity which is exponentially more efﬁcient than shallow networks, i.e., the corresponding grid tensor may be realized only by a shallow network with rectiﬁer nonlinearity of width at least 2 MT min( M, R ) T / 2 .
In order to prove the theorem we will use the standard technique of matricizations.
Simply put, by matricizing a tensor we reshape it into a matrix by splitting the indices of a tensor into two collections, and converting each one of them into one long index.
I.e., for a tensor A of order T with mode sizes being m , we split the set { 1 , . . .
, T } into two non–overlapping ordered subsets s and t , and deﬁne the matricization A ( s,t ) ∈
R M | s | × M | t | by simply reshaping (and possibly transposing) the tensor A according to s and t .
We will consider the matricization obtained by taking s odd = (1 , 3 , . . .
, T − 1) , t even =
, T ) , i.e., we split out even and odd modes.
A typical application of matricization is the following: suppose that we can upper and lower bound the ordinary matrix rank of a certain matricization using the parameters specifying each of the architectures being analyzed.
Then under the assumption that both architectures realize the same grid tensor (and thus ranks of the matricization coincide) we can compare the sizes of corresponding architectures.
In the case of generalized shallow networks with rectiﬁer nonlinearity we will use the following result (Cohen & Shashua, 2016, Claim 9).
Lemma A.1.
Let Γ (cid:96) ( X ) be a grid tensor generated by a generalized shallow network of rank R and ξ ( x, y )
=
max( x, y, 0) .
Then
[Equation]
where the ordinary matrix rank is assumed.
This result is a generalization of a well–known property of the standard CP-decomposition (i.e. if ξ ( x, y ) =
xy ), which states that for a rank R decomposition, the matrix rank of every matricization is bounded by R .
In order to prove Theorem 5.2 we will construct an example of a generalized RNN with exponentially large matrix rank of the matricization of grid tensor, from which and Lemma A.1 the statement of the theorem will follow.
Lemma A.2.
Without loss of generality assume that x i =
e i (which can be achieved since F is invertible).
Let 1 ( p,q ) denote the matrix of size p × q with each entry being 1 , I ( p,q ) denote the matrix of size p × q with I ( p,q ) ij =
δ ij ( δ being the Kronecker symbol), and b = [1 − min( M, R ) , 0 (cid:62) R − 1 ] ∈ R 1 × R .
Consider the following weight setting for a generalized RNN with ξ ( x, y )
=
max( x, y, 0) .
[Equation]
Then grid tensor Γ (cid:96) ( X ) of this RNN satisﬁes
[Equation]
where the ordinary matrix rank is assumed.
Proof.
Informal description of the network deﬁned by weights in the statement in the lemma is the following.
Given some input vector e i it is ﬁrst transformed into its bitwise negative e i , and its ﬁrst R components are saved into the hidden state.
The next block then measures whether the ﬁrst min( R, M ) components of the current input coincide with the hidden state (after again taking bitwise negative).
If this is the case, the hidden state is set 0 and the process continues.
Otherwise, the hidden state is set to 1 which then ﬂows to the output independently of the other inputs.
In other words, for all the inputs of the form X
=
( x i 1 , x i 1 , . . .
, x i T/ 2 , x i T/ 2 ) with i 1 ≤ R, . .
, i T/ 2 ≤ R we obtain that (cid:96) ( X ) = 0 , and in every other case (cid:96) ( X ) = 1 .
Thus, we obtain that (cid:2) Γ (cid:96) ( X ) (cid:3) ( s odd ,t even ) is a matrix with all the entries equal to 1 , except for min( M, R ) T / 2 entries on the diagonal, which are equal to 0 .
Rank of such a matrix is R T / 2 + 1 if R < M and M T / 2 otherwise, and the statement of the lemma follows.
Based on these two lemmas we immediately obtain Theorem 5.2.
Proof of Theorem 5.2.
Consider the example constructed in the proof of Lemma A.2.
By
Lemma A.1 the rank of the shallow network with rectiﬁer nonlinearity which is able to represent the same grid tensor is at least 2 TM min( M, R ) T / 2 .
Theorem 5.3 (Expressivity 2) .
For every value of R there exists an open set (which thus has positive measure) of generalized RNNs with rectiﬁer nonlinearity ξ ( x, y )
=
max( x, y, 0) , such that for each
RNN in this open set the corresponding grid tensor can be realized by a rank 1 shallow network with rectiﬁer nonlinearity.
Proof.
As before, let us denote by I ( p,q ) a matrix of size p ×
q
such that I ( p,q ) ij = δ ij , and by a ( p 1 ,p 2 ,...p d ) we denote a tensor of size p 1 × · · · × p d with each entry being a (sometimes we will omit the dimensions when they can be inferred from the context).
Consider the following weight settings for a generalized RNN.
[Equation]
The RNN deﬁned by these weights has the property that Γ (cid:96) ( X ) is a constant tensor with each entry being 2( MR ) T − 1 , which can be trivially represented by a rank 1 generalized shallow network.
We will show that this property holds under a small perturbation of C ( t ) , G ( t ) and F .
Let us denote each of these perturbation (and every tensor appearing size of which can be assumed indeﬁnitely small) collectively by ε .
Applying eq.
(21) we obtain (with ξ ( x, y )
=
max( x, y, 0) ).
[Equation]
where we have used a simple property connecting ⊗ ξ with ξ ( x, y )
=
max( x, y, 0) and ordinary ⊗ : if for tensors A and B each entry of A is greater than each entry of B , A ⊗ ξ B =
A ⊗ 1 .
The obtained grid tensors can be represented using rank 1 generalized shallow networks with the following weight settings.
[Equation]
where F ε is the feature matrix of the corresponding perturbed network.
B A DDITIONAL EXPERIMENTS
In this section we provide the results additional computational experiments, aimed to provide more thorough and complete analysis of generalized RNNs.
Different ξ -nonlinearities
In this paper we presented theoretical analysis of rectiﬁer nonlinearity which corresponds to ξ ( x, y )
=
max( x, y, 0) .
However, there is a number of other associative binary operators ξ which can be incorporated in generalized tensor networks.
Strictly speaking, every one of them has to be carefully explored theoretically in order to speak about their generality and expressive power, but for now we can compare them empirically.
Table 1 shows the performance (accuracy on test data) of different nonlinearities on MNIST,
CIFAR—10, and IMDB datasets for classiﬁcation.
Although these problems are not considered hard to solve, we see that the right choice of nonlinearity can lead to a signiﬁcant boost in performance.
For the experiments on the visual datasets we used T = 16 , m = 32 , R = 64 and for the experiments on the IMDB dataset we had T
= 100 , m = 50 , R = 50 .
Parameters of all networks were optimized using Adam (learning rate α = 10 − 4 ) and batch size 250 .
ξ ( x, y ) xy max( x, y, 0) ln
( e x + e y ) x
+ y (cid:112) x
2 + y
[Table]
Expressivity in the case of shared cores We repeat the expressivity experiments from Sectionin the case of equal TT–cores ( G (2) = · · · = G ( T − 1) ).
We observe that similar to the case of different cores, there always exist rank 1 generalized shallow networks which realize the same score function as generalized RNN of higher rank, however, this situation seems too unlikely for big values of R