B ENEFITS OF D EPTH FOR L ONG -T ERM M EMORY OF R ECURRENT N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The key attribute that drives the unprecedented success of modern Recurrent Neural Networks (RNNs) on learning tasks which involve sequential data, is their ever-improving ability to model intricate long-term temporal dependencies.
However, a well established measure of RNNs’ long-term memory capacity is lacking, and thus formal understanding of their ability to correlate data throughout time is limited.
Though depth efﬁciency in convolutional networks is well established by now, it does not sufﬁce in order to account for the success of deep RNNs on inputs of varying lengths, and the need to address their ‘time-series expressive power’ arises.
In this paper, we analyze the effect of depth on the ability of recurrent networks to express correlations ranging over long time-scales.
To meet the above need, we introduce a measure of the information ﬂow across time that can be supported by the network, referred to as the Start-End separation rank .
Essentially, this measure reﬂects the distance of the function realized by the recurrent network from a function that models no interaction whatsoever between the beginning and end of the input sequence.
We prove that deep recurrent networks support StartEnd separation ranks which are exponentially higher than those supported by their shallow counterparts.
Moreover, we show that the ability of deep recurrent networks to correlate different parts of the input sequence increases exponentially as the input sequence extends, while that of vanilla shallow recurrent networks does not adapt to the sequence length at all.
Thus, we establish that depth brings forth an overwhelming advantage in the ability of recurrent networks to model long-term dependencies, and provide an exemplar of quantifying this key attribute which may be readily extended to other RNN architectures of interest, e.g. variants of LSTM networks.
We obtain our results by considering a class of recurrent networks referred to as Recurrent Arithmetic Circuits (RACs), which merge the hidden state with the input via the Multiplicative Integration operation.
I NTRODUCTION
Over the past few years, Recurrent Neural Networks (RNNs) have become the prominent machine learning architecture for modeling sequential data, having been successfully employed for language modeling (Sutskever et al., 2011; Graves, 2013), neural machine translation (Bahdanau et al., 2014), speech recognition (Graves et al., 2013; Amodei et al., 2016), and more.
The success of recurrent networks in learning complex functional dependencies for sequences of varying lengths, readily implies that long-term and elaborate correlations in the given inputs are somehow supported by these networks.
However, formal understanding of the inﬂuence of a recurrent network’s structure on its expressiveness, and speciﬁcally on its ever-improving ability to integrate data throughout time ( e.g. translating long sentences, answering elaborate questions), is lacking.
An ongoing empirical effort to successfully apply recurrent networks to tasks of increasing complexity and temporal extent, includes augmentations of the recurrent unit such as Long Short Term
Memory (LSTM) networks (Hochreiter and Schmidhuber, 1997) and their variants ( e.g. Cho et al.
A parallel avenue, which we focus on in this paper, includes the stacking of layers to form deep recurrent networks (Schmidhuber, 1992).
Deep recurrent networks, which exhibit empirical superiority over shallow ones (see e.g. Graves et al.
, implement hierarchical processing of information at every time-step that accompanies their inherent time-advancing computation.
Evidence for a time-scale related effect arises from experiments (Hermans and Schrauwen, 2013) – deep recurrent networks appear to model correlations which correspond to longer time-scales than shallow ones.
These ﬁndings, which imply that depth brings forth a considerable advantage in complexity and in temporal capacity of recurrent networks, have no adequate theoretical explanation.
In this paper, we address the above presented issues.
Based on the relative maturity of depth efﬁciency results in neural networks, namely results that show that deep networks efﬁciently express functions that would require shallow ones to have a super-polynomial size ( e.g. Cohen et al.
Eldan and Shamir (2016)), it is natural to assume that depth has a similar effect on the expressiveness of recurrent networks.
Indeed, we show that depth efﬁciency holds for recurrent networks.
However, the distinguishing attribute of recurrent networks, is their inherent ability to cope with varying input sequence length.
Thus, once establishing the above depth efﬁciency in recurrent networks, a basic question arises, which relates to the apparent depth enhanced long-term memory in recurrent networks: Do the functions which are efﬁciently expressed by deep recurrent networks correspond to dependencies over longer time-scales?
We answer this question, by showing that depth provides an exponential boost to the ability of recurrent networks to model long-term dependencies.
In order to take-on the above question, we introduce in section 2 a recurrent network referred to as a recurrent arithmetic circuit (RAC) that shares the architectural features of RNNs, and differs from them in the type of non-linearity used in the calculation.
This type of connection between state-of-the-art machine learning algorithms and arithmetic circuits (also known as Sum-Product
Networks (Poon and Domingos, 2011)) has well-established precedence in the context of neural networks.
Delalleau and Bengio (2011) prove a depth efﬁciency result on such networks, and Cohen et al.
(2016) theoretically analyze the class of Convolutional Arithmetic Circuits which differ from common ConvNets in the exact same fashion in which RACs differ from more standard RNNs.
Conclusions drawn from such analyses were empirically shown to extend to common ConvNets ( e.g. Sharir and Shashua (2017); Levine et al.
Beyond their connection to theoretical models, the modiﬁcation which deﬁnes RACs resembles that of Multiplicative RNNs (Sutskever et al., 2011) and of Multiplicative Integration networks (Wu et al., 2016), which provide a substantial performance boost over many of the existing RNN models.
In order to obtain our results, we make a connection between RACs and the Tensor Train (TT) decomposition (Oseledets, 2011), which suggests that Multiplicative RNNs may be related to a generalized TT-decomposition, similar to the way Cohen and Shashua (2016) connected ReLU ConvNets to generalized tensor decompositions.
We move on to introduce in section 3 the notion of Start-End separation rank as a measure of the recurrent network’s ability to model elaborate long-term dependencies.
In order to analyze the longterm correlations of a function over a sequential input which extends T time-steps, we partition the inputs to those which arrive at the ﬁrst T / 2 time-steps (“Start”) and the last T / 2 time-steps (“End”), and ask how far the function realized by the recurrent network is from being separable w.r.t.
this partition.
Distance from separability is measured through the notion of separation rank (Beylkin and Mohlenkamp, 2002), which can be viewed as a surrogate of the L 2 distance from the closest separable function.
For a given function, high Start-End separation rank implies that the function induces strong correlation between the beginning and end of the input sequence, and vice versa.
In section 4 we directly address the depth enhanced long-term memory question above, by examining depth L = 2 RACs and proving that functions realized by these deep networks enjoy Start-End separation ranks that are exponentially higher than those of shallow networks, implying that indeed these functions can model more elaborate input dependencies over longer periods of time.
An additional reinforcing result is that the Start-End separation rank of the deep recurrent network grows exponentially with the sequence length, while that of the shallow recurrent network is independent of the sequence length.
Informally, this implies that vanilla shallow recurrent networks are inadequate in modeling correlations of long input sequences, since in contrast to the case of deep recurrent networks, the modeled dependencies achievable by shallow ones do not adapt to the actual length of the input.
Finally, we present and motivate a quantitative conjecture by which the Start-End separation rank of recurrent networks grows exponentially with the network depth.
A proof of this conjecture, which will provide an even deeper insight regarding the advantages of depth in recurrent networks, is left as an open problem.
R ECURRENT A RITHMETIC C IRCUITS
In this section, we introduce a class of recurrent networks referred to as Recurrent Arithmetic Circuits (RACs), which shares the architectural features of standard RNNs.
As demonstrated below, the operation of RACs on sequential data is identical to the operation of RNNs, where a hidden state mixes information from previous time-steps with new incoming data (see ﬁg. 1).
The two classes differ only in the type of non-linearity used in the calculation, as described by eqs. 1-3.
In the following sections, we utilize the algebraic properties of RACs for proving results regarding their ability to model long-term dependencies of their inputs.
We present below the basic framework of shallow recurrent networks (ﬁg. 1(a)), which describes both the common RNNs and the newly introduced RACs.
A recurrent network is a network that models a discrete-time dynamical system; we focus on an example of a sequence to sequence classiﬁcation task into one of the categories { 1 , ..., C } ≡
[ C ] .
Denoting the temporal dependence by t , the sequential input to the network is { x t ∈ X} Tt =1 , and the output is a sequence of class scores vectors { y t,L, Θ ∈ R C } Tt =1 , where L is the network depth, Θ denotes the parameters of the recurrent network, and T represents the extent of the sequence in time-steps.
We assume the input lies in some input space X that may be discrete (e.g. text data) or continuous (e.g. audio data), and that some initial mapping f : X → R M is preformed on the input, so that all input types are mapped to vectors f ( x t ) ∈ R M .
The function f ( · ) may be viewed as an encoding, e.g. words to vectors or images to a ﬁnal dense layer via some trained ConvNet.
The output at time t ∈
[ T ] of the shallow (depth L = 1 ) recurrent network with R hidden channels, depicted in ﬁg. 1(a), is given by:
[Equation]
where h t ∈ R R is the hidden state of the network at time t ( h 0 is some initial hidden state), Θ denotes the learned parameters W I ∈ R R × M ,
W H ∈ R R × R , W O ∈ R C × R , which are the input, hidden and output weights matrices respectively, and g is some non-linear operation.
A bias term is usually added to eq.
1, however, because it bears no effect on our analysis, we omit it for simplicity.
For common RNNs, the non-linearity is given by:
[Equation]
where σ ( · ) is typically some point-wise non-linearity such as sigmoid, tanh etc.
For the newly introduced class of RACs, g is given by:
[Equation]
where the operation (cid:12) stands for element-wise multiplication between vectors, for which the resultant vector upholds ( a (cid:12) b ) i
= a i · b i .
This form of merging the input and the hidden state by multiplication rather than addition is referred to as Multiplicative Integration (Wu et al., 2016).
The extension to deep recurrent networks is natural, and we follow the common approach (see e.g.
Hermans and Schrauwen (2013)) where each layer acts as a recurrent network which receives the hidden state of the previous layer as its input.
The output at time t of the depth L recurrent network with R hidden channels in each layer, 1 depicted in ﬁg. 1(b), is constructed by the following:
[Equation]
where h t,l ∈ R R is the state of the depth l hidden unit at time t ( h 0 ,l is some initial hidden state per layer), and Θ denotes the learned parameters.
Speciﬁcally, W I ,l ∈ R R × R ( l > 1) , W H ,l ∈ R R × R are the input and hidden weights matrices at depth l , respectively.
For l = 1 , the weights matrix which multiplies the inputs vector has the appropriate dimensions: W I , 1 ∈ R R × M .
The output weights matrix is W O ∈ R C × R as in the shallow case, representing a ﬁnal calculation of the scores for all classes 1 through C at every time-step.
The non-linear operation g determines the type of the deep recurrent network, where a common deep RNN is obtained by choosing g = g RNN (eq. 2), and a deep RAC is obtained for g = g RAC (eq. 3).
We consider the newly presented class of RACs to be a good surrogate of common RNNs.
Firstly, there is an obvious structural resemblance between the two classes, as the recurrent aspect of the calculation has the exact same form in both networks (ﬁg. 1).
In fact, recurrent networks that include
Multiplicative Integration similarly to RACs, have been shown to outperform many of the existing
RNN models (Sutskever et al., 2011; Wu et al., 2016).
Secondly, as mentioned above, arithmetic circuits have been successfully used as surrogates of convolutional networks.
The fact that Cohen and Shashua (2016) laid the foundation for extending the proof methodologies of convolutional arithmetic circuits to common ConvNets with ReLU activations, suggests that such adaptations may be made in the recurrent network analog, rendering the newly proposed class of recurrent networks all the more interesting.
In the following sections, we make use of the algebraic properties of RACs in order to obtain clear-cut observations regarding the beneﬁts of depth in recurrent networks.
T EMPORAL C ORRELATIONS M ODELED BY R ECURRENT N ETWORKS
In this section, we establish means for quantifying the ability of recurrent networks to model longterm temporal dependencies in the sequential input data.
We begin by introducing the Start-End separation-rank of the function realized by a recurrent network as a measure of the amount of information ﬂow across time that can be supported by the network.
We then tie the Start-End separation rank to the algebraic concept of grid tensors (Hackbusch, 2012), which will allow us to employ tools and results from tensorial analysis in order to show that depth provides an exponential boost to the ability of recurrent networks to model elaborate long-term temporal dependencies.
3. T HE “S TART -E ND ” S EPARATION R ANK
We deﬁne below the concept of the Start-End separation rank for functions realized by recurrent networks after T time-steps, i.e. real functions that take as input X =
( x 1 , . .
, x T ) ∈ X T .
The separation rank quantiﬁes a function’s distance from separability with respect to two disjoint subsets of its inputs.
Speciﬁcally, let ( S, E ) be a partition of input indices, such that S = { 1 , . .
, T / 2 } and E = { T / 2 + 1 , . .
, T } (we consider even values of T throughout the paper for convenience of presentation).
This implies that { x s } s ∈ S are the ﬁrst T/ 2 (“Start”) inputs to the network, and { x e } e ∈ E are the last T/ 2 (“End”) inputs to the network.
For a function y : X T → R , the Start-End separation rank is deﬁned as follows:
[Equation]
In words, it is the minimal number of summands that together give y , where each summand is separable w.r.t.
( S, E ) , i.e. is equal to a product of two functions – one that intakes only inputs from the ﬁrst T/ 2 time-steps, and another that intakes only inputs from the last T/ 2 time-steps.
The separation rank w.r.t.
a general partition of the inputs was introduced in Beylkin and
Mohlenkamp (2002) for high-dimensional numerical analysis, and was employed for various applications, e.g. chemistry (Harrison et al., 2003), particle engineering (Hackbusch, 2006), and machine learning (Beylkin et al., 2009).
Cohen and Shashua (2017) connect the separation rank to the
L 2 distance of the function from the set of separable functions, and use it to measure correlations modeled by deep convolutional networks.
Levine et al.
(2017) tie the separation rank to the family of quantum entanglement measures, which quantify correlations in many-body quantum systems.
In our context, if the Start-End separation rank of a function realized by a recurrent network is equal to 1 , then the function is separable, meaning it cannot model any interaction between the inputs which arrive at the beginning of the sequence and the inputs that follow later, towards the end of the sequence.
Speciﬁcally, if sep ( S,E ) ( y ) =
1 then there exist g s : X T/ 2 → R and g e : X T/ 2 → R such that y ( x 1 , . . .
, x T ) =
g s ( x 1 , . . .
, x
T/ 2 )
g e ( x T/ 2+1 , . . .
, x T ) , and the function y cannot take into account consistency between the values of { x 1 , . . .
, x T/ 2 } and those of { x T/ 2+1 , . . .
, x T } .
In a statistical setting, if y were a probability density function, this would imply that { x 1 , . . .
, x T/ 2 } and { x T/ 2+1 , . . .
, x T } are statistically independent.
The higher sep ( S,E ) ( y ) is, the farther y is from this situation, i.e. the more it models dependency between the beginning and the end of the inputs sequence.
Stated differently, if the recurrent network’s architecture restricts the hypothesis space to functions with low Start-End separation ranks, a more elaborate long-term temporal dependence, which corresponds to a function with a higher Start-End separation rank, cannot be learned.
In section 4 we show that deep RACs support Start-End separations ranks which are exponentially larger than those supported by shallow RACs, and are therefore much better ﬁt to model long-term temporal dependencies.
To this end, we employ in the following sub-section the algebraic tool of grid tensors that will allow us to evaluate the Start-End separation ranks of deep and shallow RACs.
3. B OUNDING THE S TART -E ND S EPARATION R ANK VIA G RID T ENSORS
We begin by laying out basic concepts in tensor theory required for the upcoming analysis.
The core concept of a tensor may be thought of as a multi-dimensional array.
The order of a tensor is deﬁned to be the number of indexing entries in the array, referred to as modes .
The dimension of a tensor in a particular mode is deﬁned as the number of values taken by the index in that mode.
If A is a tensor of order T and dimension M i in each mode i ∈
[ T ] , its entries are denoted A d 1 ...d T , where the index in each mode takes values d
i ∈
[ M i ] .
A fundamental operator in tensor analysis is the tensor product , which we denote by ⊗ .
It is an operator that intakes two tensors A ∈ R M 1 ×···× M P and B ∈ R M P +1 ×···× M P + Q , and returns a tensor A ⊗ B ∈ R M 1 ×···× M P + Q deﬁned by: ( A ⊗ B ) d 1 ...
d P + Q =
A d 1 ...d P ·B d P +1 ...
d P + Q .
An additional concept we will make use of is the matricization of A w.r.t.
the partition ( S, E ) , denoted (cid:74) A (cid:75) S,E , which is essentially the arrangement of the tensor elements as a matrix whose rows correspond to S and columns to E (formally presented in appendix C).
We consider the function realized by a shallow RAC with R hidden channels, which computes the score of class c ∈ [ C ] at time T .
This function, which is given by a recursive deﬁnition in eqs.and 3, can be alternatively written in the following closed form:
[Equation]
where the order T tensor A T, 1 , Θ c , which lies at the heart of the above expression, is referred to as the shallow RAC weights tensor , since its entries are polynomials in the network weights Θ .
Speciﬁcally, denoting the rows of the input weights matrix, W I , by a I ,α ∈ R M (or element-wise: a I ,αj =
W I α,j ), the rows of the hidden weights matrix, W H , by a H ,β ∈ R R (or element-wise: a H ,β j = W
H
β,j ), and the rows of the output weights matrix, W O , by a O ,c ∈ R R , c ∈
[ C ] (or element-wise: a O ,c j = W O c,j ), the shallow RAC weights tensor can be gradually constructed in the following fashion:
[Equation]
having set h 0 = (cid:0) W H (cid:1) † 1 , where † is the pseudoinverse operation.
In the above equation, the tensor products, which appear inside the sums, are directly related to the Multiplicative Integration property of RACs (eq. 3).
The sums originate in the multiplication of the hidden states vector by the hidden weights matrix at every time-step (eq. 1).
The construction of the shallow RAC weights tensor, presented in eq.
7, is referred to as a Tensor Train (TT) decomposition of TT-rank R in the tensor analysis community and is analogously described by a Matrix Product State (MPS) Tensor Network (see Or´us (2014)) in the quantum physics community.
See appendix A for the Tensor Networks construction of deep and shallow RACs, which provides graphical insight regarding the exponential complexity brought forth by depth in recurrent networks.
We now present the concept of grid tensors, which are a form of function discretization.
Essentially, the function is evaluated for a set of points on an exponentially large grid in the input space and the outcomes are stored in a tensor.
Formally, ﬁxing a set of template vectors x (1) , . .
, x ( M ) ∈ X , the points on the grid are the set { ( x ( d 1 ) , . . .
, x ( d T ) ) }
Md 1 ,...,d T =1 .
Given a function y ( x 1 , . . .
, x T ) , the set of its values on the grid arranged in the form of a tensor are called the grid tensor induced by y , denoted A ( y ) d 1 ,...,d T ≡ y ( x ( d 1 ) , . . .
, x ( d T ) ) .
The grid tensors of functions realized by recurrent networks, will allow us to calculate their separations ranks and establish deﬁnitive conclusions regarding the beneﬁts of depth these networks.
Having presented the tensorial structure of the function realized by a shallow RAC, as given by eqs. 6 and 7 above, we are now in a position to tie its
Start-End separation rank to its grid tensor, as formulated in the following claim:
Claim 1.
Let y T, 1 , Θ c be a function realized by a shallow RAC (ﬁg. 1(a)) after T time-steps, and let A T, 1 , Θ c be its shallow RAC weights tensor, constructed according to eq.
Assume that the network’s initial mapping functions { f d } Md =1 are linearly independent, and that they, as well as the functions g ν , g (cid:48)
ν in the deﬁnition of Start-End separation rank (eq. 5), are measurable and squareintegrable.
Then, there exist template vectors x (1) , . . .
, x ( M ) ∈
X such that the following holds:
[Equation]
where A ( y T, 1 , Θ c ) is the grid tensor of y T, 1 , Θ c with respect to the above template vectors.
Proof.
See appendix B.1.
The above claim establishes an equality between the Start-End separation rank and the rank of the matrix obtained by the corresponding grid tensor matricization, denoted (cid:74)
A ( y T, 1 , Θ c ) (cid:75) S,E , with respect to a speciﬁc set of template vectors.
Note that the limitation to speciﬁc template vectors does not restrict our results, as grid tensors are merely a tool used to bound the separation rank.
The additional equality to the rank of the matrix obtained by matricizing the shallow RAC weights tensor, will be of use to us when proving our main results below (theorem 1).
Due to the inherent use of data duplication in the computation preformed by a deep RAC (see appendix A.3 for further details), it cannot be written in a closed tensorial form similar to that of eq.
This in turn implies that the equality shown in claim 1 does not hold for functions realized by deep RACs.
The following claim introduces a fundamental relation between a function’s
StartEnd separation rank and the rank of the matrix obtained by the corresponding matricization.
This relation, which holds for all functions, is formulated below for functions realized by deep RACs:
Claim 2.
Let y T,L, Θ
c be a function realized by a depth L RAC (ﬁg. 1(b))
after T time-steps.
Then, for any template vectors x (1) , . .
, x ( M ) ∈
X it holds that:
[Equation]
where A ( y T,L, Θ c ) is the grid tensor of y T,L, Θ c with respect to the above template vectors.
Proof.
See appendix B.2.
Claim 2 will allow us to provide a lower bound on the Start-End separation rank of functions realized by deep RACs, which we show to be exponentially higher than the Start-End separation rank of functions realized by shallow RACs (to be obtained via claim 1).
Thus, in the next section, we employ the above presented tools to show that an exponential enhancement of the Start-End separation rank is brought forth by depth in recurrent networks.
Square-integrability may seem as a limitation at ﬁrst glance, as for example neurons f d ( x )
= σ ( w (cid:62) d x + b d ) with sigmoid or ReLU activation σ ( · ) , do not meet this condition.
However, since in practice our inputs are bounded ( e.g. image pixels by holding intensity values, etc), we may view functions as having compact support, which, as long as they are continuous (holds in all cases of interest), ensures square-integrability.
D EPTH E NHANCED L ONG -T ERM M EMORY IN R ECURRENT N ETWORKS
In this section, we present the main theoretical contributions of this paper.
In section 4.1, we formally present a result which exponentially separates between the memory capacity of a deep ( L = 2 ) recurrent network and a shallow ( L = 1 ) one.
Following the formal presentation of results in theorem 1, we discuss some of their implications and then conclude by sketching a proof outline for the theorem (full proof is relegated to appendix B.3).
In section 4.2, we present a quantitative conjecture regarding the enhanced memory capacity of deep recurrent networks of general depth L , which relies on the inherent combinatorial properties of the recurrent network’s computation.
We leave the formal proof of this conjecture for future work.
Theorem 1 states, that the correlations modeled between the beginning and end of the input sequence to a recurrent network, as measured by the Start-End separation rank (see section 3.1), can be exponentially more complex for deep networks than for shallow ones:
Theorem 1.
Let y T,L, Θ c be the function computing the output after T time-steps of an RAC with L layers, R hidden channels per layer, weights denoted by Θ , and initial hidden states h 0 ,l ,
l ∈
[ L ] (ﬁg. 1 with g = g RAC ).
Assume that the network’s initial mapping functions { f d } Md =1 are linearly independent.
Let sep ( S,E ) (cid:0) y T,L, Θ c (cid:1) be the Start-End separation rank of y T,L, Θ c (eq. 5).
Then, the following holds almost everywhere, i.e. for all values of Θ × h 0 ,l but a set of Lebesgue measure zero:
[Equation]
, for L = 1 (shallow network). sep ( S,E ) (cid:0) y T,L, Θ c (cid:1) ≥ min (cid:110)(cid:16)(cid:16) min { M,R } T/(cid:17)(cid:17) , M T / 2 (cid:111) , for L = 2 (deep network),
where
(cid:16)(cid:16) min { M,R } T/(cid:17)(cid:17) is the multiset coefﬁcient, given in the binomial form by (cid:0) min { M,R } + T/ 2 − 1 T/(cid:1) .
The above theorem readily implies that depth entails an enhanced ability of recurrent networks to model long-term temporal dependencies in the sequential input.
Speciﬁcally, theorem 1 indicates depth efﬁciency – it ensures us that upon randomizing the weights of a deep RAC with R hidden channels per layer, with probability 1 the function realized by it after T time-steps may only be realized by a shallow RAC with a number of hidden channels that is exponentially large.
3 Stated alternatively
, this means that almost all functional dependencies which lie in the hypothesis space of deep RACs with R hidden channels per layer, calculated after T time-steps, are inaccessible to shallow RACs with less than an exponential number of hidden channels.
Thus, a shallow recurrent network would require exponentially more parameters than a deep recurrent network, if it is to implement the same function.
The established role of the Start-End separation rank as a correlation measure between the beginning and the end of the sequence (see section 3.1), implies that these functions, which are realized by almost any deep network and can never be realized by a shallow network of a reasonable size, represent more elaborate correlations over longer periods of time.
The above notion is strengthened by the fact that the Start-End separation rank of deep RACs increases with the sequence length T , while the Start-End separation rank of shallow RACs is independent of it.
This indicates that shallow recurrent networks are much more restricted in modeling long-term correlations than the deep ones, which enjoy an exponentially increasing Start-End separation rank as time progresses.
Below, we present an outline of the proof for theorem 1 (see appendix B.3 for the full version):
Proof sketch of theorem 1.
For a shallow network, claim 1 establishes that the Start-End separation rank of the function realized by a shallow ( L = 1 ) RAC is equal to the rank of the matrix obtained by matricizing the corresponding shallow RAC weights tensor (eq. 6) according to the StartEnd partition: sep ( S,E ) (cid:0) y T, 1 , Θ c
[Equation]
Thus, it sufﬁces to prove that
[Equation]
S,
[Equation]
= R in order to satisfy bullet (1) of the theorem, as the rank is trivially upper-bounded by the dimension of the matrix, M T / 2 .
To this end, we call upon the TT-decomposition of A T, 1 , Θ c , given by eq. 7, which corresponds to the MPS Tensor Network presented in appendix A. We rely on a recent result by Levine et al.
(2017), who state that the rank of the matrix obtained by matricizing any tensor according to a partition ( S, E ) , is equal to a min-cut separating S from E in the Tensor Network graph representing this tensor.
The required equality follows from the fact that the TT-decomposition in eq.is of TT-rank R , which in turn implies that the min-cut in the appropriate Tensor Network graph is equal to R .
For a deep network, claim 2 assures us that the Start-End separation rank of the function realized by a depth L = 2 RAC is lower bounded by the rank of the matrix obtained by the corresponding grid tensor matricization:
[Equation]
Thus, proving that rank (cid:0)
[Equation]
(cid:17)(cid:17) for all of the values of parameters Θ × h 0 ,l but a set of Lebesgue measure zero, would satisfy the theorem, and again, the rank is trivially upper-bounded by the dimension of the matrix, M T / 2 .
We use a lemma proved in Sharir et al.
(2016), which states that since the entries of A ( y T,L, Θ c ) are polynomials in the deep recurrent network’s weights, it sufﬁces to ﬁnd a single example for which the rank of the matricized grid tensor is greater than the desired lower bound.
Finding such an example would indeed imply that for almost all of the values of the network parameters, the desired inequality holds.
We choose a weight assignment such that the resulting matricized grid tensor resembles a matrix obtained by raising a rank- ¯ R ≡ min { M, R } matrix to the Hadamard power of degree T/ 2 .
This operation, which raises each element of the original rank- ¯ R matrix to the power of T/ 2 , was shown to yield a matrix with a rank upper-bounded by the multiset coefﬁcient (cid:16)(cid:16)
¯ RT/ 2 (cid:17)(cid:17) (see e.g. Amini et al.
We show that our assignment results in a matricized grid tensor with a rank which is not only upper-bounded by this value, but actually achieves it.
4. I NCREASE OF M EMORY C APACITY WITH D EPTH
Theorem 1 provides a lower bound of (cid:16)(cid:16) RT/ 2 (cid:17)(cid:17) on the Start-End separation rank of depth L =recurrent networks, exponentially separating deep recurrent networks from shallow ones. By a trivial assignment of weights in higher layers, the Start-End separation rank of even deeper recurrent networks ( L > 2 ) is also lower-bounded by this expression, which does not depend on L . In the following, we conjecture that a tighter lower bound holds for networks of depth L > 2 , the form of which implies that the memory capacity of deep recurrent networks grows exponentially with the network depth:
Conjecture 1.
Under the same conditions as in theorem 1, for all values of Θ × h 0 ,l but a set of
Lebesgue measure zero, it holds for any L that:
[Equation]
We motivate conjecture 1 by investigating the combinatorial nature of the computation performed by a deep RAC.
By constructing Tensor Networks which correspond to deep RACs, we attain an informative visualization of this combinatorial perspective.
In appendix A, we provide full details of this construction and present the formal motivation for the conjecture.
Below, we qualitatively outline this combinatorial approach.
A Tensor Network is essentially a graphical tool for representing algebraic operations which resemble multiplications of vectors and matrices, between higher order tensors.
Fig.
2 shows an example of the Tensor Network representing the computation of a depth L = 3 RAC after T = 6 time-steps.
This well-deﬁned computation graph hosts the values of the weight matrices at its nodes.
The inputs { x 1 , . . .
, x T } are marked by their corresponding time-step { 1 , . . .
, T } , and are integrated in a depth dependent and time-advancing manner (see further discussion regarding this form in appendix A.3), as portrayed in the example of ﬁg. 2.
We highlight in red the basic unit in the Tensor Network which connects “Start” inputs { 1 , . . .
, T / 2 } and “End” inputs { T / 2 +1 , . .
, T } .
In order to estimate a lower bound on the Start-End separation rank of a depth L > 2 recurrent network, we employ a similar strategy to that presented in the proof sketch of the L = 2 case (see section 4.1).
Speciﬁcally, we rely on the fact that it is sufﬁcient to ﬁnd a speciﬁc instance of the network parameters Θ × h 0 ,l for which (cid:74)
A ( y T,L, Θ c ) (cid:75) S,E achieves a certain rank, in order for this rank to bound the Start-End separation rank of the network from below.
Indeed, we ﬁnd a speciﬁc assignment of the network weights, presented in appendix A.4, for which the Tensor Network effectively takes the form of the basic unit connecting “Start” and “End”, raised to the power of the number of its repetitions in the graph (bottom of ﬁg. 2).
This basic unit corresponds to a simple computation represented by a grid tensor with Start-End matricization of rank
R .
Raising such a matrix to the Hadamard power of any p ∈ Z , results in a matrix with a rank upper bounded by (cid:16)(cid:16)
R p (cid:17)(cid:17) , and the challenge of proving the conjecture amounts to proving that the upper bound is tight in this case.
In appendix A.4, we prove that the number of repetitions of the basic unit connecting “Start” and “End” in the deep RAC Tensor Network graph, is exactly equal to (cid:16)(cid:16)
[Figure]
for any depth L .
For example, in the T = 6 , L = 3 network illustrated in ﬁg. 2, the number of repetitions indeed corresponds to p =
(cid:16)(cid:16) 32 (cid:17)(cid:17) =
It is noteworthy that for L = 1 , 2 the bound in conjecture 1 coincides with the bounds that were proved for these depths in theorem 1.
Conjecture 1 indicates that beyond the proved exponential advantage in memory capacity of deep networks over shallow ones, a further exponential separation may be shown between recurrent networks of different depths.
We leave the proof of this result, which can reinforce and reﬁne the understanding of advantages brought forth by depth in recurrent networks, as an open problem.
D ISCUSSION
The notion of depth efﬁciency, by which deep networks efﬁciently express functions that would require shallow networks to have a super-polynomial size, is well established in the context of convolutional networks.
However, recurrent networks differ from convolutional networks, as they are suited by design to tackle inputs of varying lengths.
Accordingly, depth efﬁciency alone does not account for the remarkable performance of recurrent networks on long input sequences.
In this paper, we identiﬁed a fundamental need for a quantiﬁer of ‘time-series expressivity’, quantifying the memory capacity of recurrent networks.
In order to meet this need, we proposed a measure of the ability of recurrent networks to model long-term temporal dependencies, in the form of the Start-End separation rank.
The separation rank was used to quantify correlations in convolutional networks, and has roots in the ﬁeld of quantum physics.
The proposed measure adjusts itself to the temporal extent of the input series, and quantiﬁes the ability of the recurrent network to correlate the incoming sequential data as time progresses.
We analyzed the class of Recurrent Arithmetic Circuits, which are closely related to successful RNN architectures, and proved that the Start-End separation rank of deep RACs increases exponentially as the input sequence extends, while that of shallow RACs is independent of the input length.
These results, which demonstrate that depth brings forth an overwhelming advantage in the ability of recurrent networks to model long-term dependencies, were achieved by combining tools from the ﬁelds of measure theory, tensorial analysis, combinatorics, graph theory and quantum physics.
Such analyses may be readily extended to other architectural features employed in modern recurrent networks.
Indeed, the same time-series expressivity question may now be applied to the different variants of LSTM networks, and the proposed notion of Start-End separation rank may be employed for quantifying their memory capacity.
We have demonstrated that such a treatment can go beyond unveiling the origins of the success of a certain architectural choice, and leads to new insights.
The above established observation that correlations achievable by vanilla shallow recurrent network do not adapt at all to the sequence length, is an exemplar of this potential.
Moreover, practical recipes may emerge by such theoretical analyses.
The experiments preformed in Hermans and Schrauwen (2013), suggest that shallow layers of recurrent networks are related to short time-scales, e g. in speech: phonemes, syllables, words, while deeper layers appear to support correlations of longer time-scales, e g. full sentences, elaborate questions.
These ﬁndings open the door to further depth related investigations in recurrent networks, and speciﬁcally the role of each layer in modeling temporal correlations may be better understood.
Levine et al.
(2017) establish theoretical observations which translate into practical conclusions regarding the number of hidden channels to be chosen for each layer in a deep convolutional network.
The conjecture presented in this paper, by which the Start-End separation rank of recurrent networks grows exponentially with depth, can similarly entail practical recipes for enhancing their memory capacity.
Such analyses can be reinforced by experiments and lead to a profound understanding of the contribution of deep layers to the recurrent network’s memory.
Indeed, we view this work as an important step towards novel methods of matching the recurrent network architecture to the temporal correlations in a given sequential data set
A T ENSOR N ETWORK R EPRESENTATION OF R ECURRENT A RITHMETIC CIRCUITS
In this section, we expand our algebraic view on recurrent networks and make use of a graphical approach to tensor decompositions referred to as Tensor Networks (TNs).
The tool of TNs is mainly used in the many-body quantum physics literature for a graphical decomposition of tensors, and has been recently connected to the deep learning ﬁeld by Levine et al.
(2017), who constructed a deep convolutional network in terms of a TN.
The use of TNs in machine learning has appeared in an empirical context, where Stoudenmire and Schwab (2016) trained a Matrix Product State (MPS) TN to preform supervised learning tasks on the MNIST data set (LeCun et al., 1998).
The constructions presented in this section suggest a separation in expressiveness between recurrent networks of different depths, as formulated by conjecture 1.
We begin in section A.1 by providing a brief introduction to TNs.
Next, we present in section A.2 the TN which corresponds to the calculation of a shallow RAC, and tie it to a common TN architecture referred to as a Matrix Product State (MPS) (see overview in e.g. Or´us (2014)), and equivalently to the tensor train (TT) decomposition (Oseledets, 2011).
Subsequently, we present in section A.3 a TN construction of a deep RAC, and emphasize the characteristics of this construction that are the origin of the enhanced ability of deep RACs to model elaborate temporal dependencies.
Finally, in section A.4, we make use of the above TNs construction in order to formally motivate conjecture 1, according to which the Start-End separation rank of RACs grows exponentially with depth.
A. I NTRODUCTION TO T ENSOR N ETWORKS
A TN is a weighted graph, where each node corresponds to a tensor whose order is equal to the degree of the node in the graph.
Accordingly, the edges emanating out of a node, also referred to as its legs, represent the different modes of the corresponding tensor.
The weight of each edge in the graph, also referred to as its bond dimension, is equal to the dimension of the appropriate tensor mode.
In accordance with the relation between mode, dimension and index of a tensor presented in section 3.2, each edge in a TN is represented by an index that runs between 1 and its bond dimension.
Fig.
3(a) shows three examples: (1) A vector, which is a tensor of order 1 , is represented by a node with one leg.
(2) A matrix, which is a tensor of order 2 , is represented by a node with two legs.
(3) Accordingly, a tensor of order N is represented in the TN as a node with N legs.
We move on to present the connectivity properties of a TN.
Edges which connect two nodes in the TN repre- sent an operation between the two corresponding tensors.
A index which represents such an edge is called a contracted index, and the operation of contracting that index is in fact a summation over all of the values it can take.
An index representing an edge with one loose end is called an open index.
The tensor represented by the entire TN, whose order is equal to the number of open indices, can be calculated by summing over all of the contracted indices in the network.
An example for a contraction of a simple TN is depicted in ﬁg. 3(b).
There, a TN corresponding to the operation of multiplying a vector v ∈ R r 1 by a matrix M ∈ R r 2 × r 1 is performed by summing over the only contracted index, k .
As there is only one open index, d , the result of contracting the network is an order 1 tensor (a vector): u ∈ R r 2 which upholds u = M v .
Though we use below the contraction of indices in more elaborate TNs, this operation can be essentially viewed as a generalization of matrix multiplication.
A. S HALLOW RAC T ENSOR N ETWORK
The computation of the output at time T that is preformed by the shallow recurrent network given by eqs. 1 and 3, or alternatively by eqs.
6 and 7, can be written in terms of a TN.
Fig.
4(a) shows this TN, which given some initial hidden state h 0 , is essentially a temporal concatenation of a unit cell that preforms a similar computation at every time-step, as depicted in ﬁg. 4(b).
For any time t < T , this unit cell is composed of the input weights matrix, W I , contracted with the inputs vector, f ( x t ) , and the hidden weights matrix, W H , contracted with the hidden state vector of the previous time-step, h t − 1 .
The ﬁnal component in each unit cell is the 3 legged triangle representing the order 3 tensor δ ∈ R R × R × R , referred to as the δ tensor , deﬁned by:
[Equation]
[Equation] with i j ∈
[ R ] ∀ j ∈ [3] , i.e. its entries are equal to 1 only on the super-diagonal and are zero otherwise.
The use of a triangular node in the TN is intended to remind the reader of the restriction given in eq.
The recursive relation that is deﬁned by the unit cell, is given by the TN in ﬁg. 4(b):
[Equation]
where k t ∈ [ R ] .
In the ﬁrst equality, we simply follow the TN prescription and write a summation over all of the contracted indices in the left hand side of ﬁg. 4(b), in the second equality we use the deﬁnition of matrix multiplication, and in the last equality we use the deﬁnition of the δ tensor.
The component-wise equality of eq.
11 readily implies h t =
( W H h t − 1 ) (cid:12) ( W I f ( x t )) , reproducing the recursive relation in eqs.
1 and 3, which deﬁnes the operation of the shallow RAC.
From the above treatment, it is evident that the restricted δ tensor is in fact the component in the TN that yields the element-wise multiplication property.
After T repetitions of the unit cell calculation with the sequential input { x t } Tt =1 , a ﬁnal multiplication of the hidden state vector h T by the output weights matrix W O yields the output vector y T, 1 , Θ .
The tensor network which represents the order T shallow RAC weights tensor A T, 1 , Θ c , which appears in eqs.
6 and 7, is given by the TN in the upper part of ﬁg. 4(a).
In ﬁg. 4(c), we show that by a simple contraction of indices, the TN representing the shallow RAC weights tensor A T, 1 , Θ c can be drawn in the form of a standard MPS TN.
This TN allows the representation of an order T tensor with a linear (in T ) amount of parameters, rather than the regular exponential amount ( A has M T entries).
The decomposition which corresponds to this
TN is known as the Tensor Train (TT) decomposition of rank R in the tensor analysis community, its explicit form given in eq.
The presentation of the shallow recurrent network in terms of a TN allows the employment of the min-cut anal- ysis, which was introduced by Levine et al.
(2017) in the context of convolutional networks, for quantiﬁcation of the information ﬂow across time modeled by the shallow recurrent network.
This was indeed preformed in our proof of the shallow case of theorem 1.
We now move on to present the computation preformed by a deep recurrent network in the language of TNs.
A. D EEP RAC T ENSOR N ETWORK
The construction of a TN which matches the calculation of a deep recurrent network is far less trivial than that of the shallow case, due to the seemingly innocent property of reusing information which lies at the heart of the calculation of deep recurrent networks.
Speciﬁcally, all of the hidden states of the network are reused, since the state of each layer at every time-step is duplicated and sent as an input to the calculation of the same layer in the next time-step, and also as an input to the next layer up in the same time-step (see ﬁg. 1(b)).
The required operation of duplicating a vector and sending it to be part of two different calculations, which is simply achieved in any practical setting, is actually impossible to represent in the framework of TNs.
We formulate this notion in the following claim:
Claim 3.
Let v ∈ R P , P ∈ N be a vector.
v is represented by a node with one leg in the TN notation.
The operation of duplicating this node, i.e. forming two separate nodes of degree 1 , each equal to v , cannot be achieved by any TN.
Proof.
We assume by contradiction that there exists a Tensor Network φ which operates on any vector v ∈ R P and clones it to two separate nodes of degree 1 , each equal to v , to form an overall TN representing v ⊗ v .
Component wise, this implies that φ upholds ∀ v ∈ R P : (cid:80) Pi =
1 φ ijk v i = v j v k .
By our assumption, φ duplicates the standard basis elements of R P , denoted { ˆ e ( α ) } Pα =1 , meaning that ∀ α ∈ [ P ] :
P
[Equation]
By deﬁnition of the standard basis elements, the left hand side of eq.
12 takes the form φ αjk while the right hand side equals 1 only if j = k = α , and otherwise 0 .
Utilizing the δ -tensor notation presented in eq.
10, in order to successfully clone the standard basis elements, eq.
12 implies that φ must uphold φ αjk = δ αjk .
However, for v = 1 , i.e. ∀ j ∈ [ P ] : v j = 1 , a cloning operation does not take place when using this value of φ , since (cid:80) Pi =1 φ ijk v i = (cid:80) Pi [Equation] (cid:54) =
[Equation]
= [Equation] j , in contradiction to φ duplicating any vector in R P .
Claim 3 seems to pose a hurdle in our pursuit of a TN representing a deep recurrent network.
Nonetheless, a form of such a TN may be attained by a simple ‘trick’ – in order to model the duplication that is inherently present in the deep recurrent network computation, we resort to duplicating the input data itself.
By this technique, for every duplication that takes place along the calculation, the input is inserted into the TN multiple times, once for each sequence that leads to the duplication point.
This principle, which allows us to circumvent the restriction imposed by claim 3, yields the elaborate TN construction of deep RACs depicted in ﬁg. 5.
It is important to note that these TNs, which grow exponentially in size as the depth L of the recurrent network represented by them increases, are merely a theoretical tool for analysis and not a suggested implementation scheme for deep recurrent networks.
The actual deep recurrent network is constructed according to the simple scheme given in ﬁg. 1(b), which grows only linearly in size as the depth L increases, despite the corresponding
TN growing exponentially.
In fact, this exponential ‘blow-up’ in the size of the TNs representing the deep recurrent networks is closely related to their ability to model more intricate correlations over longer periods of time in comparison with their shallower counterparts, which was established in section 4.
Fig.
5 shows TNs which correspond to depth L = 2 , 3 RACs.
Even though the TNs in ﬁg. 5 seem rather convoluted and complex, their architecture follows clear recursive rules.
In ﬁg. 5(a), a depth L = 2 recurrent network is presented, spread out in time onto T = 4 time-steps.
To understand the logic underlying the input duplication process, which in turn entails duplication of entire segments of the TN, we focus on the calculation of the hidden state vector h 2 , 2 that is presented in ﬁg. 5(b).
When the ﬁrst inputs vector, f ( x 1 ) , is inserted into the network, it is multiplied by W I , 1 and the outcome is equal to h 1 , 1 .
Next, h 1 , 1 is used in two different places, as an inputs vector to layer L = 2 at time t = 1 , and as a hidden state vector in layer L = 1 for time t = 2 calculation.
Our input duplication technique inserts f ( x 1 ) into the network twice, so that the same exact h 1 , 1 is achieved twice in the TN, as marked by the red dotted line in ﬁg. 5(b).
This way, every copy of h 1 ,
In this ﬁgure, the initial condition for each layer l ∈ L , h l, 0 , is chosen such that a vector of ones will be present in the initial element-wise multiplication: ( h 0 ,l ) T = 1 T ( W H ,l ) † ,where † denotes the pseudoinverse operation.
goes to the appropriate segment of the calculation, and indeed the TN in ﬁg. 5(b) holds the correct value of h 2 , 2 :
[Equation]
The extension to deeper layers leads us to a fractal structure of the TNs, involving many self similarities, as in the L = 3 example given in ﬁg. 5(c).
The duplication of intermediate hidden states, marked in red and blue in this example, is the source of the apparent complexity of this L = 3 RAC TN.
Generalizing the above L = 1 , 2 , 3 examples, a TN representing an RAC of general depth L and of T time-steps, would involve in its structure T duplications of TNs representing RACs of depth L − 1 , each of which has a distinct length in time-steps i , where i ∈
[ T ] .
This fractal structure leads to an increasing with depth complexity of the TN representing the depth L RAC computation, which we show in the next subsection to motivate the combinatorial lower bound on the Start-End separation rank of deep RACs, given in conjecture 1.
A. A F ORMAL M OTIVATION FOR C ONJECTURE
The above presented construction of TNs which correspond to deep RACs, allows us to further investigate the effect of network depth on its ability to model long-term temporal dependencies. We present below a formal motivation for the lower bound on the Start-End separation rank of deep recurrent networks, given in conjecture 1. Though our analysis employs TNs visualizations, it is formal nonetheless – these graphs represent the computation in a well-deﬁned manner (see sections A.1-A.3). Our conjecture relies on the fact that it is sufﬁcient to ﬁnd a speciﬁc instance of the network parameters Θ × h 0 ,l for which (cid:74)
A ( y T,L, Θ c ) (cid:75) S,E achieves a certain rank, in order for this rank to be a lower bound on the Start- End separation rank of the network.
This follows from combining claim 2 and lemma 1.
Claim 2 assures us that the Start-End separation rank of the function realized by an RAC of any depth L , is lower bounded by the rank of the matrix obtained by the corresponding grid tensor matricization: sep ( S,E ) (cid:0) y T,L, Θ c (cid:1) ≥ rank (cid:0) (cid:74)
A ( y T,L, Θ c ) (cid:75) S,E (cid:1) .
Thus, one must show that rank (cid:0) (cid:74)
A ( y T,L, Θ c )
[Table]
for all of the values of parameters Θ × h 0 ,l but a set of Lebesgue measure zero, in order to establish the lower bound in conjecture 1.
Next, we rely on lemma 1, which states that since the entries of A ( y T,L, Θ c ) are polynomials in the deep recurrent network’s weights, it sufﬁces to ﬁnd a single example for which the rank of the matricized grid tensor is greater than the desired lower bound.
Finding such an example would indeed imply that for almost all of the values of the network parameters, the desired inequality holds.
In the following, we choose a weight assignment that effectively ‘separates’ between the ﬁrst layer and higher layers, in the sense that W I , 2 is of rank-1.
This is done in similar spirit to the assignment used in the proof of theorem 1, in which W I , 2 ij ≡ δ i 1 (see section B.3).
Under this simplifying assignment, which sufﬁces for our purposes according to the above discussion, the entire computation performed in deeper layers contributes only a constant factor to the matricized grid tensor.
In this case, the example of the TN corresponding to an
RAC of depth L = 3 after T = 6 time-steps, which is shown in full in ﬁg.2, takes the form shown in the upper half of ﬁg. 6.
Next, in order to evaluate rank (cid:0) (cid:74)
A ( y T,L, Θ c ) (cid:75) S,E (cid:1) , we note that graph segments which involve only indices from the “Start” set, will not affect the rank of the matrix under mild conditions on W I , 1 , W H , 1 .
Speciﬁcally, under the Start-End matricization these segments will amount to a different constant multiplying each row of the matrix.
For the example of the RAC of depth L = 3 after T = 6 time-steps, this amounts to the effective TN given in the bottom left side of ﬁg. 6.
Finally, the dependence of this TN on the indices of time- steps { T / 2 +2 , . .
, T } , namely those outside of the basic unit involving indices of time-steps { 1 , . .
, T / 2 +1 } , may only increase the resulting Start-End matricization rank. Thus, we are left with an effective TN resembling the one shown in section 4.2, where the basic unit separating “Start” and “End” indices is raised to the power of the number of its repetitions in the graph. In the following, we prove a claim according to which the number of repetitions of this basic unit in the TN graph increases exponentially with the depth of the RAC: Claim 4.
Let φ ( T, L, R ) be the TN representing the computation performed after T time-steps by an RAC with L layers and R hidden channels per layer.
Then, the number of occurrences in layer L = 1 of the basic unit connecting “Start” and “End” indices (bottom right in ﬁg. 6), is exactly (cid:16)(cid:16)
T/ 2 L − 1 (cid:17)(cid:17) .
Proof.
Let y T,L, Θ c be the function computing the output after T time-steps of an RAC with L layers, R hidden channels per layer and weights denoted by Θ . In order to focus on repetitions in layer L = 1 , we assign W I , 2 ij ≡ δ i 1 for which the following upholds 7 : [Equation]
r 1 ,...,r t 2 = (cid:32) t 2 (cid:89)
j =
W I , 1 r j d j t 2 − 1 (cid:89) j =
W H , 1 r j r j + (cid:33) , where the constant term in the ﬁrst line is the contribution of the deeper layers under this assignment, and the tensor V d 1 ...d T/ 2 , which becomes a vector under the Start-End matricization, reﬂects the contribution of the “Start” set indices.
Observing the argument of the chain of products in the above expression, (cid:80)
Rr 1 ,...,r t 2 =1 (cid:16)(cid:81) t 2 j =1 W I , 1 r j d j (cid:81) t 2 − 1 j =1 W H , 1 r j r j +1 (cid:17) , it is an order t 2 tensor, exactly given by the TN representing the computation of a depth L = 1 RAC after t 2 time-steps.
Speciﬁcally, for t 2 = T / 2 + 1 , it is exactly equal to the basic TN unit connecting “Start” and “End” indices, and for T / 2 + 1 < t 2 ≤ T it contains this basic unit.
This means that in order to obtain the number of repetition of this basic unit in φ , we must count the number of multiplications implemented by the chain of products in the above expression.
Indeed this number is equal to:
T (cid:88) t
L = T / 2 + t L (cid:88) t L − 1 = T / 2 + · · · t 3 (cid:88) t 2 = T / 2 + t 2 = (cid:18)(cid:18)
T/ 2 L − 1 (cid:19)(cid:19)
Finally, the form of the lower bound presented in conjecture 1 is obtained by considering a rank R matrix, such as the one obtained by the Start-End matricization of the TN basic unit discussed above, raised to the Hadamard power of (cid:16)(cid:16) T / 2 L − 1 (cid:17)(cid:17) .
The rank of the resultant matrix, is upper bounded by    
R (cid:18)(cid:18)
T/ 2 L − 1 (cid:19)(cid:19)     . We leave it as an open problem to prove conjecture 1, by proving that the upper bound is indeed tight in this case.
B D EFERRED PROOFS B. P ROOF OF C LAIM
We begin by proving the equality sep
( S,E ) (cid:0) y T, 1 , Θ c (cid:1)
= rank (cid:0) (cid:74)
A T, 1 , Θ c (cid:75) S,E (cid:1) .
As shown in Cohen and Shashua (2017), for any function f : R s × · · · × R s → R which follows the structure of eq.
6 with a general weights tensor A , assuming that { f d } Md =1 are linearly independent, measurable, and square-integrable (as assumed in claim 1)
, it holds that sep ( S,E ) ( f ) =
rank ( (cid:74) A (cid:75) S,E ) .
Speciﬁcally, for f = y T, 1 , Θ c and A =
A T, 1 , Θ c the above equality holds.
It remains to prove that there exists template vectors for which rank (cid:0) (cid:74)
A T [Equation]
1 , Θ c [Equation] rank [Equation]
For any given set of template vectors x (1) , . . .
, x ( M ) ∈ X , we deﬁne the matrix F ∈ R M × M such that F ij = f
j ( x ( i ) ) , for which it holds that:
[Equation]
The right-hand side in the above equation can be regarded as a linear transformation of A T, 1 , Θ c speciﬁed by the tensor operator F ⊗ · · · ⊗ F , which is more commonly denoted by ( F ⊗ · · · ⊗ F )
( A T, 1 , Θ c ) .
According to lemma 5.6 in Hackbusch (2012), if F is non-singular then rank (cid:0) (cid:74) ( F ⊗ · · · ⊗ F )
( A T, 1 , Θ c ) (cid:75)
I
[Equation]
J (cid:1) = rank (cid:0) (cid:74)
A T, 1 , Θ c (cid:75) I,J (cid:1) , for any partition ( I, J ) .
To conclude our proof, we simply note that Cohen and Shashua (2016) showed that if { f d } Md =1 are linearly independent then there exists template vectors for which F is non- singular.
(cid:3)
B. P ROOF OF C LAIM
If sep ( S,E ) (cid:0) y T,L, Θ c (cid:1) =
∞ then the inequality is trivially satisﬁed.
Otherwise, assume that sep ( S,E ) (cid:0) y T,L, Θ c (cid:1) =
K ∈ N , and let { g si , g ei }
Ki =1 be the functions of the respective decomposition to a sum of separable functions, i.e. that the following holds:
[Equation]
Then, by deﬁnition of the grid tensor, for any template vectors x (1) , . .
, x ( M ) ∈
X the following equality holds:
[Equation]
where V ν and U ν are the tensors holding the values of g sν and g eν , respectively, at the points deﬁned by the template vectors.
Under the matricization according to the ( S, E ) partition, it holds that (cid:74) V ν (cid:75) S,E and (cid:74) U ν (cid:75) S,E are column and row vectors, respectively, which we denote by v ν and u Tν .
It follows that the matricization of the grid tensor is given by:
[Equation]
which means that rank
[Equation]
B. P ROOF OF T HEOREM
In this sub-section, we follow the proof strategy that is outlined in section 4, and prove theorem 1, which shows an exponential advantage of deep recurrent networks over shallow ones in the ability to model long-term dependencies, as measured by the Start-End separation rank (see section 3.1).
In sections B.3.1 and B.3.2, we prove the bounds on the Start-End separation rank of the shallow and deep RACs, respectively, while more technical lemmas which are employed during the proof are relegated to section B.3.3.
B.3. T HE S TART -E ND S EPARATION R ANK OF S HALLOW RAC S
We consider the Tensor Network construction of the calculation carried out by a shallow RAC, given in ﬁg. 4.
According to the presented construction, the shallow RAC weights tensor (eqs. 6 and 7) is represented by a
Matrix Product State (MPS) Tensor Network (Or´us, 2014), with the following order-3 tensor building block: M k t − 1 d t k t
=
W I k t d t W H k t k t − 1 , where d t ∈ [ M ] is the input index and k t − 1 , k t ∈
[ R ] are the internal indices (see ﬁg. 4(c)).
In TN terms, this means that the bond dimension of this MPS is equal to R .
We apply the result of Levine et al.
(2017), who state that the rank of the matrix obtained by matricizing any tensor according to a partition ( S, E ) is equal to a min-cut separating S from E in the Tensor Network graph representing this tensor, for all of the values of the TN parameters but a set of Lebesgue measure zero.
In this MPS Tensor Network, the minimal cut w.r.t.
the partition ( S, E ) is equal to the bond dimension R , unless R > M T / 2 , in which case the minimal cut contains the external legs instead.
Thus, in the TN representing A T, 1 , Θ c , the minimal cut w.r.t.
the partition ( S, E ) is equal to min { R, M T / 2 } , implying rank (cid:0) (cid:74)
[Equation]
T,
[Equation]
c (cid:75)
[Equation]
S
[Equation]
(cid:1)
=
min { R, M T / 2 } for all values of the parameters but a set of Lebesgue measure zero.
The ﬁrst half of the theorem follows from applying claim 1, which assures us that the Start-End separation rank of the function realized by a shallow ( L = 1 ) RAC is equal to rank (cid:0) (cid:74)
A T, 1 , Θ c (cid:75) ) S,E (cid:1) .
(cid:3)
B.3. L OWER - BOUND ON THE S TART -E ND S EPARATION R ANK OF D EEP RAC S
For a deep network, claim 2 assures us that the Start-End separation rank of the function realized by a depth L = 2 RAC is lower bounded by the rank of the matrix obtained by the corresponding grid tensor matricization, for any choice of template vectors.
Speciﬁcally:
[Equation]
Thus, since it trivially holds that rank (cid:0) (cid:74) A ( y T,L, Θ c )
[Equation]
(the rank is smaller than the dimension of the matrix), proving that rank (cid:0) (cid:74)
A
[Equation]
y
[Equation] [Equation]
min {
[Equation] [Equation]
(cid:17)(cid:17) for all of the values of parameters Θ × h 0 ,l but a set of Lebesgue measure zero, would satisfy the theorem.
In the following, we provide an assignment of weight matrices and initial hidden states for which rank (cid:0)
[Equation]
S,E (cid:1) =
(cid:16)(cid:16) min { R,M } T/(cid:17)(cid:17) .
In accordance with claim 5, this will sufﬁce as such an assignment implies this rank is achieved for all conﬁgurations of the recurrent network weights but a set of Lebesgue measure zero.
We begin by choosing a speciﬁc set of template vectors x (1) , . . .
, x ( M ) ∈ X .
Let F ∈ R M
× M be a matrix with entries deﬁned by F ij ≡ f j ( x ( i ) ) .
According to Cohen and Shashua (2016), since { f d } Md =1 are linearly independent, then there is a choice of template vectors for which F is non-singular.
Next, we describe our assignment.
In the expressions below we use the notation δ ij = (cid:40) 1 i =
j 0 i (cid:54) =
j .
Let z ∈ R \ { 0 } be an arbitrary non-zero real number, let Ω ∈ R
+ be an arbitrary positive real number, and let Z ∈ R R × M be a matrix with entries Z ij ≡ (cid:40)
z Ω i δ ij i ≤ M i > M .
We set W I , 1 ≡ Z · ( F T ) − 1 and set W I , 2 such that its entries are W I , 2 ij ≡ δ i 1 , we set W H , 1 ≡ W H , 2 ≡ I , i.e. to the identity matrix, and additionally we set the entries of W O to W O ij = δ 1 j .
Finally, we choose the initial hidden state values so they bear no effect on the calculation, namely h 0 ,l = (cid:0) W H ,l (cid:1) − 1 1 = 1 for l = 1 , 2 .
Under the above assignment, the output for the corresponding class c after T time-steps is equal to:
[Equation]
When evaluating the grid tensor for our chosen set of template vectors, i.e.
[Equation]
) d 1 ,...,d T = y T,L, Θ c ( x ( d 1 ) , . . .
, x ( d T ) ) , we can substitute f j ( x ( i ) )
≡ F ij , and thus ( W I , 1 f ( x ( d ) ))
r =
( W I , 1 F T ) rd =
( Z · ( F T ) − 1 F T ) rd
=
Z rd .
Since we deﬁned Z such that for r ≥ min { R, M } Z rd = 0 , and denoting ¯ R ≡ min { R, M } for brevity of notation, the grid tensor takes the following form:
[Equation]
where we split the product into two expressions, the left part that contains only the indices in the start set S , i.e. d 1 , . .
, d T / 2 , and the right part which contains all external indices (in the start set S and the end set E ).
Thus, under matricization w.r.t.
the Start-End partition, the left part is mapped to a vector a ≡ (cid:114) (cid:81) T / 2 t =1 (cid:80) ¯
Rr =1 (cid:81) tj =1 Z rd j (cid:122) S,E containing only non-zero entries per the deﬁnition of Z , and the right part is mapped to a matrix B ≡ (cid:114) (cid:81)
Tt
[Equation] [Equation]
Rr
[Equation]
rd j (cid:122) S,E , where each entry of u multiplies the corresponding row of B .
This results in:
[Equation]
Since a contains only non-zero entries, diag( a ) is of full rank, and so rank (cid:0) (cid:74)
A ( y T,L, Θ c ) d 1 ,...,d T (cid:75) S,E (cid:1) = rank ( B ) , leaving us to prove that rank ( B ) =
(cid:16)(cid:16)
¯ R T / 2 (cid:17)(cid:17) .
For brevity of notation, we deﬁne N ≡ (cid:16)(cid:16)
¯ R T / 2 (cid:17)(cid:17)
To prove the above, it is sufﬁcient to show that B can be written as a sum of N rank-1 matrices, i.e. B = (cid:80)
Ni =1 u ( i ) ⊗ v ( i ) , and that { u ( i ) } Ni =1 and { v ( i ) }
Ni =1 are two sets of linearly independent vectors.
Indeed, applying claim 6 on the entries of B , speciﬁed w.r.t.
the row ( d 1 , . . .
, d T / 2 ) and column ( d T / 2 +1 , . . .
, d T ) , yields the following form:
[Equation]
∈ trajectory (cid:16) p ( T/ 2) (cid:17) ¯ R (cid:89) r =
T (cid:89)
[Equation]
   , where for all k , p ( k ) is ¯ R -dimensional vector of non-negative integer numbers which sum to k , and we explicitly deﬁne states (cid:0) ¯ R, T / 2 (cid:1) and trajectory (cid:16) p ( T / 2 ) (cid:17) in claim 6, providing a softer more intuitive deﬁnition hereinafter.
states (cid:0) ¯ R, T / 2 (cid:1) can be viewed as the set of all possible states of a bucket containing T / 2 balls of ¯ R colors, where p ( T / 2 ) r for r ∈ [ ¯ R ] speciﬁes the number of balls of the r ’th color.
trajectory (cid:16) p ( T / 2 ) (cid:17) can be viewed as all possible trajectories from a given state to an empty bucket, i.e. (0 , . . . , 0) , where at each step we remove a single ball from the bucket.
We note that the number of all initial states of the bucket is exactly (cid:12)(cid:12) states (cid:0) ¯ R, T / 2 (cid:1)(cid:12)(cid:12) =
N ≡ (cid:16)(cid:16)
¯ R T / 2 (cid:17)(cid:17) .
Moreover, since the expression in the left parentheses contains solely indices from the start set S , i.e. d 1 , . . .
, d T / 2 , while the right contains solely indices from the end set E , i.e. d T / 2 +1 , . . .
, d T , then each summand is in fact a rank-1 matrix.
Speciﬁcally, it can be written as u p ( T/ 2) ⊗ v p ( T/ 2) , where the entries of u p ( T/ 2) are represented by the expression in the left parentheses, and those of v p ( T/ 2) by the expression in the right parentheses.
We prove that the set (cid:110) u p ( T/ 2) ∈ R M T/ 2 (cid:111) p ( T/ 2) ∈ states ( ¯ R, T / 2 ) is linearly independent by arranging it as the columns of the matrix U ∈ R M
T/
× N , and showing that its rank equals to N .
Speciﬁcally, we observe the sub-matrix deﬁned by the subset of the rows of U , such that we select the row d ≡ ( d 1 , . . .
, d T / 2 ) if it holds that ∀ j, d j ≤ d j +1 .
Note that there are exactly N such rows, similarly to the number of columns, which can be intuitively understood since inside the imaginary bucket deﬁning the columns there is no meaning of order in the balls, and having imposed the restriction ∀ j, d j ≤ d j +1 on the T / 2 length tuple d , there is no longer a degree of freedom to order the ‘colors’ in d , reducing the number of rows from M T / 2 to N .
Thus, in the resulting sub-matrix, denoted by ¯ U ∈ R N × N , not only do the columns correspond to the vectors of states (cid:0) ¯ R, T / 2 (cid:1) , but also its rows, where the row speciﬁed by the tuple d , corresponds to the vector q ( T / 2 ) ∈ states (cid:0) ¯ R, T / 2 (cid:1) , such that for r ∈
[ ¯ R ] : q ( T / 2 ) r ≡ |{ j ∈ [ T / 2 ] | d j = r }| speciﬁes the amount of repetitions of the number (‘color’)
r in the given tuple.
Accordingly, for each element of ¯ U the following holds:
[Equation]
r p ( T/ 2) r (cid:32) ¯ q ( T / 2 ) r ≡ Ω r / 2 q ( T / 2 ) r ¯ p ( T / 2 ) r ≡ Ω r / 2 p
( T / 2 ) r (cid:33) ⇒ =
z (cid:68) ¯ q ( T/ 2) , ¯ p ( T/ 2) (cid:69) .
Since the elements of ¯ U are polynomial in z , then according to lemma 1, it is sufﬁcient to show that there exists a single contributor to the determinant of ¯ U that has the highest degree of z in order to ensure that the matrix is fully ranked for all values of z but a ﬁnite set.
Observing the summands of the determinant, i.e. z (cid:80) q
( T/ 2) ∈ states ( ¯ R,T/ 2 ) (cid:68) ¯ q ( T / 2 ) ,σ (¯ q ( T/ 2) ) (cid:69) , where σ is a permutation on the rows of ¯ U , and noting that states (cid:0) ¯ R, T / 2 (cid:1) is a set of non-negative numbers by deﬁnition, lemma 2 assures us the existence of a strictly maximal contributor, satisfying the conditions of lemma 1.
We prove that the set (cid:110) v p ( T/ 2) ∈ R M T/ 2 (cid:111) p ( T/ 2) ∈ states ( ¯ R, T / 2 ) is linearly independent by arranging it as the columns of the matrix V ∈ R M T/ 2 × N , and showing that its rank equals to N .
As in the case of U , we select the same sub-set of rows to form the sub-matrix ¯ V ∈ R N × N .
We show that each of the diagonal elements of ¯ V is a polynomial function whose degree is strictly larger than the degree of all other elements in its row.
As an immediate consequence, the product of the diagonal elements, i.e. (cid:81) Ni =1 ¯ V ii
( z ) , has degree strictly larger than any other summand of the determinant det( ¯ V ) , and by employing lemma 1, ¯ V of z but a ﬁnite set.
The degree of the polynomial function in each entry of ¯ V is given by:
[Equation]
The above can be formulated as the following combinatorial optimization problem.
We are given an initial state p ( T / 2 ) of the bucket of T / 2 balls of ¯ R colors and a sequence of colors d
= ( d T / 2 +1 , . . .
, d T ) .
At time-step j one ball is taken out of the bucket and yields a reward of Ω d j p ( T − j +1)
d j , i.e. the number of remaining balls of color d j times the weight Ω d j .
Finally, deg(¯ V d , p ( T/ 2) ) is the accumulated reward of the optimal strategy of emptying the bucket.
In lemma 3 we prove that there exists a value of Ω such that for every sequence of colors d , i.e. a row of ¯ V , the maximal reward over all possible initial states is solely attained at the state q ( T / 2 ) corresponding to d , i.e. q ( T / 2 ) r
= |{ j ∈ { T / 2 + 1 , . . .
, T }| d j = r }| .
Hence, deg(¯ V ii ) is indeed strictly larger than the degree of all other elements in the i ’th row.
Having proved that both U and V have rank N ≡ (cid:16)(cid:16)
¯ R T / 2 (cid:17)(cid:17) for all values of z but a ﬁnite set, we know there exists a value of z for which rank ( B ) =
N , and the theorem follows.
[Equation]
B.3. T ECHNICAL L EMMAS AND C LAIMS
In this section we prove a series of useful technical lemmas, that we have employed in our proof for the case of deep RACs, as described in section B.3.2.
We begin by quoting a claim regarding the prevalence of the maximal matrix rank for matrices whose entries are polynomial functions:
Claim 5.
Let M, N, K ∈ N , 1 ≤ r ≤ min { M, N } and a polynomial mapping A : R K → R M × N , i.e. for every i ∈
[ M ] and j ∈ [ N ] it holds that A ij : R K → R is a polynomial function.
If there exists a point x ∈ R K s.t. rank ( A ( x )) ≥ r , then the set { x ∈ R K : rank
( A ( x ))
< r } has zero measure (w.r.t.
the Lebesgue measure over R K ).
Proof.
See Sharir et al.
Claim 5 implies that it sufﬁces to show a speciﬁc assignment of the recurrent network weights for which the corresponding grid tensor matricization achieves a certain rank, in order to show this is a lower bound on its rank for all conﬁgurations of the network weights but a set of Lebesgue measure zero.
Essentially, this means that it is enough to provide a speciﬁc assignment that achieves the required bound in theorem 1 in order to prove the theorem.
Next, we show that for a matrix with entries that are polynomials in x , if a single contributor to the determinant has the highest degree of x , then the matrix is fully ranked for all values of x but a ﬁnite set:
Lemma 1.
Let A ∈ R N ×
N be a matrix whose entries are polynomials in x ∈ R .
In this case, its determinant may be written as det( A ) =
(cid:80) σ ∈ S N sgn ( σ ) p σ ( x ) , where S N is the symmetric group on N elements and p σ ( x ) are polynomials deﬁned by p σ ( x ) ≡ (cid:81)
Ni =
A iσ ( i ) ( x ) , ∀ σ ∈ S n .
Additionally, let there exist ¯ σ such that deg( p ¯ σ ( x ))
>
deg( p σ ( x )) ∀ σ (cid:54) =
¯ σ .
Then, for all values of x but a ﬁnite set, A is fully ranked.
Proof.
We show that in this case det( A ) , which is a polynomial in x by its deﬁnition, is not the zero polynomial.
Accordingly, det( A ) (cid:54) = 0 for all values of x but a ﬁnite set.
Denoting t ≡ deg( p ¯ σ ( x )) , since t > deg( p σ ( x )) ∀ σ (cid:54) =
¯ σ , a monomial of the form c · x t , c ∈ R \ { 0 } exists in p ¯ σ ( x ) and doesn’t exist in any p σ ( x ) , σ (cid:54)
=
¯ σ .
This implies that det( A ) is not the zero polynomial, since its leading term has a non-vanishing coefﬁcient sgn (¯ σ ) · c (cid:54) = 0 , and the lemma follows from the basic identity: det( A ) (cid:54) =
⇐⇒
A is fully ranked.
The above lemma assisted us in conﬁrming that the assignment provided for the recurrent network weights indeed achieves the required grid tensor matricization rank of (cid:16)(cid:16) ¯ RT/ 2 (cid:17)(cid:17) .
The following lemma, establishes a useful relation we refer to as the vector rearrangement
Lemma 2.
Let { v ( i ) }
Ni =1 be a set of N different vectors in R ¯ R such that ∀ i ∈
[ N ] , j ∈ [ ¯ R ] : v ( i ) j ≥ 0 .
Then, for all σ ∈ S N such that σ (cid:54) =
I N , where S N is the symmetric group on N , it holds that:
[Equation]
Proof.
We rely on theorem 368 in Hardy et al.
(1952), which implies that for a set of non-negative numbers { a (1) , . .
, a ( N ) } the following holds for all σ ∈ S N :
[Equation]
with equality obtained only for σ which upholds σ ( i )
=
j ⇐⇒ a ( i ) =
a ( j ) .
The above relation, referred to as the rearrangement inequality , holds separately for each component j ∈ [ ¯ R ] of the given vectors:
[Equation]
We now prove that for all σ ∈ S N such that σ (cid:54)
=
I N , ∃ ˆ j ∈ [ ¯ R ] for which the above inequality is hard, i.e. :
[Equation]
By contradiction, assume that ∃ ˆ σ (cid:54) =
I N for which ∀ j ∈
[ ¯ R ] :
[Equation]
From the conditions of achieving equality in the rearrangement inequality deﬁned in eq.
14, it holds that ∀ j ∈ [ ¯ R ] : v (ˆ σ ( i ))
j = v ( i ) j , trivially entailing: v (ˆ σ ( i ))
= v ( i ) .
Thus, ˆ σ (cid:54) =
I N would yield a contradiction to { v ( i ) } Ni
= being a set of N different vectors in R ¯ R .
Finally, the hard inequality of the lemma for σ (cid:54) =
I N is implied from eq. 15:
N
[Equation]
The vector rearrangement inequality in lemma 2, helped us ensure that our matrix of interest denoted ¯ U upholds the conditions of lemma 1 and is thus fully ranked.
Below, we show an identity that allowed us to make combinatoric sense of a convoluted expression:
Claim 6.
Let ¯ R and M be positive integers, let
Z ∈ R
¯
R
×
M be a matrix, and let A be a tensor with T modes, each of dimension M , deﬁned by A d 1 ,...,d T ≡ (cid:81) Tt = T / 2 +1 (cid:80) ¯ Rr =1 (cid:81) tj =1 Z rd j , where d 1 , . . .
, d T ∈ [ M ] .
Then, the following identity holds:
[Equation]
 
[Equation]

 , where states (cid:0) ¯ R, K (cid:1) ≡ { p ( K ) ∈
( N ∪ { 0 } ) ¯ R | (cid:80) ¯ Ri =1 p i = K } , and trajectory (cid:16) p ( K ) (cid:17) ≡ { ( p ( K − 1) , . .
, p (1) )
|∀ k ∈
[ K − 1] , ( p ( k ) ∈ states (cid:0) ¯ R, k (cid:1)
∧ ∀ r ∈
[ ¯ R ] , p ( k ) r ≤ p ( k +1) r ) } .
Proof.
We will prove the following more general identity by induction.
For any k ∈ [ T ] , deﬁne A ( k ) d 1 ,...,d T ≡ [Equation]
Tt = k
[Equation]
¯ Rr =1 (cid:81) tj =1 Z rd j , then the following identity holds:
[Equation]
[Table] [Table] The above identity coincides with our claim for k = T / 2 +
We begin with the base case of k = T , for which the set states (cid:0) ¯ R, 1 (cid:1) simply equals to the unit vectors of ( N ∪ { 0 } ) ¯ R , i.e. for each such p (1) there exists ¯ r ∈
[ ¯ R ] such that p (1) r = δ ¯ rr ≡
[Equation]
= r .
Thus, the following equalities hold:
[Equation]
By induction on k , we assume that the claim holds for A ( k +1) and prove it on A ( k ) .
First notice that we can rewrite our claim for k < T as:
[Equation]
where we simply moved the k ’th term Z p ( k ) r rd k in the right product expression to the left product.
We can also can rewrite A ( k ) as a recursive formula:
[Equation]
Then, employing our induction assumption for A ( k +1) , results in:
[Equation]

 (17)
To prove that the right hand side of eq.
17 is equal to our alternative form of our claim given by eq.
16, it is sufﬁcient to show a bijective mapping from the terms in the sum of eq.
17, each speciﬁed by a sequence (¯ r, p
( T − k ) , . .
. , p (1) ) , where ¯ r ∈ [ ¯ R ] , p ( T − k ) ∈ states (cid:0) ¯ R, T − k (cid:1) , and ( p ( T − k − 1) , . . .
, p (1) )
∈ trajectory (cid:16) p ( T − k ) (cid:17) , to the terms in the sum of eq.
16, each speciﬁed by a similar sequence ( p ( T − k +1) , p ( T − k ) , . .
, p (1) ) , where p ( T − k +1) ∈ states (cid:0) ¯ R, T − k + 1 (cid:1) and ( p ( T − k ) , . . .
, p (1) )
∈ trajectory (cid:16) p ( T − k +1) (cid:17) .
Let φ be a mapping such that (¯ r, p ( T − k ) , . .
, p (1) ) φ (cid:55)→ ( p ( T − k +1) , p ( T − k ) , . . .
, p (1) ) , where p ( T − k +1)
r ≡ p
( T − k ) r + δ ¯ rr .
φ is injective, because if φ (¯ r 1 , p ( T − k, 1) , . . .
, p (1 , 1) )
=
φ (¯ r 2 , p ( T − k, 2) , . .
, p (1 , 2) )
then for all j ∈ { 1 , . . .
, T − k + 1 } it holds
that p ( j, 1) =
p
j, 2) , and speciﬁcally for p ( T − k +1 , 1) =
p ( T − k +1 , 2) it entails that δ ¯ r 1 r =
δ ¯ r 2 r , and thus ¯ r 1 = ¯ r 2 .
φ is surjective, because for any sequence ( p ( T − k +1) , p ( T − k ) , . . .
, p (1) ) , for which it holds that ∀ j, p ( j ) ∈ ( N ∪ { 0 } )
¯ R , (cid:80) ¯
Rr =1 p ( j ) r = j , and ∀ r, p ( j ) r ≤ p ( j +1) r , then it must also holds that p ( T − k +1) r − p ( T − k ) r
= δ ¯ rr for some ¯ r , since (cid:80) ¯
Rr =1 ( p ( T − k +1) r − p ( T − k ) r ) =
( T − k + 1) − ( T − k ) =
and every summand is a non-negative integer.
Finally, lemma 3 assists us in ensuring that our matrix of interest denoted ¯ V upholds the conditions of lemmaand is thus fully ranked:
Lemma 3.
Let Ω ∈ R + be a positive real number.
For every p ( T / 2 ) ∈ states (cid:0) ¯ R, T / 2 (cid:1) (see deﬁnition in claim 6) and every d = ( d T / 2 +1 , . . .
, d T ) ∈ [ ¯ R ] T / 2 , where ∀ j, d j ≤ d j +1 , we deﬁne the following optimization problem:
[Equation]
where trajectory (cid:16) p ( T / 2 ) (cid:17) is deﬁned as in claim 6.
Then, there exists Ω such that for every such d the maximal value of f ( d , p ( T / 2 ) ) over all p ( T / 2 ) ∈ states (cid:0)
¯ R, T / 2 (cid:1) is strictly attained at ˆ p ( T / 2 ) deﬁned by ˆ p ( T / 2 ) r = |{ j ∈ { T / 2 + 1 , . . .
, T }| d j = r }| .
Proof.
We will prove the lemma by ﬁrst considering a simple strategy for choosing the trajectory for the case of f ( d , ˆ p ( T / 2 ) ) , achieving a certain reward ρ ∗ , and then showing that it is strictly larger than the rewards attained for all of the possible trajectories of any other p ( T / 2 ) (cid:54)
=
ˆ p ( T / 2 ) .
Our basic strategy is to always pick the ball of the lowest available color r .
More speciﬁcally, if ˆ p ( T / 2 )> 0 , then in the ﬁrst ˆ p ( T / 2 )time-steps we remove balls of the color 1 , in the process of which we accept a reward of Ω 1 ˆ p
( T / 2 )in the ﬁrst time-step, Ω 1 (ˆ p ( T / 2 )− 1) in the second time-step, and so on to a total reward of Ω 1 (cid:80) ˆ p ( T/ 2) 1 i
= i .
Then, we proceed to removing ˆ p ( T / 2 )balls of color 2 , and so forth.
This strategy will result in an accumulated reward of:
[Equation]
i. Next, we assume by contradiction that there exists p ( T / 2 ) (cid:54)
=
ˆ p ( T / 2 ) such that ρ ≡ f ( d , p ( T / 2 ) ) ≥ ρ ∗ .
We show by induction that this implies ∀ r, p ( T / 2 ) r ≥ ˆ p ( T / 2 ) r , which would result in a contradiction, since per our assumption p ( T / 2 ) (cid:54)
=
ˆ p ( T / 2 ) this means that there is r such that p ( T / 2 ) r > ˆ p ( T / 2 ) r , but since p ( T / 2 ) , ˆ p ( T / 2 ) ∈ states (cid:0) ¯ R, T / 2 (cid:1) then the following contradiction arises T /
=
(cid:80) ¯
Rr =1 p ( T / 2 ) r > (cid:80) ¯
Rr =1 ˆ p ( T / 2 ) r
= T / 2 .
More speciﬁcally, we show that our assumption entails that for all r starting with r = ¯ R and down to r = 1 , it holds that p ( T / 2 ) r ≥ ˆ p ( T / 2 ) r .
Before we begin proving the induction, we choose a value for Ω that upholds Ω > ( T / 2 ) 2 such that the following condition holds: for any r ∈ [ ¯ R ] , the corresponding weight for the color r , i.e. Ω r , is strictly greater than Ω r − 1 ( T / 2 ) 2 .
Thus, adding the reward of even a single ball of color r is always preferable over any possible amount of balls of color r (cid:48) < r .
We begin with the base case of r = ¯ R .
If ˆ p ( T / 2 ) =
the claim is trivially satisﬁed.
Otherwise, we assume by contradiction
that p ( T / 2 )
¯ R <
ˆ p ( T / 2 ) ¯ R .
If p ( T / 2 ) ¯ R = 0 , then the weight of the color ¯ R is not part of the total reward ρ , and per our choice of Ω it must hold that ρ < ρ ∗ since ρ ∗ does include a term of Ω ¯ R by deﬁnition.
Now, we examine the last state of the trajectory p (1) , where there is a single ball left in the bucket.
Per our choice of Ω , if p (1)¯ R = 0 , then once again ρ < ρ ∗ , implying that p (1)¯ R = 1 .
Following the same logic, for j ∈ [ p ( T / 2 ) ¯ R ] , it holds that p ( j )¯ R = j .
Thus the total contribution of the ¯ R ’th weight is at most:
[Equation]
This is because before spending all of the p ( T / 2 ) ¯ R balls of color ¯ R at the end, there are another (ˆ p ( T / 2 )
¯ R − p
( T / 2 ) ¯ R ) time-steps at which we add to the reward a value of p ( T / 2 ) ¯ R .
However, since eq. 18 is strictly less than the corresponding contribution of Ω ¯ R in ρ ∗ : Ω ¯ R (cid:80) ˆ p ( T/ 2)
i
=
i , then it follows that ρ < ρ ∗ , in contradiction to our assumption, which implies that to uphold the assumption the following must hold: p ( T / 2 ) ¯ R ≥ ˆ p ( T / 2 )
¯ R , proving the induction base.
Assuming our induction hypothesis holds for all r (cid:48)
>
r , we show it also holds for r .
Similar to our base case, if ˆ p ( T / 2 ) r = 0 then our claim is trivially satisﬁed, and likewise if p ( T / 2 ) r = 0 , hence it remains to show that the case of p ( T / 2 ) ¯ R < ˆ p ( T / 2 )
¯ R is not possible.
First, according to our hypothesis, ∀ r (cid:48) > r, p ( T / 2 ) r (cid:48) ≥ ˆ p ( T / 2 ) r (cid:48) , and per our choice of Ω , the contributions to the reward of all of the weights for r (cid:48) > r , are at most (cid:80) ¯ Rr (cid:48)
= [Equation]
+1 Ω
[Equation]
ˆ p
[Equation]
r
[Equation]
i = i , which is exactly equal to the corresponding contributions in ρ ∗ .
This means that per our choice of Ω it sufﬁces to show that the contributions originating in the color r are strictly less than the ones in ρ ∗ to prove our hypothesis.
In this optimal setting, the state of the bucket at time-step j = T / 2 − (cid:80) ¯ Rr (cid:48)
=
r +1 ˆ p ( T / 2 ) r (cid:48) must upholds p ( j ) r (cid:48)
=
ˆ p ( T / 2 ) r (cid:48) for r (cid:48) > r , and zero otherwise.
At this point, employing exactly the same logic as in our base case, the total contribution to the reward of the weight for the r ’th color is at most:
[Equation]
which is strictly less than the respective contribution in ρ ∗ .
C M ATRICIZATION D EFINITION
Suppose A ∈ R M ×···× M is a tensor of order T , and let ( I, J ) be a partition of [ T ] , i.e. I and J are disjoint subsets of [ T ] whose union gives [ T ] .
The matricization of A w.r.t.
the partition ( I, J ) , denoted (cid:74) A (cid:75)
I,J , is the M | I | -by- M | J | matrix holding the entries of A
such that A d 1 ...d T is placed in row index 1 + (cid:80) | I | t
=1 ( d i t − 1) M | I |− t and column index 1 + (cid:80) | J | t =1 ( d j t − 1) M | J |− t .