A
D EEP
R ECURRENT
N EURAL
N ETWORK
VIA
U NFOLDING R EWEIGHTED (cid:96) 1 - (cid:96) 1 M INIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep unfolding methods design deep neural networks as learned variations of optimization methods.
These networks have been shown to achieve faster convergence and higher accuracy than the original optimization methods.
In this line of research, this paper develops a novel deep recurrent neural network (coined reweighted-RNN) by unfolding a reweighted (cid:96) 1 - (cid:96) 1 minimization algorithm and applies it to the task of sequential signal reconstruction.
To the best of our knowledge, this is the ﬁrst deep unfolding method that explores reweighted minimization.
Due to the underlying reweighted minimization model, our RNN has a different soft-thresholding function (alias, different activation function) for each hidden unit in each layer.
Furthermore, it has higher network expressivity than existing deep unfolding RNN models due to the over-parameterizing weights.
Moreover, we establish theoretical generalization error bounds for the proposed reweighted-RNN model by means of Rademacher complexity.
The bounds reveal that the parameterization of the proposed reweighted-RNN ensures good generalization.
We apply the proposed reweighted-RNN to the problem of video-frame reconstruction from low-dimensional measurements, that is, sequential frame reconstruction.
The experimental results on the moving MNIST dataset demonstrate that the proposed deep reweighted-RNN signiﬁcantly outperforms existing RNN models.
I NTRODUCTION
The problem of reconstructing sequential signals from low-dimensional measurements across time is of great importance for a number of applications such as time-series data analysis, future-frame prediction, and compressive video sensing.
Speciﬁcally, we consider the problem of reconstructing a sequence of signals s t ∈ R n 0 , t = 1 , 2 , . .
, T , from low-dimensional measurements x t =
As t , where A ∈ R n × n 0 ( n (cid:28) n 0 ) is a sensing matrix.
We assume that s t has a sparse representation h t ∈ R h in a dictionary D ∈ R n 0 × h , that is, s t =
Dh t .
At each time step t , the signal s t can be independently reconstructed using the measurements x t by solving (Donoho, 2006):
[Equation]
where (cid:107) · (cid:107) p is the (cid:96) p -norm
and λ is a regularization parameter.
The iterative shrinkage-thresholding algorithm (ISTA) (Daubechies et al., 2004) solves (1) by iterating over h ( l ) t
[Equation]
1 c D T A T ( ADh
[Equation]
l
[Equation]
) t − x t )) , where l is the iteration counter, φ γ ( u ) =
sign( u )[0 , | u | − γ ]
+ is the soft-thresholding operator, γ = λc , and c is an upper bound on the Lipschitz constant of the gradient of 12 (cid:107) x t − ADh t (cid:107) 22 .
Under the assumption that sequential signal instances are correlated, we consider the following sequential signal reconstruction problem:
[Equation]
where λ 1 , λ 2 > 0 are regularization parameters and R ( h t , h t − 1 ) is an added regularization term that expresses the similarity of the representations h t and h t − 1 of two consecutive signals.
Wisdom et al.
(2017) proposed an RNN design (coined Sista-RNN) by unfolding the sequential version of
ISTA.
That study assumed that two consecutive signals are close in the (cid:96) 2 -norm sense, formally,
R ( h t , h t − 1 ) =
12 (cid:107) Dh t − FDh t − 1 (cid:107) 22 , where F ∈ R n 0 × n 0 is a correlation matrix between s t and s t − 1 .
More recently, the study by Le et al.
(2019) designed the (cid:96) 1 - (cid:96) 1 -RNN, which stems from unfolding an algorithm that solves the (cid:96) 1 - (cid:96) 1 minimization problem (Mota et al., 2017; 2016).
This is a version of Problem (2) with R ( h t , h t − 1 ) =
(cid:107) h t − Gh t − 1 (cid:107) 1 , where G ∈ R h × h is an afﬁne transformation that promotes the correlation between h t and h t − 1 .
Both studies (Wisdom et al., 2017; Le et al., 2019) have shown that carefully-designed deep RNN models outperform the generic
RNN model and ISTA (Daubechies et al., 2004) in the task of sequential frame reconstruction.
Deep neural networks (DNN) have achieved state-of-the-art performance in solving (1) for individual signals, both in terms of accuracy and inference speed (Mousavi et al., 2015).
However, these models are often criticized for their lack of interpretability and theoretical guarantees (Lucas et al., 2018).
Motivated by this, several studies focus on designing DNNs that incorporate domain knowledge, namely, signal priors.
These include deep unfolding methods which design neural networks to learn approximations of iterative optimization algorithms. Examples of this approach are LISTA (Gregor & LeCun, 2010) and its variants, including ADMM-Net (Sun et al., 2016), AMP (Borgerding et al., 2017), and an unfolded version of the iterative hard thresholding algorithm (Xin et al., 2016). LISTA (Gregor & LeCun, 2010) unrolls the iterations of ISTA into a feed-forward neural network with weights, where each layer implements an iteration: h ( l ) t = φ γ ( l ) ( W ( l ) h ( l − 1) t + U ( l ) x t ) , with
W ( l ) =
I − 1 c D T A T AD , U ( l ) = 1 c D T A T , and γ ( l ) being learned from data.
It has been shown (Gregor & LeCun, 2010; Sprechmann et al., 2015) that a d -layer LISTA network with trainable parameters Θ = { W ( l ) , U ( l ) , γ ( l ) } dl =1 achieves the same performance as the original ISTA but with much fewer iterations (i.e., number of layers).
Recent studies (Chen et al., 2018;
Liu et al., 2019) have found that exploiting dependencies between W ( l ) and U ( l ) leads to reducing the number of trainable parameters while retaining the performance of LISTA.
These works provided theoretical insights to the convergence conditions of LISTA.
However, the problem of designing deep unfolding methods for dealing with sequential signals is signiﬁcantly less explored.
In this work, we will consider a deep RNN for solving Problem (2) that outputs a sequence, ˆ s 1 , . . .
, ˆ s T from an input measurement sequence, x 1 , . . .
, x T , as following:
[Equation]
It has been shown that reweighted algorithms—such as the reweighted (cid:96) 1 minimization method by Candès et al.
(2008) and the reweighted (cid:96) 1 - (cid:96) 1 minimization by Luong et al.
(2018)—outperform their non-reweighted counterparts.
Driven by this observation, this paper proposes a novel deep RNN architecture by unfolding a reweighted- (cid:96) 1 - (cid:96) 1 minimization algorithm.
Due to the reweighting, our network has higher expressivity than existing RNN models leading to better data representations, especially when depth increases.
This is in line with recent studies (He et al., 2016; Cortes et al.
Huang et al., 2017), which have shown that better performance can be achieved by highly overparameterized networks, i.e., networks with far more parameters than the number of training samples.
While the most recent studies (related over-parameterized DNNs) consider fully-connected networks applied on classiﬁcation problems (Neyshabur et al., 2019), our approach focuses on deep-unfolding architectures and opts to understand how the networks learn a low-complexity representation for sequential signal reconstruction, which is a regression problem across time.
Furthermore, while there have been efforts to build deep RNNs (Pascanu et al., 2014; Li et al., 2018; Luo et al., 2017; Wisdom et al., 2017), examining the generalization property of such deep RNN models on unseen sequential data still remains elusive.
In this work, we derive the generalization error bound of the proposed design and further compare it with existing RNN bounds (Zhang et al., 2018; Kusupati et al., 2018).
Contributions .
The contributions of this work are as follows:
• We propose a principled deep RNN model for sequential signal reconstruction by unfolding a reweighted (cid:96) 1 - (cid:96) 1 minimization method.
Our reweighted-RNN model employs different soft-thresholding functions that are adaptively learned per hidden unit.
Furthermore, the proposed model is over-parameterized, has high expressivity and can be efﬁciently stacked.
• We derive the generalization error bound of the proposed model (and deep RNNs) by measuring Rademacher complexity and show that the over-parameterization of our RNN ensures good generalization.
To best of our knowledge, this is the ﬁrst generalization error bound for deep RNNs; moreover, our bound is tighter than existing bounds derived for shallow RNNs (Zhang et al., 2018; Kusupati et al., 2018).
• We provide experiments in the task of reconstructing video sequences from low-dimensional measurements.
We show signiﬁcant gains when using our model compared to several state-of-the-art RNNs (including unfolding architectures), especially when the depth of RNNs increases.
A DEEP RNN VIA UNFOLDING REWEIGHTED - (cid:96) 1 - (cid:96) 1 MINIMIZATION
In this section, we describe a reweighted (cid:96) 1 - (cid:96) 1 minimization problem for sequential signal reconstruction and propose an iterative algorithm based on the proximal method.
We then design a deep RNN architecture by unfolding this algorithm.
The proposed reweighted (cid:96) 1 - (cid:96) 1 minimization .
We introduce the following problem:
[Equation]
(4) where “ ◦ ” denotes element-wise multiplication, g ∈ R
h is a vector of positive weights, Z ∈ R h × h is a reweighting matrix, and G ∈ R h × h is an afﬁne transformation that promotes the correlation between h t − 1 and h t .
Intuitively, Z is adopted to transform h t to Zh t ∈ R h , producing a reweighted version of it.
Thereafter, g aims to reweight each transformed component of Zh t and Zh t − Gh t −in the (cid:96) 1 -norm regularization terms.
Because of applying reweighting (Candès et al., 2008), the solution of Problem (4) is a more accurate sparse representation compared to the solution of the (cid:96) 1 - (cid:96)minimization problem in Le et al.
(where Z = I and g
= I ).
Furthermore, the use of the reweighting matrix Z to transform h t to Zh t differentiates Problem (4) from the reweighted (cid:96) 1 - (cid:96)minimization problem in Luong et al.
where
Z = I .
The objective function in (4) consists of the differentiable ﬁdelity term f
( Zh t ) = 12 (cid:107) x t − ADZh t (cid:107)and the non-smooth term g ( Zh t ) =
λ 1 (cid:107) g ◦ Zh t (cid:107) 1 + λ 2 (cid:107) g ◦
( Zh t − Gh t − 1 ) (cid:107) 1 .
We use a proximal gradient method (Beck & Teboulle, 2009) to solve (4): At iteration l , we ﬁrst update h ( l − 1) t —after being multiplied by Z l —with a gradient descent step on the ﬁdelity term as u = Z l h ( l − 1) t − 1 c Z l ∇ f ( h ( l − 1) t ) , where ∇ f ( h ( l − 1) t ) = D T
[Equation]
T ( ADh ( l − 1)
[Equation]
− x t ) .
Then, h ( l ) t is updated as
[Equation]
where the proximal operator Φ λ 1 c g l , λ 2 c g l , Gh t − 1 ( u ) is deﬁned as
[Equation]
(6) with (cid:126)(cid:126)(cid:126) =
Gh t − 1 .
Since the minimization problem is separable, we can minimize (6) independently for each of the elements g l , (cid:126) , u of the corresponding g l , (cid:126)(cid:126)(cid:126) , u vectors.
After solving (6), we obtain Φ λ 1 c g l , λ 2 c g l , (cid:126) ( u ) [for solving (6), we refer to Proposition B.1 in Appendix B].
For (cid:126) ≥ 0 :
[Equation]
l , (cid:126)
[Equation]
and for (cid:126) < 0 :
[Equation]
ℏ − −  1     2    ℏ − +  1     2    (  )  , , ℏ  1     2     ℏ +  1     2    − +  1     2   ℏ + +  1     2    − −  1     2    (  )  , , ℏ  1     2     ℏ + −  1     2    ℏ −  1     2   (a) (cid:126) ≥ 0 .
ℏ − −  1     2    ℏ − +  1     2    (  )  , , ℏ  1     2     ℏ +  1     2    − +  1     2   ℏ + +  1     2    − −  1     2    (  )  , , ℏ  1     2     ℏ + −  1     2    ℏ −  1     2
[Figure] 1 Input Measurements x . , x
, measurement matrix A , dictionary D , afﬁne transform G , initial h h
, reweighting matrices Z 1 ,
, Z and vectors
, λ
, λ
2 Output: do h (0) t =
Gh ( d ) t − for l = 1 to d do u =
[ Z l − 1 c Z l D T A T AD ] h ( l − 1) t + 1 c
Z l D T A T x t h ( l ) t =
Φ λ 1 c g l , λ 2 c g l , Gh ( d ) t − 1 ( u ) end 9 end return
Fig. 1 depicts the proximal operators for (cid:126) ≥ 0 and (cid:126) < 0 .
Observe that different values of g l lead to different shapes of the proximal functions Φ λ 1 c g l , λ 2 c g l , (cid:126) ( u ) for each element u of u .
Our iterative algorithm is given in Algorithm 1.
We reconstruct a sequence h 1 , . .
, h T from a sequence of measurements x 1 , . . .
, x T .
For each time step t , Step 6 applies a gradient descent update for f ( Zh t − 1 ) and Step 7 applies the proximal operator Φ λ 1 c g l , λ 2 c g l , Gh ( d ) t − 1 element-wise to the result.
Let us compare the proposed method against the algorithm in Le et al.
[-, measurement matrix A , dictionary D , afﬁne transform G , initial h ( d ) 
-]
(2019)—which resulted in the (cid:96) 1 - (cid:96) 1 -RNN—that solves the (cid:96) 1 - (cid:96) 1 minimization in Mota et al.
where Z l = I and g l = I ).
In that algorithm, the update terms in Step 6, namely I − 1 c D T A T AD and 1 c D T A T , and the proximal operator in Step 7 are the same for all iterations of l .
In contrast, Algorithm 1 uses a different Z l matrix per iteration to reparameterize the update terms (Step 6) and, through updating g l , it applies a different proximal operator to each element u (in Step 7) per iteration l .
The proposed reweighted-RNN .
We now describe the proposed architecture for sequential signal recovery, designed by unrolling the steps of Algorithm 1 across the iterations l = 1 , . . .
, d (yielding the hidden layers) and time steps t = 1 , . . .
, T .
Speciﬁcally, the l -th hidden layer is given by
[Equation]
and the reconstructed signal at time step t is given by ˆ s t
= Dh ( d ) t ; where U l , W l , V are deﬁned as
[Equation]
The activation function is the proximal operator Φ
[Equation]
c g [Equation] [Equation] g
[Equation]
[Equation] ( u ) with learnable parameters λ 1 , λ 2 , c , g l (see Fig.
1 for the shapes of the activation functions).
{+3/ 𝐡 ( 𝑑 ) 𝑡 − 𝐡 (1) 𝑡 𝐡 (2) 𝑡 𝐖 2 𝐠𝐆 ... 𝐬 𝑡 𝐀 𝐱 𝑡 𝐔𝐔𝐔 𝑑 𝐬̂ 𝑡 𝐆 𝐆 𝐖𝐖 𝑑 𝐃 𝐠𝐠 𝑑 𝐡 ( 𝑑 ) 𝑡 𝐡 ( 𝑑 ) 𝑡 − 𝐡 (1) 𝑡 𝐡 (2) 𝑡 𝐖𝐖 2 𝐆 ... 𝐬 𝑡 𝐀 𝐱 𝑡 𝐔𝐔𝐔𝐬̂ 𝑡 𝐆 𝐆 𝐖𝐖𝐃 𝐡 ( 𝑑 ) 𝑡 𝐡 (1) 𝑡 − 𝐡 (1) 𝑡 𝐡 (2) 𝑡 𝐕𝐖... 𝐱 𝑡 𝐔𝐖𝐖 𝑑 𝐃 𝐡 ( 𝑑 ) 𝑡 𝐡 ( 𝑑 ) 𝑡 𝐬 𝑡 𝐬̂ 𝑡 𝐀 𝐡 (2) 𝑡 −
+}
𝐕𝐡 ( 𝑑 ) 𝑡 −1 𝐕 𝑑 𝐖𝐡𝐡 (1) 𝐡 (2)1 𝐖 2 𝐠𝐆 ... 𝐬 1 𝐀 𝐱 1 𝐔𝐔𝐔 𝑑 𝐆 𝐆 𝐖𝐖 𝑑 𝐃 𝐠𝐠 𝑑 𝐬̂𝐡 ( 𝑑 ) 𝐡 (1) 𝐡 (2)2 𝐖 2 𝐠𝐆 ... 𝐱 2 𝐔𝐔𝐔 𝑑 𝐆
𝐆
𝐖𝐖 𝑑 𝐠𝐠 𝑑 𝐬𝐬̂ 2 𝐡 ( 𝑑 ) 𝐡 ( 𝑑 ) 𝐡 ( 𝑑 ) 𝑇 − 𝐡 (1) 𝑇 𝐡 (2) 𝑇 𝐖 2 𝐠𝐆 ... 𝐱 𝑇 𝐔𝐔𝐔
𝑑 𝐆 𝐆 𝐖𝐖 𝑑 𝐠𝐠 𝑑 𝐬 𝑇 𝐬̂ 𝑇 𝐡 ( 𝑑 ) 𝑇 ... .......... 𝐀 𝐀 𝐃 𝐃 𝐖𝐖𝐖
5/14/ deepRNN.drawio 1/ x 1 s 1 A
U
W 2 h 1 ( 2 ) g 2 h 1 ( 1 )
U
W d h 1 ( d ) U d
D s 1 W 1 h
G x 2 s 2 A
U
W 2 h 2 ( 2 ) h 2 ( 1 )
U
W d h 2 ( d ) U d
D s 2 W 1 G W 1 G x T A
U
W 2 h T ( 2 ) h T ( 1 )
U
W d h T ( d ) U d
D s T W 1 G ...
s T ...
^ ^ ^ gg d g 2 gg
d
g 2 gg d ˤ , , G ̌1c ̌2c h (d)t− h [-
-](d)t− h (1)t h (2)t
W
G ... s x t
U
Uŝ t
G G
W
D h (d)t
[Table]
5/16/ deepRNN.drawio 1/ x 1 s 1 A
U
W 2 h 1 ( 2 ) g 2 h 1 ( 1 )
U
W d h 1 ( d ) U d
D s 1 W 1 h
G x 2 s 2 A
U
W 2 h 2 ( 2 ) h 2 ( 1 )
U
W d h 2 ( d ) U d
D s 2 W 1 G W 1 G x T A
U
W 2 h T ( 2 ) h T ( 1 )
U
W d h T ( d ) U d
D s T W 1 G ...
s T ...
^ ^ ^ gg d g 2 gg
d
g 2 gg d h (d)t− h (1)t h (2)t
W
W 2 g
G ... s t
A x t U
U
U d ŝ t
G
G W
W d
D gg d h (d)t h (d)t− h (1)t h (2)t
W
W 2 G ... s t
A x t U
U
Uŝ t
G
G W
W
D h (d)t h (1)t− h (1)t h (2)t
V
W... x t U
W
W d
D h (d)t h (d)t s t ŝ t
A h (2)t−1 Vh (d)t−1 V d (c) Stacked RNN.
Fig. 2(a) depicts the architecture of the proposed reweighted-RNN.
Input vectors s t , t = 1 , . .
, T are compressed by a linear measurement layer A , resulting in compressive measurements x t .
The reconstructed vectors ˆ s t , t = 1 , . . , T , are obtained by multiplying linearly the hidden representation h ( d ) t with the dictionary D . We train our network in an end-to-end fashion. During training, we minimize the loss function L ( Θ ) =
[Equation]
[Equation] using stochastic gradient descent on minibatches, where the trainable parameters are Θ = { A , D , G , h 0 , Z 1 , . .
, Z d , g 1 , . .
. , g d , c, λ 1 , λ 2 } .
We now compare the proposed reweighted-RNN [Fig. 2(a)] against the recent (cid:96) 1 - (cid:96) 1 -RNN (Le et al., 2019)
[Fig. 2(b)].
The l -th hidden layer in (cid:96) 1 - (cid:96) 1 -RNN is given by
[Equation]
The proposed model has the following advantages over (cid:96) 1 - (cid:96) 1 -RNN.
Firstly, (cid:96) 1 - (cid:96) 1 -RNN uses the proximal operator Φ λ 1 c , λ 2 c , (cid:126)(cid:126)(cid:126) ( u ) as activation function, whose learnable parameters λ 1 , λ 2 are ﬁxed across the network.
Conversely, the corresponding parameters λ 1 c g l and λ 2 c g l [see (7), (8), and Fig. 1] in our proximal operator, Φ λ [Equation]
g [Equation] [Equation] ( u ) , are learned for each hidden layer due to the reweighting vector g l ; hence, the proposed model has a different activation function for each unit per layer.
The second difference comes from the set of parameters { W l , U l } in (13) and (9).
The (cid:96) 1 - (cid:96) 1 -RNN model uses the same { W 2 , U 1 } for the second and higher layers.
In contrast, our reweighted-RNN has different sets of { W l , U l } per hidden layer due to the reweighting matrix Z l .
These two aspects [which are schematically highlighted in blue fonts in Fig. 2(a)] can lead to an increase in the learning capability of the proposed reweighted-RNN, especially when the depth of the model increases.
In comparison to a generic stacked RNN (Pascanu et al., 2014) [Fig. 2(c)], reweighted-RNN promotes the inherent data structure, that is, each vector s t has a sparse representation h t and consecutive h t ’s are correlated.
This design characteristic of the reweighted-RNN leads to residual connections which reduce the risk of vanishing gradients during training [the same idea has been shown in several works (He et al., 2016; Huang et al., 2017) in deep neural network literature].
Furthermore, in (10) and (12), we see a weight coupling of W l and U l (due to the shared components of A , D and Z ).
This coupling satisﬁes the necessary condition of the convergence in Chen et al.
(Theorem 1).
Using Theorem 2 in Chen et al.
(2018), it can be shown that reweighted-RNN, in theory, needs a smaller number of iterations (i.e., d in Algorithm 1) to reach convergence, compared to ISTA (Daubechies et al., 2004) and FISTA (Beck & Teboulle, 2009).
G ENERALIZATION ERROR BOUND
While increasing the network expressivity, the over-parameterization of reweighted-RNN raises the question of whether our network ensures good generalization.
In this section, we derive and analyze the generalization properties of the proposed reweighted-RNN model in comparison to state-ofthe-art RNN architectures.
We provide bounds on the Rademacher complexity (Shalev-Shwartz &
Ben-David, 2014) for functional classes of the considered deep RNNs, which are used to derive generalization error bounds for evaluating their generalization properties (we refer to Appendix C. for deﬁnitions of the Rademacher complexity and the generalization error bound).
Preliminaries: We consider a deep RNN as a d -layer network f ( d ) W , U ∈ F d,T : R h × R n (cid:55)→ R h with weight parameters
W
=
( W 1 , ..., W d ) and U =
( U 1 , ..., U d ) , where W l ∈ R h × h , U l ∈ R h × n .
As in Bartlett et al.
(2017); Golowich et al.
; Neyshabur et al.
(2019; 2015), we derive generalization error bounds by controlling norms of the weight matrices.
Let (cid:107) W l (cid:107) p,q = ( (cid:80)
j
( (cid:107) w l,j (cid:107) p ) q ) 1 /q deﬁne the (cid:96) p,q -norm, p, q ≥ 1 , of the weight-matrix W l , where w l,j ∈ R h is the j th row of W l .
Since we focus on deep networks with soft-thresholding-based activation units— designed by unfolding algorithms for (cid:96) 1 -norm minimization—we derive the network complexities under bounding per-unit (cid:96) 1 regularization, i.e., (cid:107) w l,j (cid:107) 1 .
We also denote (cid:107) W l (cid:107) 1 , ∞ =
max j (cid:107) w l,j (cid:107)as the maximum of the (cid:96) 1 -norms of the matrix’s rows.
We assume that the (cid:96) 1 -norm of the weights of each neuron is bounded as (cid:107) W l (cid:107) 1 , ∞ =
max j (cid:107) w l,j (cid:107) 1 ≤ α l ; similarly, (cid:107) U l (cid:107) 1 , ∞ =
max j (cid:107) u l,j (cid:107) 1 ≤ β l , where u l,j is the j th row of the matrix U l .
As shown in (9), we can write the reweighted-RNN model recursively as h (1) t = f (1) W , U ( h ( d ) t − 1 , x t ) =
Φ ( W 1 h ( d ) t − 1 + U 1 x t ) and h ( l ) t = f ( l ) W , U ( h ( d ) t − 1 , x t ) =
Φ ( W l f ( l − 1) W , U ( h ( d ) t − 1 , x t )
+ U l x t ) , where Φ ( · ) is an activation function.
For convenience, we denote the input layer as f (0) W , U
= h ( d ) t − 1 ; namely, at t = 1 , we have h ( l ) 0 ≡ h 0 .
We denote the true and training loss by L D ( f ) and L S ( f ) , respectively, where S is the training set (of size m ) drawn i.i.d.
from the distribution D .
The generalization error is L D ( f ) − L S ( f ) , with f a function from the functional class F d,T .
At time step t , we deﬁne X t ∈ R n × m as a matrix composed of m columns from the input vectors { x t,i } mi =1 .
We also deﬁne (cid:107) X t (cid:107)
[Equation]
as the maximum of the (cid:96) 2 -norms of the rows of matrix X t , and (cid:107) h 0 (cid:107) ∞ =
max j | h 0 ,j | .
Generalization error bound .
We ﬁrst derive the generalization error bound for the proposed reweighted-RNN (with T time steps) based on Rademacher complexity (see Theorem 26.5 in ShalevShwartz & Ben-David (2014) and Theorem C.1 in the Appendix).
Theorem 3.1 (Generalization error bound) .
Let F d,T : R h × R n (cid:55)→ R h denote the functional class of reweighted-RNN with T time steps and d layers, where (cid:107) W l (cid:107) 1 , ∞ ≤ α l , (cid:107) U l (cid:107) 1 , ∞ ≤ β l , and 1 ≤ l ≤ d .
Assume that the input data (cid:107) X t (cid:107) 2 , ∞ ≤ √ mB x , the initial hidden state is h 0 , and the loss function is 1-Lipschitz and bounded by η .
Then, for f ∈ F d,T and any δ > 0 , with probability at least 1 − δ over a training set S of size m drawn i.i.d. from the distribution D ,
[Equation]
D
[Equation]
(14) where
[Equation]
with Λ l deﬁned as follows: Λ l = d (cid:81) k
= l +1 α k with 0 ≤ l ≤ d − 1 and Λ d = 1 .
Proof.
The proof is given in Appendix D.
The generalization error in (14) is bounded by the Rademacher complexity, which depends on the training set S .
If the Rademacher complexity is small, the network can be learned with a small generalization error.
The bound in (15) is in the order of the square root of the network depth d multiplied by the number of time steps T .
The bound depends on the logarithm of the number of measurements n and the number of hidden units h .
It is worth mentioning that the second square root in (15) only depends on the norm constraints and the input training data, and it is independent of the network depth d and the number of time steps T under the appropriate norm constraints.
To compare our model with (cid:96) 1 - (cid:96) 1 -RNN (Le et al., 2019) and Sista-RNN (Wisdom et al., 2017), we derive bounds on their Rademacher complexities for a time step t .
The deﬁnitions of a functional class
F d,t for the t th time step of reweighted-RNN, (cid:96) 1 - (cid:96) 1 -RNN, and Sista-RNN are given in Appendix
C.2.
Let H t − 1 ∈ R h × m denote a matrix with columns the vectors of the previous hidden state { h t − 1 ,i } mi =1 , and (cid:107) H t − 1 (cid:107) 2 , ∞ = (cid:114) max k ∈{ 1 ,...,h } (cid:80) mi =1 h 2 t − 1 ,i,k ≤
[Equation]
Corollary 3.1.1.
The empirical Rademacher complexity of F d,t for reweighted-RNN is bounded as
[Equation]
with m the number of training samples and Λ l given by Λ d = 1 , Λ l =
[Equation]
l +1 α k [Equation] 0 ≤ l ≤ d − 1 .
Proof.
The proof is a special case of Theorem 3.1 for time step t .
Following the proof of Theorem 3.1, we can obtain the Rademacher complexities for (cid:96) 1 - (cid:96) 1 -RNN and
Sista-RNN:
Corollary 3.1.2.
The empirical Rademacher complexity of F d,t for (cid:96) 1 - (cid:96) 1 -RNN is bounded as: [Equation] d
[Equation]
Corollary 3.1.3.
The empirical Rademacher complexity of F d,t for Sista-RNN is bounded as:
[Equation]
By contrasting (16) with (17) and (18), we see that the complexities of (cid:96) 1 - (cid:96) 1 -RNN and Sista-RNN have a polynomial dependence on α 1 , β 1 and α 2 , β 2 (the norms of ﬁrst two layers), whereas the complexity of reweighted-RNN has a polynomial dependence on α 1 , . . .
, α d and β 1 , . .
, β d (the norms of all layers).
This over-parameterization offers a ﬂexible way to control the generalization error of reweighted-RNN.
We derive empirical generalization errors in Fig.
6 in Appendix A demonstrating that increasing the depth of reweighted-RNN still ensures the low generalization error.
E XPERIMENTAL RESULTS
We assess the proposed RNN model in the task of video-frame reconstruction from compressive measurements.
The performance is measured using the peak signal-to-noise ratio (PSNR) between the reconstructed ˆ s t and the original frame s t .
We use the moving MNIST dataset (Srivastava et al., 2015), which contains 10K video sequences of equal length (20 frames per sequence).
Similar to the setup in Le et al.
(2019), the dataset is split into training, validation, and test sets of 8K, 1K, and 1K sequences, respectively.
In order to reduce the training time and memory requirements, we downscale the frames from 64 × 64 to 16 × 16 pixels using bilinear decimation.
After vectorizing, we obtain sequences of s 1 , . . .
, s T ∈ R 256 .
Per sequence, we obtain measurements x 1 , . . .
, x T ∈ R n using a trainable linear sensing matrix A ∈ R n × n 0 , with T = 20 , n 0 = 256 and n < n 0 .
We compare the reconstruction performance of the proposed reweighted-RNN model against deepunfolding RNN models, namely, (cid:96) 1 - (cid:96) 1 -RNN (Le et al., 2019), Sista-RNN (Wisdom et al., 2017), and stacked-RNN models, that is, sRNN (Elman, 1990), LSTM (Hochreiter & Schmidhuber, 1997),
GRU (Cho et al., 2014), FastRNN (Kusupati et al., 2018) 1 , IndRNN (Li et al., 2018) and SpectralRNN (Zhang et al., 2018).
For the vanilla RNN, LSTM and GRU, the native Pytorch cell implementations were used.
The unfolding-based methods were implemented in Pytorch 2 , with
Sista-RNN and (cid:96) 1 - (cid:96) 1 -RNN tested by reproducing the experiments in Wisdom et al.
; Le et al.
For FastRNN, IndRNN, and SpectralRNN cells, we use the publically available Tensorﬂow implementations.
While Sista-RNN, (cid:96) 1 - (cid:96) 1 -RNN and reweighted-RNN have their own layer-stacking schemes derived from unfolding minimization algorithms, we use the stacking rule in Pascanu et al.
(2013) [see Fig 2(c)] to build deep networks for other RNN architectures.
Our default settings are: a compressed sensing (CS) rate of n/n 0 = 0 .
2 , d = 3 hidden layerswith h = 2 10 hidden units per layer.
In each set of experiments, we vary each of these hyperparameters while keeping the other two ﬁxed.
For the unfolding methods, the overcomplete dictionary
D ∈ R n 0 × h is initialized with the discrete cosine transform (DCT) with varying dictionary sizes of h = { 2 7 , 2 8 , 2 9 , 2 10 , 2 11 , 2 12 } (corresponding to a number of hidden neurons in the other methods).
For initializing λ 1 , λ 2 [see (2), (4)], we perform a random search in the range of [ 10 − 5 , 3.0] in the validation set.
To avoid the problem of exploding gradients, we clip the gradients during backpropagation such that the (cid:96) 2 -norms are less than or equal to 0.25.
We do not apply weight decay regularization as we found it often leads to worse performance, especially since gradient clipping is already used for training stability.
We train the networks for 200 epochs using the Adam optimizer with an initial learning rate of 0.0003, and a batch size of 32.
During training, if the validation loss does not decrease for 5 epochs, we reduce the learning rate to 0.3 of its current value.
Table 1 summarizes the reconstruction results for different CS rates n/n 0 .
The reweighted-RNN model systematically outperforms the other models, often by a large margin.
Table 2 shows similar improvements for various dimensions of hidden units.
Table 3 shows that IndRNN delivers higher reconstruction performance than our model when a small number of hidden layers ( d = 1 , 2 ) is used.
Moreover, when the depth increases, reweighted-RNN surpasses all other models.
Our network also has fewer trainable parameters compared to the popular variants of RNN.
At the default settings, reweighted-RNN, the stacked vanilla RNN, the stacked LSTM, and the stacked GRU have 4.47M, 5.58M, 21.48M, and 16.18M parameters, respectively.
[Table] [Table]
[Table] [Table]
C ONCLUSIONS
We designed a novel deep RNN by unfolding an algorithm that solves a reweighted (cid:96) 1 - (cid:96) 1 minimization problem.
Our model has high network expressivity due to per-unit learnable activation functions and over-parameterized weights .
We also established the generalization error bound for the proposed model via Rademacher complexity.
We showed that reweighted-RNN has good generalization properties and its error bound is tighter than existing RNNs in function of the number of time steps.
Experimentation on the task of sequential video-frame reconstruction shows that our model ( i ) outperforms various state-of-the-art RNNs in terms of accuracy and ( ii ) is capable of stacking many hidden layers resulting in a better learning capability than the existing unfolding methods.
A S UPPLEMENTARY EXPERIMENTS
In our experiments
[Figure]
we use the publically available Tensorﬂow implementations for FastRNN 4 ,
IndRNN 5 , and SpectralRNN 6 cells.
While Sista-RNN, (cid:96) 1 - (cid:96) 1 -RNN and reweighted-RNN have their own layer-stacking schemes derived from unfolding minimization algorithms
[Figure]
we use the stacking rule in Pascanu et al.
[see Fig 2(c)] to build deep networks for other RNN models.
Figure 3 shows the learning curves of all methods under the default setting.
It can be seen that reweighted-RNN achieves the lowest mean square error on both the training and validation sets.
It can also be observed that the unfolding methods converge faster than the stacked RNNs, with the proposed reweighted-RNN being the fastest.
More experimental results for the proposed reweighted-RNN are provided to illustrate the learning curves
[Figure]
which measure the average mean square error vs. the training epochs between the original frames and their reconstructed counterparts, with different CS rates [Fig. 4], different network depths d [Fig. 6]
[Figure]
and different network widths h
[Fig. 5]
[Figure] Since we use different frameworks to implement the RNNs used in our benchmarks, we do not report and compare the computational time for training of the models. Speciﬁcally, we rely on the Tensorﬂow implementations from the authors of Independent-RNN, Fast-RNN and Spectral RNN, while the rest is written in Pytorch.
Furthermore, even among Pytorch models, the vanilla RNN, LSTM, and GRU cells are written in CuDNN (default Pytorch implementations), so that they are signiﬁcantly faster in training than the others. This does not mean that these networks have better runtime complexities, but rather more efﬁcient implementations. However, an important comparison could be made between (cid:96) 1 - (cid:96) 1 -RNN Le et al.
(as the baseline method) and Reweighted-RNN due to their similarities in implementations. At the default settings, it takes 3,521 seconds and 2, seconds to train Reweighted-RNN and (cid:96) 1 - (cid:96) [Figure] [Figure]
[Figure]
B T HE P ROXIMAL O PERATOR FOR P ROBLEM (4)
Proposition B.1.
The proximal operator Φ λ 1 c g , λ 2 c g , (cid:126) ( u ) in (6) for the reweighted (cid:96) 1 - (cid:96) 1 minimization problem (4), for which g ( v ) =
λ 1 g | v | + λ 2 g | v − (cid:126) | , is given by
[Equation]
Proof.
We compute the proximal operator Φ λ 1 c g , λ 2 c g , (cid:126) ( u ) (19) for (cid:126) ≥ 0 , it is similar for (cid:126) < 0 .
From (6), Φ λ 1 c g , λ 2 c g , (cid:126) ( u ) is expressed by:
[Equation]
We ﬁrst consider the ∂ϕ ( v ) /∂v in v ∈ { ( −∞ , 0) , (0 , (cid:126) ) , ( (cid:126) , ∞ ) }
[Equation]
in which ∂ϕ ( v ) exists.
Taking the derivative of ϕ ( v ) in these intervals delivers
[Equation]
where sign( . ) is a sign function.
When setting ∂ϕ ( v ) /∂v
= 0 to minimize ϕ ( v )
[Equation]
we derive:
[Equation]
From (21) and (23) [Equation] we have
[Equation]
[Equation]
In the remaining range value of u , namely, − λ 1 c g − λ 2 c g ≤ u ≤ λ 1 c g − λ 2 c g and (cid:126) + λ 1 c g − λ 2 c g ≤ u ≤ (cid:126) + λ 1 c g + λ 2 c g , we prove that the minimum of ϕ ( v ) (21) is obtained when v = 0 and v = (cid:126) , respectively.
Let us rewrite ϕ ( v ) , which was deﬁned in (21), as
[Equation]
By applying the inequality | a − b | ≥ | a | − | b | , where a, b ∈ R , to (25), we obtain:
[Equation]
For − λ 1 c g − λ 2 c g ≤ u ≤ λ 1 c g − λ 2 c g , from (26), ϕ ( v ) is minimal when v = 0 , due to λ 1 c g + λ 2 c g −| u | ≥ 0 .
Similarly, for (cid:126) + λ 1 c g − λ 2 c g ≤ u ≤ (cid:126) + λ 1 c g + λ 2 c g , i.e., λ 1 c g − λ 2 c g ≤ u − (cid:126) ≤ λ 1 c g + λ 2 c g , we have ϕ ( v ) ≥ λ 1 c g | v − (cid:126) | − λ 1 c g | (cid:126) | + λ 2 c g | v − (cid:126) | + 1 2( v − (cid:126) ) 2 − | v − (cid:126)
||
u − (cid:126) | + 1 2( u − (cid:126) )≥| v − (cid:126) | (cid:16) λ 1 c g + λ 2 c g − | u − (cid:126) | (cid:17)
[Equation]
2( v − (cid:126) ) 2 − λ 1 c g
[Equation]
(cid:126) |
[Equation]
( u
[Equation]
(cid:126) ) 2 . From (27), ϕ ( v ) is minimal when v = (cid:126) , since λ 1 c g + λ 2 c g − | u − (cid:126) | ≥ 0 .
Combining these results with the result in (24), we conclude the proof.
C G ENERALIZATION AND DEEP UNFOLDING RNN S C. G ENERALIZATION ERROR BOUND DEFINITION
Notations .
Let f ( d ) W : R n (cid:55)→ R h be the function computed by a d -layer network with weight parameters W .
The network f ( d ) W maps an input sample x i ∈ R n (from an input space X ) to an output y
i ∈
R h (from an output space Y ), i.e., y i = f ( d ) W ( x i ) .
Let S denote a training set of size m , i.e., S =
( x i , y i ) } mi =1 and E ( x i , y i ) ∼ S [ · ] denote an expectation over ( x i , y i ) from S .
The set S is drawn i.i.d.
from a distribution D , denoted as S ∼ D m , over a space Z =
X
× Y .
Let F be a (class) set of functions.
Let (cid:96) : F × Z (cid:55)→ R denote the loss function and (cid:96) ◦ F = { z (cid:55)→ (cid:96) ( f, z ) : f ∈ F} .
We deﬁne the true loss and the empirical (training) loss by L D ( f ) and L S ( f ) , respectively, as follows:
[Equation]
and
[Equation]
The generalization error , which is deﬁned as a measure of how accurately a learned algorithm is able to predict outcome values for unseen data, is calculated by L D ( f ) − L S ( f ) .
Rademacher complexity .
Let F be a hypothesis set of functions (neural networks).
The empirical
Rademacher complexity of F (Shalev-Shwartz & Ben-David, 2014) for a training sample set S is deﬁned as follows:
[Equation]
where (cid:15) =
( (cid:15) 1 , ..., (cid:15) m ) ; here (cid:15)
i are independent uniformly distributed random (Rademacher) variables from {± 1 } , according to P [ (cid:15) i = 1]
=
P
[ (cid:15) i = − 1] = 1 / 2 .
The generalization error bound (Shalev-Shwartz & Ben-David, 2014) is derived based on the
Rademacher complexity deﬁned in the following theorem:
Theorem C.1.
(Shalev-Shwartz & Ben-David, 2014, Theorem 26.5)
Assume that | (cid:96) ( f, z ) | ≤ η for all f ∈ F and z .
Then, for any δ > 0 , with probability at least 1 − δ ,
[Equation]
It can be noted that the bound in (31) via the Rademacher complexity depends on the training set S , which makes it applicable to a number of learning problems, e.g., regression and classiﬁcation, under given a loss function (cid:96) .
C. N OTATION FOR D EEP U NFOLDED RNN S
In this subsection, we provide the required notation for the proposed reweighted-RNN model, (cid:96) 1 - (cid:96) 1 -RNN, and Sista-RNN, which will be used in the derivation of their respective generalization analysis.
The proposed reweighted-RNN .
Let h ( l ) t be the hidden states in layer l evolving in time step t .
We write the reweighted-RNN model recursively as h (1) t = f (1) W , U ( h ( d ) t − 1 , x t ) =
Φ ( W 1 h ( d ) t − 1 + U 1 x t ) and h ( l ) t = f ( l ) W , U ( h ( d ) t − 1 , x t ) =
Φ (cid:16) W l f ( l − 1) W , U ( h ( d ) t − 1 , x t )
+ U l x t (cid:17) , where Φ is an activation function.
The hidden state is updated as shown in (9).
The real-valued family of functions, F d,t : R h × R n (cid:55)→ R , for the functions f ( d ) W , U in layer d is deﬁned by:
[Equation]
) where α l , β l are nonnegative hyper-parameters for layer l , where 1 < l ≤ d .
In layer l = 1 , the real-valued family of functions, F 1 ,t : R h × R n (cid:55)→ R , for the functions f (1) W , U is deﬁned by:
[Equation]
33) where α 1 , β 1 are nonnegative hyper-parameters.
We denote the input layer as f (0) W , U
= h ( d ) t − 1 , in particular, at t = 1 , h ( l ) 0 ≡ h 0 .
The (cid:96) 1 - (cid:96) 1 -RNN model (Le et al., 2019).
The hidden state h ( l ) t for (cid:96) 1 - (cid:96) 1 -RNN is updated as shown in (13).
The real-valued family of functions, F d,t : R h × R n (cid:55)→ R , for the function f ( d ) W , U in layer d is deﬁned by:
[Equation]
where α 2 , β 1 are nonnegative hyper-parameters for layer l , where 1 < l ≤ d .
In layer l = 1 , the real-valued family of functions, F 1 ,t : R h × R n (cid:55)→ R , for the functions f (1) W , U is deﬁned by:
[Equation]
[Equation]
where α 1 , β 1 are nonnegative hyper-parameters.
The Sista-RNN model (Wisdom et al., 2017).
The hidden state h ( l ) t in Sista-RNN is updated by:
[Equation]
The real-valued family of functions, F d,t : R h × R n (cid:55)→ R , for the functions f ( d ) W , U in layer d is deﬁned by:
[Equation]
[Equation] where α 2 , β 1 , β 2 are nonnegative hyper-parameters.
In layer l = 1 ,
[Equation]
where α 1 , β 1 are nonnegative hyper-parameters.
D P ROOF OF T HEOREM 3.
Proof.
We consider the real-valued family of functions F d,T : R h × R n (cid:55)→ R for the functions f ( d ) W , U to update h ( d ) T in layer d , time step T , deﬁned as
[Equation]
where w d , u d are the corresponding rows from W d , U d , respectively, and α l , β l , with 1 < l ≤ d , are nonnegative hyper-parameters.
For the ﬁrst layer and the ﬁrst time step, i.e., l = 1 , t = 1 , the real-valued family of functions, F 1 , 1 : R h × R n (cid:55)→ R , for the functions f (1) W , U is deﬁned by:
[Equation]
[Equation]
where α 1 , β 1 are nonnegative hyper-parameters.
We denote the input layer as f (0) W , U
= h 0 at the ﬁrst time step.
From the deﬁnition of Rademacher complexity in (30) and the family of functions in (39) and (40), we obtain:
[Equation]
where λ > 0 is an arbitrary parameter, Eq.
(41b) follows Lemma D.1 for 1-Lipschitz Φ a long with
Inequality (62), and (41c) holds by Inequality (59).
For layer 1 ≤ l ≤ d and time step t , let us denote:
[Equation] [Equation]
where Λ l is deﬁned as follows: Λ d = 1 , Λ l = d (cid:81) k
= l +1 α k with 1 ≤ l ≤ d − 1 , and Λ 0 = d (cid:81) k
=1 α k .
Following the Hölder’s inequality in (58) in case of p = 1 and q = ∞ applied to w T l and f ( l − 1) W , U ( h t − 1 ,i , x t,i ) in (42), respectively, we get:
[Equation]
[Equation] [Equation]
Similarly, from (43)
[Equation]
we obtain:
[Equation]
m (cid:88)
[Table] = [Table]
(cid:15)
[Equation]
[Table]
∞ (cid:33)
[Equation]
(cid:32) λβ
[Equation]
(cid:12)(cid:12)(cid:12) m (cid:88) i =
[Table]
(cid:15)
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
(cid:12)(cid:12)(cid:12)(cid:33) , (46) where { τ, κ } = argmax
[Equation]
∈{ 1 ,
[Equation] [Equation]
[Equation] [Equation]
[Equation]
[Equation]
n } (cid:12)(cid:12) (cid:12)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(cid:12)(cid:12) (cid:12)
[Equation]
From (41c), (44), and (46), we get:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] m
[Equation]
i = (cid:15)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:33)(cid:35)(cid:33) ≤λ log (cid:32)
[Equation]
exp (cid:32) λα
d m
[Equation]
i = (cid:15)
[Equation]
Φ (cid:16) w T d − 1 ,k f ( d − 2) W , U ( h T − 1 ,i , x T,i ) + u T d − 1 ,k x T,i (cid:17) + λβ d m (cid:88) i = (cid:15) i x τ,i,κ (cid:33) + exp (cid:32) λα
d m (cid:88) i = (cid:15)
i Φ (cid:16) w T d − 1 ,k f ( d − 2) W , U ( h T − 1 ,i , x T,i ) + u T d − 1 ,k x T,i (cid:17) − λβ d m (cid:88) i
=
(cid:15) i x τ,i,κ (cid:33) + exp (cid:32) − λα d m (cid:88) i
= (cid:15) i Φ (cid:16) w T d − 1 ,k f ( d − 2) W , U ( h T − 1 ,i , x T,i ) + u T d − 1 ,k x T,i (cid:17)
+ λβ d m (cid:88)
i = (cid:15) i x τ,i,κ (cid:33) + exp (cid:32) − λα d m (cid:88) i
= (cid:15) i Φ (cid:16) w T d
[Equation]
1 ,k f ( d
[Equation]
2) W ,
[Equation]
( h [Equation] − 1 ,i , x T,i )
[Equation] [Equation]
,i (cid:17) − λβ
[Equation]
m (cid:88) i
= (cid:15) i x τ,i,κ (cid:33)(cid:33)(cid:35)(cid:33) ≤
[Equation]
(cid:17)(cid:35)(cid:33) (47a)
[Equation]
47b)
[Equation]
i
[Equation]
i
[Equation]
(cid:107)
[Equation]
(cid:107) [Equation] ≤ α
[Equation]
exp (cid:32) λΛm
[Equation]
i = (cid:15)
[Equation]
(cid:16) w T1 h T − 1 ,i (cid:17)(cid:33) ·
[Equation]
(cid:17)(cid:33)(cid:35)(cid:33) (47c)
[Table]
[Equation]
where (47a) holds by inequality (59), and (47b) follows by repeating the process from layer d − 1 to layer 1 for time step T .
Furthermore, (47c) is obtained as the beginning of the process for time step
T − 1 and (47d) follows inequality (58).
Proceeding by repeating the above procedure in (47e) from time step T − 1 to time step 1 , we get: m R S ( F d,T ) ≤ [Equation]
[Equation]
Let us denote µ = argmax j ∈{ 1 ,...,h } (cid:12)(cid:12)(cid:12) m
[Equation]
i =1 (cid:15) i h 0 ,j (cid:12)(cid:12)(cid:12) , from (48), we have:
[Equation]
β l Λ l (cid:17)(cid:16) Λ T 0 −Λ 0 −(cid:17) m (cid:88) i =
(cid:15)
i x τ,i,κ (cid:33) exp (cid:32) λΛ Tm (cid:88) i
= (cid:15) i h 0 ,µ (cid:33)(cid:35)(cid:33) ≤ 2 dT logλ +2 λ log (cid:32)
E (cid:15) ∈{± 1 } m (cid:34) exp (cid:32) λ (cid:16) d (cid:88) l = β l Λ l (cid:17)(cid:16) Λ T 0 −Λ 0 −(cid:17) m (cid:88) i
= (cid:15) i x τ,i,κ (cid:33) · exp (cid:32) λΛ Tm (cid:88)
i = (cid:15) i h 0 ,µ (cid:33)(cid:35)(cid:33)≤ 2 dT logλ +2 λ log
E (cid:15) ∈{± 1 } m (cid:34) exp (cid:32)
2 λ (cid:16) d (cid:88) l = β l Λ l (cid:17)(cid:16) Λ T 0 −Λ 0 −(cid:17)
m (cid:88)
i = (cid:15) i x τ,i,κ (cid:33)(cid:35) +
λ log
E (cid:15) ∈{± 1 } m (cid:34) exp (cid:32)
2 λΛ Tm (cid:88) i =
(cid:15)
i h 0 ,µ (cid:33)(cid:35) (49a) ≤ 2 dT logλ +2 λ log n (cid:88) j
=
E (cid:15) ∈{± 1 } m (cid:34) exp (cid:32)
2 λ (cid:16) d (cid:88) l = β l Λ l (cid:17)(cid:16) Λ T 0 −Λ 0 −(cid:17)
m (cid:88)
i = (cid:15) i x τ,i,j (cid:33)(cid:35)
+
λ log h (cid:88)
j =
E (cid:15) ∈{± 1 } m (cid:34) exp (cid:32)
2 λΛ Tm (cid:88) i =
(cid:15)
i h 0 ,j (cid:33)(cid:35) (49b) ≤ 2 dT logλ +2 λ log n (cid:88)
j = m (cid:89) i =
E (cid:15) ∈{± 1 } m (cid:34) exp (cid:32)
2 λ (cid:16) d (cid:88) l = β l Λ l (cid:17)(cid:16) Λ T 0 −Λ 0 −(cid:17) (cid:15)
i x τ,i,j (cid:33)(cid:35) +
λ log h (cid:88)
j = m (cid:89) i =
E (cid:15) ∈{± 1 } m (cid:34) exp (cid:32) 2 λΛ T 0 (cid:15) i h 0 ,j (cid:33)(cid:35) [Table] λ [Table]
[Table]
log
[Table]
λ
[Table]
n
[Equation]
(49d) where (49a) follows inequality (61), and (49b) holds by replacing with (cid:80) nj =1 and (cid:80) hj =1 , respectively.
In addition, (49c) follows (60), and (49d) is obtained by the following deﬁnition:
At time step t , we deﬁne X t ∈ R n × m , a matrix composed of m columns from the m input vectors { x t,i } mi =1 ; we also deﬁne (cid:107) X t (cid:107) 2 , ∞ =
(cid:114) max k ∈{ 1 ,...,n } (cid:80) mi
=1 x 2 t,i,k ≤ √ mB x , representing the maximum of the (cid:96) 2 -norms of the rows of matrix X t , and (cid:107) h 0 (cid:107) ∞ =
max j | h 0 ,j | .
Choosing λ
=
[Equation]
2 ∞ , we achieve the upper bound:
[Equation]
It can be noted that R S ( F d,T ) in (50) is derived for the real-valued functions F d,T .
For the vectorvalued functions F d,T : R h × R n (cid:55)→ R h (in Theorem 3.1), we apply the contraction lemma (Lemma D.1) to a Lipschitz loss to obtain the complexity of such vector-valued functions by means of the complexity of the real-valued functions.
Speciﬁcally, in Theorem 3.1, under the assumption of the 1-Lipschitz loss function and from Theorem C.1, Lemma D.1, we complete the proof.
D. C OMPARISON WITH EXISTING GENERALIZATION BOUNDS
Recent works have established generalization bounds for RNN models with a single recurrent layer ( d = 1 ) using Rademacher complexity [see FastRNN in Kusupati et al.
(2018)] or PAC-Bayes theory [see SpectralRNN in Zhang et al.
We re-state these generalization bounds below and apply
Theorem 3.1 with d = 1 to compare with our bound for reweighted-RNN.
FastRNN (Kusupati et al., 2018).
The hidden state h t of FastRNN is updated as follows:
[Equation]
where 0 ≤ a, b ≤ 1 are trainable parameters parameterized by the sigmoid function.
Under the assumption that a + b = 1 , the Rademacher complexity R S ( F T ) of the class F T of FastRNN (Kusupati et al., 2018), with (cid:107) W (cid:107) F ≤ α F , (cid:107) U (cid:107) F ≤ β F , and (cid:107) x t (cid:107) 2 ≤ B , is given by
[Equation]
Alternatively, under the additional assumption that a ≤
[Equation]
F
[Equation]
1) T , the bound in Kusupati et al.
(2018) becomes:
[Equation]
SpectralRNN (Zhang et al., 2018).
The hidden state h t and output y t ∈ R n y of SpectralRNN are computed as:
[Equation]
where Y ∈ R n y × h .
The generalization error in Zhang et al.
(2018) is derived for a classiﬁcation problem.
For any δ > 0 , γ > 0 , with probability ≥ 1 − δ over a training set S of size m , the generalization error (Zhang et al., 2018) of SpectralRNN is bounded by
[Equation]
where ζ = max {(cid:107) W (cid:107)
[Equation]
, 1 } max [Equation] (cid:107) U (cid:107) 22 , 1 } max {(cid:107) Y (cid:107) 22 , 1 } and ξ = max { n, n y , h } .
Reweighted-RNN .
Based on Theorem 3.1, under the assumption that the initial hidden state h
=
0 , the Rademacher complexity of reweighted-RNN with d = 1 is bounded as
[Equation]
We observe that the bound of SpectralRNN in (55) depends on T 2 , whereas the bound of FastRNN either grows exponentially with T (52) or is proportional to T (53).
Our bound (56) depends on √
T , given that the second factor in (56) is only dependent on the norm constraints α 1 , β 1 and the input training data; meaning that it is tighter than those of SpectralRNN and FastRNN in terms of the number of time steps.
D. B ACKGROUND ON R ADEMACHER COMPLEXITY CALCULUS
The contraction lemma in Shalev-Shwartz & Ben-David (2014) gives the Rademacher complexity of the composition of a class of functions with ρ -Lipschitz functions.
Lemma D.1.
(Shalev-Shwartz & Ben-David, 2014, Lemma 26.9—Contraction lemma)
Let F be a set of functions, F = { f : X (cid:55)→ R } , and Φ 1 , ..., Φ m , ρ -Lipschitz functions, namely, | Φ i ( α ) − Φ i ( β ) | ≤ ρ | α − β | for all α, β ∈ R for some ρ > 0 .
For any sample set S of m points x 1 , ..., x m ∈ X , let ( Φ ◦ f )( x i ) =
Φ ( f ( x i )) .
Then,
[Equation]
∈
[Equation]
± [Equation] m (cid:34) sup f ∈F m (cid:88) i = (cid:15)
[Equation]
i )
[Equation]
m
[Equation]
(cid:34)
[Equation]
m (cid:88)
[Equation]
=
[Equation]
(cid:15) i f
( x i ) (cid:35) , (57) alternatively, R S ( Φ ◦ F ) ≤ ρ R S ( F ) , where Φ denotes Φ 1 ( x 1 ) , ..., Φ m ( x m ) for S .
Proposition D.2.
(Mohri et al., 2018, Proposition A.1—Hölder’s inequality)
Let p, q ≥ 1 be conjugate: 1 p + 1 q = 1 .
Then, for all x , y ∈ R n , (cid:107)
[Equation]
, (58) with the equality when | y i | =
| x i | p − 1 for all i ∈ [1 , n ] .
Supporting inequalities :
(i)
If A , B are sets of positive real numbers, then:
[Equation]
(ii)
Given x ∈ R , we have:
[Equation]
(iii) Let X and Y be random variables, the Cauchy–Bunyakovsky–Schwarz inequality gives:
[Equation]
(iv) If ψ is a convex function, the Jensen’s inequality gives:
[Equation]
E A DDITIONAL EXPERIMENTS
We test our model on three popular tasks for RNNs, namely the sequential pixel MNIST classiﬁcation, the adding task, and the copy task (Le et al., 2015; Arjovsky et al., 2016; Zhang et al., 2018).
Sequential pixel MNIST and permuted pixel MNIST classiﬁcation .
This task aims to classify
MNIST images to a class label.
MNIST images are formed by a 28 × 28 gray-scale image with a label from 0 to 9.
We use the reweighted-RNN along with a softmax for category classiﬁcation.
We set d = 5 layers and h = 256 hidden units for the reweighted-RNN.
We consider two scenarios: the ﬁrst one where the pixels of each MNIST image are read in the order from left-to-right and bottom-to-top and the second one where the pixels of each MNIST image are randomly permuted.
The classiﬁcation accuracy results are shown in Fig. 7(a)
(for pixel MNIST) and Fig. 7(b) (for permuted pixel MNIST).
[Figure]
[Figure]
Adding Task .
The task inputs two sequences of length T .
The ﬁrst sequence consists of entries that are uniformly sampled from [0 , 1] .
The second sequence comprises two entries of 1 and the remaining entries of 0, in which the ﬁrst entry of 1 is randomly located in the ﬁrst half of the sequence and the second entry of 1 is randomly located in the second half.
The output is the sum of the two entrie of the ﬁrst sequence, where are located in the same posisions of the entries of 1 in the second sequence.
We also use the reweighted-RNN with d = 5 layers and h = 256 hidden units for the input sequences of length T = 300 .
Fig. 8(a) shows the mean square error versus training epoches on the validation set.
Copy task .
We consider an input sequence x ∈ A T +20 (Zhang et al., 2018), where A = { a 0 , · · · , a 9 } .
x 0 , · · · , x 9 are uniformly sampled from { a 0 , · · · , a 7 } , x T +10 = a 9 , and the remaining x i are set to a 8 .
The purpose of this task is to copy x 0 , · · · , x 9 to the end of the output sequence
y ∈
A T + given a time lag T , i.e., { y T +10 , · · · , y T +19 } ≡ { x 0 , · · · , x 9 } and the remaining y i are equal to a 8 .
We set the reweighted-RNN d = 5 layers and h = 256 hidden units for the input sequences of a time lag T = 100 .
Fig. 8(b) shows the cross entropy versus training epoches on the validation set.