On the approximation properties of recurrent encoder-decoder architectures
Abstract
Encoder-decoder architectures have recently gained popularity in sequence to sequence modelling, featuring in state-of-the-art models such as transformers.
However, a mathematical understanding of their working principles still remains limited.
In this paper, we study the approximation properties of recurrent encoder-decoder architectures.
Prior work established theoretical results for RNNs in the linear setting, where approximation capabilities can be related to smoothness and memory of target temporal relationships.
Here, we uncover that the encoder and decoder together form a particular “temporal product structure” which determines the approximation eﬃciency.
Moreover, the encoder-decoder architecture generalises RNNs with the capability to learn time-inhomogeneous relationships.
Our results provide the theoretical understanding of approximation properties of the recurrent encoder-decoder architecture, which precisely characterises, in the considered setting, the types of temporal relationships that can be eﬃciently learned.
Introduction
Encoder-decoder is an increasingly popular architecture for sequence to sequence modelling problems (Sutskever et al., 2014; Chiu et al., 2018; Venugopalan et al., 2015).
The core of this architecture is to ﬁrst encode the input sequence into a vector using the encoder and then map the vector into the output sequence through the decoder.
In particular, such architecture forms the main component in the transformer network (Vaswani et al., 2017), which has become a powerful method for modelling sequence to sequence relationships (Parmar et al., 2018;
Beltagy et al., 2020; Li et al., 2019).
The encoder-decoder family of structures diﬀer signiﬁcantly from direct application of recurrent neural networks (RNNs, Elman (1990)) and its generalisations (Hochreiter & Schmidhuber, 1997; Cho et al., 2014b) for processing sequences.
However, both architectures can be considered as modelling mappings between sequences, albeit with diﬀerent underlying structures.
Hence, a natural but unresolved question is: how are these approaches fundamentally diﬀerent?
Answering this question is not only of theoretical importance but also of practical interest.
Currently, architectural selection for diﬀerent time series modelling tasks is predominantly empirical.
Thus, it is desirable to develop a concrete mathematical framework to understand the key diﬀerences between separate architectures in order to guide practitioners in a principled way.
In this paper, we investigate the approximation properties of encoder-decoder architectures.
Approximation is one of the most basic and important problems for supervised learning.
It considers to what extent the model can ﬁt a target.
In particular, we prove a general approximation result in the linear setting, which characterises the types of temporal input-output relationships that can be eﬃciently approximated by encoder-decoder architectures.
These results reveal that such architectures essentially generalise RNNs by lifting the requirement of time-homogeneity (see Remark 3.2) in the target relationships.
Hence, it can be used to tackle a broader class of sequence to sequence problems.
Furthermore, of particular interest is the identiﬁcation of a “temporal product structure” — a precise property of the target temporal relationship that highlights another intrinsic diﬀerence between recurrent encoder-decoders and RNNs.
Our main contributions can be summarised as follows.
We prove a universal approximation result for recurrent encoder-decoder architectures in the linear setting, including the approximation rates.
We show that in the considered setting, the recurrent encoder-decoder generalises the RNNs and can approximate time-inhomogeneous relationships, which further adapt to additional temporal product structures in the target relationship.
This answers precisely how encoder-decoders are diﬀerent from RNNs, at least in the considered setting.
Organisation.
In Section 2, we review the related work on encoder-decoder architectures and general approximation theories of sequence modelling.
The approximation problem is formulated in Section 3.
Our main results, their consequences and numerical illustrations are presented in Section 4.
All the proofs and numerical details are included in appendices.
Notations.
For consistency, we adhere to the following notations.
Boldfaced letters are reserved for sequences or paths, which can be understood as functions of time.
Lower case letters can mean vectors or scalars.
Matrices are denoted by capital letters.
For α ∈ N , C α denotes the space of functions with continuous derivatives up to order- α .
Related work
We ﬁrst review some previous works on sequence to sequence modelling.
The encoderdecoder architecture ﬁrst appeared in Kalchbrenner & Blunsom (2013), where they map the input sequence into a vector using convolutional neural networks (CNNs), and then using a recurrent structure to map the vector to the output sequence.
With the ﬂexibility of manipulating the underlying structure of encoder and decoder, numerous models based on this architecture have come out thereafter.
For instance, Cho et al.
(2014b) used gated RNNs as both the encoder and decoder, while in the later work (Cho et al., 2014a), they proposed a CNN-based decoder.
In Sutskever et al.
(2014), they proposed a deep LSTM for both the encoder and decoder.
Bahdanau et al.
ﬁrst introduced the attention mechanism, which was further developed in the well-known transformer networks (Vaswani et al., 2017).
However, most of the research on encoder-decoder architectures focused on applications.
A theoretical understanding is helpful for its further improvement and development.
From the theoretical point of view, Ye & Sung (2019) studied several theoretical properties of
CNN encoder-decoders, including expressiveness, generalisation capability and optimisation landscape.
Of particular relevance to the current work is expressiveness, which considers the relationships that can be generated from the architecture.
However, this is not approximation.
Yun et al.
(2020) proved the universal approximation property of transformers for certain classes of functions, for example, permutation equivariant functions, but they did not consider the actual dynamical properties of target relationships that aﬀect approximation.
Dynamical proprieties such as memory, smoothness and low rank structures are essential, because they can precisely characterise diﬀerent temporal relationships and aﬀect the approximation capabilities of models.
Assuming the target generated from a hidden dynamical system is one approach, which is widely applied (Maass et al., 2007; Sch¨afer & Zimmermann, 2007; Doya, 1993; Funahashi & Nakamura, 1993).
In contrast, a functional-based approach is recently introduced, where the target temporal relationships are generated from functionals satisfying speciﬁc properties such as linearity, continuity, regularity and time-homogeneity (Li et al., 2021).
In Li et al.
(2021), the approximation properties of linear RNN models are studied, and the results therein show that the approximation eﬃciency is related to the memory structure.
In Jiang et al.
(2021), similar formulations are applied to investigate convolutional architectures, where the results suggest that targets with certain spectrum regularity can be well approximated by dilated CNNs.
Under this framework, the target temporal relationship that can be eﬃciently approximated is characterised by properties such as memory, smoothness and sparsity.
This enables us to make precise mathematical comparisons between diﬀerent architectures.
Our results in this work reveal that the encoder-decoders have a special temporal product structure which is intrinsically diﬀerent from other sequence modelling architectures.
Problem formulation
In this section, we precisely deﬁne the input space, output space, concept space and hypothesis space, respectively.
Functional formulation of temporal modelling.
First, we deﬁne the input and output space precisely.
A temporal sequence can be viewed as a function of time t .
The input space is deﬁned by X = C 0 (( −∞ , 0] , R d ).
This is the space of continuous functions from ( −∞ , 0] to R d vanishing at inﬁnity, where d ∈ N + is the dimension.
Denote the element in X by x = { x t ∈ R d : t ≤ 0 } , we equip X with the supremum norm k x k X := sup t ≤ 0 k x t k ∞ .
We take the outputs space as Y = C b ([0 , ∞ ) , R ), the space of bounded continuous functions from [0 , ∞ ) to R .
We consider real-valued outputs, since each dimension can be handled individually for vector-valued outputs.
The mapping between input and output sequences can be formulated as a sequence of functionals , i.e. y t =
H t ( x ), t ≥ 0.
The output y t at the time step t depends on the input sequence x .
The ground truth relation between inputs and outputs is formulated by the sequence of functionals H :=
{ H t : t ≥ 0 } .
We provide an example to illustrate the above formulation.
Given an input x , the output y is a smoothed version of x , resulting from convolving x with the Gaussian kernel g ( s ) = 1 √ 2 π exp( − s 2 2 ).
This relation can be formulated as y t =
H t ( x ) =
R ∞
g ( t + s ) x − s ds.
The RNN encoder-decoder model.
For the supervised learning problem, our goal is to use a model to learn the target relationship H .
First, we deﬁne the model.
Among all diﬀerent variants of the encoder-decoder architectures, the RNN encoder-decoder introduced in Cho et al.
(2014b) can be considered as the most simple and representative model, where the encoder and decoder are both RNNs.
We study this particular model as we try to eliminate other factors and only focus on the encoder-decoder architecture itself.
Under our setting, the simpliﬁed model of Cho et al.
(2014b) with RNNs as both encoder and decoder can be formulated as
[Equation]
where h t , g t are hidden states of the encoder and decoder respectively.
Recurrent activation functions are denoted by σ E and σ D .
Here, τ denotes the terminating time step of the encoder, and v is the summary of the input sequence, which is called as the coding vector.
The model prediction is denoted as o t ∈ R .
All the other notations are model parameters.
Equation (1) describes the following model dynamics.
First, the encoder reads the entire input x , and then summarises the input into a ﬁxed size coding vector v , which is also the last hidden state of the encoder.
Next, the coding vector is passed into the decoder as the initial state, and then the decoder produces an output at each time step.
Note that the encoder has a terminating time, and the decoder has a starting time.
This is the reason why we take the input and output as semi-inﬁnite sequences.
We study a linear, residual and continuous-time idealisation of the model dynamics (1):
[Equation]
where W ∈ R m E × m E , U ∈ R m E × d , Q ∈ R N × m E , V ∈ R m D
× m D , P ∈ R m D × N and c ∈ R m D are parameters.
m E and m D denote the width of encoder and decoder, respectively.
The coding vector v has dimension N , where we apply linear transformations to control it.
We assume h
−∞ = 0, which is the usual choice for the initial condition of RNN hidden states.
Since our goal is to investigate approximation problems over large time horizons, we are supposed to consider the stable RNN encoder-decoders, where
[Equation]
eigenvalues of V
[Equation]
real parts }
[Equation]
The hypothesis space of RNN encoder-decoder models with arbitrary widths and coding vector dimension is deﬁned as ˆ H :
=
S m E ,m D ,N ∈ N + ˆ H m E ,m D ,N , where ˆ
[Equation]
The widths m E , m D and the coding vector dimension N together control the capacity/complexity of the hypothesis space.
Note that the assumptions on eigenvalues of W and V ensure that the parameterized linear functionals are continuous.
Due to the mathematical form (5), not all functionals can be represented by RNN encoderdecoders.
To achieve a good approximation, the target functionals must possess certain structures.
We introduce the following deﬁnitions to clarify these structures.
Deﬁnition 3.1.
Let H =
{ H t : t ≥ 0 } be a sequence of functionals
For any t ≥ 0 , the functional H t is linear and continuous if for any λ 1 , λ 2 ∈ R and x 1 , x 2 ∈ X , we have H t ( λ 1 x 1 + λ 2 x 2 ) =
λ 1 H t ( x 1 ) + λ 2 H t ( x 2 ) , and k H t k := sup x ∈X , k x k X ≤ 1 | H t ( x ) | < ∞ , where k H t k denotes the induced functional norm.
For any t ≥ 0 , the functional H t is regular if for any sequence { x ( n ) } ∞ n =1 ⊂ X such that lim n →∞ x ( n ) s = 0 for almost every s ≤ 0 (Lebesgue measure), we have lim n →∞ H t ( x ( n ) )
= 0 .
For a sequence of functionals H , we deﬁne its norm by k H k :
=
Z ∞
[Equation]
Remark 3.1.
The deﬁnitions of linear and continuous functionals are standard.
One can view regular functionals as those not determined by inputs on arbitrarily small time intervals, e.g. an inﬁnitely thin spike (i.e. δ -functions).
Given the above deﬁnitions, we immediately have the following observation.
Proposition 3.1.
Let ˆ H ∈
ˆ H be a sequence of functionals in the RNN encoder-decoder hypothesis space (see (5)).
Then for any t ≥ 0 , ˆ H t ∈
ˆ H is a linear, continuous and regular functional.
Furthermore, k ˆ H t k decays exponentially as a function of t .
The proof is found in Appendix A.
This proposition characterises properties of the encoderdecoder hypothesis space.
In particular, it is diﬀerent from the RNN hypothesis space discussed in Li et al.
(2021), since the encoder-decoder is not necessarily time-homogeneous .
Remark 3.2.
A sequence of functionals H is time-homogeneous if for any t, τ ≥ 0 , H t ( x ) =
H t + τ ( x ( τ )) , with x ( τ ) s = x
s − τ for all s ∈ R .
That is, if the input is shifted to the right by τ , the output is also shifted by τ .
Temporal convolution is an example of time-homogeneous operation (recall the Gaussian convolution discussed in Section 3.
An example of timeinhomogeneous relationship is video captioning: shifts in the sequence of input video frames do not necessarily lead to corresponding shifts in the caption text sequence.
Relation with RNNs.
Here, we emphasise the diﬀerences between the encoder-decoder hypothesis space and the RNN hypothesis space discussed in Li et al.
(2021), where ˆ H (RNN) t ( x ) =
R ∞ 0 c > e W ( t + s )
Ux − s ds .
A key diﬀerence is that the encoder-decoder has a structure involving two temporal parameters t and s , while the RNN only has one depending on t + s , due to the time-homogeneity.
Owing to this diﬀerence and the fact that ˆ
H (RNN) ⊂ ˆ H , the encoder-decoder hypothesis space (5) is more general, with the extra capability to learn time-inhomogeneous relationships.
Furthermore, e V t and e Ws adapt to a temporal product structure, which is an intrinsic diﬀerence between encoder-decoders and other architectures.
We will discuss this in detail in the next section.
Approximation results
One of the most fundamental problems for supervised learning is the approximation problem.
It basically concerns the capacity of the hypothesis space to ﬁt the concept space.
In general, there are two levels of approximation problems that can be discussed.
The ﬁrst is known as the universal approximation, which considers the density of the hypothesis space in the concept space.
The second is the approximation rate, which aims to characterise quantitatively the approximation accuracy concerning the capacity/complexity of the hypothesis space (e.g. the number of trainable parameters).
In this section, both of them are developed for RNN encoder-decoders.
4. Universal approximation
We ﬁrst present the most basic density result, which states that any linear, continuous, and regular temporal relationship can be approximated by RNN encoder-decoders up to arbitrary accuracy.
The proof is found in Appendix B.
Theorem 4.1.
Let H be a sequence of linear, continuous and regular functionals deﬁned on X , and satisfy k H k < ∞ .
Then for any (cid:15) > 0 , there exists ˆ H ∈
ˆ H such that
[Equation]
Here, we highlight two important observations while deriving Theorem 4.1.
First, one can show that each sequence of functionals H ∈ H can be associated with a unique twoparameter “representation” ρ ( t, s ), such that H t ( x ) =
R ∞ 0 x >− s ρ ( t, s ) ds .
Recall the model form ˆ H t ( x ) =
R ∞ 0 x >− s ˆ ρ ( t, s ) ds , where ˆ ρ ( t, s ) :
=
[ c > e V t PQe Ws U ] > denotes the corresponding representation.
The functional approximation is then reduced to function approximation in the sense of representations, i.e. k H − ˆ H k ≤ k ρ − ˆ ρ k L 1 ([0 , ∞ ) 2 ) .
It turns out that ρ directly aﬀects the rate of approximation and gives rise to intrinsic properties.
We will discuss this in detail in Section 4.3.
In addition, we again emphasise the diﬀerences between the present work and Li et al.
In Li et al.
(2021), the target relationships are assumed to be time-homogeneous with the representation H t ( x ) =
R ∞ 0 ρ ( t + s ) x − s ds , which only depends on t + s .
However, the setting here does not assume time-homogeneity, hence implies a more general representation ρ depending on the two temporal directions t and s simultaneously.
4. General approximation rates
While the density result (Theorem 4.1) ensures the universal approximation property of the
RNN encoder-decoder, it does not identify targets that can be eﬃciently approximated.
To achieve this, we focus on approximation rates next.
We characterise the temporal structure of a target relationship by observing its responses to “constant” input signals.
Here, we consider the approximation rates for the model with “large size” coding vector, where the dimension N ≥ ¯ m := min { m E , m D } .
This is the scenario where we ﬁx the widths but take an oversized coding vector.
Theorem 4.2.
Let H be a sequence of linear, continuous and regular functionals deﬁned on X , and satisfy k H k < ∞ .
Consider the output of piece-wise constant signals
y c
i ( t, s ) =
H t ( e i 1 ( −∞ , − s ] ) , t, s ≥ 0 , i = 1 , 2 , . . .
, d , where { e i } di =1 denotes the standard basis of R d .
Assume that there exist α ∈ N + , β > 0 such that for any i = 1 , 2 , . . .
, d ,
[Equation]
Then for any m E , m D , N ∈ N + , there exists ˆ H ∈
ˆ H m E ,m D ,N such that
[Equation]
where C ( α ) , γ > 0 are both universal constants with dependence only on α and ( α, β ) , respectively, and γ :=
[Equation]
Here, the number of trainable parameters is dN ( m E + m D ) with N ≥ ¯ m .
The proof is found in Appendix C. First, note that the error bound does not depend on the coding vector size N , as long as N ≥ ¯ m .
This is because further increasing N beyond ¯ m only increases the number of trainable parameters, but does not increase the model capacity (see Remark C.2).
Only the model widths m E , m D aﬀect the approximation capabilities.
Next, we focus on the classes of target relationships that can be well approximated.
Here, α characterises the smoothness of H , and β characterises the temporal decay rates of the output responding to a constant signal under H .
This is a notion of memory in the target relationship.
The error bound (9) indicates that a sequence of target functionals can be eﬃciently approximated by the encoder-decoder if it is smooth (large α ), and has fast decayed memory (large β ).
The characterisation in smoothness and memory decay also appears in the approximation results of RNNs (Li et al., 2021), where the upper bound is C ( α ) γd βm α .
However, our results for encoder-decoders suggest extra structures, where the bound involves two (instead of one) temporal parameters together with smoothness and decay memories in both.
The two-parameter temporal dependence allows the encoder-decoder to approximate timeinhomogeneous relationships, which generalises the RNN.
This two-parameter structure further leads to adaptation to a speciﬁc low rank type of target relationships, resulting in ﬁner approximation rates as we discuss next.
4. Approximation rates via temporal product structure
Motivation of temporal product structure.
In contrast with Theorem 4.2, we next consider the model with N < ¯ m = min { m E , m D } .
In this situation, the model has fewer parameters, and we aim to characterise the target relationships by further exploiting the structure of the two-parameter representation ρ ( t, s ).
This leads to a ﬁner approximation rate by considering m E , m D , N together.
We ﬁrst motivate how the “temporal product structure” arises, and how it relates to the approximation.
Detailed discussions and proofs are found in Appendix D. For the illustration purpose, we set the input dimension d = 1.
Recall Q ∈ R N × m E , P ∈ R m D × N , then the representation ˆ ρ of the encoder-decoder functional can be rewritten as
[Equation]
This is a tensor product structure over the ( t, s ) time domain (determined by the encoder { ˆ φ n } and decoder { ˆ ϕ n } successively).
We call it the temporal product structure .
As is shown later, this structure signiﬁcantly aﬀects approximation rates.
When { ˆ φ n } and { ˆ ϕ n } are selected as the “bases” along s, t direction, respectively, N is considered as the rank of the temporal product.
We also deﬁne N as the rank of the model, which is understood as the maximum rank of temporal products that the encoder-decoder model can represent.
The rank concept of temporal relationships.
Recall that the given number of trainable parameters is dN ( m E + m D ).
Hence, a low rank model may achieve fewer trainable parameters.
When investigating relationships that can be well approximated by low rank models, a natural conjecture would be “low rank” targets.
What is the meaning of “low rank” for a temporal relationship?
It is well-known that in linear algebra, an operator is low rank means that its range space is low-dimensional.
This idea can be also applied to temporal relationships.
For a “low rank” temporal relationship, the output sequence is more “regular”, meaning that the output sequences (viewed as functions) are in a low-dimensional function space.
We provide an intuitive numerical illustration for better understanding.
[Figure]
the corresponding outputs H t ( x ). Detailed settings are found in Appendix E.1.
Figure 1 shows the outputs of a high rank (a) and a low rank (b) target relationship on the same set of random input sequences.
Diﬀerent colours refer to diﬀerent instances of inputs.
In the ﬁrst case (high rank), the temporal structure of the outputs is very complex and depends sensitively on the inputs.
However, in the second case (low rank), the output sequences are much more regular, and only macroscopic structures (e.g. scale/oﬀset) appear.
Remark 4.1.
In the research of approximation theories for temporal sequences, prior works also related a notion of rank to approximation properties of the dilated convolutional structure (Jiang et al., 2021).
Here, we emphasise that the notion of rank considered in our work is very diﬀerent from that in Jiang et al.
(2021), which mainly concerns the tensorisation of a discrete-time sequence according to the width of convolution ﬁlters.
POD as an analogue of SVD.
Now, we characterise low rank and high rank temporal relationships in a mathematical way.
We will introduce the concepts informally, and rigorous deﬁnitions and arguments can be found in Appendix D.
For a matrix, we can assess its rank by performing the singular value decomposition (SVD).
This method can be extended to the temporal relationships using proper orthogonal decomposition (POD; Liang et al. , Berkooz et al.
(1993), Chatterjee (2000)).
The basic insight is that the function ρ can be decomposed into the following form: ρ ( t, s )
=
P N 0 n =1 σ n ϕ n ( t ) φ n ( s ), where N 0 ≤ ∞ , { ϕ n } and { φ n } are orthonormal bases, and σ 1 ≥ σ 2 ≥ · · · ≥ 0 denote the singular values.
This procedure can be viewed as apply- ing SVD to an inﬁnite-dimensional space (when N 0 = ∞ ).
An analogue of Eckart–Young theorem (Eckart & Young, 1936), which characterises the best low rank approximation, also holds for POD.
It roughly states that inf rank(ˆ ρ )=
N k ρ − ˆ ρ k 2 L 2 = P N 0 n = N +1 σ 2 n .
That is, any target ρ has a rank- N best approximation, with error equalling to the tail sum of the squared singular values.
In other words, a target with fast decayed σ n (low “eﬀective rank”) has smaller approximation errors.
This forms the basis of our next result, which states that if the target relationship possesses an eﬀective low rank structure in terms of the decay of singular values, then one can achieve an eﬃcient approximation using encoder-decoder structures by limiting the size of coding vectors.
Detailed deﬁnitions for { σ n } and proofs are found in Appendix D.
Theorem 4.3.
Assume the same conditions as in Theorem 4.2.
Then for any m E , m D , N ∈
N + with N ≤ ¯ m , there exists ˆ H ∈
ˆ H m E ,m D ,N such that
[Equation]
where (cid:46) hides universal positive constants, and ¯ m = min { m E , m D } .
Here, the number of trainable parameters is dN ( m E + m D ) with N ≤ ¯ m .
This is a ﬁner approximation rate compared to Theorem 4.2, where both the widths m E , m D and the coding vector size N aﬀect the model capacity for approximation.
Besides the smoothness and memory decay, we have the additional rank structure of the target relationship, which is characterised by its singular values { σ n } .
We again focus on the class of functionals that can be well approximated.
Smoothness α and decay rate β is the same as Theorem 4.2.
The diﬀerence lies in the rank structure indicated by { σ n } : the error bound is small if { σ n } has a small tail P
¯ mn =
N +
σ 2 n .
It suggests that a target with fast decayed { σ n } or low “eﬀective rank” can be well approximated by the RNN encoder-decoder with fewer parameters.
Due to the Eckart–Young-like low rank approximation, we can appropriately select N based on the decay rate of singular values.
Here, we emphasise that the temporal product is an intrinsic structure arising from the encoder-decoder architecture.
Recall the dynamics of the encoder-decoder: it ﬁrst encodes the input sequence into a coding vector, and then decodes an entire output sequence from it.
In this sense, the coding vector is the only interaction between the input and output.
Thus, the coding vector size N is an essential measure of the model capacity concerning the dependence of outputs on inputs.
Here, we show that this concept can be formalised as a notion of rank, which can pinpoint the precise types of input-output relationships that encoder-decoder architectures are well adapted to.
Numerical illustrations.
Here, we utilise numerical examples to illustrate the above discussions.
We observe how the decay rate of singular values, the rank N 0 of the target relationships, and the model rank N aﬀect the approximation error k H − ˆ H k .
However, it is not always possible to construct the best approximation.
Instead, we perform some training steps to achieve an upper bound of the approximation error, which is consistent with our theoretical results.
In Figure 2, we train linear encoder-decoder models to learn three relationships of diﬀerent ranks determined by various decay patterns of singular values, given in (a), (b) and (c).
Diﬀerent colours denote targets with diﬀerent ranks.
From Figure 2, we have the following observations consistent with previous discussions.
First, observe that increasing the model rank N makes approximation errors smaller, as expected.
Moreover, note that when increasing N , the speeds of error decrements are diﬀerent.
If the singular values decay fast, the approximation errors also decay fast.
This implies that a target with fast decayed singular values can be approximated eﬃciently with fewer parameters (smaller N ).
In addition, for each experiment, we are able to achieve low approximation errors by choosing N (cid:28) m .
The errors will remain unchanged or decrease much more slowly when further increasing N .
This suggests that in practice, one can choose N such that it covers the major singular values of the target in order to improve the approximation eﬃciency.
10 N 0 .0 .0 .0 .0 .1 .
E rr o r × 10 −
Target Rank N 0 = 2 N 0 =
N 0 = 6 N 0 =(a) σ n = (cid:26) n − 18 , n ≤ N0 , n > N    10 N 0 .0 .1 .1 .2 .2 .
E rr o r × 10 −
Target Rank N 0 = 2 N 0 =
N 0 = 6 N 0 =(b) σ n = (cid:26) n − 1 , n ≤ N0 , n > N   30 N 0 .0 .0 .0 .0 .1 .
E rr o r × 10 −
Target Rank N 0 = ∞ (c) σ n = n −
In Figure 3, we perform experiments on the forced Lorentz 96 system (Lorenz, 1996), which parameterises a high-dimensional and nonlinear relationship between input forcing and model states.
The parameters K, J in the Lorenz 96 system control the overall complexity of the target (see Appendix E.3 for details).
We use the RNN encoder-decoder with tanh activations to learn this target.
Although our theories are developed in the linear regime, the low rank approximation phenomenon also appears in this nonlinear setting.
The error decrements saturate when increasing the coding vector size N beyond a threshold, suggesting the existence of some implicit notion of “rank” of the target nonlinear functional.
This “rank” increases with the target complexity (mainly K ).
[Figure]
Conclusion
We theoretically study the approximation properties of the RNN encoder-decoder in a linear setting.
We prove a universal approximation result for linear temporal relationships utilising encoder-decoder architectures, and show that they generalise RNNs to the timeinhomogeneous setting.
Moreover, we discover an important temporal product structure that characterises the types of input-output relationships especially suited for the eﬃcient approximation using encoder-decoders.
This elucidates the key diﬀerences between these novel architectures and classical methods for temporal modelling, and forms a basic step towards understanding the intricacies of modern deep learning.
Reproducibility statements.
Detailed proofs for theoretical results, and complete settings of numerical examples are found in the appendix.
The source code for numerical tests can be made available upon request.
Here is a quick reference:
Proposition 3.
Properties of RNN encoder-decoder functionals
Appendix A
Theorem 4.
Universal approximation theorem
Appendix B
Theorem 4.
General approximation rates
[Figure] C
Theorem 4.
Approximation rates concerning temporal product structure
[Figure] D
[Figure] [Figure] [Figure]
[Figure] Acknowledgements
ZL is supported by Peking University under BICMR mathematical scholarship. HJ is supported by National University of Singapore under PGF scholarship. QL is supported by the National Research Foundation, Singapore, under the NRF fellowship (NRF-NRFF13-20210005).
A Properties of model functionals
In this section, we prove observations of the hypothesis space reported in Proposition 3.1.
Proof of Proposition 3.1.
Recall that
[Equation]
Fix any θ =
( W, V, U, M, c ).
The linearity is obvious.
Since both W ∈ W m E and V ∈ V m D have eigenvalues with negative real parts, there exist c 1 , c 2 , c 0 1 , c 0 2 > 0, such that k e V t k ∞ ≤ c 1 e − c 2 t and k e Ws k ∞ ≤ c 0 1 e − c 0 2 s for any t, s ≥ 0, hence
[Equation]
where (cid:46) hides universal positive constants depending on parameters θ .
Therefore
[Equation]
That is, the functional ˆ H t ( · ; θ ) is bounded (i.e. continuous) with an exponentially-decayed norm (as a function of t ).
Finally, by (13) and Lebesgue’s dominated convergence theorem, the functional ˆ H t ( · ; θ ) is also regular.
The proof is completed.
B Universal approximation
In this section, we provide the proof of Theorem 4.1, i.e. the universal approximation property of RNN encoder-decoders.
As is stated in the main text, the key step is to utilise the classical representation theorem, which helps us to reduce the approximation problem of functionals to functions.
B. Preliminaries
First, we list the background deﬁnitions and notations used in the following theorems.
Let ( X, A ) be a measure space (with A as the σ -algebra of subsets of X ).
• The space X is called locally compact if for any x ∈ X , x has a compact neighbourhood.
• X is a Hausdorﬀ space if all distinct points in X are pair-wisely separable by neighbourhoods.
That is, for any x, y ∈ X , there exists a neighbourhood ∆ x of x and a neighbourhood ∆ y of y , such that ∆ x ∩ ∆ y = ∅ .
• The measure µ is called a ﬁnite measure , if it satisﬁes µ ( X ) < ∞ .
The measure ν is called a σ -ﬁnite measure , if X can be covered with at most countably many measurable sets with ﬁnite measure.
That is, there are measurable sets { A n } ∞ n =1 ⊂
A with ν ( A n ) < ∞ for all n ∈ N + , such that S ∞ n =
A
n = X .
Obviously, a ﬁnite measure is also σ -ﬁnite.
• Let X =
( −∞ , 0].
For a measure µ on the measure space (( −∞ , 0] , A ), µ is absolutely continuous with respect to the Lebesgue measure ν , if for every measurable set A , ν ( A ) =
implies µ ( A ) = 0, which is written as µ (cid:28) ν .
Denote by C 0 ( X ) the linear space of continuous functions deﬁned on X vanishing at inﬁnity.
We have the following classical representation theorem.
Theorem B.1 (Riesz-Markov-Kakutani representation theorem) .
Let X be a locally compact Hausdorﬀ space.
For any continuous linear functional ψ on C 0 ( X ) , there is a unique, regular, countably additive and signed measure µ on X , such that
[Equation]
[Equation]
with k ψ k = | µ | ( X ) .
Here, | µ | ( X ) denotes the total variation of (the signed measure) µ , which is deﬁned as | µ | ( X ) :
= sup P P ki =1 | µ ( A i ) | , where P : X =
S ki =
A i is a partition over
X with A i ∈ A for all i = 1 , 2 , · · · , k .
Proof.
Well-known, see e.g. Bogachev (2007) (CH 7.10.4).
Remark B.1.
It is straightforward to verify that | µ | ( X ) =
sup
A ∈A ( | µ ( A ) | + | µ ( A c ) | ) .
Furthermore, if µ has a density dµ/dν with respect to a countably additive, nonnegative measure ν , then we have | µ | ( X ) =
k dµ/dν k L 1 ( ν ) .
To handle signed measures, the following Jordan decomposition theorem (Bogachev, 2007) is necessary.
Theorem B.2.
Let µ be a signed measure on the measure space ( X, A ) .
Then, there are two mutually singular (non-negative) measures µ + and µ − on ( X, A ) , such that µ = µ + − µ − .
Moreover, such a pair ( µ + , µ − ) is unique.
Based on this, we have the following proposition to characterise absolutely continuous signed measures.
Proposition B.1.
If µ and ν are signed measures, then we have µ (cid:28) ν ⇔ µ + (cid:28) ν and µ − (cid:28) ν .
We also need the following Radon-Nikodym theorem (Bogachev, 2007).
Theorem B.3.
Let ( X, A , ν ) be a σ -ﬁnite measure space, and let µ be a σ -ﬁnite signed measure, such that µ (cid:28) ν .
Then there exists a unique measurable function f , such that µ ( A ) =
R A fdν for every measurable set A .
B. Proofs
Before we prove the universal approximation theorem (Theorem 4.1), we need some lemmas.
Lemma B.1.
Let { H t : t ≥ 0 } be a family of linear, continuous and regular functionals deﬁned on X .
Then there exists a integrable function ρ : [0 , ∞ )
→ R d , i.e.
[Equation]
such that
[Equation]
In particular, we have k H k = [Equation] k
[Equation]
Proof.
Obviously, ( −∞ , 0] is a locally compact Hausdorﬀ space.
For any t ≥ 0, since
H t is linear continuous, according to the Riesz-Markov-Kakutani representation theorem (Theorem B.1), there exists a unique, regular, countably additive and signed measure µ t , such that
[Equation]
with P di =1 | µ t,i | (( −∞ , 0]) =
k H t k .
We show that for any t ≥ 0, i = 1 , 2 , · · · , d , µ t,i is absolutely continuous with respect to ν (the Lebesgue measure), i.e. µ t,i (cid:28) ν .
According to Theorem B.2 and Proposition B.1, one can assume µ t,i to be non-negative without loss of generality.
Take a measurable set
A ⊂ ( −∞ , 0] with ν ( A ) = 0, the aim is to show µ t,i ( A ) = 0.
Let A
= ( −∞ , 0] \ A .
Since both A and A 0 are measurable, there exist
K n ⊂ A, K 0 n ⊂ A 0 with K n , K 0 n closed, such that µ t,i ( A \ K n ) ≤ 1 /n , µ t,i ( A 0 \ K 0 n ) ≤ 1 /n and ν ( A 0 \ K 0 n ) ≤ 1 /n for any n ∈ N + .
Fix any i ∈ { 1 , 2 , . . .
, d } , we construct the sequence of input signals { x ( n ) } ∞ n =1 as
[Equation]
which can then be continuously extended to ( −∞ , 0] by deﬁning x ( n ) s,i :=
[Equation]
We deduce that lim n →∞ x ( n ) s,i = 0 for ν -a.e.
s ≤ 0.
In fact, let S :
= { s ≤ 0 : lim n →∞ x ( n ) s,i = 0 } , we have K 0 n ⊂ S since for any s ∈ K 0 n , x ( n ) s,i = 0.
Hence, ( −∞ , 0] \ S ⊂
A ∪ ( A 0 \ K 0 n ), which gives ν (( −∞ , 0] \ S ) ≤ ν ( A )
+ ν
( A 0 \ K 0 n ) ≤ 1 /n → 0 as n → ∞ .
Due to the regularity of H t , we get lim n →∞ H t ( x ( n ) )
= 0.
By (18) and (19), we have
[Equation]
where µ t,i ( K n ) = µ t,i ( A ) − µ t,i ( A \ K n ) ∈ [ µ t,i ( A ) − 1 /n, µ t,i ( A )], and | I 1 ,n | + | I 2 ,n | ≤
R A \ K n + R A 0 \ K 0 n 1 dµ t,i ( s ) = µ t,i ( A \ K n ) + µ t,i ( A 0 \ K 0 n ) ≤ 2 /n , which gives lim n →∞ H t ( x ( n ) )
= µ t,i ( A ).
Therefore, µ t,i ( A ) = 0.
Notice that | µ t,i (( −∞ , 0]) | ≤ | µ t,i | (( −∞ , 0]) ≤ k H t k < ∞ for a.e. t ≥ 0 (since k H k = R ∞
k H t k dt < ∞ ), we get that (( −∞ , 0] , A , µ t,i ) is a ﬁnite measure space, and hence σ -ﬁnite.
Obviously, (( −∞ , 0] , A , ν ) is a σ -ﬁnite measure space.
According to the Radon-Nikodym theorem (Theorem B.3), there exists a unique measurable function ρ t,i : ( −∞ , 0] → R , such that µ t,i ( A ) =
R A ρ t,i ( s ) dν ( s ) for every measurable set A .
Hence, we have
[Equation]
with ρ ( t, s ) :
= ρ t ( − s ).
In addition, by Remark B.1, we have | µ t,i | (( −∞ , 0]) =
R
−∞ | ρ t,i ( s ) | ds , which gives
[Equation]
The proof is completed
[Equation]
Based on this representation theorem
[Equation]
the problem of functional approximation is reduced as function approximation.
That is,
[Equation] [Equation]
i.e.
[Equation]
Lemma B.2.
Let ρ ( t, s ) : [0 , ∞ )
→ R with k ρ k L 1 ([0 , ∞ ) 2 ) < ∞ .
Then for any (cid:15) > 0 , there exists a polynomial p ( u, v ) =
P mj =1 ,k =1 c jk u
j v k , such that
[Equation]
Proof.
Fix any (cid:15) > 0.
Consider the following transformation
[Equation]
This transformation preserves the norm with k ρ k L 1 ([0 , ∞ ) 2 ) =
k R k L 1 ([0 , 1] 2 ) .
First, according to the density of continuous functions in L p space (Rudin, 1987, Theorem 3.14), there exists ˜ R ∈ C ([0 , 1] 2 ), such that k R − ˜ R k L 1 ([0 , 1] 2 ) <
(cid:15)/ 2.
Next, by the density of polynomials in the space of continuous functions (Lorentz, 2005, Theorem 6), there exists a polynomial q ( u, v ) =
P mj,k =
c
jk u j v k , such that k ˜ R − q k L ∞ ([0 , 1] 2 ) < (cid:15)/ 2.
Finally, let p ( u, v ) =
uvq ( u, v ), we have
[Equation]
which completes the proof.
Now we are ready to prove the universal approximation theorem.
Proof of Theorem 4.1.
According to the representation theorem (Lemma B.1), we have
[Equation]
where k ρ k L 1 ([0 , ∞ ) 2 )
=
k H k < ∞ .
Therefore, by Lemma B.2, there exists p i ( u, v ) =
P m j,k =1 c ( i )
jk u j v k , i = 1 , 2 , . . .
, d , where m is the maximal degree of { p i } d i =1 , such that
[Equation]
Let
[Equation]
we get
[Equation] [Equation]
[Equation] [Equation]
with
[Equation]
Therefore, by (24) and (29), we have
[Equation]
which completes the proof.
C General approximation rates
In this section, the proof of Theorem 4.2 is given.
Again, by (24), the aim now is to investigate the function approximation k ρ − ˆ ρ k .
Since one can handle each spatial dimension separately (similarly with (30) and (31)), we ﬁrstly derive the estimates by assuming d = 1, and then extend the obtained results to the case of multi-dimensional inputs (for general d ∈ N + ).
Conditions on representation.
To characterise the accuracy of using the model c > e V t Me Ws u (with M := PQ ) to approximate the target ρ ( t, s ), the ﬁrst stuﬀ is to translate the conditions on the output (of piece-wise constant signals) to the representation.
Recall that y c ( t, s ) =
H t ( 1 ( −∞ , − s ] ), t, s ≥ 0, we get ρ ( t, s ) =
− dds H t ( 1 ( −∞ , − s ] ).
Hence, the assumptions on y c in Theorem 4.2 is equivalent to the following smoothness and exponential decay conditions on ρ .
That is, there exist α ∈ N + , β > 0 such that
[Equation]
[Equation]
Note that the last decay condition implies
[Equation]
for some γ > 0.
C. Basics
Let Ω ∈ R
d be a bounded set.
Deﬁne the spaces
[Equation]
some K > 0 , for all x, y ∈ Ω and | i | = α } ,
[Equation]
38) and the “norm”
[Equation]
with the shorthand k · k Ω := | · | 0 , 0 , Ω .
Theorem C.1 (Multivariate Jackson’s theorem (Schultz, 1969), Theorem 4.10) .
Let Ω ∈ R
d be a regular, 2 bounded and open set, and f ∈ C α,µ (¯Ω) for some α ∈ N , µ ∈ [0 , 1] .
Then for any n ∈ N + , we have
[Equation]
where P dn denotes the set of all polynomials with the degree of no more than n in each variable, C ( α, µ ) > 0 is a universal constant only depending on α, µ and Ω .
A commonly used case is when µ = 0.
That is, for f ∈ C α (¯Ω), we get
[Equation]
[Equation]
For any x, x 0 ∈ ¯Ω, let ˜ p
( x ) :
= p ( x ) + f ( x 0 ) − p ( x 0 ), and ˜ e ( x ) :=
f ( x ) − ˜ p
( x ).
Then ˜ p ∈ P dn , and
[Equation]
This gives the following convenient corollary.
Corollary C.1.
Let Ω ∈ R
d be a regular, bounded and open set, and f ∈ C α (¯Ω) for some α ∈ N .
Then for any n ∈ N + , there exists p ∈ P dn such that
[Equation]
where P dn denotes the set of all polynomials with the degree of no more than n in each variable, C α > 0 is a universal constant only depending on α and Ω .
C. Proofs
Now we are ready to present the proof.
Proof of Theorem 4.2.
Step 1: domain transform.
Consider the transform from the inﬁnite domain [ , ∞ ) 2 to the compact one [0 , 1] 2 :
[Equation] [Equation]
where c 0 :
= ( α + 1) /β > 0 is a ﬁxed constant.
A straightforward computation by induction shows that, for any k, l ∈ N , k + l ≤ α , and any u, v ∈ (0 , 1],
[Equation]
+ l
[Equation]
45) where C ( k, i ) , C 0 ( l, j ) are some integer constants [Equation] and ( t, s ) =
( − c 0 ln u, − c
ln
v ) is a one- to-one mapping between ( , 1] 2 and [0 , ∞ ) 2 .
By (36), we get
[Equation]
where C ( α ) > 0 is a universal constant only depending on α .
According to the decay condition (35)
[Equation]
we get
[Equation]
and similarly for u 0 , v 0 ∈ (0 , 1],
[Equation]
This gives
[Equation]
×
[Equation] [Equation] [Equation]
and
[Equation]
by (46) and (50).
Hence, R ( u, v ) ∈
C α ([0 , 1] 2 ) with bounded derivatives.
Step 2: polynomial approximation.
According to Corollary C.1 and (51), we obtain that there exists ˜ R n ∈ P 2 n , such that [Equation]
[Equation]
where C ( α ) > 0 is a universal constant only related to α .
Furthermore, let ˆ R n ( u, v ) :
= ˜
R n ( u, v ) − ˜ R n ( u, 0) − ˜ R n (0 , v ) + ˜ R n (0 , 0), we get ˆ R n ∈ P 2 n with ˆ R n ( u, 0)
=
ˆ R n (0 , v ) = 0 for any u, v ∈ [0 , 1].
By (50), we have R ( u, v ) =
for any ( u, v ) ∈ [0 , 1] × { 0 } ∪ { 0 } × [0 , 1], then
[Equation]
i.e. we can further require the approximator satisfying the zero half-boundary condition (i.e. vanishing on ( u, v ) ∈ [0 , 1] × { 0 } ∪ { 0 } × [0 , 1]) without eﬀecting the approximation accuracy.
Let ¯ m :=
min { m E , m D } , we get
[Equation]
(54) where ˜ R :=
˜ R ¯ m ∈ P 2¯ m , ˜ R ( u, v ) :
= ¯ m X i = ¯ m X j = ˜ r ij
[Equation]
v
[Equation]
Let
[Equation]
then by (24) and (54) [Equation] we have
[Equation]
That is, one can achieve an approximation accuracy scaling like (1 / ¯ m ) α with ¯ m 2 parameters [Equation]
The proof is completed.
Remark C.1.
The extension to multi-dimensional inputs (general d ∈ N + ) is found in the last paragraph of Appendix D.2.
Remark C.2.
Recall M = PQ ∈
R m D × m E with P ∈ R m D × N , Q ∈ R N × m E , we only need to investigate the case of N ≤ min { m E , m D } = ¯ m , since rank( M ) ≤ ¯ m .
D Approximation rates via temporal product structure
In this section, the proof of Theorem 4.3 is provided.
D. Proper Orthogonal Decomposition
Proper orthogonal decomposition (POD; (Liang et al., 2002), (Berkooz et al., 1993), (Chatterjee, 2000)) is a method for model reduction, which is commonly applied to numerical PDEs and ﬂuids mechanics.
It can be viewed as an extension of singular value decomposition (SVD) and principal component analysis (PCA) to inﬁnite-dimensional spaces.
Fix any R ∈ L ∞ ([0 , 1] 2 ).
3 Deﬁne the POD operator
[Equation]
Proposition D.1.
The operator K is linear, bounded, compact, self-adjoint and nonnegative.
Proof.
(i)
The linearity is obvious.
(ii) Let
[Equation]
then
[Equation]
By the Cauchy-Schwartz inequality, we get
[Equation]
which gives
[Equation]
[Equation]
Note that R ( u, v ) ∈ L ∞ ([0 , 1] 2 ) ⊂
L 2 ([0 , 1] 2 ), hence both K and R are bounded operator from L 2 [0 , 1] to itself.
(iii)
It is well-known that the Hilbert–Schmidt integral operator
[Equation]
is a compact operator from L 2 [0 , 1] to itself.
Therefore
[Equation]
which gives K = CR
Since R is bounded and C is compact, we get K is also compact.
(iv) By Fubini’s theorem, it is straightforward to verify that
[Equation]
[Equation]
(v) By Fubini’s theorem, it is straightforward to verify that
[Equation]
The proof is completed.
Combining (i)—(iv) and applying Hilbert–Schmidt’s expansion theorem, we obtain that
L 2 [0 , 1] has an orthonormal basis { φ n } n ∈N S { ψ ξ } ξ ∈ Ξ , such that
• K φ n = λ n φ n , λ n 6 = 0 for n ∈ N , and K ψ ξ = 0 for ξ ∈ Ξ, where N is a ﬁnite or countable set.
If N is not ﬁnite, we have lim n →∞ λ n = 0; • For any ψ ∈ L 2 [0 , 1], we have
[Equation]
where the second summation has at most countable non-zero terms, and
[Equation]
Here, all the series converge under the norm k · k L 2 [0 , 1] .
Without loss of generality, N = { 1 , 2 , · · · , N 0 } for N 0 ∈ N + or N 0 = + ∞ (i.e. N = N + ).
By (69), we get
[Equation]
i.e. all the eigenvalues of K are non-negative, and λ n 6 = 0 for n ∈ N implies λ n > 0, ∀ n ∈ N .
In addition, lim n →∞ λ n = 0 implies that one can index all the eigenvalues in a non-increasing sequence: λ 1 ≥ λ 2 ≥ · · · ≥ λ n ≥ · · · ≥ 0.
Then, we can present the POD estimate.
Theorem D.1.
For any R ∈ L ∞ ([0 , 1] 2 ) , we have
[Equation]
Proof.
Combining (69) and (72) gives
[Equation]
[Equation]
Similarly,
Z
[Equation]
which gives
[Equation]
Notice that R u ( v ) :
= R ( u, v ) ∈ C α [0 , 1] ⊂ L 2 [0 , 1] for any u ∈ [0 , 1].
By (70) and (76), we get
[Equation]
ξ ∈ Ξ h
[Equation]
ξ
[Equation]
L
[Equation]
[Equation]
[Equation]
ξ ∈ Ξ [Equation] L [Equation]
[Equation]
[Equation]
ξ
[Equation]
L
[Equation] [Equation] [Equation] ξ
[Equation]
Ξ
[Equation]
Hence for any N ∈ N , we have
[Equation]
[Equation]
(79) where the summation is zero by convention if the subscript is larger than the superscript.
This by (74) implies
Z
[Equation]
Here, the equality holds as a consequence of Beppo Levi’s monotone convergence lemma and Lebesgue’s dominated convergence theorem, and one has P n ∈N λ n < + ∞ .
In fact, for
N 0 = + ∞ , let S n = P nk =1 λ k , we get S n increasing (since λ k ≥ 0 for all k ∈ N ).
By (74) and (78), we have
[Equation]
which gives that S n converges as n → ∞ .
The proof is completed.
Remark D.1.
Let ϕ n ( u ) :
= h R ( u, v ) , φ n ( v ) i L 2 [0 , 1] , then we have the POD estimate
R ( u, v ) ≈ P n ϕ n ( u ) φ n ( v ) , where the error is characterised by the tail sum of eigenvalues of the POD operator.
Recall the POD operator deﬁned in (61).
We similarly deﬁne
[Equation]
where ˜ R is deﬁned as (55), i.e. the approximator constructed in the general approximation theorem before.
Obviously, as a polynomial, ˜ R ∈ L ∞ ([0 , 1] 2 ).
Hence, by Proposition D.1, ˜
K is also linear, bounded, compact, self-adjoint and non-negative.
In addition, according to
Theorem D.1, we have the following POD estimate
[Equation]
v ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
L 2 [0 , 1] du = ˜ N 0 X n = N + ˜ λ n , (83) where { ˜ λ n } ˜ N 0 n =1 are eigenvalues of ˜ K satisfying ˜ λ
1 ≥ ˜ λ 2 ≥ · · · ≥ ˜ λ n ≥ · · · ≥ 0, and { ˜ φ n } ˜ N 0 n =1 ⊂
L 2 [0 , 1] are the corresponding orthonormal eigenfunctions, i.e. ˜ K ˜ φ n = ˜ λ n ˜ φ n , ˜ λ n > 0 for n ∈ { 1 , 2 , · · · , ˜ N 0 } .
Lemma D.1.
˜ K is a ﬁnite-rank operator.
That is, ˜ N 0 ≤ ¯ m = min { m E , m D } < ∞ .
Proof.
Let ˜ ϕ n ( u ) :
= (cid:10)
˜ R ( u, v ) , ˜ φ n ( v ) (cid:11) L 2 [0 , 1] .
We ﬁrst show that both ˜ ϕ n ( u ) and ˜ φ n ( v ) are polynomials.
In fact, since ˜ R ( u, v ) =
P ¯ mi =1 P ¯ mj =1 ˜ r ij u i v j , we have
[Equation]
∂ k ∂u k ˜
[Equation]
[Equation]
X
[Equation]
≤ ¯ m X i =
k ¯ m X j = | ˜
r ij | i !
( i − k )! (cid:44) [Equation] with the convention that the summation is zero if the subscript is larger than the superscript, i.e. C 1 ( k, ¯ m ) =
for any k > ¯ m .
Let C 1 ( ¯ m ) :
= max {k ˜ R k L ∞ ([0 , 1] 2 ) , max 1 ≤ k ≤ ¯ m C 1 ( k, ¯ m ) } , then we have
[Equation]
According to Lebesgue’s dominated convergence theorem, we get by induction that
[Equation]
k
[Equation]
[Equation]
Similarly, we get [Equation] (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ¯ m X i = ¯ [Equation]
X
[Equation]
k
[Equation]
(cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ ¯ m X
i = ¯ m X j = k | ˜
r ij | j
[Equation]
and C 2 ( k, ¯ m ) =
for any k > ¯ m .
Let C 2 ( ¯ m ) :
= max {k ˜ R k L ∞ ([0 , 1] 2 ) , max 1 ≤ k ≤ ¯ m C 2 ( k, ¯ m ) } , then for k = 0 , 1 , · · · , we have
[Equation]
According to Lebesgue’s dominated convergence theorem, we get by induction that
[Equation]
That is, ˜ ϕ n , ˜ φ n ∈ C ∞ [0 , 1] for any n = 1 , 2 , · · · , ˜ N 0 .
Since ˜ R ( u, 0) =
˜ R (0 , v ) = 0 for u, v ∈ [0 , 1], we get ˜ ϕ n (0) =
˜
φ n (0) = 0.
Furthermore, we have d k du k ˜ ϕ
n ( u ) =
d k dv k ˜ φ n ( v ) =for k > ¯ m , hence ˜ ϕ
n , ˜ φ n ∈ P ¯ m .
Since { ˜ φ n } ˜ N 0 n =1 ⊂ L 2 [0 , 1] are orthonormal, we must have ˜
N 0 ≤ ¯ m < ∞ .
The proof is completed.
D. Approximation rates
Perturbation of eigenvalues.
First, we need to bound the gap between the eigenvalues { ˜ λ n } ˜ N 0 n =1 and { λ n } N 0 n =1 (corresponding to the function R deﬁned in (44)).
The following theorem is necessary.
Theorem D.2 (Courant–Fischer–Weyl min-max principle; Lax (2002)
(Chapter 28, Theorem 4)) .
Let B be a compact, self-adjoint operator on a Hilbert space H , whose positive eigenvalues are listed in a decreasing order µ 1 ≥ µ 2 ≥ · · · ≥ µ k ≥ · · · > 0 .
Then
[Equation]
where S k ⊂ H is any k -dimensional linear subspace.
Based on it, we have the following lemma to characterise the perturbation of singular values.
Lemma D.2.
For any R 1 , R 2 ∈ L ∞ ([0 , 1] 2 ) , we have the estimate
[Equation]
Proof.
According to Theorem D.2 and by (69), we have
[Equation]
Note that R R 1 − R R 2 = R R 1 − R 2 and by (64), we have
[Equation]
and similarly,
[Equation]
which completes the proof
[Equation]
Proofs.
Now we are ready to derive the ﬁnal estimate.
Proof of Theorem
[Equation]
By Lemma D.2 and (54)
[Equation]
we get
[Equation]
Combining (54)
[Equation]
(83) and (95) gives that
[Equation]
[Table]
[Equation]
vuut
[Equation]
X n
[Equation]
N
[Equation]
[Equation]
m
[Equation]
vuut ˜ N 0 X n
[Equation]
N + | ˜ λ n − λ n |
[Equation]
˜ N
[Table]
X n = N
[Table]
λ n
[Equation]
¯ m
[Equation]
+ v u ut ˜ N 0 X n = N + λ n + v u ut ˜ N 0 X n = N + (cid:12) (cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)q
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
where (cid:46) hides universal positive constants.
For the corresponding parameters, recall that ˜ ϕ n , ˜ φ n ∈ P ¯ m with ˜ ϕ n (0) =
˜
φ n (0) = 0, we can write ˜
[Equation]
for any n = 1 , 2 , · · · , ˜ N 0 .
Let
[Equation]
M = PQ with P =
[ P in ] ∈ R ¯ [Equation]
[Equation]
then we have
[Equation]
Plugging this into (96) gives
[Equation]
and the number of trainable parameters is 2 N ¯ m .
Together with Lemma D.1, the proof is completed.
Extension to multi-dimensional inputs.
The above results can be naturally extended to the general case where a d -dimensional input is given ( ∀ d ∈ N + ).
In fact, let
[Equation]
104) for some 0 < (cid:15) (cid:28) 1, then we take
[Equation] [Equation]
106) and have
[Equation]
If the form PQ (= M ) is required, it is suﬃcient to take M i
= P
i Q
i , i = 1 , 2 , · · · , d , and
P =
( P 1 , P 2 , · · · , P d ) ∈ R ¯ m × dN , Q = diag( Q 1 , Q 2 , · · · , Q d ) ∈ R
dN ×
d ¯ m .
Therefore, we obtain
[Equation]
, (109) with the number of parameters increased by d -times compared to the corresponding onedimensional setting.
D. Case analysis
Bounds under diﬀerent cases.
Now we make the comparison between (102) and (58).
Recall that { λ n } N 0 n
=1 is a positive decreased sequence (with lim n →∞ λ n = 0 and P ∞ n =1 λ n ≤ k R k 2 L 2 ([0 , 1] 2 ) by (81), if N 0 = + ∞ ), and ˜ N 0 ≤ ¯ m , we have the following cases.
• if ˜ N 0 =
o ( ¯ m ), we set N = ˜ N 0 in (102) and get the same bound as (58), but the number of parameters is only O ( ˜ N 0 ¯ m )
=
o ( ¯ m 2 ); • if ˜ N
= O ( ¯ m ), then (102) implies that
[Equation]
We are supposed to require that
[Equation]
We give the following typical examples to illustrate suﬃcient conditions to guarantee (111).
– If λ n = O ( n − r ) with r > 2 α + 1 ≥ 3 ( α ∈ N + ), since ¯ m X n = N +
[Equation]
112) we get by (111) that
[Equation]
Assume that N ∼ ¯ m δ with δ ∈ [0 , 1), then by (113) and (114), we require δ ≥ max { 2 α − 1 r − 1 , α − 1 r 2 − 1 } , i.e.
[Equation]
Meanwhile, the POD-estimate (110) achieves an accuracy scaling like (1 / ¯ m )
α −with O
( ¯ m 1+ δ ) parameters, while under the same capacity, the accuracy of (58) scales like (1 / ¯ m ) α (1+ δ ).
The former beats the latter if α − 12 > α (1+ δ ), i.e. δ < 1 − 1 /α ( α ≥ 2).
When δ = 1 − 1 /α , (115) becomes r ≥ max { 2 α 2 − 1 α − 1 , 2( α + 1) }
=
α 2 − 1 α − 1 .
That is to say, r ∗ := 2 α 2 − 1 α − 1 can be viewed as an upper bound of the critical point where the two estimates are comparable.
When r > r ∗ , the POD-estimate outperforms the non-POD-estimate, and this eﬀect gets more notable with r increasing.
In fact, when r > r ∗ , we take N = ¯ m 2 α − 1 r − 1 > ¯ m α − 1 r 2 − 1 (hence satisfying (113) and (114)), which gives an
O ((1 / ¯ m ) α − 12 ) approximation error with O ( ¯ m 1+ 2 α − 1 r − 1 ) parameters for the PODestimate, while O ( ¯ m 2 − 1 α ) trainable parameters are needed to achieve the same accuracy using the non-POD-estimate.
Since 2 α − 1 r − 1 < 1 − 1 α , we get that the
POD-estimate outperforms the non-POD-estimate.
When r → + ∞ , the number of trainable parameters for the POD-estimate is O ( ¯ m ), much better than the non-POD-estimate.
Remark D.2.
Let
[Equation]
we get
[Equation]
Recall that R ∈ C
α
[0 , 1] 2 ) ⊂
L ∞ ([0 , 1] 2 ) , we get K ∈ L ∞ ([0 , 1] 2 ) ⊂
L 2 ([0 , 1] 2 ) , and hence K can be also viewed as a Hilbert-Schmidt integral operator with the kernel K , where K is obviously symmetric as K ( v, w )
=
K ( w, v ) , and positive deﬁnite since 0 ≤ hK φ, φ i L 2 [0 , 1]
[Equation]
φ
[Equation]
φ
[Equation]
[Equation]
1] by (69).
Since the derivatives of R (up to α -order) are continuous on [0 , 1](hence bounded and integrable), we get K ∈ C
α,α ([0 , 1] 2 ) ( α -diﬀerentiable for both arguments).
According to Chang & Ha (1999) (Theorem 1), we have
[Equation]
That is to say, this general setting (only assume smoothness of the kernel) can not guarantee the suﬃcient condition provided here ( λ n = O ( n − r ) with r > 2 α 2 − 1 α − 1 ), i.e. the point that the POD-estimate outperforms the non-PODestimate requires a faster decay of the eigenvalues. – If λ n = O ( e − ωn
) with ω > 0, we get by 111) that
[Equation]
That is to say, for any α ∈ N + , ω > 0, we have N ∼ (2 α − 1)
ln ¯ m .
This implies an O ((1 / ¯ m ) α − 12 ) approximation error with O ( ¯ m ln ¯ m ) parameters for the POD-estimate, while O ( ¯ m 2 − 1 α ) parameters are needed to achieve the same accuracy using the non-POD-estimate.
– If N 0 < + ∞ (i
e. K is a ﬁnite rank operator by (71)), one can just take N = Nand get by (110) an O ((1 / ¯ m )
α − 12 ) approximation error.
For ¯ m ∈ N + suﬃciently large such that ¯ m ∼ N κ 0 for some κ (cid:29) 1, the number of trainable parameters for the POD-estimate is O ( N ¯ m ) =
O ( N κ +1), while O (cid:16) N κ (2 − 1 α )(cid:17) parameters are needed to achieve the same accuracy using the non-POD-estimate.
Obviously, if α ≥ 2, we get κ (2 − 1 α ) ≥ 32 κ (cid:29) κ + 1.
Eigenvalue approximation.
One can apply (91) in Lemma D.2 to estimate the eigenvalues { λ n } N 0 n =1 , where R 1 = R and R 2 is taken as some approximator of R , say ˆ R .
Let ˆ λ :=
λ ˆ R , we recall (91) as
[Equation]
We provide a naive method here.
That is, one can take ˆ R as a piece-wise constant approximation of R .
Fix any n ∈ N + .
Let I 0 :
= { 0 } , I i :=
( i − 1 n , in ] for i = 1 , 2 , · · · , n , then {I i ×I j } n i,j =0 is the uniform partition over [0 , 1] 2 .
Let ˆ R ( u, v ) :
= P ni,j =0 R ( i n , j n ) 1 I i ×I j ( u, v ).
Then for any φ ∈ L 2 [0 , 1], if w ∈ I k , k = 0 , 1 , · · · , n , we have ( K ˆ R φ )( w ) =
Z
Z ˆ R ( u, v ) φ ( v ) dv · ˆ R ( u, w ) du =
n X i = n X j = n X {+i 
+}
= n X j 0 =
R (cid:18) i n, j n (cid:19)
R (cid:18) in, jn (cid:19)
Z
Z0 1 I i ×I j ( u, v ) φ ( v ) dv · 1 I i 0 ×I j 0 ( u, w ) du = n X i = n X j =
R (cid:18) i n, j n (cid:19)
R (cid:18) i n, k n (cid:19)
Z
I i
Z
I j
I i ×I j ( u, v ) φ ( v ) dv · 1 I i ×I k ( u, w ) du = n X i = n X j =
R (cid:18) i n, j n (cid:19)
R (cid:18) i n, k n (cid:19) ·n
Z
I j φ ( v ) dv, (122) which is a constant only related to k .
That is, K ˆ R φ is a piece-wise constant function, i.e. K ˆ R φ ∈ span { 1 I k } nk =0 , or range( K ˆ R ) ⊂ span { 1 I k } nk =0 .
Obviously, { 1 I k } nk =0 is an orthogonal set, which implies that the operator K ˆ R is of ﬁnite rank at most n + 1.
Let ˆ R ij := 1 n R ( i − 1 n , j − 1 n ), i, j = 1 , 2 , · · · , n + 1, { σ k } n +1 k
=1 be the singular value of ˆ R :=
[ ˆ R ij ] ∈ R ( n +1) × ( n +1) and V ∈ R ( n +1) × ( n +1) with columns as the corresponding right singular vectors.
Set [ e 1 , e 2 , · · · , e n +1 ] :=
[ 1 I 0 , 1 I 1 , · · · , 1 I n ] V , then by (122), we get for l = 1 , 2 , · · · , n + 1, [Equation]
[Equation]
which gives K
[Equation]
i.e. the set { σ 2 k } n +1 k =1 collects the all the eigenvalues of K ˆ R , which can be obtained by the
SVD of ˆ R .
For the error estimate, it is straightforward to have
[Equation]
= [Equation] X i [Equation]
X
[Equation]
Z in i − 1 n
Z jn j − 1 n (cid:12)(cid:12)(cid:12)(cid:12) R ( u
[Equation]
v ) [Equation] n (cid:19)(cid:12)(cid:12)(cid:12)(cid:12) dudv ≤ n [Equation]
in i − 1 n
[Equation]
· (cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
[Equation]
i
[Equation] (cid:19)(cid:13)(cid:13)(cid:13)(cid:13)  dudv ≤ n X [Equation] (125) hence by (121), (cid:12)(cid:12) √ λ k − σ k (cid:12)(cid:12) ≤ 2 C ( α ) γ n for any n ∈ N + .
E Numerical settings
According to Lemma B.1, the target input-output temporal relationship has a Riesz representation form.
Under the discrete-time regime, we are supposed to set
H t ( x ) =[-
-]
T X s =
ρ ( t, s ) x s , (126) where T ∈ N + is the path length.
For the input, we generate 6 sequences using Gaussian i.i.d. random variables with the path length T = 30. Hence, the output (target) is H t ( x ) =
P 30 s =1 ρ ( t, s ) x s .
The high rank target has the representation
[Equation]
while the low rank target has the representation
[Equation]
E. Settings of Figure
Consider the target with the following representation
[Equation]
) where
R
[Equation]
with both { ϕ k } and { φ k } as orthonormal bases.
In this way, we construct a target with singular values { σ k } .
Under the discrete-time setting, we are supposed to use the following linear, width- m RNN encoder-decoder
[Equation]
131) where P ∈ R m × N , Q ∈ R N × m with m = m D = m E .
Recall that the approximation error is derived as
[Equation]
where ˆ R ( u, v ) :
=
P N 0 i =1 ˜ ϕ
i ( u )˜ φ i ( v ) ∈
P 2 m with ˜ ϕ n , ˜ φ n ∈ P m .
In the numerical experiments, we ﬁrst construct an R ( u, v ), and then ﬁt it with the polynomial ˆ R ( u, v ) using the method of least squares.
The norm k R − ˆ R k L 2 ([0 , 1] 2 ) is used to evaluate the approximation error.
In the particular example reported in Figure 2, we set m = 128, ϕ n ( u ) = √ 2 sin( nπu ) and φ n ( v ) = √ 2 sin( nπv ).
The singular values are taken as: (a) σ n = (cid:26) n − 18 , n ≤ N 0 0 , n > N 0 ; (b) σ n = (cid:26) n − 1 , n ≤ N 0, n > N 0 ; (c) σ n = n − 2 , with N 0 = 2 , 4 , 6 , 8.
As an inﬁnite sum, R is constructed under a ﬁnite truncation with the ﬁrst 50 terms.
E. Settings of Figure
We perform experiments on nonlinear targets to show that the insight of low rank approximation also holds in the nonlinear case.
Nonlinear target. We consider the forced Lorenz 96 system (Lorenz, 1996), which is an important example of reduced order modelling for convection dynamics, with applications in weather forecasting.
Mathematically, the system has K output variables { y k } and JK hidden variables { z j,k } with k = 1 , 2 , . . .
, K and j = 1 , 2 , . .
, J .
The parameters K, J control the number of variables in the system, and can be viewed as a complexity measure.
The input { x k } is an external temporal forcing.
The system satisﬁes the following dynamics
[Equation]
with cyclic indices y k + K = y k , z j,k + K
=
z j,k and z j + J,k
=
z j,k .
Here, we take { x k } as randomly generated input sequences with the path length 64.
We have tested for several cases with diﬀerent parameters: i) J = 6 with K = 1 , 5 , 10 , 20; ii) K = 5 with J = 5 , 15 , 25 , 100. Note that the forced Lorenz 96 system parameterizes a highly nonlinear functional.
Nonlinear model.
We learn the above system using RNN encoder-decoders with nonlinear activations, i.e.
[Equation]
where σ is the element-wise tanh activation.
Let m =
128 be the hidden dimension, N = 1 , 2 , . . .
, 32 be the size of the coding vector v , we have x s , o t , b O ∈ R K , h s , b E , b D , b 2 ∈ R m ,
W E , W D ∈ R m × m , b 1 ∈ R N , W O ∈ R m × K , Q ∈ R m × N , P ∈ R N × m .
Note that we construct the model with a ﬁxed hidden dimension m but diﬀerent N , thus only sizes of Q, P, b 1 , b 2 are varying, while sizes of other parameters remain unchanged. Training and initialisation.
We denote the model with the coding vector size N as EncDec ( N ) .
We utilise the Adam optimiser and train from EncDec (1) to EncDec (32) .
For
EncDec (1) , we use a normal random initialisation, and train for 3000 epoches until a stable error.
For EncDec ( N ) with N > 1, we use the parameters trained from EncDec ( N − 1) as the initialisation.
For the parameters Q, P, b 1 , b 2 , we pad them to match the size of EncDec ( N ) with normal distributions as initialisations.
It is shown that the low rank approximation phenomena discovered in the linear setting also appears in this nonlinear case.