E NCODING RECURRENCE INTO T RANSFORMERS
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper novelly breaks down with ignorable loss an RNN layer into a sequence of simple RNNs, each of which can be further rewritten into a lightweight positional encoding matrix of a self-attention, named the Recurrence Encoding Matrix (REM).
Thus, recurrent dynamics introduced by the RNN layer can be encapsulated into the positional encodings of a multihead self-attention, and this makes it possible to seamlessly incorporate these recurrent dynamics into a Transformer, leading to a new module, Self-Attention with Recurrence (RSA).
The proposed module can leverage the recurrent inductive bias of REMs to achieve a better sample efﬁciency than its corresponding baseline Transformer, while the self-attention is used to model the remaining non-recurrent signals.
The relative proportions of these two components are controlled by a data-driven gated mechanism, and the effectiveness of RSA modules are demonstrated by four sequential learning tasks.
I NTRODUCTION
Sequential data modeling is an important topic in machine learning, and the recurrent networks such as LSTM (Hochreiter and Schmidhuber, 1997) and GRU (Chung et al., 2014) have served as the benchmarks in this area over a long period of time.
The success mainly contributes to the variety of recurrent dynamics introduced by these models, referred to as the recurrent inductive bias.
More speciﬁcally, the dependence between any two inputs can be described by a parametric form, which heavily depends on their relative temporal locations.
However, the recurrent models are well known to suffer from two drawbacks.
The ﬁrst one is the gradient vanishing problem (Hochreiter et al., 2001), i.e. the recurrent models have difﬁculty in depicting the possibly high correlation between distant inputs.
This problem cannot be solved fundamentally by the recurrent models themselves, although it can be alleviated to some extent, say by introducing long memory patterns (Zhao et al., 2020).
Secondly, the sequential nature renders these models difﬁcult to be trained in parallel (Vaswani et al., 2017).
In practice, many techniques were proposed to improve the computational efﬁciency of recurrent models, while they all come with compromises (Luo et al., 2020; Lei et al., 2017).
In recent years, Transformer models (Vaswani et al., 2017) have been revolutionizing the ﬁeld of natural language processing by achieving the state-of-the-art performance on a wide range of tasks, such as language modeling (Kenton and Toutanova, 2019), machine translation (Dai et al., 2019) and text summarization (Liu and Lapata, 2019), etc.
They have also demonstrated a great potential in other types of sequence learning problems, for instance, time series forecasting (Zhou et al., 2021;
Li et al., 2019).
The success of Transformers is due to the fact that the similarity (or dependence) between any two tokens (or inputs) is well taken into account (Vaswani et al., 2017), and hence they can model long range dependence effortlessly.
Moreover, contrary to the recurrent models, the self-attention mechanism in Transformers is feed-forward in nature, and this enables them to be computed in parallel on the GPU infrastructure (Vaswani et al., 2017).
However, the ﬂexibility also leads to sample inefﬁciency in training a Transformer, i.e. much more samples will be needed to guarantee a good generalization ability (d’Ascoli et al., 2021).
Moreover, the chronological orders are usually ignored by Transformers since they are time-invariant, and some additional efforts, in the form of positional encoding, will be required to further aggregate the temporal information (Shaw et al., 2018; Vaswani et al., 2017; Dai et al., 2019).
In short, both recurrent and Transformer models have the pros and cons in modeling sequential data.
On one hand, due to inductive bias, the recurrent models excel at capturing the recurrent patterns even with relatively small sample sizes; see Figure 1(a).
Meanwhile, sample size is the bottleneck for
Transformers and, when there are sufﬁcient samples, Transformers are supposed to be able to depict any recurrent or non-recurrent patterns in the data; see Figure 1(b).
On the other hand, sequential
With multiple heads, the assorted REMs can replicate the temporal dynamics of an RNN layer. data have recurrent patterns more or less, and Transformers may have an improved performance if the recurrent model can be involved to handle these recurrent patterns, especially when the sample size is relatively small. Speciﬁcally, if the recurrent and non-recurrent components are separable, one then can apply a parsimonious recurrent model on the recurrent component and a Transformer on the non-recurrent one, and the sample efﬁciency can be improved comparing to the Transformer-only baseline; see illustration in Figure 1(c).
There have been various attempts in the literature to combine the two models.
Merity (2019) used the self-attention mechanism of a Transformer to process the memory from an LSTM network.
Chen et al.
(2018) mixed and matched a Transformer’s encoder with an recurrent-based decoder.
Hao et al.
(2019) introduced an additional recurrent encoder to a Transformer, while Wang et al.
(2019) stacked a recurrent layer prior to the multihead self-attention.
However, the above approaches combine both types of models in a straightforward manner, and hence they inherit both the aforementioned shortcomings of Transformer and recurrent models.
In particular, the computationally expensive sequential operation in the recurrent models limits the attractiveness of these proposals.
In an attempt to simplify the numerical calculation of RNNs, we found surprisingly that an RNN layer with linear activation can be broken down into a series of simple RNNs with scalar hidden coefﬁcients.
Each simple RNN induces a distinct recurrent pattern, and their combination forms the recurrent dynamics of the RNN layer.
Hence the calculation time can be greatly reduced by training these simple RNNs in parallel.
On top of that, it can be equivalently rewritten into the positional encodings of a multihead self-attention (MHSA).
This spontaneously inspires a solution, the multihead Self-Attention with Recurrence (RSA), to combine a Transformer with an RNN while maintaining parallel computation.
This solution enables our design to preserve the merits from both
Transformer and recurrent models, while their respective shortcomings are avoided.
Our paper makes three main contributions below.
With ignorable approximation loss, we demonstrate that an RNN layer with linear activation is equivalent to a multihead self-attention (MHSA); see Figure 2.
Speciﬁcally, each attention head can be used to recover a type of recurrent dependence structures, and multiple heads working in sync replicate the complex temporal dynamics of the RNN layer.
In the above MHSA, recurrent dynamics of the RNN layer are encapsulated entirely into the positional encodings, which we name the Recurrence Encoding Matrices (REMs).
This makes it possible to add the REMs to any existing Transformer architecture that has the self-attention mechanism, leading to the Self-Attention with Recurrence (RSA) module; see Figure 1(d).
A gated mechanism is used to control the proportion of REMs, and the gate value may also be interpreted as the proportion of the recurrent signals in the data.
Our experiments on four sequential tasks demonstrate that the proposed RSA module can effectively enhance the baseline Transformers’ forecasting power.
Moreover, from the ﬁtted gate values, we have an interesting ﬁnding that time series data have stronger recurrent signals than those of the regular languages, while the code or natural languages have much weaker recurrent signals.
1. R ELATED WORKS
Adding recurrence to Transformers Feedback Transformer (Fan et al., 2021) introduces the memory vectors to aggregate information across layers, and uses them to update the next token in a recursive manner.
Other papers provide alternative ways to think of recurrence for Transformer models.
One example is the Universal Transformer (Dehghani et al., 2018), which considers recurrence along depth.
Speciﬁcally, it shares the parameters between consecutive layers in the encoder or decoder block.
Another inﬂuential line of research applies the recurrent models and the self-attention to learn coarse-grained and ﬁne-grained temporal information of a sequence, respectively.
For instance, the Transformer-XL (Dai et al., 2019) introduces a recurrent state between sequentially ordered segments in consecutive batches.
Temporal Latent Bottleneck (TLB) (Didolkar et al., 2022) and Block-Recurrent Transformer (BRT) (Hutchins et al., 2022) further divide each segment into smaller chunks. Then, information within each chunk is summarized into a few state vectors, which are updated in a recurrent-style. Different from these works, the proposed RSA uses both self-attention and recurrent networks to model the dependence between individual tokens, i.e. it explores recurrence from different perspectives and temporal granularity; see illustration in Figure 5.
As a result, the
RSA can be incorporated into these architectures to further improve the performance, and we use Transformer-XL, TLB and BRT as our baseline models for the natural language modeling tasks in Section 4.3.
The proposed REMs can provide complementary positional encoding in the RSA module.
Essentially, its recurrent pattern can convey relative location information.
This property is inherited from the distance-awareness of the RNN.
Compared to the relative positional encoding (RPE) in the existing literature, it is more parsimonious and interpretable.
The fully learnable RPE proposed by Shaw et al.
(2018) requires O ( T 2 d ) parameters, where T is the number of tokens and d is the token dimension.
Later, different variants are introduced by Huang et al.
and Dai et al.
(2019), which reduce the number of parameters to O
( Td ) and O ( d ) , respectively.
In contrast, our REMs have parametric forms of merely one or two parameters.
Time series forecasting Recent literature sees a surge in deep learning models designed specially for time series forecasting tasks, which are not Transformer-based.
For instance, DLinear (Zeng et al., 2022) and Autoformer (Wu et al., 2021) decompose the series into trend and non-trend components.
Autoformer and SCINet (Liu et al., 2021) further exploit the temporal dependence structure of time series via the autocorrelation function and multi-level convolution operations, respectively.
FiLM (Zhou et al., 2022a) and FEDformer (Zhou et al., 2022b) transform the series to the frequency domain to capture meaningful signals while removing noises.
This paper, however, differs from these literature in both target and method.
We aim to add the recurrent inductive bias to general-purpose
Transformers, and hence its performance rely heavily on the baseline Transformer that it modiﬁes.
R ELATIONSHIP BETWEEN RNN AND MULTIHEAD SELF - ATTENTION
This section demonstrates that, without loss of much generality, an RNN layer can be approximated by a series of simple RNNs with scalar (hidden) coefﬁcients, which can be further represented in the form of a multihead self-attention.
2. B REAKING DOWN AN RNN LAYER
Consider an RNN layer with the input variables { x t ∈ R d in , 1 ≤ t ≤ T } , and it has the form of h t
=
g ( W h h t − 1 + W x x t + b ) , where g ( · ) is the activation function, h t ∈ R
d is the output or hidden variable with h 0 = 0 , b ∈ R d is the bias term, W h ∈ R d × d and W x ∈ R
d
×
d in are weights.
When the activation function is linear, i.e. g ( x ) = x , the RNN becomes
[Equation]
, or equivalently
[Equation]
where the bias term b is suppressed for simplicity.
Although it has a feedforward form, the RNN cannot be trained in parallel, and this is mainly caused by the recurrent weights W h .
This section makes an effort to block diagonalize W h such that the RNN at (1) can be broken down into a sequence of simple RNNs with scalar (hidden) coefﬁcients.
Lemma 1 (Theorem 1 in Hartﬁel (1995)) .
Real matrices with R distinct nonzero eigenvalues are dense in the set of all d × d real matrices with rank at most R , where 0 < R ≤ d .
Suppose that the weight matrix W h has rank R ≤ d .
By Lemma 1, without loss of much generality, we can assume that the nonzero eigenvalues of W h are all distinct.
Speciﬁcally, W h has r real nonzero eigenvalues λ 1 , . . .
, λ r , and s pairs of complex nonzero eigenvalues λ r +1 , . . .
, λ r +2 s , where ( λ r +2 k − 1 , λ r +2 k )
= ( γ k e iθ k , γ k e − iθ k ) for 1 ≤ k ≤ s , i represents the imaginary unit, and R = r + 2 s .
As a result, from Horn and Johnson (2012), we have the Jordan decomposition in real form,
W h
=
BJB − 1 , where B ∈ R d × d is invertible and
J ∈ R
d × d is a block diagonal matrix.
It holds that W jh = BJ j B − 1 for all j ≥ 1 , and we can then break down the recurrence induced by W h into that of the p × p block matrices in J with p = 1 or 2.
Similar to (1), we deﬁne three types of RNNs with linear activation below,
[Equation]
and h C2 t ( γ, θ ) =
(cid:80) t − 1 j =1 γ j sin( jθ ) W C2 x x t − j , where the ﬁrst one corresponds to the real eigenvalues, i.e. the 1 × 1 block matrices in J , while the last two correspond to the complex eigenvalues, i.e. the 2 × 2 block matrices in J .
Note that each of the three RNNs has the recurrent weights of λ or ( γ, θ ) , and its form with a nonlinear activation function is given in the Appendix.
Proposition 1.
Suppose that W h has rank R = r
+ 2 s ≤ d , and its eigenvalues are deﬁned above.
Let h 0 ,t
= W x x t , and then the RNN with linear activation at (1) can be equivalently rewritten into
[Equation]
2. A N EQUIVALENT MHSA REPRESENTATION
Consider the RNN of { h R t } , and let X = ( x 1 , . .
, x T ) (cid:48)
∈ R T ×
d in be an input matrix consisting of T tokens with dimension d in , where the transpose of a matrix A is denoted by A (cid:48) throughout this paper.
We ﬁrst give the value matrix V by projecting X with a linear transformation, i.e. V = XW V with
W V = W R (cid:48) x ∈ R d in × d , and the relative positional encoding matrix is set to
[Equation]
f 1 (
[Equation]
· · · f
[Equation] [Equation]
[Equation] [Equation]
[Equation]
  , (3) where f t ( λ ) =
λ t for 1 ≤ t ≤ T − 1 .
As a result, the ﬁrst RNN at (2) can be represented into a self-attention (SA) form,
[Equation]
where Q and K are zero query and key matrices, respectively.
We call P Rmask ( λ ) the recurrence encoding matrix (REM) since it summarizes all the recurrence in { h R t } .
For the RNN of { h C1 t } , the REM is denoted by P C1mask ( γ, θ ) , which has the form of (3) with f t ( λ ) being replaced by f t ( γ, θ ) =
γ t cos( tθ ) for 1 ≤ t ≤ T − 1 , and the value matrix has the form of
V = XW V with W V = W C1 (cid:48) x ∈ R d in × d .
Similarly, for the RNN of { h C2 t } , the REM, P C2mask ( γ, θ ) , has the form of (3) with f t ( λ ) being replaced by f t ( γ, θ ) =
γ t sin( tθ ) for 1 ≤ t ≤ T − 1 , and the value matrix is deﬁned as V = XW V with W V = W C2 (cid:48) x ∈ R d in × d .
Thus, these two RNNs at (2) can also be represented into SA forms,
[Equation]
)] V with i = 1 or 2 , where query and key matrices Q and K are both zero.
Finally, for the remaining term in Proposition 1, h 0 ,t depends on x t only, and there is no interdependence involved.
Mathematically, we can represent it into a SA with the identity relative positional encoding matrix and zero query and key matrices.
Proposition 2.
If the conditions of Proposition 1 hold, then the RNN with linear activation at (1) can be represented into a multihead self-attention (MHSA) with r + 2 s + 1 heads, where the query and key matrices are zero, and relative positional encoding matrices are { P Rmask ( λ k ) , 1 ≤ k ≤ r } , { P C1mask ( γ k , θ k ) , P C2mask ( γ k , θ k ) , 1 ≤ k ≤ s } and an identity matrix, respectively.
The three simple RNNs at (2) provide different temporal decay patterns: h R t provides the regular exponential decay induced by the real eigenvalues λ k ’s, and { h C1 t , h C2 t } provide the cyclical damped cosine or sine decay induced by the pair of complex eigenvalues ( γ k e iθ k , γ k e − iθ k ) ’s. These temporal decay patterns are further summarized into the regular REM, P Rmask ( λ ) , and cyclical REMs,
P C1mask ( γ, θ ) and P C2mask ( γ, θ ) , respectively; see Figure 2 for the illustration.
From Proposition 2, the combination of these three types of patterns forms the recurrent dynamics of the RNN layer at (1).
This leads to a parameter complexity of O ( Rd 2 ) , and it is slightly larger than that of the
RNN at (1) since R is usually much smaller than d (Prabhavalkar et al., 2016).
Moreover, the MHSA representation in Proposition 2 gives us a chance to make use of parallel matrix calculation on the
GPU hardware; see Appendix D.3 for an illustration of the computational efﬁciency of REMs.
E NCODING RECURRENCE INTO SELF - ATTENTION
While the query and key matrices are set to zero in the MHSA representation at Proposition 2, they play a central role in a standard Transformer. This motivates us to propose the Self-Attention with Recurrence (RSA) module to seamlessly combine the strengths of RNNs and Transformers: [Equation]
for each head, where P is a regular or cyclical REM, and σ ( µ ) ∈ [0 , 1] is a gate with σ being the sigmoid function and µ being the learnable gate-control parameter.
Figure 1(d) provides a graphical illustration of one RSA head.
Note that the REMs in Section 2 are all lower triangular matrices, which correspond to unidirectional RNNs. In the meanwhile, for non-causal sequential learning tasks (Graves et al., 2005; Zhou et al., 2016), bidirectional RNNs are usually applied, and accordingly we can deﬁne the unmasked versions of REMs. Speciﬁcally, the regular REM is P Runmask ( λ ) =
P Rmask ( λ ) + [ P Rmask ( λ )] (cid:48) , and the cyclical
REMs are P Ciunmask ( γ, θ ) =
P Cimask ( γ, θ ) + [ P Cimask ( γ, θ )] (cid:48) with i = 1 and 2.
In practice, these REMs will be explosive when | λ | > 1 or | γ | > 1 .
To avoid this problem, we further bound these two parameters by transformations in this paper, i.e. λ = tanh( η ) and γ = sigmoid ( ν ) , and these notations then become P R k ( η ) ,
P C1 k
( ν, θ ) and P C2 k
( ν, θ ) with k ∈ { mask , unmask } accordingly.
Sequential data more or less have recurrent patterns, which can be captured by REMs. In the meanwhile, the remaining non-recurrent patterns can be modeled by the baseline Transformer via softmax ( QK (cid:48) ) .
The learnable gate σ ( µ ) is used to measure the proportion or strength of recurrent patterns.
When the sample size is relatively small, it is also possible to use REMs to approximate a part of non-recurrent patterns to obtain a better bias-variance trade-off, leading to a higher value of σ ( µ ) .
On one hand, comparing with the baseline Transformer, the inclusion of REMs will lead to a signiﬁcant improvement of sample efﬁciency in modeling the recurrent patterns, and hence a higher accuracy in prediction can be expected from the proposed RSA.
On the other hand, comparing with the recurrent models, the non-recurrent patterns can be taken care of by the ﬂexible Transformers, and hence the representation power of the RSA module is as good as that of the baseline Transformer.
In addition, the REM can act as a complementary positional encoding scheme to further provide relative location information.
For the multihead RSA, the gate-control parameter µ only varies across layers, while the parameters controlling the matrix P vary across all heads and layers.
Initialization. Two types of parameters are introduced in (4).
The gate-control parameter µ is initialized in the interval [ − 3 , 3] for all the layers.
The second type of parameters are ( λ, γ, θ ) , which determine the recurrent patterns.
To encourage REMs to be non-zero and well-diversiﬁed, we initialize λ ’s at different heads to spread out between [ − 2 , − 1] ∪ [1 , 2] and ν ’s to spread out between [1 , 2] , and θ is initialized at π/ 4 .
Dilated REM variants.
The dilated REMs can be further obtained by considering the block matrix formed by the ﬁrst T columns and ﬁrst T rows of P ⊗ I d , where d is the dilating factor.
In fact, the dilated REMs can encapsulate the recurrence patterns of the dilated RNNs (Chang et al., 2017); see Proposition 4 in the Appendix.
They describe potentially periodic recurrence over a long-term timespan, and can signiﬁcantly enrich the temporal dynamics of our RSA.
Hyperparameters k i and d .
In each multihead RSA layer with H heads, the number of the six types of REMs, namely one regular, two cyclical, one dilated regular and two dilated cyclical REMs, are denoted by k 1 − k 6 , respectively.
Since it holds that (cid:80) 6 i =1 k i
=
H , we can apply constrained hyperparameter search to optimize over the choices of k i ’s. For simplicity, we can set k 2 = kand k 5 = k 6 , which indicates that the cyclical REMs come in pairs.
The search for d , on the other hand, can be guided by various data analytic tools.
For instance, d can be observed from the autocorrelation plots as the seasonal period length for time series data (Cryer and Chan, 2008); while for the language-related data, d can be heuristically deduced from the recurrence plots (Webber et al., 2016). Multihead RSA operations.
We ﬁrst choose masked or unmasked REMs according to the nature of the task, and then select the hyperparameters including the dilating factor d and the numbers of the six types of REMs ( k 1 , . .
. , k 6 ) .
Each RSA head is calculated by (4) with a different REM.
Finally, a linear layer is applied to combine the output from all heads, and it is then followed by layer-normalization and dropout selectively.
More discussions on REMs. Each type of REMs is basically a masked or unmasked linear aggregation of tokens, and we may alternatively consider a more general Toeplitz, or even a fully learnable, matrix P at (4) such that all kinds of temporal decay patterns can be automatically captured. Although more ﬂexible than REMs, it will need O ( T ) or O ( T 2 ) additional parameters, where T is the sequence length, while each REM requires merely one or two parameters. Note that the proposed RSA at (4) also includes a standard self-attention, which is ﬂexible enough for all remaining patterns from the ﬁtted REMs, and hence it may not be necessary to consider a more general yet less efﬁcient structure for REMs.
On the other hand, the REMs may fail to capture the possible nonlinearity in the data since they come from a linear RNN, while the self-attention component in the RSA module can remedy the problem to some extent; see Appendix B for both theoretical and empirical evidences.
E XPERIMENTS
This section contains four sequential modeling tasks and, for each task, we modify some popular
Transformer baselines by adding the REMs to their attention weights via a gated mechanism as in (4).
The modiﬁed models are named with the preﬁx “RSA-”.
The trained gate σ ( µ ) controls the percentage of weights allocated to the REMs, and it hence provides a rough measure for recurrent signals in the data.
From Figure 3, we may argue that time series data have the strongest recurrent signals, followed by the regular language (Yu, 1997) and ﬁnally the code or natural languages.
The gate values also vary across different layers of Transformer models, indicating that the proportion of recurrent signals may change as the data propagate through layers.
Moreover, the data-driven gated mechanism can help maintain the optimal allocation between REM and the self-attention to achieve better sample efﬁciency, and improved performance can be observed uniformly.
All experiments are conducted on Nvidia V100 32GB GPUs.
4. T IME SERIES FORECASTING
Time series have many unique attributes that can be leveraged to boost forecasting accuracy, one of which is the strong recurrent signals; see Section 1.1 for a brief review of state-of-the-art (SOTA) models.
Although Transformer-based models are not the current SOTA models, it is still of interest to explore how their performance can be further improved by the proposed RSA module.
[Table]
ETT m 10.
W ea t h e r0.
[Table]
Our experiments are performed on two public benchmark datasets: the ETT 1 dataset is comprised of seven features related to the electric power long-term deployment, where {ETTh 1 , ETTh 2 } are recorded by the hour and ETTm 1 is recorded by 15-minute intervals; and the Weather 2 dataset contains twelve climate indicators collected every 1 hour over a 4-year period.
For the baseline models, we adopt Informer (Zhou et al., 2021),
LogSparse Transformer (Li et al., 2019) and Transformer-XL (Dai et al., 2019).
In particular, to accommodate the cases where the attention map is non-square, we extend the recurrent patterns and adjust the shape of our REMs; see details in Appendix D.2.
All hyperparameters in baseline models are set to the optimal setting in Zhou et al.
, and we also follow their train/val/test division and training schemes to conduct our experiments.
Table 1 summarizes the mean squared error (MSE) and mean absolute error (MAE) for the three baselines against their RSA counterparts, averaged across 5 repetitions.
Overall, the revised models show signiﬁcant improvement in performance over their baselines.
Moreover, it is interesting to see that the REMs can act as effective substitutions for positional encoding.
For Transformer-XL in particular, although we remove their learnable relative positional encoding, superior performance is attained by RSA-XL while the total number of parameters is reduced.
We further provide a boxplot for the ﬁtted gate values at different attention layers of RSA-Informer on the four datasets in Figure 3.
Since more than 70% of weights are allocated to the REMs, it numerically veriﬁes that the recurrent signals dominate the non-recurrent ones for time series data.
4. R EGULAR LANGUAGE LEARNING
Regular languages are intimately related to the linear recurrence sequences, such as Fibonacci numbers (Kotek and Makowsky, 2010).
Some works report that Transformers have difﬁculty in generalizing the rules of regular languages (Bhattamishra et al., 2020; Tran et al., 2018; Hahn, 2020).
Experiments are conducted to show that the proposed RSA module can leverage the recurrent inductive bias of the
REMs to improve Transformer’s sample efﬁciency, and subsequently its generalization ability.
For the experiments, we deliberately choose six types of regular languages on which the Transformer model has been shown to perform poorly, namely Parity, Tomita 3, 5 & 6 (Tomita, 1982), D 2 and
D 4 (Bhattamishra et al., 2020).
We use the decoder-only vanilla Transformer model as the baseline, and modify its self-attention layers into our RSA forms under four hyperparameter settings (detailed in Table 2).
As a comparison, another benchmark model (denoted by Transformer (cid:63) ) is obtained by replacing the absolute sinusoidal positional encoding with the learnable relative positional encoding (Dai et al., 2019).
To evaluate their generalization ability, the test set is further separated into two
[Table]
bins, where bin 1 contains longer test samples than bin 0.
Following Bhattamishra et al.
(2020), we recast the prediction task into a classiﬁcation one, where the prediction is deemed a success only when the output values at all positions are correct, and a failure otherwise.
From Table 2, it can be seen that the vanilla Transformer model fails to grasp the recurring rules of these languages.
With assistance of the learned relative positional encoding, they can learn better on D 2 , D 4 and Tomita 3, but not the others.
However, the RSA-Transformer not only achieves near 100% accuracy on D 2 , D 4 and Tomita 3, but also gains a signiﬁcant boost in performance on the other three languages.
In particular, Figure 3 shows that approximately 60% of weights are allocated to the REMs, implying strong recurrent signals in the datasets.
We further provide some visualization of the REMs under Cases I – IV and their interpretaions in Appendix E.2.
4. C ODE AND NATURAL LANGUAGE MODELING
Different from regular languages, the recurrence relationship in programming or natural languages is weaker and harder to interpret.
Nevertheless, they have a wide range of applications on which the
Transformers are serving as the baseline models.
Our experiments show that the baseline models can improve the performance with inclusion of a small percentage of REMs.
We ﬁrst conduct a defect detection task based on the C language dataset provided by Zhou et al.
(2019), which is a binary classiﬁcation task to evaluate whether a code is vulnerable to external attacks.
The pretrained CodeT5 (Wang et al., 2021) 3 is adapted to this downstream task.
Speciﬁcally, we use CodeT5-small as our baseline model with 12 layers and 8 heads, and directly add the REMs on top of the self-attention with fully learnable RPE via a gated mechanism.
The gate-control parameter µ is initialized at − 3 and we ﬁne-tune both the baseline and our RSA models forepochs with early stopping.
It can be observed from the boxplot in Figure 3 that the gate values at different layers remain close to the initialization, with only 5% of weights allocated to the REMs.
Although the recurrent signals in the code language is apparently much weaker than those in time series or regular language, our RSA-CodeT5-small is still able to achieve a signiﬁcant improvement against the CodeT5-small baseline as shown in Table 3(a).
The RSA-CodeT5-small model has only about 320 additional parameters over the baseline with 66 million parameters, which is equal to 5 × 10 − 4 % .
More surprisingly, it surpasses the current SOTA result of 65.78%, which is achieved by the CodeT5-base model with a total number of 220 million parameters (Wang et al., 2021).
Due to limited resources, we have not yet ﬁne-tuned RSA-CodeT5-base model.
The natural language modeling task is conducted on three standard NLP datasets, namely the wordlevel WikiText-
(Merity et al., 2017),
Enwik8 (LLC., 2009) and Text8 (LLC., 2009).
We use three baseline models: Transformer-XL, Temporal Latent Bottlenck (TLB) and Block Recurrent
Trasnformer (BRT), and their corresponding RSA models are formed by directly adding the REMs on top of some of the self-attention heads via a gated mechanism. For Wikitext-103, all transformers are 16-layer with 12 heads.
In the meantime, for both Enwik8 and Text8 there are 14 layers and 8 heads.
The details of model structures can be referred to Appendix E.3. We follow the training schemes of Transformer-XL, and the results are reported in Table 3(b).
Overall, it can be seen that RSA models can achieve better performance than their corresponding baselines.
From Figure 3, we can see that the REMs at the ﬁrst three self-attention layers contribute more substantially for Text comparing to the other two datasets, which may provide an explanation for RSA models achieving a more signiﬁcant improvement on this particular dataset. [Table] 4. A BLATION STUDIES ON THE SCALING OF MODEL AND DATA SIZES
This section presents two ablation studies on Text8 by scaling the model and data sizes, respectively.
The ﬁrst ablation study compares the performance of three models, namely the baseline TransformerXL, RSA-XL and XL with Fully Learnable Linear Aggregation (FLLA-XL), under various model sizes. FLLA-XL is provided as another baseline model for RSA-XL, where the REM is replaced by a fully learnable matrix. The results presented in the left panel of Figure 4 shows that RSA-XL is more parameter-efﬁcient than both Transformer-XL and FLLA-XL across all model scales. The second ablation investigates how Transformer-XL and RSA-XL perform for different sizes of training data, i.e. the proportion of training data ranges from 5% to 100%.
The middle and right panels in Figure 4 jointly verify the sample efﬁciency of the RSA design.
As the available training samples shrink, the learnable gate will automatically increase the weights of the parsimonious REMs, and this subsequently helps RSA-XL to maintain a consistently higher sample efﬁciency than the baseline XL model.
C ONCLUSION AND DISCUSSION
By formulating RNN into an MHSA form, we novelly propose an RSA module to incorporate the recurrent dynamics of an RNN into a Transformer.
Speciﬁcally, the lightweight REMs are combined with the self-attention weights via a gated mechanism, maintaining a parallel and efﬁcient computation on the GPU infrastructure.
Experiments on four sequential learning tasks show that the proposed RSA module can boost the sample efﬁciency of the baseline Transformer, supported by signiﬁcantly improved performance.
In addition, with the ﬁtted gate values to measure the recurrent signal strength in the data, it can be observed that the code or natural languages have much smaller proportion of recurrent patterns, which further explain the superiority of Transformer models in these two data domains, while the inclusion of REMs still can boost the baseline performance.
This paper can be extended along three directions below.
Firstly, while six typical examples of
REMs are introduced, we may also consider other types of REM designs, such as the non-distinct eigenvalues of the RNN weight matrix W h , and long-memory ﬁlters operated on top of the RNNs (Zhao et al., 2020).
Secondly, the proposed RSA module can be further applied to other types of sequential learning tasks such as video learning (Bulat et al., 2021) and skeleton-based human motion detection (Shu et al., 2021).
Finally, it is of interest to incorporate the REMs into models other than
Transformers, or even to apply it directly as a standalone recurrent unit.
R EPRODUCIBILITY STATEMENT
To ensure the reproducibility and completeness of this paper, we include the Appendix with ﬁve main sections.
Appendix A discusses the relationship between RSA and some important related works.
Appendix B provides a theoretical and empirical study on the performance gap between linear and nonlinear RNNs.
Appendix C contains the complete proofs for all propositions presented in this paper.
An efﬁcient method to compute the REMs is detailed in Appendix D, together with an extension to non-square REMs and an empirical study on the time efﬁciency of the proposed method.
The experiments in the paper are reproducible with additional implementation details provided in
Appendix E. We also include the hyperparameter settings for all results contained in Tables 1 and 3.
We will make the codes available to the public.
A PPENDIX
This Appendix contains ﬁve sections.
The ﬁrst section provides an illustrative plot for multiscale recurrence in various Transformer models.
The theoretical and empirical gaps between the linear and nonlinear RNNs are studied in the second section.
The third section presents Proposition 3, and then gives the detailed proofs for Propositions 1 – 3.
An efﬁcient method to compute the REMs is detailed in Section 4, and we further extend it to non-square REMs and conduct an empirical study on the training time per batch for the RSA models against their baselines.
Section 5 provides additional implementation details for the experiments, additional ablation studies and visualization of the learned patterns. A M ULTISCALE RECURRENCE
For sequential learning tasks, the recurrence relationship can be well observed at different levels of temporal granularity.
This feature has inspired new designs to add recurrence to Transformers from varying scales; see illustration in Figure 5.
Transformer-XL (XL) (Dai et al., 2019) partitions a long input sequence into segments, and places them into consecutive batches.
A segment-level recurrence is then introduced by using a cached hidden state to pass historical information iteratively to the next segment.
Temporal Latent Bottleneck (TLB) (Didolkar et al., 2022) further divides the segment within one batch into smaller chunks, and then adopts the state vectors to aggregate both high-level information across layers and temporal information across chunks. The chunk-level recurrence helps create a slow information stream in the sequence to learn a more condensed representation. Block-Recurrent Transformer (BRT) (Hutchins et al., 2022) also establishes recurrence across chunks (or blocks), while their recurrent states are layer-speciﬁc and updated with an LSTM-style gated design. As a comparison, the proposed RSA follows the RNN to account for the recurrence between individual inputs.
In other words, it models the token-level recurrence, which is at the most ﬁnegrained scale.
Subsequently, it can be easily incorporated into the aforementioned coarser-grained designs, and may potentially bring beneﬁts to their performance.
For illustration, we use XL, TLB and BRT as our baseline models in Section 4.3.
B T HEORETICAL AND EMPIRICAL GAPS BETWEEN LINEAR AND NONLINEAR RNN S B. T HEORETICAL GAP
This subsection theoretically evaluates prediction errors when a linear RNN model is used to train the data generated by a nonlinear RNN.
For simplicity, we consider 1D case only, and many-to-many
RNNs are assumed.
Speciﬁcally, the nonlinear RNN model used to generate the data has the recursive form of g t
[Equation]
( u t ) with u t = w ∗ h g t − 1 + w ∗ x x t + b ∗ , (5) and σ h ( · ) is the activation function satisfying
[Equation]
Note that many commonly used activation functions, including Tanh and Sigmoid, satisfy the above condition.
We further consider an additive error ε t , i.e. y t =
g t + ε t , where ε t has mean zero and a ﬁnite variance denoted by γ .
For the generated data { y t } , we train a linear RNN model,
[Equation]
where the parameters θ = ( w h , w x , b ) .
Then the mean squared prediction error can be deﬁned as
[Equation]
and its theoretical bound is provided in the following proposition.
Proposition 3.
Suppose that E ( u 2 t ) ≤ α and E ( u 2 t u 2 s ) ≤ β for all t, s ∈ Z .
If | w ∗ h | < 1 and the condition at (6) holds, then
[Equation]
where the ﬁrst part is due to the misspeciﬁcation of using the linear activation to approximate σ h ( · ) .
Proof of Proposition 3.
Let θ ∗ =
( w ∗ h , w ∗ x , b ∗ ) , and denote h t = h t ( θ ∗ ) for all t ∈ Z , i.e.
[Equation]
[Equation]
By the deﬁnition of e pred , it holds that
[Equation]
where the equality comes from E ( ε t )
= 0 and var ( ε t )
=
γ .
By applying second-order Taylor expansion at zero and from (5) and (6), we have
[Equation]
where
[Equation]
and (cid:101) u lies between u t and zero.
This, together with (7), leads to g t − h t
=
σ h (0) + w ∗ h ( g t
[Equation]
1 − h t − 1 )
+ R t (0) .
Let δ t =
g t − h t , and it then holds that
[Equation]
where the second equality is obtained by applying the ﬁrst equality recursively.
As a result, by the condition that | σ h (0) | < 1 , E ( u 2 t ) ≤ α and E ( u 2 t u 2 s ) ≤ β for all t, s ∈ Z , we can show that
[Equation]
where ξ = (cid:80) ∞ j =0 | w ∗ h | j .
If | w ∗ h | < 1 , we have ξ = (1 −| w ∗ h | ) − 1 .
This, together with (8), accomplishes the proof.
B. E MPIRICAL GAP
This subsection conducts a synthetic experiment to evaluate the performance of linear RNNs when there is nonlinearity in the data.
We ﬁrst generate the data by using a two-layer nonlinear RNN model with the form of z
[Equation]
i
[Equation]
t and
[Equation]
with i = 1 and 2, where z (0) t = x t , z ( i ) t ∈ R 2 for 1 ≤ i ≤ 3 , σ h ( · ) is a nonlinear activation function, and 0 ≤ α ≤ 1 is the weight of nonlinearity.
An additive error is further assumed, i.e. y t =
z (2) t + ε t , where { x t } and { ε t } are independent and follow the standard multivariate normal distribution.
Three nonlinear functions are considered for σ h ( · ) , Tanh, Sigmoid and ReLU.
As α increases from 0 to 1, the data generating process gradually changes from a strictly linear RNN to a nonlinear one, i.e. α essentially controls the proportion of nonlinearity involved.
The sequence { y t , 1 ≤ t ≤ T } is ﬁtted separately by a linear RNN, a nonlinear RNN with the corresponding activation, and a linear RNN combined with a self-attention, i.e. the proposed RSA.
Speciﬁcally, we generate a sequence of length 10000 and then divide it into 100 segments, each of length 100.
In each segment, we train with the ﬁrst 99 observations and calculate the prediction error for the last observation.
The Adam optimizer is adopted for training, and the training procedure will be terminated when the training loss drops by less than 10 − 5 .
The mean squared prediction error (MSPE) averaged over 100 segments are denoted by e L pred , e NL pred and e RSA pred for the three models, respectively.
Using nonlinear RNNs as the benchmark, the MSPE ratio for the linear RNN or the
RSA is deﬁned as
[Equation]
where i ∈ { L , RSA } .
Figure 6 presents the MSPE ratios for three types of activation functions.
It can be seen that, when α = 1 , nonlinear RNNs perform the best, while linear RNN suffers from misspeciﬁcation error.
Alternatively, when α = 0 , the opposite can be observed.
Moreover, as α increases, i.e. there are more nonlinearity, it is expected that linear RNNs become less favorable, while the proposed RSA can remedy the problem to some extent.
Especially when α > 0 .
6 , the RSA consistently achieves better prediction performance than the pure linear RNN.
C P ROPOSITION 3 AND PROOFS FOR ALL P ROPOSITIONS C. P ROPOSITION 3 FOR DILATED RNN S
For some positive integer d , let ¯ P R mask be the block matrix formed by the ﬁrst T columns and the ﬁrst
T rows of P Rmask ⊗ I d .
And ¯ P C i mask can be deﬁned similarly for i = 1 or 2 .
Consider a dilated RNN (Chang et al., 2017) with the dilating factor d .
It has the form h t
=
g
( W h h t − d + W x x t + b ) where g ( · ) is the activation function, h t ∈ R
d is the output or hidden variable with h 0 = 0 , b ∈ R d in is the bias term, W h ∈ R d × d and W x ∈ R
d
×
d in are weights.
When the activation function is linear, i.e. g ( x ) = x , the RNN becomes
[Equation]
where the bias term b is suppressed for simplicity.
We have the following proposition.
Proposition 4.
If the conditions of Proposition 1 hold for W h in (10), then the RNN with linear activation at (10) can be represented into a multihead self-attention (MHSA) with r + 2 s +heads, where the query and key matrices are zero, and relative positional encoding matrices are { ¯ P Rmask ( λ k ) , 1 ≤ k ≤ r } , { ¯ P C1mask ( γ k , θ k ) , ¯ P C2mask ( γ k , θ k ) , 1 ≤ k ≤ s } and an identity matrix, respectively.
Proof.
The proof follows directly from Propositions 1 and 2.
C. P ROOF FOR P ROPOSITION
Let W h be a d × d real matrix with distinct eigenvalues, and from Chapter 3 of Horn and Johnson (2012), we have the Jordan decomposition, W h
=
BJB − 1 , where B ∈ R d × d is invertible, and
J ∈ R d × d has a real Jordan form, J = diag { λ 1 , . . .
, λ r , C 1 , . . .
, C s , 0 } with
[Equation]
Then,
[Equation]
all j ≥ 1 , where G R k ’s, G C1 k ’s and G C2 k ’s are d
×
d real matrices determined jointly by B and B − 1 .
Let h 0 ,t
= W x x t
and then,
[Equation]
where h R
[Equation]
=
[Equation]
=
[Equation] x x [Equation] − j or equivalently, h R t =
[Equation]
t
[Equation]
g ( · ) being the identity function and h C 1 t
[Equation]
=1 γ j sin
[Equation]
jθ ) G C2 W x x t − j which can be obtained via the recursive relationships,
[Equation]
with G being G C1 or G C2 , respectively.
For a more general form, we can further assume that the activation function g ( · ) is nonlinear for the simple RNNs at (11) and (12).
C. P ROOF FOR P ROPOSITION
Using the SA form, we can represent the three types of the RNNs by ( h R1 ( λ k ) , · · · , h R1 ( λ k ))
=
SA R k ( X ) and ( h C
i 1 ( γ k , θ k ) , · · · , h C i 1 ( γ k , θ k )) = SA C ik ( X ) for i = 1 or 2 .
Therefore, the ﬁrst term in Proposition 1 can be represented as
[Equation]
where W o =
( I d , . .
, I d ) (cid:48) ∈ R rd ×
d with I d being d -dimensional identity matrix.
Similarly the
MHSA for second and third term in Proposition 1 is given by MHSA
[Equation]
concat
[Equation]
(cid:33)
[Equation]
where W o =
( I d , . .
, I d ) (cid:48)
∈ R sd × d .
And we deﬁne the additional head as SA 0 ( X ) =
( h 0 , 1 , . . .
, h 0 ,T ) (cid:48) =
[softmax( QK (cid:48) )
+ I ] V with
W Q = W K = 0 and W V = W (cid:48)
x .
Combine with (13) and (14), we have
MHSA ( X ) = concat [ { SA R k ( X ) } 1 ≤ k ≤ r , { SA C1 k ( X ) , SA C2 k ( X ) } 1 ≤ k ≤ s , SA 0 ( X )]
W o (15) where W o =
( I d , . .
, I d ) (cid:48)
∈ R
( r +2 s +1)
d × d .
D C OMPUTATION OF THE REM S D. E FFICIENT COMPUTATION OF THE REM S
For any matrix A , let cos ( A ) and sin ( A ) refer to applying the cosine and sine functions to each element of A .
To compute the REMs, we ﬁrst construct the exponent matrix L that has a Toeplitz form:
[Equation]
 
Denote the square matrix whose elements are all equal to λ as λ .
Similarly, we deﬁne γ and θ for γ and θ , respectively.
Extending the conventional power notation, we let λ L = pow ( λ , L ) and γ L = pow ( γ , L ) , i.e., ( λ L ) i,
j = λ
L i,j i,j and ( γ L ) i,j = γ
L i,j i,j .
Then we can calculate the REMs by
P = [Equation] for regular REM ,
D
[Equation] for cyclical REMs with 1 ≤ i ≤ 2 , where D R = λ L , D C1 =
γ L (cid:12) cos ( θ (cid:12) L ) , D C2 =
γ L (cid:12) sin ( θ (cid:12) L ) and (cid:12) is the Hadamard product.
Such design allows us to circumvent redundant computations and achieve parallelization as much as possible.
To avoid extremely large or small values, we mask the entries that have power larger than a preset number K .
In practice, we set K = 200 .
Moreover, we can easily extend the above calculation to dilated REMs by substituting L with L d , where ( L d ) i,j = I { L i,j mod d =0 } L i,j /d and d is the dilating factor.
The remaining operations are unchanged.
D. F LEXIBLE ADAPTATION TO DIFFERENT ATTENTION SHAPES
In some cases, the query and key have different time spans with an overlapping period; examples including the cross-attention and all attention in the Transformer-XL.
We show in this section that
our
REMs can be adapted for such cases.
Typically, the historical information will be mostly stored in the query matrix, and the new information will be included in the key and the value matrices.
Assume that the query matrix contains the tokens from time 1 to time T , and the key matrix contains that from time T − K to time T + L , where K > 0 is the overlapping period length and L > 0 is the length of new information.
Then the REM is of size ( L + K + 1) × T and takes the form of     ···
T − K −
T − K ···
T −
T
T − K f T − K − 1 ( β ) f T − K − 2 ( β ) · · · f 1 ( β ) · · · f K − 1 ( β ) f K ( β )
T − K + f T − K − 2 ( β ) f T − K − 3 ( β ) · · · f 2 ( β ) f 1 ( β ) · · · f K − 2 ( β ) f K − 1 ( β ) .
T −f T − 2 ( β ) f T − 3 ( β ) · · · f K ( β ) f K − 1 ( β ) · · · f 1 ( β )
T f T − 1 ( β ) f T − 2 ( β ) · · · f K +1 ( β ) f K ( β ) · · · f 1 ( β ) T + f T ( β ) f T − 1 ( β ) · · · f K +2 ( β ) f K +1 ( β ) · · · f 2 ( β )
f 1 ( β ) .
T + L f T + L − 1 ( β ) f T + L − 2 ( β ) · · · f K + L
+
β ) f K + L ( β ) · · · f L +1 ( β ) f L ( β )    , which maintains the recurrent patterns while considering the shift in the relative time locations, where β = λ for regular REM or ( γ, θ ) for cyclical REMs.
And we can extend to the dilated REMs similarly.
D. T HE COMPUTATIONAL EFFICIENCY OF THE REM S FOR VARIOUS T RANSFORMER BASELINES
We ﬁrst conduct an empirical study on the Weather dataset to evaluate the computational time of the
REMs for RSA-Informer, RSA-LogSparse and RSA-XL.
Speciﬁcally, we record the training time per batch for the RSA models, while simultaneously we exclude the REMs from the models and record the training time again.
The difference between the two gives the additional computation time due to the REMs, as shown by the green bar in Figure 7.
It can be seen that the calculations involving the
REMs account for less than 20% of the total training time in general.
In the next experiment, we further compare the computational time between the RSA models and their respective baseline models.
Note that the original Informer and LogSparse Transformer use absolute positional encoding (APE), while Transformer-XL adopts a learned relative positional encoding (RPE).
These positional encoding schemes are excluded from the RSA models.
The training time per batch for all the models are reported in Figure 8.
We can observe a small increase in computational time for RSA-Informer and RSA-LogSparse against their baselines, while that for RSA-XL remains nearly unchanged.
This implies that the REMs requires a bit more computation effort than the APEs, and the same computation effort as the RPE introduced by Transformer-XL.
E A DDITIONAL EXPERIMENTS AND RESULTS E. T IME SERIES FORECASTING
We apply the same time series datasets as in the Informer paper (Zhou et al., 2021), and the details are given below for completeness.
The ﬁrst dataset is the Electricity Transformer Temperature (ETT).
It contains the records for seven electric power indicators over a two-year period from two separated counties in China.
Zhou et al.
(2021) created three separate datasets ETTh 1 and ETTh 2 for hourly level, and ETTm 1 for the ﬁfteen-minute level.
The train/val/test is 12/4/4 months.
The second dataset consists of climatological data over a four-year period from 2010 to 2013 for various locations in
US.
The train/val/test is 28/10/10 months.
To ensure the quality of our baselines models, we make reference to the following two github repositories, namely Informer2020 4 and transformer-xl 5 .
Since
Transformer-XL requires a sequential order for training, testing and validation sets, we generate the training samples by sliding window starting from different points of the original time series following the practice in Salinas et al.
In this task, we let the REMs substitute the original positional encoding scheme in the RSA models.
Moreover, since the distilling operation in the original Informer design is likely to distort the relative positions, we exclude it from both our Informer baseline and the RSA-Informer.
The hyperparameters in RSA are ﬁne tuned via grid search.
Table 5 reports the ﬁne-tuned hyperparameter settings considered in Table 1 of the main paper.
To provide additional information about the standard deviation, we provide a more detailed version of
Table 1 of the main paper, given in Table 4. [Table]
ETT m
[Table]
W ea t h e r
[Table]
[Table]
pred window [Table] [Table] Ablation on the hyperparameters To shed light on the importance of different recurrent dynamics, we provide an ablation study with RSA-Informer on ETTh 1 .
The model has a total of 8 heads, and we set k 2 = k 3 to some positive integer s and k 5 = k 6 to some positive integer ¯ s .
In particular, when ¯ s = 0 , we also take k 4 =
and hence there are not any dilated REMs, leading to k 1 = 8 − 2 s .
Otherwise, we take k 4 = 4 − 2¯ s and k 1 = 4 − 2 s .
It can be observed from Table 6 that the inclusion of dilation and the cyclical patterns can signiﬁcantly improve the forecasting performance.
The concurrence of dilated and non-dilated RSA heads allows the model to capture both the short- and long-term recurrent dynamics.
Gate-control parameter learning To study how much proportion of the RSA output is accounted for by the REMs, Figure 9 plots the change in its relative importance i.e. σ ( µ ) during training for all the attention layers, starting from different initializations.
As the learning rate drops rapidly, all [Table] [Table] ETT
[Table]
[Table] W ea t h e r(0,0,0,0,4,4) [24,24,24,24] (8,0,0,0,0,0) - (0,2,2,0,2,2) [48,24,48,24] curves are ﬂattened towards the end. The relative positional attention is the most effective for the cross-attention layer, and the least effective for the self-attention layer in the decoder, whose recurrent patterns may be disrupted by zero paddings.
We observe a decrease in MSE as the initialization increases from − 1 to 1 , which provides an empirical support for initializing at µ = 1 .
E. R EGULAR LANGUAGE LEARNING
The six types of regular language datasets are obtained from the github repository TransformerFormal-Language .
6 We brieﬂy describe the rules of these languages:
• The Parity language contains the numbers { 0 , 1 } with an even number of 1’s.
For example, the string 0110 belongs to the language but 0111 does not.
• The D n language is deﬁned on the alphabets { a , b }.
The D 1 language can be written using the regular expression ( ab ) (cid:63) , where “ (cid:63) ” represents the Kleene star operation.
Then for n ≥ 1 , D n can be expressed by the recursive relationship D n = ( aD n − 1 b ) (cid:63) .
We choose Dand D 4 , where D 2 =
( a ( ab ) (cid:63) b ) and D 4 = ( a ( a ( a ( ab ) (cid:63) b ) (cid:63) b ) (cid:63) b ) (cid:63) .
• The Tomita languages also use the numbers { 0 , 1 } and include seven different grammars.
In our experiment, we choose Tomita 3, 5 and 6.
In Tomita 3, consecutive 1’s must be of an odd number and consecutive 0 ’s must be of an even number.
In Tomita 5, the total numbers of 1 ’s and 0 ’s must both be even.
Finally, Tomita 6 requires the difference between the total number of 1 ’s and 0 ’s to be a multiple of 3.
In this experiment, we adopt a simple setting with 3 layers and 5 heads for all models.
The embedding size is set to 20.
During training, the Adam optimizer is initialized at 0.005 with a 0.5 × decay every
5 epochs. And all models are run for 25 epochs without early stopping. Both the training set and bin 0 contain samples with length in the range of [2 , 50] for Parity and Tomita, and [2 , 100] for D n , while bin 1 contains longer test samples with length in the range of [51 , 100] for Parity and Tomita, and [101 , 200] for D n . The models are trained on 10K samples and tested on 2K samples per bin for Parity and Tomita, and trained on 5K samples with 1K test samples per bin for D n .
Following
Bhattamishra et al.
(2020), we train the models as below: starting from the ﬁrst position of an input sequence, the model outputs a string of possible characters for the next position, and this continues until the sequence is exhausted.
The prediction task is recast into a classﬁcation task, where the prediction is deemed a success if and only if the output values at each position are correct, and a failure otherwise.
From Table 2, we can see that the Transformer-only models may require much more training samples to learn these patterns.
Moreover, when the training samples are shorter than the test samples, they will have difﬁculty in extending the learned patterns to longer sequences, even with absolute or learned relative positional encoding.
Meanwhile, REMs can easily extend its learned patterns to any sequence length, which may account for the better generalization ability of RSA-Transformers.
To help interpret the results in Table 2, we provide some visualizations of the learned recurrent patterns on different languages in Figure 10.
Learning Parity requires the model to count the number of 1’s, and then decide whether it is an even number.
Its recurrent pattern at (a) mainly consists of alternating {− 1 , 1 } that leads to regular oscillations when encountering an even or odd number of 1’s.
Tomita 5 has an additional requirement that the number of 0 is also even, and hence its learning can be assisted by dilated patterns; see (b) for illustration.
Although Tomita 6 is more complicated, the regular cyclical patterns at (c) seems sufﬁcient for it to learn well.
This gives us a glimpse of how the RSA-Transformer manages to learn the regular languages.
E. C ODE AND NATURAL LANGUAGE MODELING
For RSA-CodeT5-small, our code is based on the original CodeT5, which can be referred to in the repository https://github.com/salesforce/CodeT5 .
For the RSA-XL models adopted for the natural language modeling, only some of the heads are replaced with the RSA heads, while the remaining ones are still the original XL heads.
In the RSA-BRT and baseline BRT models, the recurrent layer is put at the third layer counted from the top, i.e., the recurrent layer is at layer 12 for 14-layer models and at layer 14 for 16-layer models.
The nonrecurrent layers are of the same structure as RSA-XL and XL respectively.
In Table 7, we report the detailed hyperparameters settings for the RSA and their baseline models used in Table 3, where “n_RSA_heads" refers to the number of heads that are changed to the RSA heads.
[Table]