A P RIMAL -D UAL F RAMEWORK FOR T RANSFORMERS AND N EURAL N ETWORKS
A BSTRACT
Self-attention is key to the remarkable success of transformers in sequence modeling tasks including many applications in natural language processing and computer vision.
Like neural network layers, these attention mechanisms are often developed by heuristics and experience.
To provide a principled framework for constructing attention layers in transformers, we show that the self-attention corresponds to the support vector expansion derived from a support vector regression problem, whose primal formulation has the form of a neural network layer.
Using our framework, we derive popular attention layers used in practice and propose two new attentions:
1) the Batch Normalized Attention (Attention-BN) derived from the batch normalization layer and 2) the Attention with Scaled Head
(Attention-SH) derived from using less training data to fit the SVR model.
We empirically demonstrate the advantages of the Attention-BN and Attention-SH in reducing head redundancy, increasing the model’s accuracy, and improving the model’s efficiency in a variety of practical applications including image and time-series classification.
I NTRODUCTION
Transformer models (Vaswani et al., 2017) have achieved impressive success with state-of-the-art performance in a myriad of sequence processing tasks, including those in computer vision (Dosovitskiy et al., 2021; Liu et al., 2021; Touvron et al., 2020; Ramesh et al., 2021;
Radford et al., 2021; Arnab et al., 2021;
Liu et al., 2022; Zhao et al., 2021; Guo et al., 2021), natural language processing (Devlin et al., 2018; Al-Rfou et al., 2019; Dai et al., 2019; Child et al., 2019; Raffel et al., 2020; Baevski &
Auli, 2019; Brown et al., 2020; Dehghani et al., 2018), reinforcement learning (Chen et al., 2021;
Janner et al., 2021), and other important applications (Rives et al., 2021; Jumper et al., 2021; Zhang et al., 2019; Gulati et al., 2020; Wang & Sun, 2022).
Transformers can also effectively transfer knowledge from pre-trained models to new tasks with limited supervision (Radford et al., 2018; 2019; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019).
The driving force behind the success of transformers is the self-attention mechanism (Cho et al., 2014; Parikh et al., 2016; Lin et al., 2017), which computes a weighted average of feature representations of the tokens in the sequence with the weights proportional to similarity scores between pairs of representations.
The weights calculated by the self-attention determine the relative importance between tokens and thus capture the contextual representations of the sequence (Bahdanau et al., 2014; Vaswani et al., 2017; Kim et al., 2017).
It has been argued that the flexibility in capturing diverse syntactic and semantic relationships is critical for the success of transformers (Tenney et al., 2019; Vig & Belinkov, 2019; Clark et al., 2019).
1. B ACKGROUND : S ELF -A TTENTION
For a given input sequence X :=
[ x 1 , · · · , x N ] ⊤ ∈ R N × D x of N feature vectors, self-attention transforms X into the output sequence H in the following two steps:
Step 1.
The input sequence X is projected into the query matrix Q , the key matrix K , and the value matrix V via three linear transformations
[Equation]
where W Q , W K ∈ R D × D x , and W V ∈ R D v × D x are the weight matrices.
We denote Q :=
[ q 1 , · · · , q N ] ⊤ , K :=
[ k 1 , · · · , k N ] ⊤ , and V :=
[ v 1 , · · · , v N ] ⊤ , where the vectors q i , k i , v i for i = 1 , · · · , N are the query, key, and value vectors, respectively.
Step 2.
The output sequence H :=
[ h 1 , · · · , h N ] ⊤ is then computed as follows
[Equation]
where the softmax function is applied to each row of the matrix QK ⊤ / √ D .
The matrix A := softmax (cid:16)
QK ⊤ √ D (cid:17) ∈ R N × N and its component a ij for i, j = 1 , · · · , N are called the attention matrix and attention scores, respectively.
For each query vector q i for i = 1 , · · · , N , an equivalent form of
Eqn.
(1) to compute the output vector
h
i is given by
[Equation]
The self-attention computed by Eqn.
(1) and (2) is called the scaled dot-product or softmax attention.
In our paper, we call a transformer that uses this attention the softmax transformer.
The structure that the attention matrix A learns from training determines the ability of the self-attention to capture contextual representation for each token.
Additionally, a residual connection can be added to the output of the self-attention layer, h i
=
x
i + (cid:80) Nj =1 softmax (cid:16) q ⊤ i k j / √ D (cid:17) v j .
Multi-head Attention (MHA).
In MHA, multiple heads are concatenated to compute the final output.
This MHA mechanism allows transformers to capture more diverse attention patterns and increase the capacity of the model.
Let H be the number of heads and W multi O
[Equation]
be the projection matrix for the output where W 1 O , . . .
, W HO ∈
R D v × D v .
The MHA is defined as
MultiHead ( { H } Hs =1 ) =
Concat (
[Equation] [Equation]
W
[Equation]
Despite their remarkable success, most attention layers are developed based on heuristic approaches, and a coherent principled framework for synthesizing attention layers has remained elusive.
1. C ONTRIBUTION
We derive the self-attention as the support vector expansion of a given support vector regression (SVR) problem.
The primal representation of the regression function has the form of a neural network layer.
Thus, we establish a primal-dual connection between an attention layer in transformers and a neural network layer in deep neural networks.
Our framework suggests a principled approach to developing an attention mechanism: Starting from a neural network layer and a support vector regression problem, we derive the dual as a support vector expansion to attain the corresponding attention layer.
We then employ this principled approach to invent two novel classes of attentions: the Batch Normalized Attention (Attention-BN) derived from the batch normalization layer in deep neural networks and the Attention with Scaled Heads (Attention-SH) resulting from solving the support vector regression model with less amount of training data.
Our contribution is three-fold.
We derive self-attention as a support vector expansion that solves a SVR problem, thus providing a principled primal-dual framework to study and develop self-attentions.
We re-derive popular attentions, such as the linear attention (Katharopoulos et al., 2020), the sparse attention (Child et al., 2019)
and the multi-head attention (Vaswani et al., 2017), from our proposed framework.
We develop two new attention mechanism: the Batch Normalized Attention (Attention-BN) and the Attention with Scaled Heads (Attention-SH) using our proposed framework.
We empirically demonstrate that 1) the Attention-BN significantly outperforms the baseline softmax and linear attention and 2) the Attention-SH performs better while being more efficient than the same baselines on a variety of practical tasks including image and time-series classification.
P RIMAL -D UAL I NTERPRETATION OF S ELF -A TTENTION
We first provide a primal-dual interpretation of self-attention as a support vector regression problem in Section 2.1.
Based on that primal-dual framework, we derive popular attention mechanisms as the support vector expansion in Section 2.2.
Finally, we introduce two new attention mechanisms in
Section 2.3, the Attention-BN and Attention-SH.
2. A TTENTION AS A S UPPORT V ECTOR R EGRESSION M ODEL
In this section, we derive self-attention from a support vector regression problem.
Suppose we are given a training data { ( k 1 , y 1 ) , . . .
, ( k N , y N ) } ⊂ K × Y , where K = R D and Y = R D v .
Here, k 1 , . .
, k N are attention keys in self-attention, and y 1 , . . .
, y N are the training targets.
We consider the function f , taking the form
[Equation]
where x ∈ K = R D , Φ( x ) =
[ ϕ 1 ( x ) , . . .
ϕ D ϕ ( x )] ∈ R D ϕ , W = [ w 1 , . .
. , w D v ] ⊤ ∈ R D
v × D ϕ , b ∈ R D v , and h ( x ) is a vector-scalar function.
We fit the function f to the training data { ( k 1 , y 1 ) , . . .
, ( k N , y N ) } with an L 2 regularization on W , i.e., a ridge regression, by solving the following convex optimization problem:
[Equation]
1 ,...,N
[Equation]
d
[Equation]
d
[Equation]
subject to
[Equation]
The Eqn. 5 implies that there exists a function f that can approximates all pairs ( k j , y j ) with ϵ precision.
The additional slack variables ξ j , ˜ ξ j relax this assumption and allows some of the training set data points to have the training error greater than ϵ just as in the soft-margin SVM (Cortes &
Vapnik, 1995; Sch ¨ olkopf et al., 2002).
C > 0 is a constant determining the trade between the complexity penalizer (cid:80) D v d =1 ∥ w d ∥ 2 , i.e., the flatness of f , and the amount up to which deviations larger than ϵ are tolerated.
In order to derive the self-attention from the support vector regression defined by the optimization problem 5, the key idea to construct the Lagrangian from Eqn. 5 and find the representation of the w d , d = 1 , . . .
, D v , in terms of the dual variables.
We define the Lagrangian function as follows:
[Equation]
where η j , ˜ η j , α j and ˜ α
j are Lagrange multipliers.
These dual variables have to satisfy positivity constraints, i.e., η j ( d ) , ˜ η j ( d ) , α j ( d ) , ˜ α j ( d ) ≥ 0 , ∀ j = 1 , . .
N, ∀ d = 1 , . .
, D v .
It follows from the saddle point condition that the partial derivatives of the Lagrangian function L with respect to the primal variables (cid:16) w d , b ( d ) , { ξ j ( d ) , ˜ ξ j ( d ) } N j =1 (cid:17) , d = 1 , . . .
, D v , have to vanish for optimality, namely, we have:
[Equation]
Let v j =
[ α j (1) − ˜
α j (1) h ( k j ) , . .
, α j ( D v ) − ˜ α j (
[Equation]
[Equation] [Equation]
) ] ⊤ , j = 1 , . .
N
[Equation]
and substitute Eqn. 8 into Eqn.
[Equation]
we obtain the following support vector expansion of the linear basis function f :
[Equation]
Remark 1 Notice that from Eqn.
9 and the conditions
η j ( d ) , ˜ η j ( d ) , α j ( d ) , ˜ α j ( d ) ≥ 0 , we can prove that α j ( d ) , ˜ α j ( d ) ∈
[0 , C ] .
Furthermore, we can show that α j ( d ) ∗ ˜ α j ( d ) =
(Smola &
Sch¨olkopf, 2004; Sch¨olkopf et al., 2002).
As a result, v j ( d ) ∈ (cid:104) − C h ( k j ) , C h ( k j ) (cid:105) , d = 1 , . . .
, D v .
Deriving Softmax Attention.
Choosing the appropriate h ( x ) and Φ( x ) allows us to derive the popular softmax attention given in Eqn. 1 and 2.
In particular, if we choose h ( x ) :
= (cid:80) Nj Φ( x ) T Φ( k j ) ,
Eqn. 10 becomes
[Equation]
We then select Φ( x ) =
(cid:16) a (0) l 0 , a (1)1 , . . .
, a (1) l 1 , . . .
, a ( t ) 1 , . . .
, a ( t ) l t , . . .
(cid:17) where l t = (cid:0) D + t − 1 t (cid:1) and
[Equation]
Since exp
[Equation]
then Eqn. 27 becomes
[Equation]
Let x = q i , b = 0 and relax the boundness constraint of v j in Remark 1.
Eqn. 30 becomes Eqn.
2 of the softmax attention (Vaswani et al., 2017).
We summarize our results in the following theorem.
Theorem 1 (Softmax Attention as a Support Vector Expansion)
Given the function f defined in
Eqn. 4 with h ( x ) := (cid:80) Nj Φ( x ) T Φ( k j ) and the support vector regression problem defined in Eqn.
5, we set b = 0 , choose Φ( x ) as in Eqn. 28, and relax the boundness constraint of the variables v j =
[ α j (1) − ˜
α j (1) h ( k j ) , . .
, α j ( D v ) − ˜ α
j ( D v ) h ( k j ) ] ⊤ , where α j and ˜ α
j are dual variables of Eqn.
5, j = 1 , . . .
, N .
Then, the support vector expansion of f derived from Eqn.
5 has the form of a softmax attention
[Equation]
Remark 2 Since b is set to 0, the centering constraint of α j and ˜ α j in Eqn.
7 can be ignored.
Remark 3 Theorem 1 and its derivation can be easily extended to capture the full form of the softmax attention with the residual connection, the query matrix projection W Q , the key matrix projection
W K , and the value matrix projection W V .
We include this result in Appendix F.
Remark
The primal representation of the function f as in Eqn. 4 has the form of a neural network layer where W is the weight, b is the bias term, Φ( x ) is the input, and h ( x ) is the normalization term.
Thus, an attention layer and a neural network layer are primal-dual of each other.
A principled approach to developing an attention mechanism.
The observation in Remarksuggests a principled way to construct an attention layer: Starting from a neural network layer and a support vector regression problem, we derive the dual as a support vector expansion to attain the corresponding attention layer.
Using this approach, we derive popular attention mechanisms in
Section 2.2 and propose our new attention mechanisms in Section 2.3.
2. D ERIVING P OPULAR A TTENTION M ECHANISMS AS THE S UPPORT V ECTOR E XPANSION
In this section, we derive popular attentions such as the linear attention (Katharopoulos et al., 2020), the sparse attention (Child et al., 2019), and the multi-head attention (Vaswani et al., 2017).
2.2. L INEAR A TTENTION
The Eqn. 27, which is obtained when choosing h ( x ) := (cid:80) Nj Φ( x ) T Φ( k j ) , already matches the formula of the linear attention.
Here, we can let b = 0 as above and select the function Φ that results in a positive similarity function, e.g. Φ( x ) = elu ( x ) + 1 , as in (Katharopoulos et al., 2020).
2.2. S PARSE A TTENTION
The sparse attention (Child et al., 2019) can be derived by fitting the function f in Eqn.
4 using a different subset { (
k m x (1) ,
y m x (1) ) , . .
, ( k m x ( M ) , y m x ( M ) ) } of training data { ( k 1 , y 1 ) , . . .
, ( k N , y N ) } for each input data x , where M x = { m x (1) , . . .
, m x ( M ) } ⊂ { 1 , . . .
, N } .
The support vector expansion of f is then given by
[Equation]
where 1 M x ( j ) =
[ j ∈ M x ] := (cid:26)
if j ∈ M x 0 otherwise .
Note that the subsets M x are different for different x .
When letting x = q i where q i , i = 1 , . .
, N , are the query vectors and choosing Φ , h, b as in Section 2.1, we can obtain the sparse attention in (Child et al., 2019) where the binary matrix
M =
[Equation]
i
[Equation]
i
[Equation]
1 becomes the sparse masking matrix.
2.2. M ULTI - HEAD A TTENTION (MHA)
The MHA can be derived by solving multiple support vector regression problems and then linearly combining their outputs.
In particular, given H training datasets { ( k 11 , y 11 ) , . . .
, ( k 1 N , y 1 N ) } , . .
, { ( k H 1 , y H 1 ) , . . .
, ( k HN , y HN ) } ⊂ K × Y , where K = R D and Y =
R D v .
We define the function f applied on the input vector x
= [ x 1 , . .
, x H ] as follows
[Equation]
where each function f s ( x s )
[Equation]
is fitted to the training dataset { ( k s 1 , y s 1 ) , . . .
, ( k sN , y sN ) } .
Following the same derivation and choosing { Φ s , h s , b s } Hs =1 as in Section 2.1, we can rewrite f ( x ) in terms of the support vector expansions of the individual functions f s ( x s ) , which are the individual softmax attentions
[Equation]
) Comparing Eqn. 18 and Eqn. 3, we see that Eqn.
18 computes the MHA when choosing x s = q si where q si , i = 1 , . .
, N , are the query vectors at the s th head.
2. D ERIVING N EW A TTENTION M ECHANISMS : B ATCH N ORMALIZED A TTENTION AND M ULTIRESOLUTION H EAD A TTENTION
In this section, we employ our primal-dual framework to develop new attention mechanisms.
In particular, we derive: 1) the Batch Normalized Attention from employing the batch normalization (Ioffe & Szegedy, 2015); and 2) the Attention with Scaled Heads from using different amounts of training data.
By 1) and 2), we demonstrate that new attentions can be invented by modifying the primal neural network layer and the support vector regression problem in our framework, respectively .
2.3. B ATCH N ORMALIZED A TTENTION
We incorporate the batch normalization into the primal form of the function f in Eqn. 4.
Given a training data { ( k 1 , y 1 ) , . . .
, ( k N , y N ) } ⊂ K × Y , where K = R D and Y = R D v as in Section 2.1, the resultant f is defined as follows
[Equation]
where
[Equation]
Here, d = 1 , . . .
, D , and the mean subtraction and division by the standard deviation is performed element-wise along the feature dimension of x .
Following the same derivation as in Section 2.1, we derive the following support vector expansion of f
[Equation]
Here,
[Equation]
⊤ , where α j and ˜ α
j are the dual variables, j = 1 , . . .
, N .
Same as in Section 2.1, in Eqn. 21, we choose Φ as in Eqn.
28, h ( x ) :
= (cid:80) Nj Φ( x ) T Φ( k j ) , and b = 0 to obtain the the Batch Normalized Attention, which is defined as follows.
Definition 1 (Batch Normalized Attention)
Given a set of the key and value vectors { k j , v j } Nj =1 , for each query vector q i , i = 1 , . . .
, N , the Batch Normalized Attention (Attention-BN) computes the corresponding output vector h i of the query q i by the following attention formula:
[Equation]
where
[Equation]
The Effect of Normalization.
Expanding the dot product in the Attention-BN (see Appendix E),
Eqn. 22 becomes
[Equation]
Eqn. 24 implies that in the Attention-BN, the similarity between the query q i and the key k j is adjusted by the similarity between the key k j and all the keys k j ′ , j ′
= 1 , . .
. , N .
In particular, if the key k j is too similar to other keys, the query q i will attend to it less and vice versa.
2.3. A TTENTION WITH S CALED H EADS
The Attention with Scaled Heads , named Attention-SH, is derived based on the derivation of the MHA in Section 2.2.3.
The key idea underlying the Attention-SH is to train multiple support vector regression problems using different amounts of training data.
In particular, the
Attention-SH follows Eqn. 17 in Section 2.2.3 and defines the same regression function f as the
MHA.
However, the Attention-SH fits the function f s , s = 1 , . . .
, H , in Eqn.
17 with training sets { ( k 11 , y 11 ) , . .
, ( k 1 N 1 , y 1 N 1 ) } , . .
, { ( k H 1 , y H 1 ) , . . .
, ( k HN H , y HN H ) } ⊂ K × Y of different sizes
N 1 , . .
N H , where K = R D and Y =
R D v .
The resultant support vector expansion yields the formula of the Attention-SH as in the following definition.
Definition 2 (Attention with Scaled Heads)
Given H sets of the key and value vectors { k 1 j , v 1 j } N 1 j =1 , . . .
, { k H j , v Hj } N H j =1 , for each set of H query vectors q 1 i , . . .
, q Hi , i = 1 , . .
, N , the
Attention with Scaled Heads (Attention-SH) computes the corresponding output vector h i of the queries q 1 i , . .
q Hi by the following attention formula:
[Equation]
[Table]
Remark
For a given input sequence X :=
[ x 1 , · · · , x N ] ⊤ ∈ R N × D x of N feature vectors in self-attention, in order to generate the sets of { k sj , v sj } N s j =1 at the scale s th , we can downsample the input X before projecting into the key matrix K and the value matrix V .
There are multiples approaches to downsampling X , such as using the average-pooling, max-pooling, 1-D convolution, or K-means clustering.
In this paper, we employ the average-pooling to downsample X .
Linear Attention with Batch Normalization and Scaled Heads.
The Attention-BN/SH can be extended to use with the linear attention.
In particular, in the Linear Attention-BN/SH, we replace the softmax kernel in Eqn.
22 and Eqn.
by the linear kernel, respectively.
E XPERIMENTAL R ESULTS
In this section, we empirically demonstrate the advantages of our Attention-BN, Attention-SH, and their combination (Attention-BN+SH) over the baseline softmax attention on the UEA timeseries classification benchmark (Bagnall et al., 2018), the Long Range Arena benchmark (Tay et al., 2021), and the image classification task on the Imagenet dataset (Deng et al., 2009; Russakovsky et al., 2015).
We aim to show that: (i) Attention-BN significantly outperforms the softmax baseline across tasks; (ii) Attention-SH achieves better or comparable accuracy while saving computation and memory compared to the baseline; (iii) Attention-BN+SH, which combines both Attention-BN and Attention-SH, results in the best model performance in term of accuracy and efficiency; (iv) all our proposed models help reduce the redundancy in multi-head attention and benefit learning of the long-term dependency in long input sequences; (v) Attention-BN and Attention-SH can be applied on other attention mechanisms beyond the softmax attention.
When combined with the linear attention (Katharopoulos et al., 2020), the resultant Linear Attention-BN and Linear Attention-SH yield similar advantages mentioned in (i), (ii), (iii) and (iv) over the baseline linear attention.
In our experiments, we compare the proposed models with the baseline softmax and linear attentions of the same configuration.
For the Attention-BN and Attention-BN+SH, we observe that recentering queries and keys alone is sufficient for improving the model performance.
In addition, weighting µ with a constant β , as in Eqn. 26 in the Appendix, enables the Attention-BN/BN+SH to adjust the effect of normalization to the attention score and help increase the accuracy.
Our results are averaged over 5 runs.
Details on datasets, models, and training are provided in Appendix A.
UEA Time Series Classification.
Table 1 compares the accuracy of the Attention-BN and AttentionSH with the baseline softmax attention on 10 tasks in the UEA Time Series Classification benchmark (Bagnall et al., 2018).
Both Attention-BN and Attention-SH significantly outperform the softmax baseline on most tasks and on average among all tasks.
When combining two models, the resulting Attention-BN+SH yields the best accuracy with more than 1% overall improvement over the softmax baseline.
Notably, the Attention-SH and Attention-BN+SH are much more efficient than the baseline since they need much fewer keys and values in computing the attention output.
The efficiency advantage of the Attention-SH/BN+SH is discussed and analyzed in detail in Section 4.
Long Range Arena (LRA) benchmark.
In this experiment, we verify the advantage of our methods over the softmax baseline on tasks that involve very long sequences (e.g., the sequence length can be up to 4K) in the LRA benchmark (Tay et al., 2021).
Those tasks require the model to capture long-range dependency in the input sequence.
The summarized results in Table 2 indicate significant improvements of Attention-BN/SH/BN+SH over the baseline softmax attention.
Same as in the UEA
[Table] [Table]
Time Series experiment, on this LRA benchmark, Attention-BN and Attention-SH both outperform the softmax attention on most five tasks.
Moreover, Attention-BN+SH, which combines these two attention mechanisms, results in the most accurate models on average across tasks.
Specifically, for the retrieval task, the most challenging task with the largest sequence length in the LRA benchmark,
Attention-BN+SH achieve a remarkable improvement of more than 1 .
5% over the baseline.
Image Classification on Imagenet.
We corroborate the advantage of our proposed attention over the baseline softmax attention when scaled up for the large-scale ImageNet image classification task.
We summarize the results in Table 3.
The Deit model (Touvron et al., 2021) equiped with the
Attention-BN yields better performance than the softmax baseline.
Meanwhile, Attention-SH/BN+SH
Deit perform on par with the baseline while being more efficient.
These results, together with other results above justify the benefits of our proposed methods across various tasks and data modalities, proving the effectiveness of our primal-dual approach to develop new attentions.
E MPIRICAL A NALYSIS
Efficiency Analysis.
The Attention-BN+SH not only improves the accuracy of the model remarkably but also help reduce the computational and memory cost significantly.
Fig.1 presents the efficiency benefits of our Attention-BN+SH trained on the retrieval task when the model dimension D and sequence lengths N grow.
The efficiency advantage of our model increase as N increase.
In addition, the scaled-up models (with large D ) remains significantly more efficient than the baseline.
When the model dimension is 64 and sequence length is 4096, which is the standard configuration of the task, the model’s FLOPS, in both training and inference, reduce almost 25% , whereas the reductions for memory usage in training and testing are 31 .
9% and 47 .
3% , respectively.
Notably, this efficient model also outperforms the baseline with more than 1 . 5% improvement in accuracy.
These results prove the benefits of applying the Attention-BN+SH for long-sequence tasks and large-scale models.
New Attentions Helps Reduce Head Redundancy.
We compute the average L 2 distances between heads to analyze the attention diversity.
Given our trained models for the retrieval task, the layeraverage mean and standard deviation of distances between heads are reported in Table 4.
All our introduced attentions attain greater L 2 distances compared to the baseline, reducing the risk of learning redundant heads.
In particular, Attention-SH has the highest head difference, indicating the model’s attention patterns are most spread out between heads.
[Table]
Combining Attention-BN and Attention-SH with Other Attentions.
Our methods can be extended to combine with other attention mechanisms.
We study the Linear Attention-BN/SH/BN+SH, that combine the Attention-BN/SH/BN+SH with the linear attention (Katharopoulos et al., 2020) as explained at the end of Section 2.3.
We summarize our results in Table 5 in Appendix B.1.
R ELATED W ORK
Interpretation of Attention Mechanism.
Recent works have focused on understanding the attention mechanism in transformers from different perspectives.
(Tsai et al., 2019) considers attention as a weighted moving average over the inputs via a smoothing kernel.
(Nguyen et al., 2022) draws a connection between self-attention and nonparametric kernel regression.
With this understanding, the work explores better regression estimators, e.g. the generalized Fourier nonparametric regression estimator, to improve transformers.
In addition, (Cao, 2021) then shows that the linear transformer (Katharopoulos et al., 2020) corresponds to a Petrov-Galerkin projection (Reddy, 2004) and proves that the softmax normalization in the softmax attention is sufficient but not necessary.
Other works that employ ordinary/partial differential equations to provide an interpretation for attention include (Lu et al., 2019;
Sander et al., 2022).
From a probabilistic perspective, (Tang & Matteson, 2021; Gabbur et al., 2021; Zhang & Feng, 2021) propose Gaussian mixture model frameworks to study the self-attention in transformers.
Using graph-structured learning and message passing in graphical models is another attempt at understanding the attention mechanism Wang et al.
Shaw et al.
(2018); Kreuzer et al. (2021).
Optimization perspectives of attention mechanisms are recently explored.
(Sander et al., 2022) connects transformers with an optimization process across iterations by specifically constructing the core energy function.
(Sahiner et al., 2022) derive finitedimensional convex equivalence of attentions that can be solved for global optimality.
Different from these approaches, our primal-dual framework focuses on deriving attention as the dual expansion of a primal neural network layer via solving a support vector regression problem.
This framework allows us to not only explain many different types of attention mechanisms but also create new ones.
Efficient Transformers.
Recently, efficient transformers have been studied (Roy et al., 2021).
Among them are sparse transformers which incorporate sparse structures into the attention matrix (Parmar et al., 2018; Liu et al., 2018; Qiu et al., 2019; Child et al., 2019; Beltagy et al., 2020).
Another class of efficient transformers are models that aim to have better coverage by integrating different access patterns (Child et al., 2019; Ho et al., 2019), which can also be learned from the data (Kitaev et al., 2020; Roy et al., 2021; Tay et al., 2020).
An emerging body of work is proposed to distill and prune the model, including (Sanh et al., 2019; Sun et al., 2019; Voita et al., 2019; Sajjad et al., 2020).
In other works, a side memory module is utilized in order to access multiple tokens simultaneously (Lee et al., 2019; Sukhbaatar et al., 2019; Asai & Choi, 2020; Beltagy et al., 2020).
Low-rank and kernelization methods have been proposed to improve the efficiency of self-attention calculation (Tsai et al., 2019; Wang et al., 2020; Katharopoulos et al., 2020; Choromanski et al., 2021;
Shen et al., 2021; Peng et al., 2021).
Our Attention-SH/BN+SH is orthogonal to these methods.
C ONCLUDING R EMARKS
In this paper, we derive self-attention as a support vector expansion that solves a support vector regression (SVR) problem and provide a principled primal-dual framework to analyze and synthesize attention mechanisms.
We then use our framework to invent two new attention mechanisms, the
Batch Normalized Attention (Attention-BN) and the Attention with Scaled Heads (Attention-SH), that improve the accuracy and the efficiency of the baseline softmax attention.
In our work, we approximate and learn the dual variables α j and ˜ α
j using the value vector v j , j = 1 , . .
, N in self-attention.
It is natural to include more inductive biases and structures of those dual variables from solving the dual optimization problem of SVR into the value vectors v j .
Furthermore, extending our framework to explain the attention modules that compute the attention using neural network layers or convolutional neural network layers applying on the input feature, such as the Convolutional Block
Attention Module (Woo et al., 2018), is an interesting research direction.
We leave these exciting research ideas as future work.
A CKNOWLEDGEMENTS
This material is based on research sponsored by the NSF under Grant# 2030859 to the Computing
Research Association for the CIFellows Project (CIF2020-UCLA-38).
SJO acknowledges support from the ONR N00014-20-1-2093 and N00014-20-1-2787 and the NSF DMS 2208272 and 1952339.
RGB acknowledges support from the NSF grants CCF-1911094, IIS- and IIS-1730574;
ONR grants N00014-18-12571, N00014-20-1-2534, and MURI N00014-20-1-2787; AFOSR grant
FA9550-22-1-0060; and a Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047.
ALB acknowledges support from the NSF grants DMS-2152717 and DMS-1952339.
NH acknowledges support from the NSF IFML 2019844 and the NSF AI Institute for Foundations of Machine Learning.
A A DDITIONAL DETAILS ON THE EXPERIMENTS
This section provides datasets, models, and training details for experiments in Section 3.
As mentioned in Section 3, for Attention-BN models, recentering queries and keys alone is sufficient for accuracy improvement, and we weight the mean µ in Eqn 22 with a constant β .
Hence Eqn 22 is simplified to:
[Equation]
In our experiments, we consider the constant β in Attention-BN/BN+SH and the different downsampling scales in Attention-SH/SH+BN as hyper-parameters to finetune.
All of our experiments are conducted on a server with 4 NVIDIA A100 GPUs.
A. UEA TIME SERIES CLASSIFICATION
Datasets and metrics The benchmark (Bagnall et al., 2018) consists of 30 datasets.
Following (Wu et al., 2022), we choose 10 datasets, which vary in input sequence lengths, the number of classes, and dimensionality, to evaluate our models on temporal sequences.
We report the test accuracy as evaluation for the benchmark.
Models and baselines The experiment setups and configurations for the softmax/linear baseline and our models are the same as in (Wu et al., 2022) 1 (for the PEMS-SF, SelfRegulationSCP2,
UWaveGestureLibrary datasets) and (Zerveas et al., 2021) 2 (for the other tasks).
In all models, the number of heads is 8, whereas the model dimension and number of transformer layers are varied.
For
Attention-SH/SH+BN, we downsample keys and values by the factor of 2, after every two successive heads.
A. L ONG RANGE ARENA BENCHMARK
Datasets and metrics We adopt the tasks: Listops (Nangia & Bowman, 2018), byte-level IMDb reviews text classification (Maas et al., 2011), byte-level document retrieval (Radev et al., 2013),
CIFAR-10 image classification (Krizhevsky et al., 2009) and the Pathfinder challenge (Linsley et al., 2018) in the LRA benchmark for our experiments.
They consist of long sequences of length 2 K , 4 K , 4 K , 1 K , and 1 K respectively.
The evaluation protocol and metric are the same as in (Tay et al., 2021).
Models and baselines All our models and softmax/linear baselines follow the same architecture and configuration as in (Zhu et al., 2021) 3 .
Each model consists of two layers and 64 embedding dimensions.
While one head at each layer remains intact, the keys and values of the other heads are halved in our Attention-SH/SH+BN experiments.
A. I MAGE C LASSIFICATION ON I MAGENET
Datasets and metrics The ImageNet dataset (Deng et al., 2009; Russakovsky et al., 2015) consists of 1 .
M training images and 50 K validation images.
The task is to classify 1000 categories.
Topand top-5 accuracies are reported.
Models and baselines Our baseline is DeiT-tiny model (Touvron et al., 2021) withtransformer layers, 4 attention heads per layer, and the model dimension of 192.
For model setting and setting and configuration, we follow (Touvron et al., 2021) 4 .
The downsampling scales in
Attention-SH/BN+SH models are [1 , 1 , 2 , 4] for 4 heads at each layer, respectively.
[Table] [Table]
[Table]
B A DDITIONAL E XPERIMENTAL R ESULTS B. UEA T IME S ERIES C LASSIFICATION USING THE
L INEAR A TTENTION -
BN
/SH/
BN+
SH
Table 5 summarizes the comparison between the Linear Attention-BN/SH/BN+SH and the baseline
Linear Attention on the UEA Time Series Classification task.
The Linear Attention-BN/SH/BN+SH achieve better accuracy than the Linear Attention baseline while being more efficient.
B. C ONVOLUTION A TTENTION
Table 6 demonstrates the advantage of Attention-Conv2D (Def. 3, Section G) over softmax Deit on the ImageNet image classification task.
Furthemore, as shown in Table 7, the Attention-Conv1D (Def. 4, Section G) outperforms the baseline softmax attention on 5 tasks of the LRA benchmark (Tay et al., 2021).
B. A DDITIONAL EXPERIMENTS ON THE UEA T IMESERIES C LASSIFICATION BENCHMARK AND THE UCR T IME S ERIES R EGRESSION A RCHIVE
In this section, we further demonstrate the advantage of our Attention-BN/SH/BN+SH on additional 15 tasks in the UEA Time Series Classification benchmark and on 6 tasks in the UCR Time Series
Regression benchmark.
The results in Table 8 and 9 show that our Attention-BN and AttentionSH+BN outperform the baseline softmax transformers significantly on both of these benchmarks, while the attention-SH has comparable performance with the baseline but being more effiicient.
B. UEA T IME S ERIES C LASSIFICATION USING THE S PARSE A TTENTION -BN/SH/BN+SH
Table 10 summarizes the comparison between the Sparse Attention-BN/SH/BN+SH and the Sparse
Attention baseline on a subset of the UEA Time Series Classification benchmark.
Our models when combined with Sparse Attention achieve significantly better accuracy than the Sparse Attention baseline while the Sparse Attention-SH/BN+SH are more efficient (See Fig.
3 and Fig. 4 in
Appendix C).
[Table] [Table]
[Table] [Table]
B. A TTENTION -BN/BN+SH WITH LEARNABLE β
We experiment with our Attention-BN/BN+SH with learnable β on the retrieval task.
Table 11 shows that learning β does not improve much over setting β to be a hyperparameter
[Table]
C A DDITIONAL R ESULTS ON E FFICIENCY A NALYSIS
This section provides more efficiency analysis on our models.
Attention-SH.
Fig.2 shows the efficiency benefits of our Attention-SH when trained on the retrieval task.
Same as in the case of Attention-SH+BN, the efficiency benefits of our Attention-SH over the baseline Softmax attention grows when N and D increase.
Sparse Attention-SH/BN+SH.
Fig.3 and Fig.4 show that the efficiency advantages of our Sparse
Attention-BN+SH and Sparse Attention-SH, respectively, increase as the model dimension D and sequence length N grow.
All models are trained on the LRA retrieval task.
In addition to the efficiency advantage, the Sparse Attention-BN+SH also significantly outperforms the Sparse Attention baseline in terms of accuracy in this task (79.86% vs. 78.20%) while the Sparse Attention-SH achieves a comparable result to the baseline.
More accuracy advantages of the Sparse Attention-BN/SH/BN+SH over the Sparse Attention baseline are given in Table 10.
D D ERIVING S OFTMAX A TTENTION .
Choosing the appropriate h ( x ) and Φ( x ) allows us to derive the popular softmax attention given in
Eqn. 1 and 2.
In particular, if we choose h ( x ) :
= (cid:80) Nj Φ( x ) T Φ( k j ) , Eqn.
10 becomes
[Equation]
We then select Φ( x ) =
(cid:16) a
[Equation] [Equation]
[Equation]
where l t = (cid:0) D + t
[Equation]
1 t
[Equation]
and
[Equation] [Equation]
Since
[Equation]
then Eqn. 27 becomes
[Equation]
Let x = q i , b = 0 and relax the boundness constraint of v j in Remark 1.
Eqn. 30 becomes Eqn.
2 of the softmax attention (Vaswani et al., 2017)
[Equation]
E B ATCH N ORMALIZED A TTENTION : D ERIVATION OF E QN .h i =
[Equation]
[Table]
[Equation]
F A TTENTION WITH THE R ESIDUAL C ONNECTION AND M ATRIX P ROJECTIONS
In this supplement, we first discuss attention with the residual connection and matrix projections in
Appendix F.
Suppose we are given a training data { ( x 1 , y 1 ) , . . .
, ( x N , y N ) } ⊂ X × Y , where X =
R D x and
Y = R D v .
Here, x 1 , . . .
, x N are the training inputsd, and y 1 , . . .
, y N are the training targets.
In order to derive the attention with the residual connection and query, key, and value matrix projections, we define the function f as follows
[Equation]
where x ∈ X
=
R D x , W proj =
[ w proj 1 , . . .
, w proj D ] ⊤ ∈ R D
× D x , Φ( · )
=
[ ϕ 1 ( · ) , . . . , ϕ D ϕ ( · )] : R D →
R D ϕ , W = [ w 1 , . .
. , w D v ] ⊤ ∈ R D
v × D ϕ , b ∈ R D v , and h ( x ) is a vector-scalar function.
We fit the function f to the training data { ( x 1 , y 1 ) , . . .
, ( x N , y N ) } with an L 2 regularization on W and
W proj by solving the following convex optimization problem:
[Equation]
subject to
[Equation]
The Lagrangian of the optimization problem 33 is given by
[Equation]
Similar to the derivation in Section 2.1, the partial derivatives of L 1 with respect to the primal variable w d , d = 1 , . .
, D v , have to vanish for optimality, which leads to
[Equation]
Note that here we only find the form of the optimal solution for W = [ w 1 , . . .
, w D v ] ⊤ .
The optimal value of W proj can then be found by optimization algorithm such as the (stochastic) gradient descent when training the transformer.
Let v j =
[ α
[Equation] [Equation]
−
[Equation]
α
[Equation] [Equation]
, α j ( D v ) − ˜ α
j ( D v ) h ( x
[Equation]
) ] ⊤ , j = 1 , . .
, N , we obtain the following support vector expansion of the function f :
[Equation]
Here, the support vector expansion of f already includes a residual connection.
The softmax attention can then be derived by selecting h ( x ) :
= (cid:80) Nj Φ( W proj x ) T Φ( W proj x j ) and choosing Φ as in
Eqn.
28 in Section 2.1.
Note that in Eqn. 36, { x j } Nj =1 and x are the training samples and test sample, respectively.
In order to derive the key, query, and value matrix projections in attention, we can then relax Eqn. 36 by letting W proj x j = W K x j , W proj x = W Q x , v j = W V x j and choosing the test sample x among the training samples { x j } N j =1 .
Remark 6 Here, for self-attention, we choose the test sample x among the training samples { x j } Nj = to compute the attention score of a token to other tokens in the same sequence.
For cross-attention where a token in a sequence attends to tokens in another sequence, this constraint can be removed.
G 2D-C ONVOLUTION A TTENTION
In this section, we discuss attention with 2D-convolution.
Suppose we are given a training data { ( x train, y train) , . . .
, ( x trainN H × N W , y trainN H × N W ) } ⊂ X × Y , where X =
R D x and Y =
R D v .
Here, x train, . . .
, x trainN H × N W are the training inputs, and y train, . . .
, y trainN H × N W are the training targets.
Let X train ∈ R N H × N W × D
x
be the 3D-tensor of training inputs, where X train ( h, w, d ) =
x trainN W × ( h − 1)+ w ( d ) .
Given a new set of inputs { x 1 , . . .
, x N H × N W } ⊂ X and the corresponding 3D-tensor X ∈ R N H
× N W × D x of these inputs, where X ( h, w, d ) =
x N W × ( h − 1)+ w ( d ) .
We consider the function f applying on the 3D-tensor X and taking the following form
[Equation]
where Conv2D is the depth-wise 2D-convolution (Howard et al., 2017), with the kernel size s × s and identical kernel channels, applied on the input tensor X .
Here, the last dimension of X , i.e., D x , is the depth.
Also, Φ( x ) =
[ ϕ 1 ( x ) , . . .
ϕ D ϕ ( x )] ∈ R D ϕ , W = [ w 1 , . .
. , w D v ] ⊤ ∈
R D v × D ϕ , b ∈ R D v , and h is a vector-scalar function.
We fit the function f to the training data { ( x train, y train) , . . .
, ( x trainN H × N W , y trainN H × N W ) } with an L 2 regularization on W , i.e., a ridge regression, by solving the following convex optimization problem: minimize
[Equation]
subject to 
[Equation]
The Lagrangian of the optimization problem 38 is given by
[Equation] [Equation]
( d ) (cid:17) −
N H × N W (cid:88) j =
D v (cid:88)
d = (cid:16) η j ( d ) ξ j ( d ) + ˜ η j ( d )˜ ξ j ( d ) (cid:17) −
N H × N W (cid:88) j =
D v (cid:88) d = α j ( d ) (cid:32) ϵ + ξ j ( d ) − y trainj
( d ) + w ⊤ d
Φ( Flatten ( Conv2D ( X train , s ))( j ))
h ( x trainj )
+ b ( d ) (cid:33) −
N H × N W (cid:88) j =
D v (cid:88) d = ˜ α j ( d ) (cid:32) ϵ + ˜ ξ j ( d ) + y trainj ( d ) − w ⊤ d
Φ( Flatten ( Conv2D ( X train , s ))( j ))
h ( x trainj ) − b ( d ) (cid:33) , (39)
Similar to the derivation in Section 2.1 in the main text, the partial derivatives of L with respect to the primal variable w d , d = 1 , . .
, D v , have to vanish for optimality, which leads to ∂ w d L =
w d −
N H × N W (cid:88)
j = ( α j ( d ) − ˜ α j ( d ))
Φ( Flatten ( Conv2D ( X train , s ))
( j ))
h ( x trainj ) =(40)
⇒ w d =
N H × N W (cid:88)
j = ( α j ( d ) − ˜ α j ( d ))
Φ( Flatten ( Conv2D ( X train , s ))
( j ))
h ( x trainj ) .
Let v
[Equation] [Equation]
, j = 1 , . .
. , N H × N W , and substitute Eqn.
41 into
Eqn. 38, we obtain the following support vector expansion of the linear basis function f :
[Equation]
where A ij := Φ( Flatten ( Conv2D ( X , s ))
( i ))
⊤ Φ( Flatten ( Conv2D ( X train , s ))
( j )) .
Same as in Section 2.1, we set b s = 0 .
To derive the softmax normalization in attention, we choose h ( x i ) :
= (cid:80) Nj =
1 A ij and select Φ as in Eqn.
Let the training inputs { x train, . . .
, x trainN H × N W } ⊂ X be the attention keys { k 1 , . . .
, k N H × N W } ⊂ K , where K = R D , in self-attention.
Also, let the new inputs { x 1 , . . .
, x N H × N W } ⊂
X be the attention queries { q 1 , . . .
, q N H × N W } ⊂ K in self-attention.
We define the 2D-Convolution Attention (Attention-Conv2D) as follows:
Definition 3 (2D-Convolution Attention)
Given a set of the key and value vectors { k j , v j } N H × N W j
=
, and a set of the query vectors { q i } N H
× N W i = .
Denote the key tensor and the query tensor by K ∈
R N H × N W × D and Q ∈ R N H × N W × D , respectively, where K ( h, w, d ) =
k N W × ( h − 1)+ w ( d ) and
Q ( h, w, d ) =
q N W × ( h − 1)+ w ( d ) .
The 2D-Convolution Attention (Attention-Conv2D) computes the corresponding output vector h i of the query q i by the following attention formula:
[Equation]
where the Conv 2 D ( · , s ) is the depth-wise 2D-convolution (Howard et al., 2017) with the kernel size s × s and identical kernel channels.
Remark 7 (Convolutional Projection for Attention in the Convolutional vision Transformer)
The convolutional projections used in the Convolutional vision Transformer (CvT) (Wu et al., 2021) can be derived from Eqn.
42 by letting the training input tensor X train to be the 2D input matrix of size N × D x
of the self-attention layer (see Section 1.1 in the main text) reshaped into a 3D tensor of size N H
× N W × D x where N = N H
×
N W .
Here, to avoid confusion, we denote the input of the self-attention layer by X input and its reshaped version by Reshape2D ( X input ) .
We then replace the depth-wise 2D-convolution by the depth-wise separable 2D-convolution in (Wu et al., 2021) and remove the constraint that the kernels have identical channels.
In order to derive the convolutional projections for the keys, queries, and values in CvT, for i, j = 1 , . . .
, N , we let k j = Flatten ( Conv2D ( X train , s ))
( j ))
=
Φ( Flatten ( Conv2D ( Reshape2D ( X input ) , s, W K ))
( j ) , q i =
Flatten ( Conv2D ( X , s ))
( i ))
=
Φ( Flatten ( Conv2D ( Reshape2D ( X input ) , s, W Q ))
( i ) , v j = Flatten ( Conv2D ( X train , s ))
( j ))
=
Φ( Flatten ( Conv2D ( Reshape2D ( X input ) , s, W V ))( j ) .
Here, we specify the kernel/filter W K , W Q , and W V to emphasize that the convolutional projections in CvT uses different kernels to compute keys, queries, and values in self-attention.
Eqn.matches the convolutional projects in CvT. By choosing h and Φ similar to above, we can derive the convolutional attention in CvT.
H 1D-C ONVOLUTION A TTENTION
Following the derivation for the Attention-Conv2D in Appendix G above, we can derive the 1DConvolution Attention (Attention-Conv1D) in a similar way by letting X train ∈ R N × D x and
X ∈ R N × D x be 2D-matrices of training inputs and new inputs, respectively, and by replacing
Conv2D by Conv1D, which is the depth-wise 1D-convolution, with the kernel size s × 1 and identical kernel channels, applied on the input tensor X .
Here, the last dimension of X , i.e., D x , is the depth.
We define the 1D-Convolution Attention (Attention-Conv1D) as follows:
Definition 4 (1D-Convolution Attention)
Given a set of the key and value vectors { k j , v j } Nj =1 , and a set of the query vectors { q i } matrix matrix
[ k 1 , . . .
, k N ] ⊤ ∈ R N × D and Q :=
[ q 1 , . . . , q N ] ⊤ ∈ R N × D , respectively.
The 1D-Convolution
Attention (Attention-Conv1D) computes the corresponding output vector h i of the query q i by the following attention formula:
[Equation]
where the Conv 1 D ( · , s ) is the depth-wise 1D-convolution with the kernel size s × 1 and identical kernel channels.
I A TTENTION WITH B ATCH N ORMALIZATION AND SCALED HEADS
The
Attention-BN+SH combines both the
Attention-BN and
Attention-SH.
The
Attention-BN+SH fits the function f s , s = 1 , . . .
, H , in Eqn.
17 with training sets { ( k 11 , y 11 ) , . .
, ( k 1 N 1 , y 1 N 1 ) } , . .
, { ( k H 1 , y H 1 ) , . . .
, ( k HN H , y HN H ) } ⊂ K × Y of different sizes
N 1 , . .
N H , where K = R D and Y =
R D v .
The function f s is defined as:
[Equation] [Equation]
where
[Equation]
Following the same derivation as in Section 2.1, we derive the following support vector expansion of
[Equation]
Here, v sj =
[Equation]
(cid:21) ⊤ , where α sj and ˜ α sj are the dual variables, j = 1 , . . .
, N .
Same as in Section 2.1, in Eqn. 48, we choose Φ as in Eqn.
28, h s ( x ) :
= (cid:80) N
s j Φ( x ) T Φ( k sj ) , and b s = 0 to obtain the Batch Normalized Attention with Scaled Heads (Attention-BN+SH), which is defined as follows:
Definition 5 (Batch Normalized Attention with Scaled Heads)
Given H sets of the key and value vectors { k 1 j , v 1 j } N 1 j =1 , . . .
, { k Hj , v Hj } N H j =1 , for each set of H query vectors q 1 i , . . .
, q Hi , i = 1 , . .
, N , the Batch Normalized Attention with Scaled Heads (Attention-BN+SH) computes the corresponding output vector h
i of the queries q 1 i , . . .
, q Hi by the following attention formula:
[Equation]
where
[Equation]
Following the same Remark 5 in Section 2.3.2, given input sequence X :=
[ x 1 , · · · , x N ] ⊤ ∈ R N × D x of N feature vectors in self-attention, in order to generate the sets of { k sj , v sj } N s j =1 at the scale s th , we can downsample the input X before projecting into the key matrix K and the value matrix V .
In this paper, we use the average-pooling to downsample X .
As in the same case of Attention-BN, for Attention-BN+SH, recentering queries and keys alone are sufficient for accuracy improvement, and we weight the mean µ in Eqn 49 with a constant β .
Hence Eqn. 49 is simplified to:
[Equation]
[Table] [Table]
[Table]
J H YPERPARAMETERS
In this section
[Table]
we provide the hyper-parameters for our best models.
J. UEA T IME S ERIES C LASSIFICATION AND R EGRESSION
For these two benchmarks, use the set of downsampling factors s = [1 , 1 , 2 , 2 , 4 , 4 , 8 , 8] for AttentionSH/BN+SH and Linear/Sparse Attention-SH/BN+SH models trained on the UEA benchmark.
Table 13 and Table 12 provide the values of β used for our best Attention-BN/BN+SH and Linear
Attention-BN/BN+SH, Sparse Attention-BN/BN+SH models trained on subsets of the two benchmarks.
J. L ONG R ANGE A RENA BENCHMARK
For all 5 tasks of the LRA benchmark, we set the downsampling factors s of Attention-SH/BN+SH,
Linear/Sparse Attention-SH/BN+SH is [1 , 2] and kernel size of Attention-Conv1D models is 5.
In addition, Table 14 provides the values β of Attention-BN/BN+SH models trained on the benchmark.
J. I MAGENET C LASSIFICATION
This task’s β of Attention-BN/BN+SH is 1.
Attention-SH/BN+SH has the downsampling factor of [1 , 1 , 2 , 4] , and the kernel size of Attention-Conv2D is (2 , 2) .