FourierFormer: Transformer Meets Generalized Fourier Integral Theorem
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Multi-head attention empowers the recent success of transformers, the state-of-theart models that have achieved remarkable success in sequence modeling and beyond.
These attention mechanisms compute the pairwise dot products between the queriesand keys, which results from the use of unnormalized Gaussian kernels with theassumption that the queries follow a mixture of Gaussian distribution.
There is noguarantee that this assumption is valid in practice.
In response, we ﬁrst interpretattention in transformers as a nonparametric kernel regression.
We then proposethe FourierFormer, a new class of transformers in which the dot-product kernelsare replaced by the novel generalized Fourier integral kernels.
Different from thedot-product kernels, where we need to choose a good covariance matrix to capturethe dependency of the features of data, the generalized Fourier integral kernels canautomatically capture such dependency and remove the need to tune the covariancematrix.
We theoretically prove that our proposed Fourier integral kernels can efﬁciently approximate any key and query distributions.
Compared to the conventionaltransformers with dot-product attention, FourierFormers attain better accuracyand reduce the redundancy between attention heads.
We empirically corroboratethe advantages of FourierFormers over the baseline transformers in a variety ofpractical applications including language modeling and image classiﬁcation.
Introduction
Transformers [72] are powerful neural networks that have achieved tremendous success in manyareas of machine learning [35, 66, 31] and become the state-of-the-art model on a wide rangeof applications across different data modalities, from language [20, 1, 15, 10, 52, 4, 7, 18] toimages [21, 38, 68, 53, 49, 24], videos [3, 39], point clouds [84, 26], and protein sequence [55, 29].
In addition to their excellent performance on supervised learning tasks, transformers can alsoeffectively transfer the learned knowledge from a pretraining task to new tasks with limited or nosupervision [50, 51, 20, 82, 37].
At the core of transformers is the dot-product self-attention, whichmainly accounts for the success of transformer models [11, 46, 36].
This dot-product self-attentionlearn self-alignment between tokens in an input sequence by estimating the relative importance of agiven token with respect to all other tokens.
It then transform each token into a weighted average ofthe feature representations of other tokens where the weight is proportional to a importance scorebetween each pair of tokens.
The importance scores in self-attention enable a token to attend to othertokens in the sequence, thus capturing the contextual representation [5, 72, 33].
1.1 Self-Attention
Given an input sequence X :=
[ x 1 , · · · , x N ] > 2 R N ⇥ D x of N feature vectors, self-attentioncomputes the output sequence H from X as follows:
Step 1:
Projecting the input sequence into different subspaces.
The input sequence X is transformed into the query matrix Q , the key matrix K , and the value matrix V via three linear
Submitted to
Do
distribute.
transformations
[Equation]
where W Q , W K 2 R D ⇥ D x , and W V 2 R D v ⇥ D x are the weight matrices.
We denote Q :=[ q 1 , · · · , q N ] > , K :=
[ k 1 , · · · , k N ] > , and V :=
[ v 1 , · · · , v N ] > , where the vectors q i , k i , v i for 37 i = 1 , · · · , N are the query, key, and value vectors, respectively.
Step 2:
Computing the output as a weighted average.
The output sequence H :=
[ h 1 , · · · , h N ] > 39 is then given by
[Equation]
where the softmax function is applied to each row of the matrix ( QK > ) / p D .
For each query vectorq i , i = 1 , · · · , N , Eqn.
(1) can be written in the vector form to compute the output vector h
i asfollows
[Equation]
The matrix A 2 R N ⇥ N and its component a ij for i, j = 1 , · · · , N are the attention matrix andattention scores, respectively.
The self-attention computed by equations (1) and (2) is called the dotproduct attention or softmax attention.
In our paper, we refer a transformer that uses this attention asthe baseline transformer with the dot-product attention or the dot-product transformer.
The structureof the attention matrix A after training governs the ability of the self-attention to capture contextualrepresentation for each token.
Multi-head Attention
Each output sequence H forms an attention head.
Multi-head attentionconcatenates multiple heads to compute the ﬁnal output.
Let H be the number of heads and
W O 2 R HD v ⇥ HD v be the projection matrix for the output.
The multi-head attention is deﬁned as
MultiHead
[Equation]
The capacity of the attention mechanism and its ability to learn diverse syntactic and semanticrelationships determine the success of transformers [67, 73, 14, 74, 27].
However, equations (1)and (2) implies that the dot-product attention assumes the features ( q i 1 , . .
. , q iD ) in q i , as well asthe features ( k j 1 , . .
. , q jD ) in k j , are independent.
Thus, the dot-product attention fail to capture thecorrelations between these features, limiting its representation capacity and inhibit the performanceof transformers on practical tasks where there is no guarantee that independent features can learnedfrom complex data.
One solution to capture correlations between features q i and k j is to introducecovariance matrices into the formulation of the dot-product attention with the cost of signiﬁcantlyincreasing of the computational complexity.
Also, choosing good covariance matrices is difﬁcult.
1.2 Contribution
In this paper, we ﬁrst establish a correspondence between self-attention and nonparametric kernelregression.
Under this new perspective of self-attention, we explain the limitation of the dot-productself-attention that it may fail to capture correlations between the features in the query and keyvectors.
We then leverage the generalized Fourier integral theorems, which can automatically capturethese correlations, and derive the generalized Fourier integral estimators for the nonparametricregression problem.
Using this new density estimator, we propose the FourierFormer, a novelclass of transformers that can capture correlations between features in the query and key vectors ofself-attention.
In summary, our contribution is three-fold:
We derive the formula of self-attention from solving a nonparametric kernel regressionproblem, thus providing a nonparametric regression interpretation to study and furtherdevelop self-attention.
We develop the generalized Fourier integral estimators for the nonparametric regressionproblem and provide theoretical guarantees for these estimator.3.
We propose the FourierFormer
Finally, we empirically show that the FourierFormer attains signiﬁcantly better accuracy than thebaseline transformer with the dot-product attention on a variety of tasks including the WikiTextlanguage modeling and ImageNet image classsiﬁcation.
We also demonstrate in our experiments that
FourierFormer helps reduce the redundancy between attention heads.
Organization We structure this paper as follows:
In Section 2, we present the correspondencebetween self-attention and nonparametric kernel regression.
In Section 3, we discuss the generalized
Fourier integral estimators and deﬁne the FourierFormer.
We validate and empirically analyze theadvantages of FourierFormer in Section 4.
We discuss related works in Section 5.
The paper ends withconcluding remarks.
Technical proofs and more experimental details are provided in the Appendix.
Notation For any N 2 N , we denote [ N ] = { 1 , 2 , . . .
, N } .
For any D  1 , L 1 ( R D ) denotes thespace of real-valued functions on R D that are integrable.
For any two sequences { a N } N  1 , { b N } N  1 , 89 we denote a
N = O ( b N ) to mean that a N  Cb N for all N  1 where C is some universal constant.
A Nonparametric Regression Interpretation of Self-attention
In this section, we establish the connection between self-attention and nonparametric kernel regression.
In particular, we derive the self-attention in equation (2) as a nonparametric kernel regression inwhich the key vectors k j and value vectors v j are training inputs and training targets, respectively,while the query vectors
q i and the output vectors
h i form a set of new inputs and their correspondingtargets that need to be estimated, respectively, for i, j = 1 , · · · , N .
In general, we can view thetraining set { k j , v j } for j 2 [ N ] to come from the following nonparametric regression model :
[Equation]
where " 1 , . .
. , " N are independent noises such that E ( " j ) = 0 .
Furthermore, we consider a randomdesign setting where the key vectors k 1 , k 2 , . . .
, k N are i.i.d.
samples from the distribution thatadmits p as density function.
By an abuse of notation, we also denote p as the joint density where thekey and value vectors ( v 1 , k 1 ) , . . .
, ( v N , k N ) are i.i.d.
samples from.
Here, f is a true but unknownfunction and we would like to estimate it.
Nadaraya–Watson estimator Our approach to estimate the function f is based on
Nadaraya–Watson’s nonparametric kernel regression approach [45].
In particular, from the nonparametric regression model (3), we have E [ v j | k j ] = f
( k j ) for all j 2 [ N ] .
Therefore, it is sufﬁcient toestimate the conditional distribution of the value vectors given the key vectors.
Given the densityfunction p of the key vectors and the joint density p of the key and value vectors, for any pair ofvectors ( v , k ) generate from model (3) we have
[Equation]
The formulation (4) of the conditional expectation indicates that as long as we can estimate the jointdensity function p ( v , k ) and the marginal density function p ( v ) , we are able to obtain an estimationfor the conditional expectation and thus for the function f .
This approach is widely known as
Nadaraya–Watson’s nonparametric kernel regression approach.
Kernel density estimator To estimate p ( v , k ) and p ( k )
, we employ the kernel density estimationapproach [56, 47].
In particular, by using the isotropic Gaussian kernel with bandwidth  , we havethe following estimators of p ( v , k ) and p ( k ) :
[Equation]
[Equation]
where '  ( . ) is the isotropic multivariate Gaussian density function with diagonal covariance matrix 2 I D .
Given the kernel density estimators (5), we obtain the following estimation of the function f :
[Equation]
Connection between Self-Attention and nonparametric regression By plugging the query vectorsq i into the function b f  in equation (6), we obtain that
[Equation]
If we further assume that the keys k j are normalized, which is usually done in practice to stabilizethe training of transformers [61], the value of b f  ( q i ) in equation (6) then becomes
[Equation]
When we choose  2 = p D where D is the dimension of q i and k j , equation (8) matches equation (2) of self-attention, namely, b f  ( q i )
=
h i .
Thus, we have shown that self-attention performsnonparametric regression using isotropic Gaussian kernels.
Remark 1 The assumption that k j is normalized is to recover the pairwise dot-product attention intransformers.
In general, this assumption is not necessary.
In fact, the isotropic Gaussian kernel inequation (7) is more desirable than the dot-product kernel in equation (8) of the pairwise dot-productattention since the former is Lipschitz while the later is not Lipschitz [32].
The Lipschitz constrainthelps improve the robustness of the model [13, 71, 2] and stabilize the model training [43].
Limitation of Self-Attention From our nonparametric regression interpretation, self-attention isderived from the use of isotropic Gaussian kernels for kernel density estimation and nonparametricregression estimation, which may fail to capture the complex correlations between D featuresin q i and k j [77, 28].
Using multivariate Gaussian kernels with dense covariance matrices canhelp capture such correlations; however, choosing good covariance matrices is challenging andinefﬁcient [76, 63, 9].
In the following section, we discuss the Fourier integral estimator and its useas a kernel for computing self-attention in order to overcome these limitations.
FourierFormer: Transformer via Generalized Fourier Integral Theorem
In the following, we introduce generalized integral theorems that are able to capture the complexinteractions among the features of the queries and keys.
We then apply these theorems to densityestimation and nonparametric regression problems.
We also establish the convergence rates of theseestimators.
Given these density estimators, we introduce a novel family of transformers, named
FourierFormer , that integrates the generalized Fourier integral theorem into the dot-product attentionstep of the standard transformer.
3.1 (Generalized) Fourier Integral Theorems and Their Applications
The Fourier integral theorem is a beautiful result in mathematics [81, 6] and has been recently usedin nonparametric mode clustering, deconvolution problem, and generative modeling [28].
It is acombination of Fourier transform and Fourier inverse transform.
In particular, for any functionp 2 L 1 ( R D ) , the Fourier integral theorem is given by
[Equation]
where k = ( k 1 , . . . , k D ) and y =
( y 1 , . .
. , y D ) .
Equation (9) suggests that p R ( k ) :
= y can be used as an estimator of the function p .
Beneﬁts of the Fourier integral over Gaussian kernel There are two important beneﬁts of theestimator p R : (i) it can automatically preserve the correlated structure lying within p even when p is very complex and high dimensional function.
It is in stark contrast to the standard kernel estimatorbuilt based on multivariate Gaussian kernel where we need to choose good covariance matrix in themultivariate Gaussian kernel to guarantee such estimator to work well.
We note that as the standardsoft-max Transformer is constructed based on the multivariate Gaussian kernel, the issue of choosinggood covariance matrix in dot-product transformer is inevitable; (ii) The product of sinc kernels inthe estimator p R does not decay to a point mass when R ! 1 .
It is in stark difference from themultivariate Gaussian kernel estimator, which converges to a point mass when the covariance matrixgoes to 0.
It indicates that p R is a non-trivial estimator of the function p .
Finally, detailed illustrationsof these beneﬁts of the Fourier integral over Gaussian kernel in density estimation and nonparametricregression problems, which we have just shown to have connection to the self-attention in transformer,can be found in Section 8 in [28].
Generalized Fourier integral estimator Borrowing the above beneﬁts of Fourier integral estimatorp R , in the paper we would like to consider a generalization of that estimator, named generalized
Fourier integral estimator , which is given by:
[Equation]
where A :=
R
R  ⇣ sin
[Equation]
z ⌘ dz and  : R !
R is a given function.
When  ( k ) =
k for allk 2 R D , the generalized Fourier integral estimator p  R becomes the Fourier integral estimator p R .
Under appropriate conditions on the function  (see Theorem 1 in Section 3.1.1 and Theorem 3 in
Appendix A.1), the estimator p  R converges to the true function p , namely,
[Equation]
We name the above limit as generalized Fourier integral theorem .
Furthermore, the estimator p  R alsoinherits similar aforementioned beneﬁts of the Fourier integral estimator p R .
Therefore, we will usethe generalized Fourier integral theorem as a building block for constructing density estimators andnonparametric regression estimators, which are crucial to develop the FourierFormer in Section 3.2.
3.1.1 Density Estimation via Generalized Fourier Integral Theorems
We ﬁrst apply the generalized Fourier integral theorem to the density estimation problem.
To ease thepresentation, we assume that k 1 , k 2 , . . .
, k N 2 R D are i.i.d.
samples from a distribution admittingdensity function p where D  1 is the dimension.
Inspired by the generalized Fourier integraltheorem, we obtain the following generalized Fourier density estimator p  N,R of p as follows:
[Equation]
where A =
R
R  ⇣ sin( z ) z ⌘ dz and k i =
( k i 1 , . . . , k iD ) for all
i 2 [ N ] .
To quantify the error betweenthe generalized Fourier density estimator p  n,R and the true density p , we utilize mean integratedsquared errors (MISE) [80], which is given by:
[Equation]
We start with the following bound on the MISE between p  n,R and p .
Theorem 1 Assume that
[Equation]
sin( z ) /z )
z j dz = 0 for all j 2 [ m ] and
[Equation]
(sin [Equation]
|
[Equation] 1 for some m 2 N .
Then, there exist universal constants C and C’ depending on d and A such that
[Equation]
Proof of Theorem 1 is in Appendix B.1.
A few comments are in order.
First, by choosing Rto balance the bias and variance in the bound of MISE in Theorem 1, we have the optimal R as
R = O
( N 1 / ( D + m +1) ) .
With that choice of R , the MISE rate of p  N,R is O
( N  ( m +1) /
( D + m +1) ) .
Second, when  ( z ) =
z l for l  4 and z 2 R , the assumptions in Theorem 1 are satisﬁed whenm = 1 .
Under this case, the MISE rate of p  N,R is O
( N  2 / ( D +2) ) .
However, these assumptionsdo not satisfy when  ( z ) =
z l and l 2 { 1 , 2 , 3 } , which is due to the limitation of the current prooftechnique of Theorem 1 that is based on Taylor expansion of the estimator p  n,R .
To address the limitation of the Taylor expansion technique, we utilize the Plancherel theorem in
Fourier analysis to establish the MISE rate of p  N,R when  ( z ) =
z l and l 2 { 1 , 2 , 3 } .
The details ofthe theoretical analyses for such setting are in Appendix A.
3.2 FourierFormer: Transformers with Fourier Attentions
Motivated by the preservation of the correlated structure of the function from the generalized Fourierintegral theorem as well as the theoretical guarantees of density estimators, in this section we adaptthe nonparametric regression interpretation of self-attention in Section 2 and propose the generalized
Fourier nonparametric regression estimator in Section 3.2.1.
We also establish the convergenceproperties of that estimator.
Then, based on generalized Fourier nonparametric regression estimator,we develop the Fourier Attention and its corresponding FourierFormer in Section 3.2.2.
3.2.1 Nonparametric Regression via Generalized Fourier Integral Theorem
We now discuss an application of the generalized Fourier integral theorems to the nonparametricregression setting (3), namely, we assume that ( v 1 , k 1 ) , . . .
, ( v N , k N ) are i.i.d.
samples from thefollowing nonparametric regression model:
[Equation]
where " 1 , . .
. , " N are independent noises such that E ( " j ) =
and the key vectors k 1 , k 2 , . .
k N arei.i.d.
samples from p .
Given the generalized Fourier density estimator (12), following the argument in
Section 2, the Nadaraya–Watson estimator of the function f based on the generalized Fourier densityestimator is given by:
[Equation]
The main difference between the generalized Fourier nonparametric regression estimator f N,R inequation (14) and the estimator b f  in equation (6) is that the estimator f N,R utilizes the generalized
Fourier density estimator to estimate the conditional distribution of the value vectors given the keyvectors instead of the isotropic Gaussian kernel density estimator as in b f  .
As we highlighted in
Section 3, an important beneﬁt of the generalized Fourier density estimator is that it can capture thecomplex dependencies of the features of the value vectors and the key vectors while the Gaussiankernel needs to have good covariance matrix to do that, which is computationally expensive inpractice.
We now have the following result establishing the mean square error (MSE) of f N,R .
Theorem 2 Assume that
[Equation]
for all 1 
[Equation]

[Equation]
and
[Equation]
< [Equation] for any m +1  j  2 m +2 for some m 2 N .
Then, for any k 2 R D , there exist universal constants 221 C 1 , C 2 , C 3 , C 4 such that the following holds:
[Equation]
, where J ( R ) = 1
[Equation]
⌘ .
Here, the outer expectation is taken withrespect to the key vectors k 1 , . . .
, k N and the noises " 1 , . . .
, " N .
Proof of Theorem 2 is in Appendix B.3.
A few comments with Theorem 2 are in order.
First, bychoosing R to balance the bias and variance in the bound of the MSE of the nonparametric generalized
Fourier estimator f N,R , we have the optimal radius R as R =
O ( N 1 2( m +1)+ D ) .
With that choice ofthe optimal radius R , the rate of f N,R is O
( N  2( m +1)
D +2( m +1) ) .
Second, when  ( z ) =
z l for l  6 , the assumption on the function  of Theorem 2 is satisﬁed with m = 1 .
Under this case, the rate of f N,Rbecomes O ( N  4 D +4 ) .
In Appendix A, we also provide the rate of f N,R when  ( z ) =
z l for somel  5 , which includes the original Fourier integral theorem.
3.2.2 FourierFormer
Given the generalized Fourier nonparametric regression estimator f N,R in equation (14), by pluggingthe query values q 1 , . . .
, q N into that function, we obtain the following deﬁnition of the Fourierattention:
Deﬁnition 1 (Fourier Attention)
A Fourier attention is a multi-head attention that does nonparametric regression using the generalized Fourier nonparametric regression estimator f N,R .
The outputˆ h
i of the Fourier attention is then computed as
[Equation]
Given the Fourier Attention in Deﬁnition 1, we then give the deﬁnition of FourierFormer as follows.
Deﬁnition 2 (FourierFormer)
A FourierFormer is a transformer that uses Fourier attention tocapture dependency between tokens in the input sequence and the correlation between features ineach token.
Remark 2 (The Nonnegativity of the Fourier Kernel)
The density estimation via generalized
Fourier integral theorem in Section 3.1.1 does not require the generalized Fourier density estimator to be nonnegative.
However, empirically, we observe that negative density estimator can causeinstability in training the FourierFormer.
Thus, in FourierFormer, we choose the function  to be anonnegative function to enforce the density estimator to be nonnegative.
In particular, we choose  tobe power functions of the form  ( x ) = x 2 m , where m is an positive integer.
Note that when m =
2and m = 4 , the kernels in our generalized Fourier integral estimators are the well-known Fejer-de la
Vallee Poussin and Jackson-de la Vallee Poussin kernels [17].
3.3 An Efﬁcient Implementation of the Fourier Attention
The Fourier kernel is implemented efﬁciently in the C++/CUDA extension developed by Pytorch [48].
The idea is similar to the function cdist [48], which computes the p-norm distance between each pair of the two collections of row vectors.
In our case, we aim to compute kernel functions that represent a Fourier attention in Deﬁnition 1.
The core of this implementation is the following Fourier metric function d f :
[Equation]
We directly implement d f as a torch.autograd.
Function [48] in which we provide an efﬁcientway to compute forward and backward function ( d f and gradient of d f ).
While the implementationof the forward function is straight forward, the backward function is more tricky since we need tooptimize the code to compute the gradient of d f w.r.t to variables q , k , and R all at once.
We candevelop the backward function with highly parallel computation by exploiting GPU architecture andutilizing the reduction technique.
The computational time is comparable to function cdist ; thus, our
FourierFormer implementation is as computationally time-efﬁcient.
Experimental Results
In this section, we numerically justify the advantage of FourierFormer over the baseline dot-producttransformer on two large-scale tasks: language modeling on WikiText-103 [41] (Section 4.1) andimage classiﬁcation on ImageNet [19, 57] (Section 4.2).
We aim to show that: (i) FourierFormerachieves better accuracy than the baseline transformer on a variety of practical tasks with differentdata modalities, and (ii)
FourierFormer helps reduce head redundancy compared to the baselinetransformer (Section 4.3).
Throughout the section, we compare FourierFormers with the baseline dot-product transformersof the same conﬁguration.
In all experiments, we made the constant R in Fourier attention (see
[Table]
equation (15)) to be a learnable scalar and set choose the function  ( x ) = x 4 (see Remark 2).
All ofour results are averaged over 5 runs with different seeds.
More details on the models and training areprovided in Appendix C. We also provide additional experimental results in Appendix D.
Modeling on [-WikiText-
-]WikiText-
Datasets and metrics WikiText-103 is a collection of articles from Wikipedia, which have longcontextual dependencies.
The training set consists of about 28 K articles containing 103 M runningwords; this corresponds to text blocks of about 3600 words.
The validation and test sets have 218 Kand 246 K running words, respectively.
Each of them contains 60 articles and about 268 K words.
Ourexperiment follows the standard setting [41, 61] and splits the training data into L -word independentlong segments.
For evaluation, we use a batch size of 1, and process the text sequence with a slidingwindow of size L .
The last position is used for computing perplexity (PPL) except in the ﬁrst segment,where all positions are evaluated as in [1, 61].
Models and baselines Our implementation is based on the public code by [61].
We use theirsmall and medium models in our experiments.
In particular, for small models, the key, value, andquery dimension are set to 128, and the training and evaluation context length are set to 256.
Formedium models, the key, value, and query dimension are set to 256, and the training and evaluationcontext length are set to 384.
In both conﬁgurations, the number of heads is 8, the feed-forward layerdimension is 2048, and the number of layers is 16.
Results We report the validation and test perplexity (PPL) of FourierFormer versus the baselinetransformer with the dot-product attention in Table 1.
FourierFormers attain much better PPL than thebaselines in both small and medium conﬁgurations.
For the small conﬁguration, the improvements of
FourierFormer over the baseline are 1.29 PPL in validation and 1.44 PPL in test.
For the mediumconﬁguration, these improvements are 1.39 PPL in validation and 1.59 PPL in test.
These resultssuggest that the advantage of FourierFormer over the baseline dot-product transformer grows with themodel’s size.
This meets our expectation because larger models has larger query and key dimensions,e.g. the language model with medium conﬁguration in this experiment has the query and keydimension of 256 versus 128 as in the language model with small conﬁguration.
Since the advantageof FourierFormer results from the property that FourierFormer can capture correlation betweenfeatures in query and key vectors, the larger the query and key dimensions are, the more advantage
FourierFormer has.
4.2 Image Classiﬁcation on ImageNet
Datasets and metrics The ImageNet dataset [19, 57] consists of 1 .
M training images and 50 Kvalidation images.
For this benchmark, the model learns to predict the category of the input imageamong 1000 categories.
Top-1 and top-5 classiﬁcation accuracies are reported.
Models and baselines We use the DeiT-tiny model [69] with 12 transformer layers, 4 attention headsper layer, and the model dimension of 192.
To train the models, we follow the same setting andconﬁguration as for the baseline [69].
Results We summarize our resuls in Table 2.
Same as in the language modeling experiment, for thisimage classiﬁcation task, the Deit model equipped with FourierFormer signiﬁcantly outperforms thebaseline Deit dot-product transformer in both top-1 and top-5 accuracy.
This result suggests that theadvantage of FourierFormer over the baseline dot-product transformer holds across different datamodalities.
[Table] [Table]
4.3 FourierFormer Helps Reducing Head Redundancy
[Table] To study the diversity between attention heads, given the model trained for the WikiText-103 languagemodeling task, we compute the average L 2 distance between heads in each layer.
We show thelayer-average mean and variance of distances between heads in Table 3.
Results in Table 3 showsthat
FourierFormer obtains greater L 2 distance between attention heads than the baseline transformerwith the dot-product attention and thus helps reduce the head redundancy.
Note that we use the smallconﬁguration as speciﬁed in Section 4.1 for both models.
Related Work
Interpretation of Attention Mechanism in Transformers Recent works have tried to gain anunderstanding of transformer’s attention from different perspectives.
[70] considers attention asapplying kernel smoother over the inputs.
Extending this kernel approach, [30, 12, 78] linearize thesoftmax kernel in dot-product attention and propose a family of efﬁcient transformers with linearcomputational and memory complexity.
[8] then shows that these linear transformers are comparableto a Petrov-Galerkin projection [54], suggesting that the softmax normalization in the dot-productattention is sufﬁcient but not necessary.
Other works provide an understanding of attention intransformers via ordinary/partial differential equation include [40, 59].
In addition, [65, 25, 83] relateattentions in transformers to a Gaussian mixture models.
Several works also connect the attentionmechanism to graph-structured learning and message passing in graphical models [79, 62, 34].
Ourwork focuses on deriving the connection between self-attention and nonparametric kernel regressionand exploring better regression estimator, such as the generalized Fourier nonparametric regressionestimator, to improve the performance of transformers.
Redundancy in Transformers [16, 42, 22] show that neurons and attention heads in the pre-trainedtransformer are redundant and can be removed when applied on a downstream task.
By studyingthe contextualized embeddings in pre-trained networks, it has been demonstrated that the learnedrepresentations from these redundant models are highly anisotropic [44, 23].
Furthermore, [60, 64, 75,58] employ knowledge distillation and sparse approximation to enhance the efﬁciency of transformers.
Our FourierFormer is complementary to these methods and can be combined with them.
Concluding Remarks
In this paper, we establish the correspondence between the nonparametric kernel regression and theself-attention in transformer.
We then develop the generalized Fourier integral estimators and proposethe FourierFormer, a novel class of transformers that use the generalized Fourier integral estimators toconstruct their attentions for efﬁciently capturing the correlations between features in the query andkey vectors.
We theoretically prove the approximation guarantees of the generalized Fourier integralestimators and empirically validate the advantage of FourierFormer over the baseline transformerwith the dot-product attention in terms of accuracy and head redundancy reduction.
It is interestingto incorporate robust kernels into the nonparametric regression framework of FourierFormer toenhance the robustness of the model under data perturbation and adversarial attacks.
A limitation of
FourierFormer is that it still has the same quadratic computational and memory complexity as thebaseline transformer with the dot-product attention.
We leave the development of the linear versionof FourierFormer that achieves linear computational and memory complexity as future work.
It isworth noting that there is no potential
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A] Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 6(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] [Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[N/A]
[Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]