FourierFormer: Transformer Meets Generalized Fourier Integral Theorem
Abstract
Multi-head attention empowers the recent success of transformers, the state-of-theart models that have achieved remarkable success in sequence modeling and beyond.
These attention mechanisms compute the pairwise dot products between the queries and keys, which results from the use of unnormalized Gaussian kernels with the assumption that the queries follow a mixture of Gaussian distribution.
There is no guarantee that this assumption is valid in practice.
In response, we first interpret attention in transformers as a nonparametric kernel regression.
We then propose the FourierFormer, a new class of transformers in which the dot-product kernels are replaced by the novel generalized Fourier integral kernels.
Different from the dot-product kernels, where we need to choose a good covariance matrix to capture the dependency of the features of data, the generalized Fourier integral kernels can automatically capture such dependency and remove the need to tune the covariance matrix.
We theoretically prove that our proposed Fourier integral kernels can efficiently approximate any key and query distributions.
Compared to the conventional transformers with dot-product attention, FourierFormers attain better accuracy and reduce the redundancy between attention heads.
We empirically corroborate the advantages of FourierFormers over the baseline transformers in a variety of practical applications including language modeling and image classification.
Introduction
Transformers [83] are powerful neural networks that have achieved tremendous success in many areas of machine learning [40, 76, 36] and become the state-of-the-art model on a wide range of applications across different data modalities, from language [23, 1, 18, 13, 62, 4, 8, 21] to images [24, 43, 78, 63, 59, 27], videos [3, 44], point clouds [97, 31], and protein sequence [65, 34].
In addition to their excellent performance on supervised learning tasks, transformers can also effectively transfer the learned knowledge from a pretraining task to new tasks with limited or no supervision [60, 61, 23, 94, 42].
At the core of transformers is the dot-product self-attention, which mainly accounts for the success of transformer models [14, 56, 41].
This dot-product self-attention learn self-alignment between tokens in an input sequence by estimating the relative importance of a given token with respect to all other tokens.
It then transform each token into a weighted average of the feature representations of other tokens where the weight is proportional to a importance score between each pair of tokens.
The importance scores in self-attention enable a token to attend to other tokens in the sequence, thus capturing the contextual representation [6, 83, 38].
1.1 Self-Attention
Given an input sequence X :=
[ x 1 , · · · , x N ] ⊤ ∈ R N × D x of N feature vectors, self-attention computes the output sequence H from X as follows:
Step 1:
Projecting the input sequence into different subspaces.
The input sequence X is transformed into the query matrix Q , the key matrix K , and the value matrix V via three linear transformations
[Equation]
where W Q , W K ∈ R D × D x , and W V ∈ R D v × D x are the weight matrices.
We denote Q :=
[ q 1 , · · · , q N ] ⊤ , K :=
[ k 1 , · · · , k N ] ⊤ , and V :=
[ v 1 , · · · , v N ] ⊤ , where the vectors q i , k i , v i for i = 1 , · · · , N are the query, key, and value vectors, respectively.
Step 2:
Computing the output as a weighted average.
The output sequence H :=
[ h 1 , · · · , h N ] ⊤ is then given by
[Equation]
where the softmax function is applied to each row of the matrix ( QK ⊤ ) / √ D .
For each query vector q i , i = 1 , · · · , N , Eqn.
(1) can be written in the vector form to compute the output vector
h
i as follows
[Equation]
The matrix A ∈ R N × N and its component a ij for i, j = 1 , · · · , N are the attention matrix and attention scores, respectively.
The self-attention computed by equations (1) and (2) is called the dotproduct attention or softmax attention.
In our paper, we refer a transformer that uses this attention as the baseline transformer with the dot-product attention or the dot-product transformer.
The structure of the attention matrix A after training governs the ability of the self-attention to capture contextual representation for each token.
Multi-head Attention
Each output sequence H forms an attention head.
Multi-head attention concatenates multiple heads to compute the final output.
Let H be the number of heads and
W O ∈ R HD v × HD v be the projection matrix for the output.
The multi-head attention is defined as
[Equation]
The capacity of the attention mechanism and its ability to learn diverse syntactic and semantic relationships determine the success of transformers [77, 84, 17, 85, 32].
However, equations (1) and (2) implies that the dot-product attention assumes the features ( q i 1 , . .
. , q iD ) in q i , as well as the features ( k j 1 , . .
. , q jD ) in k j , are independent.
Thus, the dot-product attention fail to capture the correlations between these features, limiting its representation capacity and inhibit the performance of transformers on practical tasks where there is no guarantee that independent features can learned from complex data.
One solution to capture correlations between features q i and k j is to introduce covariance matrices into the formulation of the dot-product attention with the cost of significantly increasing of the computational complexity.
Also, choosing good covariance matrices is difficult.
1.2 Contribution
In this paper, we first establish a correspondence between self-attention and nonparametric kernel regression.
Under this new perspective of self-attention, we explain the limitation of the dot-product self-attention that it may fail to capture correlations between the features in the query and key vectors.
We then leverage the generalized Fourier integral theorems, which can automatically capture these correlations, and derive the generalized Fourier integral estimators for the nonparametric regression problem.
Using this new density estimator, we propose the FourierFormer, a novel class of transformers that can capture correlations between features in the query and key vectors of self-attention.
In summary, our contribution is three-fold:
We derive the formula of self-attention from solving a nonparametric kernel regression problem, thus providing a nonparametric regression interpretation to study and further develop self-attention.
We develop the generalized Fourier integral estimators for the nonparametric regression problem and provide theoretical guarantees for these estimator.
We propose the FourierFormer whose attentions use the generalized Fourier integral estimators to capture more efficiently correlations between features in the query and key vectors.
Finally, we empirically show that the FourierFormer attains significantly better accuracy than the baseline transformer with the dot-product attention on a variety of tasks including the WikiText language modeling and ImageNet image classsification.
We also demonstrate in our experiments that
FourierFormer helps reduce the redundancy between attention heads.
Organization We structure this paper as follows:
In Section 2, we present the correspondence between self-attention and nonparametric kernel regression.
In Section 3, we discuss the generalized
Fourier integral estimators and define the FourierFormer.
We validate and empirically analyze the advantages of FourierFormer in Section 4.
We discuss related works in Section 5.
The paper ends with concluding remarks.
Technical proofs and more experimental details are provided in the Appendix.
Notation For any N ∈ N , we denote [ N ] = { 1 , 2 , . . .
, N } .
For any D ≥ 1 , L 1 ( R D ) denotes the space of real-valued functions on R D that are integrable.
For any two sequences { a N } N ≥ 1 , { b N } N ≥ 1 , we denote a N = O ( b N ) to mean that a N ≤ Cb N for all N ≥ 1 where C is some universal constant.
A Nonparametric Regression Interpretation of Self-attention
In this section, we establish the connection between self-attention and nonparametric kernel regression.
In particular, we derive the self-attention in equation (2) as a nonparametric kernel regression in which the key vectors k j and value vectors v j are training inputs and training targets, respectively, while the query vectors q i and the output vectors
h i form a set of new inputs and their corresponding targets that need to be estimated, respectively, for i, j = 1 , · · · , N .
In general, we can view the training set { k j , v j } for j ∈ [ N ] to come from the following nonparametric regression model :
[Equation]
where ε 1 , . .
. , ε N are independent noises such that E ( ε j )
= 0 .
Furthermore, we consider a random design setting where the key vectors k 1 , k 2 , . . .
, k N are i.i.d.
samples from the distribution that admits p as density function.
By an abuse of notation, we also denote p as the joint density where the key and value vectors ( v 1 , k 1 ) , . . .
, ( v N , k N ) are i.i.d.
samples from.
Here, f is a true but unknown function and we would like to estimate it.
Nadaraya–Watson estimator Our approach to estimate the function f is based on
Nadaraya–Watson’s nonparametric kernel regression approach [50].
In particular, from the nonparametric regression model (3), we have E [ v j | k j ] = f
( k j ) for all j ∈ [ N ] .
Therefore, it is sufficient to estimate the conditional distribution of the value vectors given the key vectors.
Given the density function p of the key vectors and the joint density p of the key and value vectors, for any pair of vectors ( v , k ) generate from model (3) we have
[Equation]
The formulation (4) of the conditional expectation indicates that as long as we can estimate the joint density function p ( v , k ) and the marginal density function p ( v ) , we are able to obtain an estimation for the conditional expectation and thus for the function f .
This approach is widely known as
Nadaraya–Watson’s nonparametric kernel regression approach.
Kernel density estimator To estimate p ( v , k ) and p ( k )
, we employ the kernel density estimation approach [66, 57].
In particular, by using the isotropic Gaussian kernel with bandwidth σ , we have the following estimators of p ( v , k ) and p ( k ) :
[Equation]
where φ σ ( . ) is the isotropic multivariate Gaussian density function with diagonal covariance matrix σ 2 I D .
Given the kernel density estimators (5)
[Equation]
we obtain the following estimation of the function f :
[Equation]
Connection between Self-Attention and nonparametric regression By plugging the query vectors q i into the function (cid:98) f
σ in equation (6)
[Equation]
we obtain that
[Equation]
If we further assume that the keys k j are normalized
[Equation]
which is usually done in practice to stabilize the training of transformers [71]
[Equation]
the value of (cid:98) f σ ( q i ) in equation (6) then becomes
[Equation]
When we choose σ 2 = √ D where D is the dimension of q i and k j , equation (8) matches equation (2) of self-attention, namely, (cid:98) f σ ( q i )
=
h i .
Thus, we have shown that self-attention performs nonparametric regression using isotropic Gaussian kernels.
Remark 1 The assumption that k j is normalized is to recover the pairwise dot-product attention in transformers.
In general, this assumption is not necessary.
In fact, the isotropic Gaussian kernel in equation (7) is more desirable than the dot-product kernel in equation (8) of the pairwise dot-product attention since the former is Lipschitz while the later is not Lipschitz [37].
The Lipschitz constraint helps improve the robustness of the model [16, 81, 2] and stabilize the model training [48].
Limitation of Self-Attention From our nonparametric regression interpretation, self-attention is derived from the use of isotropic Gaussian kernels for kernel density estimation and nonparametric regression estimation, which may fail to capture the complex correlations between D features in q i and k j [88, 33].
Using multivariate Gaussian kernels with dense covariance matrices can help capture such correlations; however, choosing good covariance matrices is challenging and inefficient [87, 73, 11].
In the following section, we discuss the Fourier integral estimator and its use as a kernel for computing self-attention in order to overcome these limitations.
FourierFormer: Transformer via Generalized Fourier Integral Theorem
In the following, we introduce generalized integral theorems that are able to capture the complex interactions among the features of the queries and keys.
We then apply these theorems to density estimation and nonparametric regression problems.
We also establish the convergence rates of these estimators.
Given these density estimators, we introduce a novel family of transformers, named
FourierFormer , that integrates the generalized Fourier integral theorem into the dot-product attention step of the standard transformer.
3.1 Generalized Fourier Integral Theorems and Their Applications
The Fourier integral theorem is a beautiful result in mathematics [92, 7] and has been recently used in nonparametric mode clustering, deconvolution problem, and generative modeling [33].
It is a combination of Fourier transform and Fourier inverse transform.
In particular, for any function p ∈
L 1 ( R D ) , the Fourier integral theorem is given by p ( k ) =
[Equation]
sin(
[Equation] [Equation]
y
[Equation] [Equation]
( k
[Equation]
− y j ) p ( y ) d y , (9) where k = ( k 1 , . . . , k D ) , y =
( y 1 , . .
. , y D ) , s =
( s 1 , . . .
, s D ) , and R is the radius.
The detailed derivation of Equation (9) is in Appendix B.3.
Equation (9) suggests that p R ( k ) :=
[Equation]
p
[Equation]
y
[Equation]
d y can be used as an estimator of the function p .
Benefits of the Fourier integral over Gaussian kernel There are two important benefits of the estimator p R : (i) it can automatically preserve the correlated structure lying within p even when p is very complex and high dimensional function.
It is in stark contrast to the standard kernel estimator built based on multivariate Gaussian kernel where we need to choose good covariance matrix in the multivariate Gaussian kernel to guarantee such estimator to work well.
We note that as the standard soft-max Transformer is constructed based on the multivariate Gaussian kernel, the issue of choosing good covariance matrix in dot-product transformer is inevitable; (ii) The product of sinc kernels in the estimator p R does not decay to a point mass when R → ∞ .
It is in stark difference from the multivariate Gaussian kernel estimator, which converges to a point mass when the covariance matrix goes to 0.
It indicates that p R is a non-trivial estimator of the function p .
Finally, detailed illustrations of these benefits of the Fourier integral over Gaussian kernel in density estimation and nonparametric regression problems, which we have just shown to have connection to the self-attention in transformer, can be found in Section 8 in [33].
Generalized Fourier integral estimator Borrowing the above benefits of Fourier integral estimator p R , in the paper we would like to consider a generalization of that estimator, named generalized
Fourier integral estimator , which is given by:
[Equation]
where A := (cid:82) R ϕ (cid:16) sin( z ) z (cid:17) dz and ϕ : R → R is a given function.
When ϕ ( k ) =
k for all k ∈ R D , the generalized Fourier integral estimator p ϕR becomes the Fourier integral estimator p R .
Under appropriate conditions on the function ϕ (see Theorem 1 in Section 3.1.1 and Theorem 3 in
Appendix C.1), the estimator p ϕR converges to the true function p , namely,
[Equation]
We name the above limit as generalized Fourier integral theorem .
Furthermore, the estimator p ϕR also inherits similar aforementioned benefits of the Fourier integral estimator p R .
Therefore, we will use the generalized Fourier integral theorem as a building block for constructing density estimators and nonparametric regression estimators, which are crucial to develop the FourierFormer in Section 3.2.
3.1.1 Density Estimation via Generalized Fourier Integral Theorems
We first apply the generalized Fourier integral theorem to the density estimation problem.
To ease the presentation, we assume that k 1 , k 2 , . .
. , k N ∈ R D are i.i.d.
samples from a distribution admitting density function p where D ≥ 1 is the dimension.
Inspired by the generalized Fourier integral theorem, we obtain the following generalized Fourier density estimator p ϕN,R of p as follows:
[Equation]
where A = (cid:82) R ϕ (cid:16) sin( z ) z (cid:17) dz and k i =
( k i 1 , . . . , k iD ) for all i ∈ [ N ] .
To quantify the error between the generalized Fourier density estimator p ϕn,R and the true density p , we utilize mean integrated squared errors (MISE) [91], which is given by:
[Equation]
We start with the following bound on the MISE between p ϕn,R and p .
Theorem 1 Assume that (cid:82) R ϕ (sin( z ) /z )
z j dz = 0 for all j ∈ [ m ] and (cid:82) R | ϕ (sin( z ) /z )
||
z | m +1 dz < ∞ for some m ∈ N .
Then, there exist universal constants C and C ′ depending on d and A such that
[Equation]
Proof of Theorem 1 is in Appendix D.1.
A few comments are in order.
First, by choosing R to balance the bias and variance in the bound of MISE in Theorem 1, we have the optimal R as
R = O
( N 1 / ( D + m +1) ) .
With that choice of R , the MISE rate of p ϕN,R is O ( N − ( m +1) /
( D + m +1) ) .
Second, when ϕ ( z ) =
z l for l ≥ 4 and z ∈ R , the assumptions in Theorem 1 are satisfied when m = 1 .
Under this case, the MISE rate of p ϕN,R is O ( N − 2 /
( D +2) ) .
However, these assumptions do not satisfy when ϕ ( z ) =
z l and l ∈ { 1 , 2 , 3 } , which is due to the limitation of the current proof technique of Theorem 1 that is based on Taylor expansion of the estimator p ϕn,R .
To address the limitation of the Taylor expansion technique, we utilize the Plancherel theorem in
Fourier analysis to establish the MISE rate of p ϕN,R
when ϕ ( z ) =
z l and l ∈ { 1 , 2 , 3 } .
The details of the theoretical analyses for such setting are in Appendix C.
3.2 FourierFormer: Transformers with Fourier Attentions
Motivated by the preservation of the correlated structure of the function from the generalized Fourier integral theorem as well as the theoretical guarantees of density estimators, in this section we adapt the nonparametric regression interpretation of self-attention in Section 2 and propose the generalized
Fourier nonparametric regression estimator in Section 3.2.1.
We also establish the convergence properties of that estimator.
Then, based on generalized Fourier nonparametric regression estimator, we develop the Fourier Attention and its corresponding FourierFormer in Section 3.2.2.
3.2.1 Nonparametric Regression via Generalized Fourier Integral Theorem
We now discuss an application of the generalized Fourier integral theorems to the nonparametric regression setting (3), namely, we assume that ( v 1 , k 1 ) , . . .
, ( v N , k N ) are i.i.d.
samples from the following nonparametric regression model:
[Equation]
where ε 1 , . .
. , ε N are independent noises such that E ( ε j )
=
and the key vectors k 1 , k 2 , . .
, k N are i.i.d.
samples from p .
Given the generalized Fourier density estimator (12), following the argument in
Section 2, the Nadaraya–Watson estimator of the function f based on the generalized Fourier density estimator is given by:
[Equation]
The main difference between the generalized Fourier nonparametric regression estimator f N,R in equation (14) and the estimator (cid:98) f σ in equation (6) is that the estimator f N,R utilizes the generalized
Fourier density estimator to estimate the conditional distribution of the value vectors given the key vectors instead of the isotropic Gaussian kernel density estimator as in (cid:98) f σ .
As we highlighted in
Section 3, an important benefit of the generalized Fourier density estimator is that it can capture the complex dependencies of the features of the value vectors and the key vectors while the Gaussian kernel needs to have good covariance matrix to do that, which is computationally expensive in practice.
We now have the following result establishing the mean square error (MSE) of f N,R when D v = 1 .
Theorem 2 Assume that (cid:82) R ϕ (cid:16) sin( z ) z (cid:17)
z j dz = 0 for all 1 ≤ j ≤ m and (cid:82) R (cid:12)(cid:12)(cid:12) ϕ (cid:16) sin( z ) z (cid:17)(cid:12)(cid:12)(cid:12)
| z | j dz < ∞ for any m + 1 ≤ j ≤ 2 m + 2 for some m ∈ N .
Then, for any k ∈ R D , when D v = 1 there exist universal constants C 1 , C 2 , C 3 , C 4 such that the following holds:
[Equation]
, where J ( R ) =
1 − 1 p 2 ( k ) (cid:16)
C
[Equation] [Equation]
(cid:17) .
Here, the outer expectation is taken with respect to the key vectors k 1 , . . .
, k N and the noises ε 1 , . .
. , ε N .
Proof of Theorem 2 is in Appendix D.3.
A few comments with Theorem 2 are in order.
First, by choosing R to balance the bias and variance in the bound of the MSE of the nonparametric generalized
Fourier estimator f N,R , we have the optimal radius R as R =
O ( N 1 2( m +1)+ D ) .
With that choice of the optimal radius R , the rate of f N,R is O ( N − 2( m +
D +2( m +1) ) .
Second, when ϕ ( z ) =
z l for l ≥ 6 , the assumption on the function ϕ of Theorem 2 is satisfied with m = 1 .
Under this case, the rate of f N,R becomes O ( N − 4 D +4 ) .
In Appendix C, we also provide the rate of f N,R when ϕ ( z ) =
z l for some l ≤ 5 , which includes the original Fourier integral theorem.
3.2.2 FourierFormer
Given the generalized Fourier nonparametric regression estimator f N,R in equation (14), by plugging the query values q 1 , . . .
, q N into that function, we obtain the following definition of the Fourier attention:
Definition 1 (Fourier Attention)
A Fourier attention is a multi-head attention that does nonparametric regression using the generalized Fourier nonparametric regression estimator f N,R .
The output ˆ h i of the Fourier attention is then computed as
[Equation]
Given the Fourier Attention in Definition 1, we then give the definition of FourierFormer as follows.
Definition 2 (FourierFormer)
A FourierFormer is a transformer that uses Fourier attention to capture dependency between tokens in the input sequence and the correlation between features in each token.
Remark 2 (The Nonnegativity of the Fourier Kernel)
The density estimation via generalized
Fourier integral theorem in Section 3.1.1 does not require the generalized Fourier density estimator to be nonnegative.
However, empirically, we observe that negative density estimator can cause instability in training the FourierFormer.
Thus, in FourierFormer, we choose the function ϕ to be a nonnegative function to enforce the density estimator to be nonnegative.
In particular, we choose ϕ to be power functions of the form ϕ ( x ) = x 2 m , where m is an positive integer.
Note that when m =and m = 4 , the kernels in our generalized Fourier integral estimators are the well-known Fejer-de la
Vallee Poussin and Jackson-de la Vallee Poussin kernels [20].
3.3 An Efficient Implementation of the Fourier Attention
The Fourier kernel is implemented efficiently in the C++/CUDA extension developed by Pytorch [58].
The idea is similar to the function cdist [58], which computes the p-norm distance between each pair of the two collections of row vectors.
In our case, we aim to compute kernel functions that represent a Fourier attention in Definition 1.
The core of this implementation is the following Fourier metric function d f :
[Equation]
We directly implement d f as a torch.autograd.
Function [58] in which we provide an efficient way to compute forward and backward function ( d f and gradient of d f ).
While the implementation
[Table]
of the forward function is straight forward, the backward function is more tricky since we need to optimize the code to compute the gradient of d f w.r.t to variables q , k , and R all at once.
We can develop the backward function with highly parallel computation by exploiting GPU architecture and utilizing the reduction technique.
The computational time is comparable to function cdist ; thus, our
FourierFormer implementation is as computationally time-efficient.
Experimental Results
In this section, we numerically justify the advantage of FourierFormer over the baseline dot-product transformer on two large-scale tasks: language modeling on WikiText-103 [46] (Section 4.1) and image classification on ImageNet [22, 67]
(Section 4.2), time series classification on the UEA benchmark [5] (Section 4.3), and reinforcement learning on the D4RL Benchmark [29] (Section 4.4), and the machine translation on the IWSLT’ 14 De-En [10] (Section 4.5).
We aim to show that: (i)
FourierFormer achieves better accuracy than the baseline transformer on a variety of practical tasks with different data modalities, and (ii)
FourierFormer helps reduce head redundancy compared to the baseline transformer (Section 4.6).
Throughout the section, we compare FourierFormers with the baseline dot-product transformers of the same configuration.
In all experiments, we made the constant R in Fourier attention (see equation (16)) to be a learnable scalar and set choose the function ϕ ( x ) =
x 4 (see Remark 2).
All of our results are averaged over 5 runs with different seeds.
The details on the models and training are provided in Appendix A. Moreover, additional experiments results are provided in Appendix E. Our PyTorch code with documentation can be found at https://github.com/minhtannguyen/FourierFormer_NeurIPS.
4.1 Language Modeling on WikiText-
We report the validation and test perplexity (PPL) of FourierFormer versus the baseline transformer with the dot-product attention in Table 1.
FourierFormers attain much better PPL than the baselines in both small and medium configurations.
For the small configuration, the improvements of
FourierFormer over the baseline are 1.29 PPL in validation and 1.44 PPL in test.
For the medium configuration, these improvements are 1.39 PPL in validation and 1.59 PPL in test.
These results suggest that the advantage of FourierFormer over the baseline dot-product transformer grows with the model’s size.
This meets our expectation because larger models has larger query and key dimensions, e.g. the language model with medium configuration in this experiment has the query and key dimension of 256 versus 128 as in the language model with small configuration.
Since the advantage of FourierFormer results from the property that FourierFormer can capture correlation between features in query and key vectors, the larger the query and key dimensions are, the more advantage FourierFormer has.
4.2 Image Classification on ImageNet
In the Imagenet classification task, we illustrates the benefits of Fourierformers in different data modalities.
We summarize our models’ results in Table 2.
Same as in the language modeling experiment, for this image classification task, the Deit model equipped with FourierFormer significantly outperforms the baseline Deit dot-product transformer [79] in both top-1 and top-5 accuracy.
This result suggests that the advantage of FourierFormer over the baseline dot-product transformer holds across different data modalities.
4.3 UEA Time Series Classification
To evaluate Fourierformers on temporal sequences, we compare the accuracy of the our models and the baseline softmax transformers trained on 10 datasets in the the UEA Time Series Classification
Archive benchmark [5].
We summarize our results in Table 3.
We observe show that Fourierformers outperforms softmax baselines in 7 out of 10 tasks and yields significantly better accuracy than the softmax transformer on average, showing the our models benefits when trained on temporal data.
[Table]
[Table] [Table]
4.4 Reinforcement learning on the D4RL benchmark
We also examine the performance of our Fourierformers in reinforcement learning.
In Table
[Table]
we verify the advantage of decision FourierFormer over the baseline decision transformer [12] on the continuous control tasks from the D4RL benchmark [29].
The decision FourierFormer is the decision transformer with the Fourier attention instead of the softmax attention.
On this benchmark, our decision FourierFormer significantly outperforms the baseline decision transformer on 8 out oftasks and on average across tasks.
Each experiment result averaged over 5 runs with different random seeds.
We follow the architecture and training configuration from [93].
4.5 Machine Translation on IWSLT’ 14 De-En
We demonstrate the performance of Fourierformer on the IWSLT’ 14 De-En [10] neural machine translation task, which has different inputs’ the sequence lengths.
Table 5 shows that the FourierFormer achieves better BLUE scores than the softmax baseline.
4.6 FourierFormer Helps Reducing Head Redundancy
To study the diversity between attention heads
[Table]
given the model trained for the WikiText-103 language modeling task, we compute the average L 2 distance between heads in each layer.
We show the layer-average mean and variance of distances between heads in Table 6.
Results in Table 6 shows
[Table] [Table]
that FourierFormer obtains greater L 2 distance between attention heads than the baseline transformer with the dot-product attention and thus helps reduce the head redundancy.
Note that we use the small configuration as specified in Section 4.1 for both models.
Related Work
Interpretation of Attention Mechanism in Transformers Recent works have tried to gain an understanding of transformer’s attention from different perspectives.
[80] considers attention as applying kernel smoother over the inputs.
Extending this kernel approach, [35, 15, 52, 89, 54] linearize the softmax kernel in dot-product attention and propose a family of efficient transformers with linear computational and memory complexity.
[9] then shows that these linear transformers are comparable to a Petrov-Galerkin projection [64], suggesting that the softmax normalization in the dot-product attention is sufficient but not necessary.
Other works provide an understanding of attention in transformers via ordinary/partial differential equation include [45, 69].
In addition, [51, 75, 30, 96, 53] relate attentions in transformers to a Gaussian mixture models.
Several works also connect the attention mechanism to graph-structured learning and message passing in graphical models [90, 72, 39].
Our work focuses on deriving the connection between self-attention and nonparametric kernel regression and exploring better regression estimator, such as the generalized
Fourier nonparametric regression estimator, to improve the performance of transformers.
Redundancy in Transformers [19, 47, 25] show that neurons and attention heads in the pre-trained transformer are redundant and can be removed when applied on a downstream task.
By studying the contextualized embeddings in pre-trained networks, it has been demonstrated that the learned representations from these redundant models are highly anisotropic [49, 26].
Furthermore, [70, 74, 86, 68] employ knowledge distillation and sparse approximation to enhance the efficiency of transformers.
Our FourierFormer is complementary to these methods and can be combined with them.
Concluding Remarks
In this paper, we establish the correspondence between the nonparametric kernel regression and the self-attention in transformer.
We then develop the generalized Fourier integral estimators and propose the FourierFormer, a novel class of transformers that use the generalized Fourier integral estimators to construct their attentions for efficiently capturing the correlations between features in the query and key vectors.
We theoretically prove the approximation guarantees of the generalized Fourier integral estimators and empirically validate the advantage of FourierFormer over the baseline transformer with the dot-product attention in terms of accuracy and head redundancy reduction.
It is interesting to incorporate robust kernels into the nonparametric regression framework of FourierFormer to enhance the robustness of the model under data perturbation and adversarial attacks.
A limitation of
FourierFormer is that it still has the same quadratic computational and memory complexity as the baseline transformer with the dot-product attention.
We leave the development of the linear version of FourierFormer that achieves linear computational and memory complexity as future work.
It is worth noting that there is no potential negative societal impacts of FourierFormer.
Acknowledgements
This material is based on research sponsored by the AFOSR MURI FA9550-18-1-0502, the ONR grant N00014-20-1-2093, the MURI N00014-20-1-2787, and the NSF under Grant# 2030859 to the
Computing Research Association for the CIFellows Project (CIF2020-UCLA-38).
NH acknowledges support from the NSF IFML 2019844 and the NSF AI Institute for Foundations of Machine Learning.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See Section(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Supplement
to "FourierFormer: Transformer Meets Generalized Fourier Integral Theorem"
In the supplementary material, we collect proofs, additional theories, and experiment results deferred from the main text.
In Appendix C, we provide additional theoretical results for generalized Fourier density estimator and for generalized Fourier nonparametric regression estimator.
We provide proofs of key results in the main text and additional theories in Appendix D. We present experiment details in Appendix A while including additional experimental results in Appendix E.
A Experiment Details
This section provides the details of the model and training for experiments in Section 4.
All of our experiments are conducted on a server with 4 NVIDIA A100 GPUs.
A.1 Language Modeling
Datasets and metrics WikiText-103 is a collection of articles from Wikipedia, which have long contextual dependencies.
The training set consists of about 28 K articles containing 103 M running words; this corresponds to text blocks of about 3600 words.
The validation and test sets have 218 K and 246 K running words, respectively.
Each of them contains 60 articles and about 268 K words.
Our experiment follows the standard setting [46, 71] and splits the training data into L -word independent long segments.
For evaluation, we use a batch size of 1, and process the text sequence with a sliding window of size L .
The last position is used for computing perplexity (PPL) except in the first segment, where all positions are evaluated as in [1, 71].
Models and baselines Our implementation is based on the public code by [71].
We use their small and medium models in our experiments.
In particular, for small models, the key, value, and query dimension are set to 128, and the training and evaluation context length are set to 256.
For medium models, the key, value, and query dimension are set to 256, and the training and evaluation context length are set to 384.
In both configurations, the number of heads is 8, the feed-forward layer dimension is 2048, and the number of layers is 16.
In our experiments on WikiText-103 in Section 4.1, we let R be a learnable scalar initialized toand choose ϕ ( x ) = x 4 .
The same setting is used for all attention units in the model; each unit has a different R. We observe that by setting R to be a learnable vector [ R 1 , . . .
, R D ] ⊤ , the FourierFormer gains advantage in accuracy but with the cost of the increase in the number of parameters.
When R is a vector [ R 1 , . .
, R D ] ⊤ , the equation of the Fourier Attention is given by
[Equation]
We provide an ablation study for the effect of R and ϕ in Section E.
A.2 Image Classification
Datasets and metrics The ImageNet dataset [22, 67] consists of 1 .
M training images and 50 K validation images.
For this benchmark, the model learns to predict the category of the input image among 1000 categories.
Top-1 and top-5 classification accuracies are reported.
Models and baselines We use the DeiT-tiny model [79] with 12 transformer layers, 4 attention heads per layer, and a model dimension of 192.
To train the models, we follow the same setting and configuration as for the baseline [79].
Similar to the setting for language modeling, in our experiments on ImageNet image classification, we set R to be a learnable scalar initialized to 1 and choose ϕ ( x ) = x 4 .
Different attention units have different R.
A.3 UEA Time Series Classification
Following [93], we choose 10 out of 30 datasets in the benchmark [5], which vary in input sequence lengths, the number of classes, and dimensionality, to evaluate our models on temporal sequences.
The test accuracy is reported as an evaluation for the benchmark.
Models and baseline For all experiments in this task, we adapt the setups and configurations as in [93] 3 (for the PEMS-SF, SelfRegulationSCP2, UWaveGestureLibrary datasets) and [95](for the other tasks).
The number of heads is 8 in all models, whereas the model dimension and number of transformer layers are varied.
A.4 Reinforcement learning on the D4RL benchmark
Datasets and metrics In the D4RL benchmark [29], which consists of the continuous control tasks for offline reinforcement learning, we choose HalfCheetah, Hopper, and Walker as experiment environments and Medium-Expert, Medium, and Medium-Replay as behavior policies.
This selection is adapted from [93].
Models and baseline The models trained on this benchmark has the same configuration as in [93], with 3 transformer layers and 4 heads per layer.
In our D4RL experiments, we choose ϕ = x 4 and the initial value of the learnable scalar R to be 1.
A.5 Machine Translation
Datasets and metrics
The IWSLT’14 De-En dataset consists of 170 K training sentence pairs, 7 K validation pairs, and 7 K test pairs.
In this task, the model does the translation from German to
English.
To measure the performance of the trained model, the BLEU score [55] is used
Models and baselines The architecture of the Fourierformer and the baseline containstransformer layers with 4 heads per layer.
Our implementation is based on the public code https://github.com/pytorch/fairseq/tree/main/examples/translation.
In our Fourierformer models, we choose ϕ ( x ) =
x 2 and the initialization R init = 1 . 0 .
B Background B.1 Kernel Density Estimation
Kernel density estimation (KDE) is the application of kernel smoothing for probability density estimation, i.e., a non-parametric method to estimate the probability density function of a random variable based on kernels as weights.
Let ( x 1 , x 2 , . . .
, x n ) be i.i.d.
samples drawn from some univarite distribution with an unknown density f at any given point x .
We are interested in estimating the shape of this function f .
Its kernel density estimator is
[Equation]
where K is the kernel and h > 0 is a smoothing parameter called the bandwidth.
A kernel with subscript h is called the scaled kernel and defined as K h ( x ) =
1 /hK
( x/h ) .
B.2 Nonparametric Kernel Regression
Kernel regression is a nonparametric technique to estimate the conditional expectation of a random variable.
The objective is to find a non-linear relation between a pair of random variables X and Y.
In any nonparametric regression, the conditional expectation of a variable Y relative to a variable X may be written:
[Equation]
where m is an unknown function.
Nadaraya–Watson kernel regression Nadaraya–Watson kernel regression estimates m as a locally weighted average, using a kernel as a weighting function.
The Nadaraya–Watson estimator is given by
[Equation]
[Equation]
where K h is a scaled kernel with a bandwidth h .
B.3 Fourier Integral Theorem
The Fourier integral theorem [92, 7] has been used in nonparametric mode clustering, deconvolution problem, and generative modeling [33].
It is a combination of Fourier transform and Fourier inverse transform.
In particular, for any function p ∈
L 1 ( R D ) , the Fourier integral theorem is given by
[Equation]
where x = ( x 1 , . .
, x D ) ,
y = ( y 1 , . . .
, y D ) , s =
( s 1 , . . .
, s D ) , and R is the radius.
Here, the first equality in equation (20) is due to
[Equation]
and the final equality in equation (20) is due to
[Equation]
for all y ∈ R D .
Equation (20) suggests that p R ( x ) := 1 π D (cid:82) R D (cid:81)
Dj =1 sin( R ( y j − x j ))
( y j − x j ) p ( y ) d y can be used as an estimator of the function p .
C Additional Theoretical Results
In this section, we provide additional theoretical results for generalized Fourier density estimator in
Appendix C.1 and for generalized Fourier nonparametric regression estimator in Appendix C.2.
C.1 Generalized Fourier density estimator
We now establish the MISE rate of p ϕN,R in equation (12) when ϕ ( z ) =
z l and l ∈ { 1 , 2 } .
We consider the following tail bounds on the Fourier transform of the true density function p as follows.
Definition 3 (1)
We say that p is supersmooth of order
α if we have universal constants C 1 and Csuch that the following inequalities hold for almost surely x ∈ R D :
[Equation]
Here, (cid:98) p denotes the Fourier transform of the function p .
The function p is ordinary smooth of order β
if there exists universal constant c such that the following inequality holds for almost surely x ∈ R D :
[Equation]
The notions of supersmoothness and ordinary smoothness had been used widely in deconvolution problems [28] and density estimation problems [20, 82, 33].
The supersmooth condition is satisfied when the function p is Gaussian distribution or Cauchy distribution while the ordinary smooth condition is satisfied when the function p is Laplace distribution and Beta distribution.
Based on the smoothness conditions in Definition 3, we have the following result regarding the mean-square integrated error (MISE) of the function generalized Fourier density estimator (12) (see equation (13) for a definition of MISE) when ϕ ( z ) =
z l and l ∈ { 1 , 2 } .
Theorem 3 (a) When ϕ ( z ) =
z , the following holds:
• (Supersmooth setting)
If the true density function p is supersmooth function of order α for some α > 0 , then there exists universal constants ¯ C 1 , ¯ C 2 , and ¯ C 3 such that as long as R ≥ ¯ C 1 we have
[Equation]
• (Ordinary smooth setting)
If the true density function p is ordinary smooth function of order β for some β > 1 , then there exists universal constants ¯ c such that
[Equation]
(b) When ϕ ( z ) =
z 2 , the following holds • (Supersmooth setting)
If the true density function p is supersmooth function of order α for some α > 0 , then there exists universal constants C ′ 1 and C ′ 2 such that as long as R ≥ C ′we have
[Equation]
• (Ordinary smooth setting)
If the true density function p is ordinary smooth function of order β for some β > 3 , then there exists universal constants c ′ such that
[Equation]
Proof of Theorem 3 is in Appendix D.2.
A few comments with the results of Theorem 3 are in order.
When ϕ ( z ) =
z : As part (a) of Theorem 3 indicates, when the function p is supersmooth, by choosing the radius R to balance the bias and variance, we have the optimal R as R
= (cid:16) log( N ) ¯ C 3 (cid:17) 1 /α
and
the
MISE rate of the generalized Fourier density estimator p ϕN,R becomes O (cid:16) log( N ) D/α N
[Equation]
It indicates that, the MISE rate of p ϕN,R is parametric when the function p is supersmooth.
On the other hand, when the function p is ordinary smooth, the optimal R becomes R
= O
( N 1 D + β − 1 ) and the MISE rate becomes O (cid:16)
N − β − 1 D + β − 1 (cid:17) .
It is slower than the MISE rate when the function p is supersmooth.
When ϕ ( z ) =
z 2 : The results of part (b) of Theorem 3 demonstrate that the upper bounds for the
MISE rate of the generalized Fourier density estimator p ϕN,R is similar for both the supersmooth and ordinary smooth settings.
The optimal radius R = O (cid:16) N 1 D +2 (cid:17) and the MISE rate of the estimator is
O (cid:16) N − 2 D +2 (cid:17) .
C.2 Generalized Fourier nonparametric regression estimator
In this appendix, we provide additional result for the mean square error (MSE) rate of the generalized
Fourier nonparametric regression estimator f N,R in equation (14) when ϕ ( z ) =
z , namely, the setting of the Fourier integral theorem.
The results when ϕ ( z ) =
z l for l ∈ { 2 , 3 , 4 , 5 } are left for the future work.
When ϕ ( z ) =
z , the MSE rate of f N,R had been established in Theorem 9 of Ho et al.
[33] when the function p is supersmooth function.
Here, we restate that result for the completeness.
Theorem 4 Assume that the function p is supersmooth function of order α for some α > 0 and sup k ∈ R D | p ( k ) | < ∞ .
Furthermore, we assume that the function f in the nonparametric regression model (3) is such that sup k ∈ R D | f 2 ( k ) p ( k ) | < ∞ and | (cid:99) f.p ( t ) | ≤ C 1 Q ( | t 1 | , | t 2 | , . .
, | t D | ) exp   − C 
D (cid:88)
j
= | t
j | α   
 , where (cid:99) f.p ( t ) is the Fourier transform of the function f.p , C 1 and C 2 are some universal constants, and Q ( | t 1 | , | t 2 | , . .
, | t D | ) is some polynomial function of | t 1 | , . . .
, | t D | with non-negative coefficients.
Then, we can find universal constants C 3 , C 4 , C 5 such that as long as R ≥ C 3 we have
[Equation]
where deg ( Q ) denotes the degree of the polynomial function Q ,
[Equation]
Proof of Theorem 4 is similar to the proof of Theorem 9 of Ho et al.
[33]; therefore, it is omitted.
The result of Theorem 4 indicates that the optimal radius R = (cid:16) log( N ) 2 C 2 (cid:17) 1 /α and the MSE rate of the generalized Fourier nonparametric regression estimator f N,R is O (cid:16) log( N ) D/α N
[Equation]
D Proofs
In this Appendix, we provide proofs for key results in the paper and in Appendix C.
D.1 Proof of Theorem
Recall that, k 1 , k 2 , . . .
, k N ∈ R D are i.i.d.
samples from the density function p .
In equation (12), the generalized Fourier density estimator of p 0 is given by:
[Equation]
, where A = (cid:82) R ϕ (cid:16) sin( z ) z (cid:17) dz , k i = ( k i 1 , . . . , k iD ) , and k = ( k 1 , . . . , k D ) .
Direct calculation demonstrates that
[Equation]
An application of Taylor expansion up to the m -th order indicates that
[Equation]
where α = ( α 1 , . . . , α d ) , | α | = (cid:80) dj
=1 α j , and ¯ R ( k , y ) is Taylor remainder admitting the following form:
[Equation]
Plugging equations (22) and (23) into equation (21), we find that
[Equation]
1 ≤| α |≤ m
R | α | α !
(cid:90)
R D
D (cid:89) j
= ϕ (cid:18) sin( y j ) y j
(cid:19) d (cid:89)
j = ( − y j ) α j ∂ | α | p ∂ k
α (
[Equation] [Equation]
+
A D (cid:88) | β | = m + m +
R m +1 β !
(cid:90)
R D
D (cid:89) j
= ϕ (cid:18) sin( y j ) y j
(cid:19)
[Equation]
j
[Equation] [Equation]
j ) β
j
[Equation]
[Equation] [Equation]
β (cid:18)
[Equation]
R (cid:19) d
y dt.
According to the hypothesis that (cid:82) R ϕ
[Equation]
sin
[Equation]
z )
z
[Equation]
z j dz = 0 for all 1 ≤ j ≤ m , we obtain that
[Equation]
for any α = ( α 1 , . . .
, α d ) such that 1 ≤ | α | ≤ m .
Collecting the above results, we arrive at | E [ p
[Equation]
,R ( k )] − p ( k ) |
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Since the function p ∈ C m +
( R D ) , we can find positive constant M such that ∥ ∂ m +1 p ∂ k β ( k ) ∥ ∞ ≤ M for all β = ( β 1 , . .
. , β d ) such that | β | = m + 1 .
Therefore, we find that
[Equation]
E
[Equation] [Equation]
p
[Equation]
A
[Equation]
For any β = ( β 1 , . . .
, β D )
such that | β | = m + 1 , an application of the AM-GM inequality indicates that (cid:81)
Dj =1 | y
j | β j ≤ m ( (cid:80) Dj =1 | y j | m +1 ) .
Hence, putting these results together leads to | E [
[Equation]
(cid:19)(cid:12)(cid:12)(cid:12)(cid:12)  
D (cid:88)
j
[Equation]
| y
[Equation]
| m +
[Equation]
 
d y .
From the hypothesis, we have (cid:82) R (cid:12)(cid:12)(cid:12) ϕ (cid:16) sin( z ) z (cid:17)(cid:12)(cid:12)(cid:12)
| z | m +1 dz < ∞ .
As a consequence, we can find a universal constant C depending on A and d such that
[Equation]
for all k ∈ R D .
Bounding the variance: We now move to bound the variance of p ϕN,R ( k ) .
Indeed, direct computation indicates that
[Equation]
where the variance and the expectation are taken with respect to K
=
( K . 1 , . . .
, K .d )
∼ p .
As
[Equation]
∞ , there exists a universal constant C ′ depending on A and D such that
[Equation]
As a consequence
[Equation]
we obtain the conclusion of the theorem.
D.2 Proof of Theorem
From the Plancherel theorem, we obtain that
[Equation]
24) where (cid:98) p ϕN,R and (cid:98) p are respectively the Fourier transforms of p N,R and p .
From the definition of generalized Fourier density estimator p ϕN,R in equation (12), it is clear that
[Equation]
for any t = ( t 1 , . . .
, t D ) ∈ R D where we define K R ( y ) :
= 1 π (cid:82) R Rϕ (cid:16) sin( Rx ) Rx (cid:17) exp( iyx ) dx for any y ∈ R .
To ease the presentation, we denote ¯ K R ( t ) :
= (cid:81)
Dj =1 K R ( t j ) and φ N ( t ) =
1 N (cid:80) Ni
=1 exp( i t ⊤ k i ) for any t =
( t 1 , t 2 , . .
. , t D ) ∈ R D .
Based on these notations, we can rewrite
[Equation]
Direct calculation shows that E k N 1 [ φ N ( t )] = (cid:98) p ( t ) for any t ∈ R D where k N 1 :=
( k 1 , . . .
, k n ) .
Furthermore, we have
[Equation]
Collecting the above results
[Equation]
we have the following equations:
[Equation]
Combining the results from equations (24) and (25), we find that
[Equation]
D.2.
When ϕ ( z
[Equation]
We first consider the setting when ϕ ( z )
=
z , namely, the setting of the Fourier integral theorem.
Under this setting, direct computation indicates that
[Equation]
Given the smoothness assumptions on the function p , we have two settings on that function.
Supersmooth setting of the function p : When the function p is supersmooth density, we have
[Equation]
where C 1 and C 2 are some universal constants.
Therefore, we find that
[Equation]
where B i := { t ∈ R D : | t i | ≥ R } .
We now proceed to bound (cid:82) B i exp (cid:16) − C 2 (cid:16)(cid:80)
Dj =1 | t
j | α (cid:17)(cid:17)
d t for all i ∈
[ D ] .
Indeed, we have that
[Equation]
When α ≥ 1 , we have that
[Equation]
When α ∈ (0 , 1)
[Equation]
then we find that
[Equation]
When the R is such that R α ≥
[Equation]
1 − α ) C 2 α
[Equation]
the above inequality becomes
[Equation]
Collecting the above results
[Equation]
we arrive at
[Equation]
Plugging the inequality (28) into the inequality (31)
[Equation]
there exists universal constant C 3 depending on α and D such that
[Equation]
On the other hand
[Equation]
we also have
[Equation] [Equation]
Combining the results from equations (29) and (30), we obtain that
MISE
[Equation]
p
[Equation]
As a consequence, we obtain the conclusion of Theorem 3 under the supersmooth setting of the function p and ϕ ( z ) =
z .
Ordinary smooth setting of the function p : The proof of Theorem 3 when the function p is ordinary smooth also proceeds in the similar fashion as that when p is supersmooth.
In particular, we have
[Equation]
where B i := { t ∈ R D : | t i | ≥ R } .
By simple algebra, we obtain that
[Equation]
Putting the above results together leads to
[Equation]
where c 1 is some universal constant.
Similar to the supersmooth setting, we also can bound the variance 1 N (cid:82)
[Equation]
p
[Equation]
t
[Equation]
t under the ordinary smooth setting as follows:
[Equation]
Combining the results from equations (32) and (23), we obtain that
[Equation]
where c 2 is a universal constant.
As a consequence, we obtain the conclusion of Theorem 3 under the ordinary smooth setting of the function p and ϕ ( z ) =
z .
D.2.2 When ϕ ( z ) =
z
When ϕ ( z ) =
z 2 , which corresponds to the Féjer integral setting, we find that
[Equation]
Given the formulation of the function ¯ K R , we first bound 1 N (cid:82) R D (1 − | (cid:98) p ( t ) | 2 ) ¯ K 2 R ( t ) d t .
Indeed, direct calculation shows that
[Equation]
Now, we proceed to upper bound (cid:82) R D (cid:98) p 2 ( t )(1 − ¯ K R ( t )) 2 d t .
We have two settings of the function p .
Supersmooth setting of the function p :
Given the above formulation of the function ¯ K R , we have
[Equation]
By using the similar argument as when ϕ ( x ) = x , when p is supersmooth function, we obtain that
[Equation]
[Equation]
where C ′ 1 and C ′ 2 are universal constants.
On the other hand, we have
[Equation]
where ¯ C 1 is some universal constant.
Here, i 1 , . .
, i m in the sum satisfy that they are pairwise different and 1 ≤ i 1 , . . .
, i m ≤ D .
Now, simple calculations indicate that
[Equation]
where ¯ C 2 is some universal constant.
Combining the results from equations (37) and (38)
[Equation]
there exists universal constant ¯ C 3 depending on D such that
[Equation]
Plugging the inequalities (36) and (39) to equation (35) leads to the following bound
[Equation]
Combining the results from equations (34) and (40)
[Equation]
we have
[Equation]
As a consequence
[Equation]
we obtain the conclusion of Theorem 3 when ϕ ( z ) =
z 2 and the function p is supersmooth function.
Ordinary smooth setting of the function p : Using similar proof argument as that of the supersmooth setting of the function p
[Equation]
as β > 3 , we find that
[Equation]
(41) where c, c 1 , c 2 are universal constants.
Combining the inequalities (34) and (41), we obtain the conclusion of Theorem 3 under the ordinary smooth setting of the function p and ϕ ( z ) =
z 2 .
D.3 Proof of Theorem
Our proof strategy is to first bound the bias of f N,R ( k ) and then establish an upper bound for the variance of f N,R ( k ) for each k ∈ R D .
D.3.1 Upper bound on the bias
Recall that in equation (14)
[Equation]
we define f N,R ( k ) as follows:
[Equation]
where p ϕN,R ( k ) is generalized Fourier density estimator in equation (12) while a N,R ( k ) is defined as follows:
[Equation]
Simple algebra leads to
[Equation]
Therefore
[Equation]
via an application of Cauchy-Schwarz inequality we obtain that
[Equation]
[ f N,R ( k )] − f ( k ))
[Equation]
≤
[Equation]
where the second inequality is due to the standard inequality E 2 ( XY )
≤ E ( X 2 ) E
( Y 2 ) for all the random variables
X, Y .
According to the assumptions of Theorem 2 and the result of Theorem
[Equation]
we have
[Equation]
where C 1 and C 2 are some universal constants in Theorem 1.
Now
[Equation]
we proceed to bound | E [ a N,R ( k ) − f ( k ) p N,R ( k )] | .
Direct calculation demonstrates that
[Equation]
An application of Taylor expansion up to the m -th order indicates that
[Equation] [Equation]
where α = ( α 1 , . . . , α d ) , | α | = (cid:80) dj
=1 α j , and ¯ R 1 ( k , y ) , R 2 ( k , y ) are Taylor remainders admitting the following forms:
[Equation]
Combining equations (46) and (47), we obtain that
[Equation]
As we have (cid:82) R ϕ (cid:16) sin( z ) z (cid:17) z j dz = 0 for all 1 ≤ j ≤ m
[Equation]
plugging the equation in the above display to equation (45) leads to
[Equation]
where B 1 , B 2 , B 3 , B 4 are defined as follows:
[Equation]
D
[Equation]
B
[Equation]
= 1 A D (cid:90)
R D
[Equation]
ϕ
[Equation]

[Equation]
B
[Equation]
1 A D (cid:90)
R D
[Equation]
(cid:89)
j =
[Equation] [Equation]
d y
[Equation]
Since we have (cid:82) R (cid:12)(cid:12)(cid:12) ϕ (cid:16) sin( z ) z (cid:17)(cid:12)(cid:12)(cid:12) | z | j dz < ∞ for any m + 1 ≤ j ≤ 2 m + 2 and p 0 , f ∈ C m +
( R d ) , we find that as long as R ≥ ¯ c for some given constant ¯ c | B 1 | ≤ 1 A D (cid:88) m +
≤| α | + | β |≤ 2 m
R
[Equation]
α !
β ! (cid:90)
R D
D (cid:89)
[Equation]
where c 1 is some universal constant depending on A , D , and ¯ c .
Furthermore, we find that
[Equation]
where c 2 is some universal constant depending on A , d , and ¯ c .
Similarly, we also can demonstrate that B 3 ≤ c 3 /R m
+1 and B 4 ≤ c 4 /R 2( m +1) for some universal constants c 3 and c 4 .
Putting the above results together, we arrive at the following bound:
[Equation]
Plugging the results from equations (44) and (48) to equation (43), we obtain that
[Equation]
N
[Equation] [Equation]
D.3.2 Upper bound on the variance
Now, we study the variance of f N,R ( k ) .
By taking variance both sides of the equation (42), we obtain that
[Equation]
Upper bound of T 2 :
To upper bound T 2 , we utilize the following lemma.
Lemma 1 Assume that the function ϕ and p 0 satisfy the assumptions of Theorem 1.
Furthermore, ϕ ( z ) ≤ C as long as | z | ≤ 1 for some universal constant C .
Then, for almost all k ∈ R D , there exist universal constants C ′ such that
[Equation]
Proof of Lemma 1 is given in Appendix D.4.
Now given the result of Lemma 1, we denote B as the event such that
[Equation]
where C ′ is a universal constant in Lemma 1.
Then, we obtain P ( B ) ≥ 1 − δ .
Hence, we have the following bound with T 2 :
[Equation]
, where c ′ is some universal constant and the final inequality is based on the inequalities: P ( B c ) ≤ δ and ( p ( k ) − p ϕN,R ( k ))
2 ≤ 2( p 2 ( k ) + ( p ϕN,R ) 2 ( k )) ≤ 2 (cid:16) p 2 ( k ) + C D R 2 D A D (cid:17) where C is a universal constant such that ϕ ( z ) ≤ C when | z | ≤ 1 .
By choosing δ such that δ =
R D N ( p 2 ( k )+ C D R 2 D /A D ) , we obtain that
[Equation]
for some universal constant c ′′ when R is sufficiently large.
Upper bound of T 1 : As v i =
f ( k i )
+ ϵ
i for all i ∈
[ N ] , direct calculation shows that
[Equation]
An application of Cauchy-Schwarz inequality leads to
[Equation]
Since we have E (cid:20)(cid:16) 1 N (cid:80) Ni
=
Z i (cid:17) 2 (cid:21) ≤ 1 N E (cid:2) Z 21 (cid:3) + E 2 [ Z 1 ] for any i.i.d.
samples Z 1 , . .
, Z N , we obtain that
[Equation]
where the outer expectation is taken with respect to X =
( X . 1 , . . .
, X .d )
∼ p .
From the result in equation (48), we have
[Equation]
where c ′ is some universal constant.
In addition, an application of Cauchy-Schwarz inequality leads to
[Equation]
Since we have (cid:82) R ϕ 2 (sin( z ) /z )
dz < ∞
[Equation]
it indicates that we can find a universal constant c ′′ such that
[Equation]
Putting the above results together
[Equation]
we obtain that
[Equation]
Similarly, since E ( ϵ i ) = 0 and var( ϵ i ) =
σ 2 for all i ∈
[ N ]
[Equation]
we have
[Equation]
where c ′′′ is some universal constant.
Combining the results from equation (52) and equation (53), we find that
[Equation]
where C is some universal constant.
Plugging the bounds of T 1 and T 2 from equations (51) and (54) into equation (50), when R ≥ C ′ where C ′ is some universal constant, we have
[Equation]
where C ′ 1 , C ′ 2 , C ′ 3 are some universal constants.
Combining the results with bias and variance in equations (49) and (55)
[Equation]
we obtain the following bound:
[Equation]
As a consequence, we obtain the conclusion of the theorem.
D.4 Proof of Lemma
Invoking triangle inequality, we obtain that
[Equation]
If we denote v i = R D
A D (cid:81)
Dj =1 ϕ (cid:16) sin( R ( k j − k ij )
R ( k j − k ij ) (cid:17) for all i ∈ [ N ] , then as sin( R ( k j − k ij ) /
( R ( k j − k ij )) ≤ 1 for all j ∈
[ D ] we have | v i | ≤ C D R D /A
D for all i ∈
[ N ] where C is the constant such that ϕ ( z ) ≤ C when | z | ≤ 1 .
Furthermore, from the proof of Theorem 1 we have var( v i ) ≤ C ′ R D where C ′ > 0 is some universal constant.
Given these bounds of v i and var( v i ) , for any t ∈ (0 , C ′′ ]
Bernstein’s inequality shows that
[Equation]
By choosing t = ¯ C (cid:112) R D log(2 /δ ) /N , where ¯ C is some universal constant, we find that
[Equation]
From the result of Theorem 1, there exists universal constant c such that
[Equation]
Plugging the bounds (57) and (58) into the triangle inequality (56), we obtain the conclusion of the lemma.
E Additional Experimental Results E.1 Effect of ϕ
Using the WikiText-103 language modeling as a case study, we analyze the effect of ϕ ( x ) on the performance of FourierFormer.
In particular, we set ϕ
( x ) = x k and compare the performance of
FourierFormer for k = 1 , 2 , 3 , 4 and 6 .
We keep other settings the same as in our experiments in
[Table] [Table]
Section 4.1.
We summarize our results in Table 7.
We observe that for odd values of k such as k = 1 , 3 , the training diverges, confirming that negative density estimator cause instability in training
FourierFormer (see Remark 3.1).
For even values of k such as k = 2 , 4 , 6 , we observe that the greater value of k results in better valid and test PPL.
However, the gap between k = 4 and k = 6 is smaller compared to the gap between k = 2 and k = 4 , suggesting that using k > 4 does not add much advantage in terms of accuracy.
We have also studied other choices of ϕ that are nonnegative functions such as ϕ ( x ) =
| x | , ReLU ( x ) , and sigmoid ( x ) .
Those functions yield worse results than ϕ ( x ) = x 2 m .
We summarize these results in Table 7.
E.2 Effect of the Initialization of R
In this section, we study the effect of the initialization value of R on the performance of FourierFormer when trained for the WikiText-103 language modeling and summarize our results in Table 8.
Here we choose R to be learnable scalars as in experiments described in our main text.
Other settings are also the same as in our experiments in Section 4.1.
We observe that when R is initialized too small (e.g.
R init = 0 . 1 ) or too big (e.g. R init = 4 ), the PPL of the trained FourierFormer decreases.
R init = 1 , 2 ,yield best results.
We also study the performance of the FourierFormer when R is chosen to be a learnable vector, R =
[ R 1 , . . .
, R D ] ⊤ .
We report our result in the last row of Table 8.
FourierFormer with R be learnable vectors achieves better PPLs than FourierFormer with R be learnable scalars of the same setting.
As we mentioned in Section A, this advantage comes with an increase in the number of parameters in the model.
Finally, from our experiments, we observe that making R a learnable parameter yields better PPLs than making R a constant and selecting its value via a careful search.
E.3 Efficiency Analysis
We have included quantitative results on the runtime and GPU memory usage of the FourierFormer versus the baseline softmax transformer in Table 9.
E.4 Synthetic Examples for Density Estimation and Nonparametric Regression via The Generalized Fourier Integral Theorem
We empirically confirm Theorem 1 for density estimation and Theorem 2 for nonparametric regression using the Generalized Fourier Integral Theorem in this section.
In Figure 1, we show that the generalized Fourier density estimator can approximate (A) 1-D and (B) 2-D Gaussian distribution
[Table]
[Figure]
[Table]
[Figure] [Figure]
with a dense covariance matrix well, which further verify Theorem 1.
In Figure 2, we show that the generalized Fourier nonparametric regression estimator can approximate the function that maps from a random variable to another random variable, which further verify Theorem 2.
In particular, for the density estimation experiments, we sample 100000 data points from the 1-D and 2-D Gaussian distribution and estimate the density for 1000 uniformly sampled test points.
The mean square errors (MSE) are 1 .
29 × 10 − 5 and 2 .
42 × 10 − 5 for the 1-D and 2-D case, respectively.
For the non-parametric regression task, we build a training dataset with 90000 correlated normally distributed samples and choose a 3-degree polynomial as the ground truth function.
The MSE between ground truth labels and predictions is 0.06.