Original: Chefs’ Random Tables: Non-Trigonometric Random Features
Revised: Chefs’ Random Tables: Non-Trigonometric Random Features

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We introduce chefs’ random tables (CRTs), a new class of non-trigonometricrandom features (RFs) to approximate Gaussian and softmax-kernels.
Revised: We introduce chefs’ random tables (CRTs), a new class of non-trigonometric random features (RFs) to approximate Gaussian and softmax-kernels.

Original: CRTs arean alternative to standard random kitchen sink (RKS) methods, which inherentlyrely on the trigonometric maps [46].
Revised: CRTs are an alternative to standard random kitchen sink (RKS) methods, which inherently rely on the trigonometric maps [46].

Original: We present variants of CRTs where RFs arepositive, a key requirement for applications in recent low-rank Transformers [15].
Revised: We present variants of CRTs where RFs are positive, a key requirement for applications in recent low-rank Transformers [15].

Original: Further variance reduction is possible by leveraging statistics which are simple tocompute.
Revised: Further variance reduction is possible by leveraging statistics which are simple to compute.

Original: One instantiation of CRTs, the optimal positive random features (OPRFs),is to our knowledge the first RF method for unbiased softmax-kernel estimationwith positive and bounded RFs, resulting in exponentially small tails and muchlower variance than its counterparts.
Revised: One instantiation of CRTs, the optimal positive random features (OPRFs), is to our knowledge the first RF method for unbiased softmax-kernel estimation with positive and bounded RFs, resulting in exponentially small tails and much lower variance than its counterparts.

Original: As we show, orthogonal random featuresapplied in OPRFs provide additional variance reduction for any dimensionality d(not only asymptotically for sufficiently large d , as for RKS).
Revised: As we show, orthogonal random features applied in OPRFs provide additional variance reduction for any dimensionality d (not only asymptotically for sufficiently large d , as for RKS).

Original: We test CRTs onmany tasks ranging from non-parametric classification to training Transformers fortext, speech and image data, obtaining new state-of-the-art results for low-rank text
Revised: We test CRTs on many tasks ranging from non-parametric classification to training Transformers for text, speech and image data, obtaining new state-of-the-art results for low-rank text

Original: Transformers, while providing linear space and time complexity of the attention.
Revised: Transformers, while providing linear space and time complexity of the attention.

Original:  Introduction & related work
Revised:  Introduction & related work

Original: The idea that nonlinear mappings of the random-weight linear combinations of data features canbe used to linearize various nonlinear similarity functions transformed kernel methods.
Revised: The idea that nonlinear mappings of the random-weight linear combinations of data features can be used to linearize various nonlinear similarity functions transformed kernel methods.

Original: This led tothe development of Random Kitchen Sinks (RKSs) techniques; and the new field of scalable kernelalgorithms, introduced in the paper trilogy [44, 45, 46] was born.
Revised: This led to the development of Random Kitchen Sinks (RKSs) techniques; and the new field of scalable kernel algorithms, introduced in the paper trilogy [44, 45, 46] was born.

Original: RKSs were subsequently usedin many applications, ranging from kernel and function-to-function regression [1, 33, 41], SVMalgorithms [50] to operator-valued and semigroup kernels [36, 62], neural networks [25, 61, 11, 27]and even differentially-private ML algorithms [9], as well as (very recently) nonparametric adaptivecontrol [4].
Revised: RKSs were subsequently used in many applications, ranging from kernel and function-to-function regression [1, 33, 41], SVM algorithms [50] to operator-valued and semigroup kernels [36, 62], neural networks [25, 61, 11, 27] and even differentially-private ML algorithms [9], as well as (very recently) nonparametric adaptive control [4].

Original: Random features (RFs) are a subject of much theoretical analysis [34, 63, 51, 48].
Revised: Random features (RFs) are a subject of much theoretical analysis [34, 63, 51, 48].

Original: To approximate shift invariant (e.g. Gaussian, Cauchy or Laplace) and softmax kernels, RKSs rely onthe trigonometric nonlinear mappings provided directly by Bochner’s Theorem [36].
Revised: To approximate shift invariant (e.g. Gaussian, Cauchy or Laplace) and softmax kernels, RKSs rely on the trigonometric nonlinear mappings provided directly by Bochner’s Theorem [36].

Original: Trigonometric
Revised: Trigonometric

Original: RFs provide strong concentration results (e.g. uniform convergence, see Claim 1 in [45]), but sufferfrom a weakness that was noted recently – they are not guaranteed to be positive.
Revised: RFs provide strong concentration results (e.g. uniform convergence, see Claim 1 in [45]), but suffer from a weakness that was noted recently – they are not guaranteed to be positive.

Original: This makes themunsuitable for approximating softmax-attention in scalable Transformers relying on implicit attentionvia random features [15].
Revised: This makes them unsuitable for approximating softmax-attention in scalable Transformers relying on implicit attention via random features [15].

Original: As noted in [15], trigonometric features lead to unstable training, as theyyield poor approximations of the partition functions applied to renormalize attention and involvingseveral small softmax kernel values.
Revised: As noted in [15], trigonometric features lead to unstable training, as they yield poor approximations of the partition functions applied to renormalize attention and involving [Figure] several small softmax kernel values.

Original: To address this, [15] proposed a new method for unbiasedsoftmax kernel estimation with positive RFs, the so-called FAVOR+ mechanism ( F ast A ttention
Revised: To address this, [15] proposed a new method for unbiased softmax kernel estimation with positive RFs, the so-called FAVOR+ mechanism ( F ast A ttention

Original: V ia O rthogonal R andom Positive Features), as opposed to FAVOR using trigonometric RFs (as in [Figure]
Revised: V ia O rthogonal R andom Positive Features), as opposed to FAVOR using trigonometric RFs (as in [13]).

Original: Positive random features guarantee that the denominator in self-attention is a sum of positivenumbers, hence it cannot be negative or too small.
Revised: Positive random features guarantee that the denominator in self-attention is a sum of positive numbers, hence it cannot be negative or too small.

Original: Unfortunately FAVOR+ features are not bounded, and worse, the moment generating function of thecorresponding softmax kernel estimator is not finite.
Revised: Unfortunately FAVOR+ features are not bounded, and worse, the moment generating function of the corresponding softmax kernel estimator is not finite.

Original: Consequently, no concentration results beyondthose involving second moment methods (variance bounds) have been provided for FAVOR+.
Revised: Consequently, no concentration results beyond those involving second moment methods (variance bounds) have been provided for FAVOR+.

Original: Despiteactive research on constructing new RFs for implicit attention in Transformers [12, 17], the followingquestions of great practical importance remained open:
Revised: Despite active research on constructing new RFs for implicit attention in Transformers [12, 17], the following questions of great practical importance remained open:

Original: Does there exist an unbiased estimator of the softmax/Gaussian kernel relying on positive andsimultaneously bounded random features?
Revised: Does there exist an unbiased estimator of the softmax/Gaussian kernel relying on positive and simultaneously bounded random features?

Original: Can it be efficiently constructed?
Revised: Can it be efficiently constructed?

Original: We answer both questions affirmatively in this paper, introducing a new mechanism called optimalpositive random features (OPRFs).
Revised: We answer both questions affirmatively in this paper, introducing a new mechanism called optimal positive random features (OPRFs).

Original: We propose other RF methods that, as OPRFs, do not applytrigonometric functions and provide positivity.
Revised: We propose other RF methods that, as OPRFs, do not apply trigonometric functions and provide positivity.

Original: We call this new set of RF mechanisms chefs’ randomtables (CRTs, see Figure 1-left).
Revised: We call this new set of RF mechanisms chefs’ random tables (CRTs, see Figure 1-left).

Original: The new OPRF-based method for fast self-attention approximation,applying in addition block-orthogonal random projections, is referred to as FAVOR++ .
Revised: The new OPRF-based method for fast self-attention approximation, applying in addition block-orthogonal random projections, is referred to as FAVOR++ .

Original: We compute the variance of OPRF-based estimators (see Theorem 3.1 & 3.2) and show that they canprovide e 60 x variance reduction for Gaussian/softmax kernel estimation (see Figure 1).
Revised: We compute the variance of OPRF-based estimators (see Theorem 3.1 & 3.2) and show that they can provide e 60 x variance reduction for Gaussian/softmax kernel estimation (see Figure 1).

Original: We give thefirst exponentially small upper bounds for tails of the Gaussian/softmax kernel estimators relyingon positive RFs, leveraging boundedness of OPRFs (see Theorem 4.2).
Revised: We give the first exponentially small upper bounds for tails of the Gaussian/softmax kernel estimators relying on positive RFs, leveraging boundedness of OPRFs (see Theorem 4.2).

Original: Consequently, using OPRFswe give the first uniform convergence results for softmax attention approximation with positive RFs(Theorem 4.3).
Revised: Consequently, using OPRFs we give the first uniform convergence results for softmax attention approximation with positive RFs (Theorem 4.3).

Original: We show that orthogonal random projections combined with OPRFs (leading to
Revised: We show that orthogonal random projections combined with OPRFs (leading to

Original: FAVOR++) provably reduce the variance of OPRFs for any dimensionality d (see Theorem 4.1) asopposed to only asymptotically for d large enough which is the case for RKSs.
Revised: FAVOR++) provably reduce the variance of OPRFs for any dimensionality d (see Theorem 4.1) as opposed to only asymptotically for d large enough which is the case for RKSs.

Original: Finally, we provideextensive empirical evaluation in Section 5, for Transformers (text, image and speech domains),establishing new state-of-the-art results for low-rank attention Transformers for text.
Revised: Finally, we provide extensive empirical evaluation in Section 5, for Transformers (text, image and speech domains), establishing new state-of-the-art results for low-rank attention Transformers for text.

Original:  Prerequisites 
Revised:  Prerequisites 2.1 The definition of random features

Original: 2.1 The definition of random features

Original: [Figure]

Original: Let x ,
Revised: Let x ,

Original: y ∈ R
Revised: y ∈ R

Original: d be real vectors and K ( x , y ) =
Revised: d be real vectors and K ( x , y ) =

Original: exp( − 12 ∥ x − y ∥ 2 ) be a Gaussian kernel where ∥ · ∥denotes the L 2 -norm.
Revised: exp( − 12 ∥ x − y ∥ 2 ) be a Gaussian kernel where ∥ · ∥ denotes the L 2 -norm.

Original: By random features (RFs) for the Gaussian kernel we denote two functionsf (1) ( ω, x ) , f (2) ( ω, y ) :
Revised: By random features (RFs) for the Gaussian kernel we denote two functions f (1) ( ω, x ) , f (2) ( ω, y ) :

Original: R d × R d → C where ω is a random vector from some distribution p ( ω ) on
Revised: R d × R d → C where ω is a random vector from some distribution p ( ω ) on

Original: R d .
Revised: R d .

Original: Functions f ( · ) ( ω, x ) satisfy the following:
Revised: Functions f ( · ) ( ω, x ) satisfy the following:

Original: [Equation]
Revised: [Equation]

Original: (1) for all x , y ∈ R d where Re ( · ) denote the real part of a complex number ( Im ( · ) for the imaginarypart).
Revised: (1) for all x , y ∈ R d where Re ( · ) denote the real part of a complex number ( Im ( · ) for the imaginary part).

Original: The decomposition (1) can be used for an unbiased approximation of the linear operator
Revised: The decomposition (1) can be used for an unbiased approximation of the linear operator

Original: K =
Revised: K =

Original: ( K ( x ( i ) , y ( j ) ))
Revised: ( K ( x ( i ) , y ( j ) ))

Original: L,Li,j =1 ∈ R L × L where x ( i ) , y ( j ) ∈ R d .
Revised: L,Li,j =1 ∈ R L × L where x ( i ) , y ( j ) ∈ R d .

Original: Such linear operators emerge in variousapplications, e.g. kernel SVM [45], kernel regression [38, 59] or Transformers [15] (see Section 2.3).
Revised: Such linear operators emerge in various applications, e.g. kernel SVM [45], kernel regression [38, 59] or Transformers [15] (see Section 2.3).

Original: For any c ∈ R L , evaluating K c naively would result in O ( dL 2 ) time complexity which is prohibitivelyexpensive for large L .
Revised: For any c ∈ R L , evaluating K c naively would result in O ( dL 2 ) time complexity which is prohibitively expensive for large L .

Original: Instead, we can use the Monte Carlo approximation: draw i.i.d.
Revised: Instead, we can use the Monte Carlo approximation: draw i.i.d.

Original: samplesω 1 , . .
Revised: samples ω 1 , . .

Original: ω M ∼ p ( ω ) , where M ≪ L , and compute for 1 ≤ i ≤ L : 
Revised: ω M ∼ p ( ω ) , where M ≪ L , and compute for 1 ≤ i ≤ L : ( K c ) i = L (cid:88) j =

Original: [Equation]
Revised: [Equation]

Original: Therefore, K c can be approximated by first precomputing { (cid:80)
Revised: Therefore, K c can be approximated by first precomputing { (cid:80)

Original: Lj =1 f (2) ( ω m , y ( j ) )
Revised: Lj =1 f (2) ( ω m , y ( j ) )

Original: c j } Mm =1 and thenevaluating (2) in O ( dML ) total time.
Revised: c j } Mm =1 and then evaluating (2) in O ( dML ) total time.

Original: Precision of this approximation can be theoretically bounded[45, 15].
Revised: Precision of this approximation can be theoretically bounded [45, 15].

Original: In this manuscript, we will use the variance Var p ( ω ) Re (cid:0) f (1) ( ω, x ) f (2) ( ω, y ) (cid:1) of (1) tojudge the precision of the Monte Carlo approximation.
Revised: In this manuscript, we will use the variance Var p ( ω ) Re (cid:0) f (1) ( ω, x ) f (2) ( ω, y ) (cid:1) of (1) to judge the precision of the Monte Carlo approximation.

Original: Since samples ω 1 , . . .
Revised: Since samples ω 1 , . . .

Original: , ω M are i.i.d., the number M of RFs controls the tradeoff between the total variance Var ω 1 ,...,ω M ( . . . )
Revised: , ω M are i.i.d., the number M of RFs controls the tradeoff between the total variance Var ω 1 ,...,ω M ( . . . )

Original: =
Revised: =

Original: 1 M Var p ( ω ) ( . . . )
Revised: 1 M Var p ( ω ) ( . . . )

Original: (inversely proportional to M ) and number of computations (directly proportional to M ).
Revised: (inversely proportional to M ) and number of computations (directly proportional to M ).

Original: The softmax kernel is defined as: K sfm ( x , y ) =
Revised: The softmax kernel is defined as: K sfm ( x , y ) =

Original: exp( x T y ) , and can be easily derived from the Gaussian kernel K as follows:
Revised: exp( x T y ) , and can be easily derived from the Gaussian kernel K as follows:

Original: K sfm ( x , y ) = exp( ∥ x ∥ 2 / 2) K
Revised: K sfm ( x , y ) = exp( ∥ x ∥ 2 / 2) K

Original: ( x , y ) exp( ∥ y ∥ 2 / 2) .
Revised: ( x , y ) exp( ∥ y ∥ 2 / 2) .

Original: Thus, in particular,any RF mechanism for the Gaussian kernel immediately transfers to the corresponding one for thesoftmax kernel and vice versa.
Revised: Thus, in particular, any RF mechanism for the Gaussian kernel immediately transfers to the corresponding one for the softmax kernel and vice versa.

Original: Thus from now on, unless explicitly stated otherwise, the estimatorswe consider are approximating the Gaussian kernel.
Revised: Thus from now on, unless explicitly stated otherwise, the estimators we consider are approximating the Gaussian kernel.

Original: 2.2 Existing trigonometric and positive random feature methods
Revised: 2.2 Existing trigonometric and positive random feature methods

Original: Here we summarize existing RFs for Gaussian kernel estimation.
Revised: Here we summarize existing RFs for Gaussian kernel estimation.

Original: Trigonometric RFs (TrigRFs) , thecore of RKSs [45] and FAVOR [13], are defined as follows:
Revised: Trigonometric RFs (TrigRFs) , the core of RKSs [45] and FAVOR [13], are defined as follows:

Original: f (1)trig ( ω, x ) =
Revised: f (1)trig ( ω, x ) =

Original: exp(i ω ⊤ x ) , f (2)trig ( ω, y ) =
Revised: exp(i ω ⊤ x ) , f (2)trig ( ω, y ) =

Original: exp( − i ω ⊤ y ) , p trig ( ω ) ∼ N ( 0 d , I d ) where i denotes an imaginary unit (as opposed to the indexnotation i ), 0 d ∈ R d is a vector of zeros and I d ∈ R d × d is an identity matrix.
Revised: exp( − i ω ⊤ y ) , p trig ( ω ) ∼ N ( 0 d , I d ) where i denotes an imaginary unit (as opposed to the index notation i ), 0 d ∈ R d is a vector of zeros and I d ∈ R d × d is an identity matrix.

Original: The variance of these
Revised: The variance of these

Original: RFs has the following form [15]: Varp
Revised: RFs has the following form [15]: Var

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: =

Original: [Equation]

Original: , y )

Original: [Equation]
Revised: [Equation]

Original: Positive RFs (PosRFs) [15], the key ingredient of the FAVOR+ mechanism, are defined as follows:
Revised: Positive RFs (PosRFs) [15], the key ingredient of the FAVOR+ mechanism, are defined as follows:

Original: [Equation]
Revised: [Equation]

Original: x − ∥ x ∥ 2 ) , p pos ( ω ) ∼ N ( 0 d , I d ) .

Original: Their name is due to thefact that f (1)pos ( ω, x ) , f (2)pos ( ω, y ) are always positive real numbers.
Revised: Their name is due to the fact that f (1)pos ( ω, x ) , f (2)pos ( ω, y ) are always positive real numbers.

Original: PosRF variance has the form [15]:
Revised: PosRF variance has the form [15]:

Original: [Equation]
Revised: [Equation]

Original: 2.3 Random features for scalable Transformers
Revised: 2.3 Random features for scalable Transformers

Original: One recent application of RFs is in the area of scalable Transformers for processing long sequences[15].
Revised: One recent application of RFs is in the area of scalable Transformers for processing long sequences [15].

Original: Let L be the length of the sequence.
Revised: Let L be the length of the sequence.

Original: Interactions between elements in Transformers areimplemented via the self-attention mechanism .
Revised: Interactions between elements in Transformers are implemented via the self-attention mechanism .

Original: Given three matrices Q , K , V ∈ R L × d , the selfattention mechanism returns the following result:
Revised: Given three matrices Q , K , V ∈ R L × d , the selfattention mechanism returns the following result:

Original: [Equation]
Revised: [Equation]

Original: where 1 L ∈ R L is a vector of all ones, x ( i )
Revised: where 1 L ∈ R L is a vector of all ones, x ( i )

Original: =
Revised: =

Original: d − 1 / 4 Q i, : ( i ’th row of Q ) and y ( j ) =
Revised: d − 1 / 4 Q i, : ( i ’th row of Q ) and y ( j ) =

Original: d − 1 / 4 K j, : ,1 ≤ i, j ≤ L .
Revised: d − 1 / 4 K j, : , 1 ≤ i, j ≤ L .

Original: We deduce that computing (3) reduces to applying the linear operator K sfm to d + 1vectors: 1 L , V : , 1 , ..., V : ,d .
Revised: We deduce that computing (3) reduces to applying the linear operator K sfm to d +vectors: 1 L , V : , 1 , ..., V : ,d .

Original: Hence, when L is large, RF approximation similar to (2) but for the
Revised: Hence, when L is large, RF approximation similar to (2) but for the

Original: K sfm ( · , · ) kernel can reduce the computational complexity from O ( dL 2 ) to O ( dML ) .
Revised: K sfm ( · , · ) kernel can reduce the computational complexity from O ( dL 2 ) to O ( dML ) .

Original: Importantly, when the approximation (2) with the replacement K → K sfm can take negative and/ornear-zero values, training is unstable since this approximation emerges in the denominator (inversed)term diag( K sfm 1 L ) in (3).
Revised: Importantly, when the approximation (2) with the replacement K → K sfm can take negative and/or near-zero values, training is unstable since this approximation emerges in the denominator (inversed) term diag( K sfm 1 L ) in (3).

Original: One way to address this is to restrict f (1) ( ω, x ) and f (2) ( ω, y ) to always map into strictly positive numbers R + .
Revised: One way to address this is to restrict f (1) ( ω, x ) and f (2) ( ω, y ) to always map into strictly positive numbers R + .

Original: This is where PosRFs introduced in 2.2 are particularlyrelevant.
Revised: This is where PosRFs introduced in 2.2 are particularly relevant.

Original:  Chefs’ Random Tables
Revised:  Chefs’ Random Tables

Original: We are ready to present our mechanism of chefs’ random tables.
Revised: We are ready to present our mechanism of chefs’ random tables.

Original: All proofs are in the Appendix.
Revised: All proofs are in the Appendix.

Original: 3.1 Generalized exponential RFs (GERFs) & optimal positive RFs (OPRFs)
Revised: 3.1 Generalized exponential RFs (GERFs) & optimal positive RFs (OPRFs)

Original: Our first goal will be to generalize both trigonometric and positive RFs.
Revised: Our first goal will be to generalize both trigonometric and positive RFs.

Original: Then we will focus on onespecial case of this generalization, that will directly lead to the FAVOR++ mechanism.
Revised: Then we will focus on one special case of this generalization, that will directly lead to the FAVOR++ mechanism.

Original: We will be looking for RFs of the following generalized exponential form for p GE ( ω ) ∼ N ( 0 d , I d ) :
Revised: We will be looking for RFs of the following generalized exponential form for p GE ( ω ) ∼ N ( 0 d , I d ) :

Original: [Equation]
Revised: [Equation]

Original: where A, B, C, D ∈ C and s ∈ {− 1 , +1 } .
Revised: where A, B, C, D ∈ C and s ∈ {− 1 , +1 } .

Original: It can be seen that A = 0 , B = i , C = 0 , D = 1 , s = − 1corresponds to trigonometric RFs and A = 0 , B = 1 , C = − 1 , D = 1 , s = 1 corresponds to positive
Revised: It can be seen that A = 0 , B = i , C = 0 , D = 1 , s = −corresponds to trigonometric RFs and A = 0 , B = 1 , C = − 1 , D = 1 , s = 1 corresponds to positive

Original: RFs.
Revised: RFs.

Original: The next theorem describes the conditions under which f ( . )
Revised: The next theorem describes the conditions under which f ( . )

Original: GE can be used to approximate the
Revised: GE can be used to approximate the

Original: Gaussian kernel.
Revised: Gaussian kernel.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: p GE ( ω ) and f ( . )GE , defined in (4), satisfy (1) if
Revised: p GE ( ω ) and f ( . )GE , defined in (4), satisfy (1) if

Original: [Equation]
Revised: [Equation]

Original: where √· and n √· denotes a principal root if the argument is complex.
Revised: where √· and n √· denotes a principal root if the argument is complex.

Original: Hence, A and s can be treated as free parameters and B , C , D as dependent ones.
Revised: Hence, A and s can be treated as free parameters and B , C , D as dependent ones.

Original: The variance ofthese RFs can be expressed through A and s as follows:
Revised: The variance of these RFs can be expressed through A and s as follows:

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Let Re (1 − 8 A ) > 0 .
Revised: Let Re (1 − 8 A ) > 0 .

Original: The variance of (2) using p GE ( ω ) , f ( . )
Revised: The variance of (2) using p GE ( ω ) , f ( . )

Original: GE is given as
Revised: GE is given as

Original: [Equation]
Revised: [Equation]

Original: where α 1 =(cid:18)(cid:113)
Revised: where α [Equation] =

Original: [Equation]
Revised: [Equation]

Original: While it is unclear how to find a global minimum of the objective (6) with respect to A ∈ C ,
Revised: While it is unclear how to find a global minimum of the objective (6) with respect to A ∈ C ,

Original: Re (1 − 8 A ) > 0 and s ∈ {− 1 , +1 } , we observe that it’s possible to find an optimum when werestrict A to be a real number and fix s = +1 .
Revised: Re (1 − 8 A ) > 0 and s ∈ {− 1 , +1 } , we observe that it’s possible to find an optimum when we restrict A to be a real number and fix s = +1 .

Original: Theorem 3.3 (Minimum variance) .
Revised: Theorem 3.3 (Minimum variance) .

Original: When s =
Revised: When s =

Original: +1 ,
Revised: +1 ,

Original: A is restricted to be a real number and ∥ x + y ∥ 2 >0 , the variance (6) is minimized when A = (1 − 1 /ρ ∗ ) / 

Revised: A is restricted to be a real number and ∥ x + y ∥ 2 > 0 , the variance (6) is minimized when A = (1 − 1 /ρ ∗ ) /

Original: where 0 < ρ ∗ < 1 [-,ρ ∗ = (cid:18)(cid:113) (2 ∥ x + y ∥ 2 + d ) 2 + 8 d ∥ x + y ∥ 2 − 
 ∥ x + y ∥ 2 − d (cid:19) / (cid:0) 
 ∥ x + y ∥-]
Revised: where 0 < ρ ∗ < 1 [-,ρ ∗ = (cid:18)(cid:113) (2 ∥ x + y ∥ 2 + d ) 2 + 8 d ∥ x + y ∥ 2 − 
 ∥ x + y ∥ 2 − d (cid:19) / (cid:0) 
 ∥ x + y ∥-],

Original: [Equation]
Revised: [Equation]

Original: Note: One can show that A < 0 for x ̸ = − y thus the corresponding estimator is bounded since theterm A ∥ ω ∥ 2 prevails over linear terms Bω ⊤ x and sBω ⊤ y in (4).
Revised: Note: One can show that A < 0 for x ̸ = − y thus the corresponding estimator is bounded since the term A ∥ ω ∥ 2 prevails over linear terms Bω ⊤ x and sBω ⊤ y in (4).

Original: When ∥ x + y ∥ → 0 then ρ ∗ → 1and thus A → 0 .
Revised: When ∥ x + y ∥ → 0 then ρ ∗ →and thus A → 0 .

Original: Therefore for x = − y the mechanism reduces to PosRF described in Sec. 
Revised: Therefore for x = − y the mechanism reduces to PosRF described in Sec. 2.

Original: 2.2as expected, since for x = − y PosRFs provide perfect estimation (variance equal to zero).
Revised: as expected, since for x = − y PosRFs provide perfect estimation (variance equal to zero).

Original: Largervalues of ∥ x + y ∥ lead to larger gains coming from the new mechanism.
Revised: Larger values of ∥ x + y ∥ lead to larger gains coming from the new mechanism.

Original: From (5) it can be inferred that B, C, D are real when A is real and s = +1 .
Revised: From (5) it can be inferred that B, C, D are real when A is real and s = +1 .

Original: Hence, f (1) ( ω, x ) ,f (2) ( ω, y ) are positive real numbers in this case.
Revised: Hence, f (1) ( ω, x ) , f (2) ( ω, y ) are positive real numbers in this case.

Original: Furthermore, s = +1 , A = 0 corresponds to positive RFs.
Revised: Furthermore, s = +1 , A = 0 corresponds to positive RFs.

Original: Therefore, we refer to RFs with A defined according to (7) as optimal positive RFs(OPRFs).
Revised: Therefore, we refer to RFs with A defined according to (7) as optimal positive RFs (OPRFs).

Original: Figure 1-right illustrates the analytical variance reduction achieved via OPRFs.
Revised: Figure 1-right illustrates the analytical variance reduction achieved via OPRFs.

Original: In practice, we are given sets { x ( i ) } , { y ( j ) } instead of a single pair x , y .
Revised: In practice, we are given sets { x ( i ) } , { y ( j ) } instead of a single pair x , y .

Original: For this reason, in (6,7),we can use the averages of ∥ x ( i ) ∥ 2 , ∥ y ( j ) ∥ 2 , ∥ x ( i ) + s y ( j ) ∥ 2 instead of ∥ x ∥ 2 , ∥ y ∥ 2 , ∥ x + s y ∥ 2 .
Revised: For this reason, in (6,7), we can use the averages of ∥ x ( i ) ∥ 2 , ∥ y ( j ) ∥ 2 , ∥ x ( i ) + s y ( j ) ∥ 2 instead of ∥ x ∥ 2 , ∥ y ∥ 2 , ∥ x + s y ∥ 2 .

Original: This heuristic is based on the assumption that all { x ( i ) } and { y ( j ) } are homogeneous and ∥ x ( i ) ∥ 2,∥ y ( j ) ∥ 2 , ∥x ( i ) + s y ( j ) ∥ 2 are tightly concentrated around their mean.
Revised: This heuristic is based on the assumption that all { x ( i ) } and { y ( j ) } are homogeneous and ∥ x ( i ) ∥ 2 , ∥ y [Equation]

Revised: j )

Revised: ∥ 2 , ∥ x ( i ) + s y ( j ) ∥ 2 are tightly concentrated around their mean.

Original: Computing averages of ∥ x ( i ) ∥ 2 ,∥ y ( j ) ∥ 2 takes O ( Ld ) time.
Revised: Computing averages of ∥ x ( i ) ∥ 2 , ∥ y ( j ) ∥ 2 takes O ( Ld ) time.

Original: Using the formula below, the average of ∥ x ( i ) + s y ( j ) ∥ 2 can be computedwith the same complexity:
Revised: Using the formula below, the average of ∥ x ( i ) + s y ( j ) ∥ 2 can be computed with the same complexity:

Original: [Equation]
Revised: [Equation]

Original: The closed-form solution for real A and s =
Revised: The closed-form solution for real A and s =

Original: +1 allows O (1) -time optimization of (6) after precomputing these statistics.
Revised: +1 allows O (1) -time optimization of (6) after precomputing these statistics.

Original: In the general case we can rely on numerical optimization of (6) with respectto A ∈ C and s ∈ {− 1 , +1 } .
Revised: In the general case we can rely on numerical optimization of (6) with respect to A ∈ C and s ∈ {− 1 , +1 } .

Original: Using precomputed statistics, each evaluation of (6) takes O (1) time.
Revised: Using precomputed statistics, each evaluation of (6) takes O (1) time.

Original: As long as the total number of these evaluations is O ( LM ( d + n )) , where n is the number of K or
Revised: As long as the total number of these evaluations is O ( LM ( d + n )) , where n is the number of K or

Original: K sfm evaluations ( n = d + 1 in Section 2.3), it does not affect the total complexity.
Revised: K sfm evaluations ( n = d + 1 in Section 2.3), it does not affect the total complexity.

Original: The next class of mechanisms, if implemented straightforwardly, does not give positive-valued RFsbut, as we explain in Section 3.2.3, can be easily transformed to variants providing positivity.
Revised: The next class of mechanisms, if implemented straightforwardly, does not give positive-valued RFs but, as we explain in Section 3.2.3, can be easily transformed to variants providing positivity.

Original: 3.2 Discretely-induced random features (DIRFs)
Revised: 3.2 Discretely-induced random features (DIRFs)

Original: Take a discrete probabilistic distribution p ( ω ) where ω 1 , . .
Revised: Take a discrete probabilistic distribution p ( ω ) where ω 1 , . .

Original: ω d are i.i.d.
Revised: ω d are i.i.d.

Original: with P ( ω l = k ) =
Revised: with P ( ω l = k ) =

Original: p k , [Equation] ∞ k =

Revised: p k , (cid:80) ∞ k =

Original: p k
Revised: p k

Original: = 1 and p k > 0 for k ∈ { 0 } ∪ N .
Revised: = 1 and p k > 0 for k ∈ { 0 } ∪ N .

Original: Note that, by Taylor series expansion of exp( · ) ,
Revised: Note that, by Taylor series expansion of exp( · ) ,

Original: [Equation]
Revised: [Equation]

Original: where X l = x ω l l ( ω l !)
Revised: where X l = x ω l l ( ω l !)

Original: − 12 p − 12 ω l , Y l = y ω
Revised: − 12 p − 12 ω l , Y l = y ω

Original: l l ( ω l !)
Revised: l l ( ω l !)

Original: − 12 p − 12 ω l .
Revised: − 12 p − 12 ω l .

Original: Thus we can define discretely-induced randomfeatures providing Gaussian kernel estimation as follows:
Revised: Thus we can define discretely-induced random features providing Gaussian kernel estimation as follows:

Original: [Equation]
Revised: [Equation]

Original: Different instantiations of the above mechanism are given by different probabilistic distributions{ p k } .
Revised: Different instantiations of the above mechanism are given by different probabilistic distributions { p k } .

Original: We will consider two prominent special cases: (a) Poisson, and (b) geometric distributions.
Revised: We will consider two prominent special cases: (a) Poisson, and (b) geometric distributions.

Original: 3.2.1 Poisson random features (PoisRFs)
Revised: 3.2.1 Poisson random features (PoisRFs)

Original: If { p k } is a Poisson distribution, i.e. p k = e − λ λ k /k ! , k ∈ { 0 } ∪ N , then the corresponding RFs aredefined as: f 
Revised: If { p k } is a Poisson distribution, i.e. p k = e − λ λ k /k ! , k ∈ { 0 } ∪ N , then the corresponding RFs are defined as: f (1)pois ( ω, x ) =

Original: [Equation]
Revised: f (2)pois ( ω, x ) = f pois ( ω, x ) = e λd/ 2 −∥ x ∥ 2 / 2 (cid:81) dl =1 x ω l l λ − ω l / 2 .

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: Variance of (2) with p pois , f pois is given by 
Revised: Variance of (2) with p pois , f pois is given by Var

Original: [Equation]
Revised: [Equation]

Original: The exp argument in (11) is convex as a function of λ > 0 .
Revised: The exp argument in (11) is convex as a function of λ > 0 .

Original: By setting its derivative to zero, we findthat λ ∗ =
Revised: By setting its derivative to zero, we find that λ ∗ =

Original: d − 1 / 2 ( (cid:80) dl
Revised: d − 1 / 2 ( (cid:80) dl

Original: =1 x 2 l y 2 l ) 1 / 2 gives the minimum of (11).
Revised: =1 x 2 l y 2 l ) 1 / 2 gives the minimum of (11).

Original: When, instead of a single pair x , y , sets { x ( i ) } , { y ( j ) } are provided, we can use the same homogeneityassumption as in Section 3.1 and substitute the average of (cid:80) dl
Revised: When, instead of a single pair x , y , sets { x ( i ) } , { y ( j ) } are provided, we can use the same homogeneity assumption as in Section 3.1 and substitute the average of (cid:80) dl

Original: =1 ( x ( i ) l ) 2 ( y ( j ) l ) 2 over 1 ≤ i, j ≤ Linstead of (cid:80) dl
Revised: =1 ( x ( i ) l ) 2 ( y ( j ) l ) 2 over 1 ≤ i, j ≤ L instead of (cid:80) dl

Original: =1 x 2 l y 2 l .
Revised: =1 x 2 l y 2 l .

Original: This average can be computed efficiently in O ( Ld ) time as follows:
Revised: This average can be computed efficiently in O ( Ld ) time as follows:

Original: [Equation]
Revised: [Equation]

Original: After computing this statistic, we can calculate λ ∗ in O (1) time using the analytic formula.
Revised: After computing this statistic, we can calculate λ ∗ in O (1) time using the analytic formula.

Original: 3.2.2 Geometric random features (GeomRFs)
Revised: 3.2.2 Geometric random features (GeomRFs)

Original: If { p k } is a geometric distribution, i.e. p k = p (1 − p ) k , k ∈ { 0 } ∪ N , for a parameter 0 <p < 1 , then the corresponding RFs are defined as:f (1)geom ( ω, x ) = f (2)geom ( ω, x ) = f geom ( ω, x )
Revised: If { p k } is a geometric distribution, i.e. p k = p (1 − p ) k , k ∈ { 0 } ∪ N , for a parameter 0 < p < 1 , then the corresponding RFs are defined as:

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.5.
Revised: Theorem 3.5.

Original: The variance of (2) with p geom , f geom is given as
Revised: The variance of (2) with p geom , f geom is given as

Original: [Equation]
Revised: [Equation]

Original: where I 0 ( · ) is the modified Bessel function of the first kind of order 0 .
Revised: where I 0 ( · ) is the modified Bessel function of the first kind of order 0 .

Original: Again as for the previously described mechanisms, when sets { x ( i ) } , { y ( j ) } are given, we can useaverages of | x ( i ) l y ( j ) l | , 1 ≤ l ≤ d , instead of | x l y l | in (13) assuming homogeneity of x ( i ) ’s and y ( j ) ’s.
Revised: Again as for the previously described mechanisms, when sets { x ( i ) } , { y ( j ) } are given, we can use averages of | x ( i ) l y ( j ) l | , 1 ≤ l ≤ d , instead of | x l y l | in (13) assuming homogeneity of x ( i ) ’s and y ( j ) ’s.

Original: Each out of d averages can be computed in O ( L ) time as follows:
Revised: Each out of d averages can be computed in O ( L ) time as follows:

Original: [Equation]
Revised: [Equation]

Original: After precomputation of these statistics, evaluation of (11) takes O ( d ) time.
Revised: After precomputation of these statistics, evaluation of (11) takes O ( d ) time.

Original: A numerical optimizationcan be used to minimize (11) with respect to p .
Revised: A numerical optimization can be used to minimize (11) with respect to p .

Original: As long as the number of variance evaluations is
Revised: As long as the number of variance evaluations is

Original: O ( LM (1 + n/d )) , the total complexity estimate is not affected.
Revised: O ( LM (1 + n/d )) , the total complexity estimate is not affected.

Original: 3.2.3 Making discretely-induced RFs positive
Revised: 3.2.3 Making discretely-induced RFs positive

Original: As can be inferred from Eq. 10, DIRFs are positive when all elements of x and y are positive.
Revised: As can be inferred from Eq. 10, DIRFs are positive when all elements of x and y are positive.

Original: Ifthis is not the case, and positive-valued RFs are needed, e.g. in applications involving scalable
Revised: If this is not the case, and positive-valued RFs are needed, e.g. in applications involving scalable

Original: Transformers, one way to make them positive is to take some vector c ∈ R d such that c l < x l , y l .
Revised: Transformers, one way to make them positive is to take some vector c ∈ R d such that c l < x l , y l .

Original: An example of such a vector is given by c l = min i min( x ( i ) l , y ( i ) l ) − ϵ where ϵ > 0 is a smallconstant.
Revised: An example of such a vector is given by c l = min i min( x ( i ) l , y ( i ) l ) − ϵ where ϵ > 0 is a small constant.

Original: Next, define (cid:98) x
Revised: Next, define (cid:98) x

Original: ( i ) =
Revised: ( i ) =

Original: x ( i ) − c , (cid:98) y
Revised: x ( i ) − c , (cid:98) y

Original: ( j ) =
Revised: ( j ) =

Original: y ( j ) − c .
Revised: y ( j ) − c .

Original: Then, clearly, (cid:98) x ( i ) − (cid:98) y
Revised: Then, clearly, (cid:98) x ( i ) − (cid:98) y

Original: ( j ) =
Revised: ( j ) =

Original: x ( i ) − y ( j ) ,
Revised: x ( i ) − y ( j ) ,

Original: K ( (cid:98) x
Revised: K ( (cid:98) x

Original: ( i ) , (cid:98) y
Revised: ( i ) , (cid:98) y

Original: ( j ) )
Revised: ( j ) )

Original: =
Revised: =

Original: K ( x ( i ) , y ( j ) ) and RFs can be used on (cid:98) x
Revised: K ( x ( i ) , y ( j ) ) and RFs can be used on (cid:98) x

Original: ( i ) , (cid:98) y
Revised: ( i ) , (cid:98) y

Original: ( j ) which have positive entries.
Revised: ( j ) which have positive entries.

Original: Werefer to these variants of PoisRFs and GeomRFs as PoisRF+ and GeomRF+ respectively.
Revised: We refer to these variants of PoisRFs and GeomRFs as PoisRF+ and GeomRF+ respectively.

Original:  Additional theoretical results & FAVOR++
Revised:  Additional theoretical results & FAVOR++

Original: Interestingly, as in the case of the PosRF mechanism from [15], OPRFs also benefit from applyingblock-orthogonal ensembles of projections ω (see Appendix 8.1 and [15] for the exact definition).
Revised: Interestingly, as in the case of the PosRF mechanism from [15], OPRFs also benefit from applying block-orthogonal ensembles of projections ω (see Appendix 9.1 and [15] for the exact definition).

Original: We show below that orthogonal RFs reduce the variance of GERFs with A ∈ R , s = 1 (which is thecase for OPRF) for any d > 0 :
Revised: We show below that orthogonal RFs reduce the variance of GERFs with A ∈ R , s = 1 (which is the case for OPRF) for any d > 0 :

Original: Theorem 4.1 (Orthogonal OPRFs) .
Revised: Theorem 4.1 (Orthogonal OPRFs) .

Original: If Var( (cid:98) K ort M ( x , y )) denotes the variance of the orthogonal GERFestimator (cid:98) K ort M ( x , y ) of the Gaussian kernel with A ∈ R , s = 1 in (4) at x , y ∈ R d using M RFsand Var( (cid:98) K iid M ( x , y )) is the analogous expression with i.i.d.
Revised: If Var( (cid:98) K ort M ( x , y )) denotes the variance of the orthogonal GERF estimator (cid:98) K ort M ( x , y ) of the Gaussian kernel with A ∈ R , s = 1 in (4) at x , y ∈ R d using M RFs and Var( (cid:98) K iid M ( x , y )) is the analogous expression with i.i.d.

Original: samples, then for some C ( ∥ x + y ∥ ) ≥ 0 :
Revised: samples, then for some C ( ∥ x + y ∥ ) ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original: Note: The analogous inequality can be obtained for TrigRFs only in the asymptotic sense (for d largeenough, see Theorem 3.8 in [14]).
Revised: Note: The analogous inequality can be obtained for TrigRFs only in the asymptotic sense (for d large enough, see Theorem 3.8 in [14]).

Original: One of the key properties used in the proof of Theorem 4.1 ispositivity of RFs.
Revised: One of the key properties used in the proof of Theorem 4.1 is positivity of RFs.

Original: We conclude that positive-valued RFs are particularly well suited for the quasi
Revised: We conclude that positive-valued RFs are particularly well suited for the quasi

Original: Monte-Carlo methods based on the orthogonal ensembles.
Revised: Monte-Carlo methods based on the orthogonal ensembles.

Original: Analogously to FAVOR+ [15], we refer tothe self-attention approximation mechanism based on orthogonal OPRFs as FAVOR++ .
Revised: Analogously to FAVOR+ [15], we refer to the self-attention approximation mechanism based on orthogonal OPRFs as FAVOR++ .

Original: We now provide strong concentration results for the OPRF-based estimators critically relying onthe boundedness of OPRFs.
Revised: We now provide strong concentration results for the OPRF-based estimators critically relying on the boundedness of OPRFs.

Original: Boundedness of OPRFs is due to A = (1 − 1 /ρ ∗ ) / 8 < 0 even when∥ x + y ∥ 2 is substituted by average (8) in (7).
Revised: Boundedness of OPRFs is due to A = (1 − 1 /ρ ∗ ) / 8 < 0 even when ∥ x + y ∥ 2 is substituted by average (8) in (7).

Original: To the best of our knowledge, these are the firstsuch results for positive-valued RFs.
Revised: To the best of our knowledge, these are the first such results for positive-valued RFs.

Original: Denote by L the Legendre Transform of the random variable
Revised: Denote by L the Legendre Transform of the random variable

Original: Z = f (1)GE ( ω, x ) f (2)GE ( ω, y ) for f ( · )GE as in (4) with A, B, C, D ∈ R defining GERFs.
Revised: Z = f (1)GE ( ω, x ) f (2)GE ( ω, y ) for f ( · )GE as in (4) with A, B, C, D ∈ R defining GERFs.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Suppose A < 0 .
Revised: Suppose A < 0 .

Original: The following is true for any ϵ > 0 : P [ | (cid:98) K iid M ( x , y ) − K ( x , y )
Revised: The following is true for any ϵ > 0 : P [ | (cid:98) K iid M ( x , y ) − K ( x , y )

Original: | ≥ ϵ ] ≤2 exp( − Mϵ 2 2 exp( ∥ x ∥ 2 + ∥ y ∥
Revised: | ≥ ϵ ] ≤ 2 exp( − Mϵ 2 2 exp( ∥ x ∥ 2 + ∥ y ∥

Original: 2 2 A )) .
Revised: 2 2 A )(1 − 4 A ) − d 2 ) .

Original: Furthermore, for the orthogonal variant we have: P [ (cid:98) K ort M ( x , y ) − K ( x , y ) ≥ ϵ ] ≤ exp( − M L ( K ( x , y ) + ϵ )) and L ( K ( x , y ) + ϵ )) > 0 .
Revised: Furthermore, for the orthogonal variant we have: P [ (cid:98) K ort M ( x , y ) − K ( x , y ) ≥ ϵ ] ≤ exp( − M L ( K ( x , y ) + ϵ )) and L ( K ( x , y ) + ϵ )) > 0 .

Original: Finally, below we provide the first result regarding uniform convergence for attention approximationin the efficient low-rank Transformers (Section 2.3).
Revised: Finally, below we provide the first result regarding uniform convergence for attention approximation in the efficient low-rank Transformers (Section 2.3).

Original: Theorem 4.3 (Uniform convergence for attention approximation) .
Revised: Theorem 4.3 (Uniform convergence for attention approximation) .

Original: Assume that rows of Q and K from(3) come from the L 2 -ball of radius R > 0 .
Revised: Assume that rows of Q and K from (3) come from the L 2 -ball of radius R > 0 .

Original: Denote by (cid:98) K sfm the approximation of K sfm from (3) via the GERF-mechanism with A < 0 using M independent random projections.
Revised: Denote by (cid:98) K sfm the approximation of K sfm from (3) via the GERF-mechanism with A < 0 using M independent random projections.

Original: Then ∥K sfm − (cid:98) K sfm ∥ ∞ ≤ ϵwith any constant probability when M = Ω(Γ dϵ 2 log( γρϵ )) , where Γ = exp( − 3 R 2 √ dA ) , ρ = √ 2 Rd − 14and γ =  (cid:113) 4Γ ( R 2 √ d + d 2 ) (for A as in the definition of the OPRFs).
Revised: Then ∥K sfm − (cid:98) K sfm ∥ ∞ ≤ ϵ with any constant probability when M = Ω((1 − 4 A ) d 2 Γ dϵ 2 log( γρϵ )) , where: Γ = exp( − 3 R 2 √ dA ) , ρ = √ 2 Rd − 14 , γ = (1 − 4 A ) d 2 (cid:113) 4Γ( R 2 √ d + d 2 ) (for A as in the OPRFs definition).

Original:  Experiments
Revised:  Experiments

Original: We present an extensive empirical evaluation of CRTs.
Revised: We present an extensive empirical evaluation of CRTs.

Original: Additional details and results for eachexperiment can be found in the Appendix 8.10.
Revised: Additional details and results for each experiment can be found in the Appendix 9.10.

Original: 5.1 Comparing variance of CRTs for Gaussian kernel estimation
Revised: 5.1 Comparing variance of CRTs for Gaussian kernel estimation

Original: In this initial experiment, we sample synthetic pairs of vectors x , y and evaluate variance of CRTsbased on the analytic formulas (6,11,13).
Revised: In this initial experiment, we sample synthetic pairs of vectors x , y and evaluate variance of CRTs based on the analytic formulas (6,11,13).

Original: Our goal is to check whether there are scenarios when thenewly introduced RF mechanisms have smaller variance than existing TrigRF and PosRF methods.
Revised: Our goal is to check whether there are scenarios when the newly introduced RF mechanisms have smaller variance than existing TrigRF and PosRF methods.

Original: We set d = 64 which is standard in e.g. Transformer applications (Section 2.3).
Revised: We set d = 64 which is standard in e.g. Transformer applications (Section 2.3).

Original: We use four differentregimes for drawing x , y : normal corresponds to x , y sampled from N ( 0 d , σ 2 I d ) , sphere corresponds to x , y sampled uniformly on a sphere σ S d − 1 , heterogen corresponds to x and y sampledfrom two heterogeneous distributions: N ( 0 d , σ 2 I d ) and N ( σ 1 d , σ 2 I d ) and mnist corresponds tox , y being random images from MNIST dataset [19] resized to 8 × 8 , scaled by σ > 0 and flattened.
Revised: We use four different regimes for drawing x , y : normal corresponds to x , y sampled from N ( 0 d , σ 2 I d ) , sphere corresponds to x , y sampled uniformly on a sphere σ S d − 1 , heterogen corresponds to x and y sampled from two heterogeneous distributions: N ( 0 d , σ 2 I d ) and N ( σ 1 d , σ 2 I d ) and mnist corresponds to x , y being random images from MNIST dataset [19] resized to 8 × 8 , scaled by σ > 0 and flattened.

Original: In many scenarios (see Figure 2), CRTs outperform TrigRF and PosRF baselines.
Revised: In many scenarios (see Figure 2), CRTs outperform TrigRF and PosRF baselines.

Original: Among otherimprovements, GERF gives more than e 80 , e 125 , e 10 times variance reduction compared to TrigRFin normal , heterogen and mnist when σ = 1 .
Revised: Among other improvements, GERF gives more than e 80 , e 125 , e 10 times variance reduction compared to TrigRF in normal , heterogen and mnist when σ = 1 .

Original: OPRF and GeomRF+ give more than e 75 , e 125 , e 7times variance reduction compared to PosRF in normal , heterogen and mnist when σ = 1 .
Revised: OPRF and GeomRF+ give more than e 75 , e 125 , etimes variance reduction compared to PosRF in normal , heterogen and mnist when σ = 1 .

Original: 5.2 Comparing CRTs in the non-parametric classification
Revised: 5.2 Comparing CRTs in the non-parametric classification

Original: Our next experiment is a non-parametric classification where probabilities are predicted by kernelregression [38, 59] with the Gaussian kernel.
Revised: Our next experiment is a non-parametric classification where probabilities are predicted by kernel regression [38, 59] with the Gaussian kernel.

Original: Training data consists of objects o (1) , . . .
Revised: Training data consists of objects o (1) , . . .

Original: , o ( L ) ∈ R dwith corresponding one-hot encoded labels r (1) , . . .
Revised: , o ( L ) ∈ R d with corresponding one-hot encoded labels r (1) , . . .

Original: , r ( L ) ∈ R n .
Revised: , r ( L ) ∈ R n .

Original: The predicted label distribution for[Table] the new object o ∗ is defined as r ∗ =
Revised: The predicted label distribution for the new object o ∗ is defined as r ∗ =

Original: (cid:80) Li =1 K ( σ o ∗ , σ o ( i ) )
Revised: (cid:80) Li =1 K ( σ o ∗ , σ o ( i ) )

Original: r ( i ) / (cid:80) Li
Revised: r ( i ) / (cid:80) Li

Original: =1 K ( σ o ∗ , σ o ( i ) )
Revised: =1 K ( σ o ∗ , σ o ( i ) )

Original: where σ > 0is a hyperparameter tuned on the validation set.
Revised: where σ > [Table] is a hyperparameter tuned on the validation set.

Original: Using the RF approximation for the kernel as in(2), we, with O ( nLM ) preprocessing, can approximate r ∗ in O ( nM ) time per example instead of
Revised: Using the RF approximation for the kernel as in (2) [Table] we, with O ( nLM ) preprocessing, can approximate r ∗ in O ( nM ) time per example instead of

Original: O ( nL ) for the exact computation.
Revised: O ( nL ) for the exact computation.

Original: Since the predicted class is argmax 1 ≤ l ≤ n r ∗ , we can ignore the denominator term and, therefore, usenon-positive RFs.
Revised: Since the predicted class is argmax 1 ≤ l ≤ n r ∗ , we can ignore the denominator term and, therefore, use non-positive RFs.

Original: We evaluate on classification benchmarks from UCI Repository [24] (Table 1).
Revised: We evaluate on classification benchmarks from UCI Repository [24] (Table 1).

Original: Thebest results are achieved by new RF mechanisms, with GeomRF and OPRF performing particularlywell.
Revised: The best results are achieved by new RF mechanisms, with GeomRF and OPRF performing particularly well.

Original: OPRF shows the best average performance, therefore our recommendation for practitioners is toopt for this method.
Revised: OPRF shows the best average performance, therefore our recommendation for practitioners is to opt for this method.

Original: For the same reason, we focus on the FAVOR++ variant (OPRF with orthogonalrandom projections for attention approximation) in our Transformer experiments below.
Revised: For the same reason, we focus on the FAVOR++ variant (OPRF with orthogonal random projections for attention approximation) in our Transformer experiments below.

Original: 5.3 FAVOR++ in scalable Transformers 
Revised: 5.3 FAVOR++ in scalable Transformers 5.3.1 Natural language processing

Original: 5.3.1 Natural language processing

Original: In this setting, we test different low-rank attention Transformers on the General Language Understanding Evaluation (GLUE) benchmark [57], consisting of 8 different natural language understandingtasks with the sequence length ranging from 32 to 128.
Revised: In this setting, we test different low-rank attention Transformers on the General Language Understanding Evaluation (GLUE) benchmark [57], consisting of 8 different natural language understanding tasks with the sequence length ranging from 32 to 128.

Original: We used the same training parametersas mentioned in [20] (see Appendix 8.10.3 for details).
Revised: We used the same training parameters as mentioned in [20] (see Appendix 9.10.3 for details).

Original: We compared FAVOR+ [15], ELU [31]and ReLU [15] variants of the Performers [15] against a FAVOR++ variant and report the resultsin Table 2.
Revised: We compared FAVOR+ [15], ELU [31] and ReLU [15] variants of the Performers [15] against a FAVOR++ variant and report the results in Table 2.

Original: We find that FAVOR++ outperforms all these low-rank Transformers in most GLUEtasks.
Revised: We find that FAVOR++ outperforms all these low-rank Transformers in most GLUE tasks.

Original: In particular, FAVOR++ outperforms FAVOR+ on all GLUE tasks, demonstrating downstreameffectiveness of the variance reduction of the softmax kernel estimation.
Revised: In particular, FAVOR++ outperforms FAVOR+ on all GLUE tasks, demonstrating downstream effectiveness of the variance reduction of the softmax kernel estimation.

Original: Furthermore, warm-startingwith pre-trained BERT-base model checkpoint [20] ( Uptrain FAVOR++ in Table 2), further improvesperformance demonstrating backward-compatibility of FAVOR++ with the exact softmax kernel.
Revised: Furthermore, warm-starting with pre-trained BERT-base model checkpoint [20] ( Uptrain FAVOR++ in Table 2), further improves performance demonstrating backward-compatibility of FAVOR++ with the exact softmax kernel.

Original: 5.3.2 Speech modelling
Revised: 5.3.2 Speech modelling

Original: We compare FAVOR++ with FAVOR+ on speech models with the LibriSpeech ASR corpus ([42]).
Revised: We compare FAVOR++ with FAVOR+ on speech models with the LibriSpeech ASR corpus ([42]).

Original: We apply both to approximate attention blocks in the 17 -layer Conformer-Transducer encoder ([26])of only 4 attention heads and use the word error rate (WER) metric – a standard way to evaluate[Table] speech models.
Revised: We apply both to approximate attention blocks in the 17 -layer Conformer-Transducer encoder ([26]) of only 4 attention heads and use the word error rate (WER) metric – a standard way to evaluate speech models.

Original: In both cases FAVOR++ outperforms FAVOR+, as shown in Figure 3. 
Revised: In both cases FAVOR++ outperforms FAVOR+, as shown in Figure 3. The WER

Original: The WERimprovement for FAVOR++ is substantial: 2.49% for M = 8 and 3.05% for M = 16 with anegligible O ( Ld )
Revised: [Table] improvement for FAVOR++ is substantial: 2.49% for M = 8 and 3.05% for M = 16 with a negligible O ( Ld )

Original: ≪
Revised: ≪

Original: O ( LMd ) overhead for computing (8) compared to FAVOR+.
Revised: O ( LMd ) overhead for computing (8) compared to FAVOR+.

Original: 5.3.3 Vision Transformers
Revised: 5.3.3 Vision Transformers

Original: To further showcase the need for more accurate softmax kernel approximation, we compare theperformance of FAVOR+ and FAVOR++ on ImageNet ([18]).
Revised: To further showcase the need for more accurate softmax kernel approximation, we compare the performance of FAVOR+ and FAVOR++ on ImageNet ([18]).

Original: We inject both mechanisms to theattention modules of Vision Transformers (ViT [21]).
Revised: We inject both mechanisms to the attention modules of Vision Transformers (ViT [21]).

Original: In Figure 4, we show the results of trainingfrom scratch and uptraining from the MAE checkpoint [29].
Revised: In Figure 4, we show the results of training from scratch and uptraining from the MAE checkpoint [29].

Original: We see that, as opposed to FAVOR+,
Revised: We see that, as opposed to FAVOR+,

Original: FAVOR++ is more stable and is able to improve performance especially for uptraining, demonstratingbackward-compatibility with the exact softmax kernel.
Revised: FAVOR++ is more stable and is able to improve performance especially for uptraining, demonstrating backward-compatibility with the exact softmax kernel.

Original: Finally, we compare the computational complexity of FAVOR+ and FAVOR++.
Revised: Finally, we compare the computational complexity of FAVOR+ and FAVOR++.

Original: In Figure 4, the rightplot shows the number of steps per second as a function of sequence length L on the same hardware.
Revised: In Figure 4, the right plot shows the number of steps per second as a function of sequence length L on the same hardware.

Original: We see that attention modules using FAVOR+ and FAVOR++ have very similar computation time(both provide linear attention).
Revised: We see that attention modules using FAVOR+ and FAVOR++ have very similar computation time (both provide linear attention).

Original: Moreover, for sequence lengths above 1000 , training a regular ViTmodel became increasingly difficult due to out-of-memory errors.
Revised: Moreover, for sequence lengths above 1000 , training a regular ViT model became increasingly difficult due to out-of-memory errors.

Original:  Limitations of this work & broader impact
Revised:  Limitations of this work & broader impact

Original: Several of the mechanisms proposed in this paper can be further extended, potentially leading toeven more accurate algorithms.
Revised: Several of the mechanisms proposed in this paper can be further extended, potentially leading to even more accurate algorithms.

Original: For instance, it remains an open question how to choose theoreticallyoptimal parameters for GERF and GeomRF mechanisms.
Revised: For instance, it remains an open question how to choose theoretically optimal parameters for GERF and GeomRF mechanisms.

Original: Furthermore, DIRFs can benefit fromoptimizing the discrete distributions defining them (to minimize variance of the estimation) ratherthan choosing them a priori.
Revised: Furthermore, DIRFs can benefit from optimizing the discrete distributions defining them (to minimize variance of the estimation) rather than choosing them a priori.

Original: Our methods should be used responsibly, given rising concerns regardingthe carbon footprint of training massive Transformer models and other societal issues [49, 8, 3, 60].
Revised: Our methods should be used responsibly, given rising concerns regarding the carbon footprint of training massive Transformer models and other societal issues [49, 8, 3, 60].

Original:  Conclusion
Revised:  Conclusion

Original: We presented a new class of RF mechanisms called chefs’ random tables (CRTs) including methodsproviding positivity and boundedness of random features – two key properties for new applications of
Revised: We presented a new class of RF mechanisms called chefs’ random tables (CRTs) including methods providing positivity and boundedness of random features – two key properties for new applications of

Original: RFs in Transformer training.
Revised: RFs in Transformer training.

Original: We provided comprehensive theoretical results and extensive empiricalevaluation, resulting in particular in new state-of-the-art low-rank attention Transformers for text.
Revised: We provided comprehensive theoretical results and extensive empirical evaluation, resulting in particular in new state-of-the-art low-rank attention Transformers for text.

Revised:  Acknowledgements

Revised: V. L. acknowledges support from the Cambridge Trust and DeepMind.

Revised: V. L. was part-time employed by Google while a PhD student.

Revised: A.W. acknowledges support from a Turing AI Fellowship under

Revised: EPSRC grant EP/V025279/1, The Alan Turing Institute and the Leverhulme Trust via CFI.

Original: Checklist
Revised: Checklist

Original:  and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes (b) Did  describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 
Revised: [Yes] See Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeSection 6.(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] See Section 6. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See statementsof Theorems 3.1, 3.2, 3.3, 3.4, 3.5, 4.1, 4.3.(b)
Revised: [Yes] See statements of Theorems 3.1, 3.2, 3.3, 3.4, 3.5, 4.1, 4.3.

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendices 8.2,8.3, 8.4, 8.5, 8.6, 8.7, 8.8, 8.9.3.
Revised: [Yes] See Appendices 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We include the part of the code that is not confidential, the core CRT variant: FAVOR++ mechanism.
Revised: [Yes] We include the part of the code that is not confidential, the core CRT variant: FAVOR++ mechanism.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5 and Appendix 8.10.
Revised: [Yes] See Section 5 and Appendix 9.10.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Figure 2 and Table 3 in the Appendix.
Revised: [Yes] See Figure 2 and Table 3 in the Appendix.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We mention that in the Appendix8.10.
Revised: [Yes] We mention that in the Appendix 9.10.

Original: We don’t report the total amount of compute, because that would violate internalpolicies of our organisation.
Revised: We don’t report the total amount of compute, because that would violate internal policies of our organisation.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] We use standard publicly available datasets and frameworks.
Revised: [N/A] We use standard publicly available datasets and frameworks.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: We only use standard publicly available datasets.
Revised: We only use standard publicly available datasets.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: We only standard publicly available datasets.
Revised: We only standard publicly available datasets.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: We didn’t use crowdsourcing (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: We didn’t use crowdsourcing (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: ] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]