H YBRID R ANDOM F EATURES
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a new class of random feature methods for linearizing softmax and
Gaussian kernels called hybrid random features (HRFs) that automatically adapt the quality of kernel estimation to provide most accurate approximation in the deﬁned regions of interest.
Special instantiations of HRFs lead to well-known methods such as trigonometric (Rahimi & Recht, 2007) or (recently introduced in the context of linear-attention Transformers) positive random features (Choromanski et al., 2021b).
By generalizing Bochner’s Theorem for softmax/Gaussian kernels and leveraging random features for compositional kernels, the HRFmechanism provides strong theoretical guarantees - unbiased approximation and strictly smaller worst-case relative errors than its counterparts.
We conduct exhaustive empirical evaluation of HRF ranging from pointwise kernel estimation experiments, through tests on data admitting clustering structure to benchmarking implicit-attention Transformers (also for downstream Robotics applications), demonstrating its quality in a wide spectrum of machine learning problems.
I NTRODUCTION & R ELATED W ORK
Consider the softmax and Gaussian kernel functions K : R d × d → R deﬁned as follows:
[Equation]
These two are prominent examples of functions used in the so-called kernel methods (Gretton et al., 2005; Zhang et al., 2018) and beyond, i.e. in softmax-sampling (Blanc & Rendle, 2018).
Random features (RFs, Rahimi & Recht, 2007; Liu et al., 2020; Peng et al., 2021) yield a powerful mechanism for linearizing and consequently scaling up kernel methods with dot-product kernel decompositions disentangling x from y in the formulae for kernel value K( x , y )
≈ φ ( x ) (cid:62) φ ( y ) via data-agnostic probabilistic (random feature) maps φ : R d → R m .
The tight relationship between softmax and Gaussian kernels given by the transformation K gauss ( x , y ) =
exp( − (cid:107) x (cid:107)
[Equation]
( x , y ) exp( − (cid:107) y
[Equation]
22 2 ) provides a mapping of any random feature vector φ SM ( u ) for the softmax kernel to the corresponding one φ gauss ( u ) =
exp( − (cid:107) u
(cid:107) 2 2 2 )
φ SM ( u ) for the Gaussian kernel, thus we will focus on the former kernel.
The classic random feature map mechanism φ trig m for the softmax kernel, obtained from
Bochner’s Theorem applied to the Gaussian kernel (Rahimi & Recht, 2007), is of the form: [Equation]
[Equation]
where m stands for the number of random features and ω 1 , ..., ω m iid ∼ N (0 , I d ) .
The above (common in most downstream use-cases) method for linearizing softmax/Gaussian kernels was recently shown to fail in some of the new impactful applications of scalable kernel methods such as implicit-attention Transformer architectures called Performers (Choromanski et al., 2021b).
We denote by MSE the mean squared error of the estimator (i.e. its variance since all estimators considered in this paper are unbiased).
The above mechanism struggles to accurately approximate close-to-zero kernel values, as characterized by the particularly large MSE in that region.
This is a crucial problem since most of the entries of the attention matrices in Transformers’ models are very small and the approximators need to be particularly accurate there.
Otherwise the renormalizers computed to make attention matrices row-stochastic (standard attention normalization procedure in
Transformers) would be estimated imprecisely, potentially even by negative values.
The solution to this problem was presented in FAVOR+ mechanism (Choromanski et al., 2021b), where a new positive random feature map for unbiased softmax kernel estimation was applied:
[Equation]
Even though, very accurate in estimating small softmax kernel values (which turns out to be crucial in making RFs work for Transformers training), this mechanism is characterized by larger MSE for large kernel values.
In several applications of softmax kernels (in particular Transformers, where attention matrices typically admit sparse combinatorial structure with relatively few but critical large entries and several close-to-zero ones or softmax sampling) the algorithm needs to process simultaneously very small and large kernel values.
The following natural questions arise:
Is it possible to get the best from both the mechanisms to obtain RFs-based estimators particularly accurate for both very small and large softmax kernel values ?
Furthermore, can those estimators be designed to have low variance in more generally pre-deﬁned regions ?
We give afﬁrmative answers to both of the above questions by constructing a new class of random feature maps techniques called hybrid random features or HRFs.
Theoretical methods used by us to develop HRFs: (a) provide a unifying perspective, where trigonometric random features from
Bochner’s Theorem and novel mechanisms proposed in (Choromanski et al., 2021b) are just special corollaries of the more general result from complex analysis, (b) integrate in the original way several other powerful probabilistic techniques such as Goemans-Willimson method (Goemans &
Williamson, 2004) and random features for the compositional kernels (Daniely et al., 2017).
We provide detailed theoretical analysis of HRFs, showing in particular that they provide strictly more accurate worst-case softmax kernel estimation than previous algorithms and lead to computational gains.
We also conduct their thorough empirical evaluation on tasks ranging from pointwise kernel estimation to downstream problems involving training Transformer-models or even end-toend robotic controller-stacks including attention-based architectures.
Related Work: The literature on different random feature map mechanisms for Gaussian (and thus also softmax) kernel estimation is voluminous.
Most focus has been put on reducing the variance of trigonometric random features from (Rahimi & Recht, 2007) via various Quasi Monte Carlo (QMC) methods, where directions and/or lengths of Gaussian vectors used to produce features are correlated, often through geometric conditions such as orthogonality (Choromanski et al., 2017;
Rowland et al., 2018; Yu et al., 2016; Choromanski et al., 2019; Choromanski & Sindhwani, 2016).
Our HRFs do not compete with those techniques (and can be in fact easily combined with them) since rather than focusing on improving sampling mechanism for a given approximation algorithm, they provide a completely new algorithm.
The new application of random features for softmax kernel in Transformers proposed in (Choromanski et al., 2021b)
led to the fruitful research on the extensions and limitations of these methods.
In (Schlag et al., 2021) random features were replaced by sparse deterministic constructions (no longer approximating softmax kernel).
In (Luo et al., 2021) it was observed that combining L 2 -normalization of queries and keys for variance reduction of softmax kernel estimation with FFT-based implementations of relative position encoding and FAVOR+ mechanism from Performers helps in training.
Trigonometric random features were applied for softmax sampling in (Rawat et al., 2019).
Several other techniques such as Nystr¨om method (Yang et al., 2012; Williams & Seeger, 2000; Rudi et al., 2015) were proposed to construct datadependent feature representations.
Even though, as we show in Sec. 2.4, certain instantiations of the HRF mechanism clearly beneﬁt from some data analysis, our central goal remains an unbiased estimation of the softmax/Gaussian kernels which is no longer the case for those other techniques.
H YBRID R ANDOM F EATURES 2. P RELIMINARIES
Whenever we do not say explicitly otherwise, all presented lemmas and theorems are new.
We start with the following basic deﬁnitions and results.
Deﬁnition 2.1 (Kernel with a Random Feature Map Representation) .
We say that a kernel function
K : R d × R d → R admits a random feature (RF) map representation if it can be written as
[Equation]
for some ξ i : R d × R d → R , and where ω is sampled from some probabilistic distribution Ω ∈
P ( R d ) .
The corresponding random feature map, for a given m ∈ N , is deﬁned as
[Equation]
where φ im = ( ξ i ( u , ω 1 ) , ..., ξ i ( u , ω m )) (cid:62) , (cid:63) stands for vertical concatenation, and ω 1 , ..., ω m iid ∼ Ω .
Random feature maps can be used to unbiasedly approximate corresponding kernels, as follows:
[Equation]
Using the above notation, a trigonometric random feature φ trig m from Equation 2 can be encoded as applying l = 2 , ξ 1 ( u , ω ) = sin( ω (cid:62) u ) , ξ 2 ( u , ω ) =
cos( ω (cid:62) u ) .
Similarly, positive random features can be encoded as taking l = 2 and ξ 1 ( u , ω ) = 1 √ 2 exp( ω (cid:62) u ) , ξ 2 ( u , ω ) = 1 √ 2 exp( − ω (cid:62) u ) .
The following result from (Choromanski et al., 2021b) shows that the mean squared error (MSE) of the trigonometric estimator is small for large softmax kernel values and large for small softmax kernel values, whereas an estimator applying positive random features behaves in the opposite way.
Denote by (cid:100) SM trig m ( x , y ) an estimator of SM( x , y ) for x , y ∈ R d using trigonometric RFs and ω 1 , ..., ω m iid ∼ N (0 , I d ) .
Denote by (cid:100) SM ++ m ( x , y ) its analogue using positive RFs.
We have:
Lemma 2.2 (positive versus trigonometric RFs) .
Take ∆ = x − y , z
= x + y , f 1 ( u ) =
(2 m ) − 1 exp( u 2 )SM − 2 ( x , y ) , f 2 ( u ) = (2 m ) − 1 exp( u 2 )SM 2 ( x , y ) , f 3 ( u ) =
(1 − exp( − u 2 )) 2 .
The MSEs of these estimators are:
[Equation]
2. T HE A LGORITHM
We are ready to present the mechanism of Hybrid Random Features (HRFs).
Denote by E =
( (cid:100) SM k
( x , y ))
p +1 k
=1 a list of estimators of SM( x , y ) (the so-called base estimators ) and by Λ
= ( (cid:98) λ k
( x , y )) pk =1 a list of estimators of { λ k ( x , y ) } pk =1 for some functions
λ k : R d × R d
→ [0 , 1] , constructed independently from E .
Take the following estimator of SM( x , y ) :
[Equation]
In the next section, we explain in detail how base estimators are chosen.
We call (cid:100) SM E , Λ ( x , y ) a hybrid random feature (HRF) estimator of SM( x , y ) parameterized by E , Λ .
The role of the λcoefﬁcients is to dynamically (based on the input ( x , y ) ) prioritize or deprioritize certain estimators to promote those which are characterized by lower variance for a given input.
Note that if elements of E are unbiased estimators of SM( x , y ) , then trivially (cid:100) SM E , Λ ( x , y ) is also an unbiased estimator of SM( x , y ) .
Assume that each (cid:100) SM k ( x , y ) is of the form (cid:100)
SM k ( x , y ) =
( φ k 1 ,m ( x )) (cid:62) φ k 2 ,m ( y ) for φ kj,m ( u ) = 1 √ m φ 1 ,kj,m ( u ) (cid:63) ...
(cid:63) φ t k ,k j,m ( u ) , t k > 0 and φ 1 ,kj,m , ..., φ t k ,k j,m : R d
→ R m , where j ∈ { 1 , 2 } .
Assume also that λ k
( x , y ) can be written as:
[Equation]
for some scalars a k ∈ R , distribution Ω ∈ P ( R d ) (where P ( R d stands for the set of probabilistic distributions on R d ), mappings ξ ik , η ik : R d × R d → R and that the corresponding estimator (cid:98) λ k
( x , y )
= (cid:98) λ kn ( x , y ) of λ k ( x , y ) is of the form:
[Equation]
y ) (10) for ρ kj,n ( u ) = 1 √ n ρ 1 ,kj,n ( u ) (cid:63) ...
(cid:63) ρ l k ,k j,n ( u ) , ρ i,kj,n ( u ) =
( f ij,k ( u , τ 1 ) , ..., f ij,k ( u , τ n )) (cid:62) , τ 1 , ..., τ n ∼ Ω , and where j ∈ { 1 , 2 } .
Linearization of the λ -coefﬁcients given by Equation 9 is crucial to obtain linearization of the hybrid estimators, and consequently random feature map decomposition.
We denote a hybrid estimator using n random features for its base estimators and m to approximate λ -coefﬁcients as (cid:100) SM hyb m,n .
Furthermore, for two vectors u , v , we denote their vectorized outerproduct by u ⊗ v .
Finally, we denote by (cid:81) (cid:63) vectors-concatenation.
Estimator (cid:100) SM hyb m,n ( x , y ) can be rewritten as a dot-product of two (hybrid) random feature vectors, as the next lemma shows.
Lemma 2.3.
The HRF estimator (cid:100) SM hyb m,n ( x , y ) satisﬁes (cid:100) SM hyb m,n ( x , y )
=
Ψ 1 ( x ) (cid:62) Ψ 2 ( y ) , where Ψ j for j ∈ { 1 , 2 } is given as Ψ j ( z )
=
Ψ 1 j ( z ) (cid:63) Ψ 2 j ( z ) (cid:63) Ψ 3 j ( z ) (cid:63) Ψ 4 j ( z ) and:
[Equation]
Ψ
[Equation]
Bipolar estimators: A prominent special case of the general hybrid estimator deﬁned above is the one where: E =
( (cid:100) SM
++ ( x , y ) , (cid:100)
SM trig ( x , y )) .
Thus consider the following estimator (cid:100) SM hyb m,n :
[Equation]
The question arises whether (cid:100) SM hyb m,n deﬁned in such a way can outperform both (cid:100) SM ++ m and (cid:100)
SM trig m .
That of course depends also on the choice of λ : R d × R d → R .
If we consider a (common) normalized setting where all input vectors have the same norm, we can rewrite λ ( x , y ) ad
λ ( θ x , y , r ) , where θ x , y is an angle between x and y and (cid:107) x (cid:107) =
(cid:107) y (cid:107) =
r .
By our previous analysis we know that (cid:100) SM ++ m becomes perfect for θ = π and (cid:100)
SM trig m becomes perfect for θ = 0 .
That suggests particularly simple linear dependence of λ on θ to guarantee vanishing variance for both the critical values: θ = 0 and θ = π .
It remains to show that such a λ -coefﬁcient can be linearized.
It turns out that this can be done with a particularly simple random feature map mechanism, as we will show later, leading to the so-called angular hybrid variant (see: Section 2.4).
2. C HOOSING BASE ESTIMATORS FOR HRF S : C OMPLEX E XPONENTIAL E STIMATORS
In this section, we explain how we choose base estimators in Equation 8.
We denote by i a complex number such that i 2 = − 1 .
The following lemma is a gateway to construct unbiased base estimators.
Lemma 2.4.
Let ω ∼ N (0 , I d ) .
Then for every z = ( z 1 , ..., z d ) (cid:62) ∈ C d the following holds:
[Equation]
For a complex vector z = ( z 1 , ..., z d ) ∈ C d , we denote z 2 = (cid:80) di
=1 z 2 i .
Consider z
= Ax + ( A (cid:62) )
− 1 y for an invertible (in C d × d ) matrix A ∈ C d × d .
Using Lemma 2.4, we get:
[Equation]
SM
[Equation] [Equation]
Thus for Ψ m M ( u ) def =
[Equation]
)(exp
[Equation]
) , ..., exp
[Equation]
Mu )) (cid:62) and ω i ∼ N (0 , I d ) :
SM
[Equation]
x
[Equation] [Equation]
[Equation] [Equation]
[Equation]
We obtain the new random feature map mechanism providing unbiased estimation of the softmax kernel.
In general it is asymmetric since it applies different parameterization for x and y , i.e. one using A , the other one ( A (cid:62) ) − 1 .
Asymmetric random features is a simple generalization of the model using the same φ for both x and y , presented earlier.
The resulting estimators (cid:100) SM cexp m ( x , y ) , that we call complex exponential (CE) will serve as base estimators, and are deﬁned as:
[Equation]
[Equation]
For A = iI d , CE-estimator becomes trigonometric and for A
= I d , it becomes (cid:100) SM
++ m ( x , y ) .
Note also that an estimator based on this mechanism has variance equal to zero if
[Equation]
In fact we can relax that condition.
It is easy to check that for the variance to be zero, we only need:
[Equation]
and
[Equation]
2. A DAPTING HRF ESTIMATORS : HOW TO INSTANTIATE HYBRID RANDOM FEATURES
We will use complex exponential estimators from Section 2.3 as base estimators in different insantiations of HRF estimators (that by deﬁnition admit random feature map decomposition).
The key to the linearization of the HRF estimators is then Equation 9 which implies linearization of the shifted versions of λ -coefﬁcients (Equation 10) and consequently - desired random feature map decomposition via the mechanism of compositional kernels (details in the Appendix).
The questions remains how to construct λ -coefﬁcients to reduce variance of the estimation in the desired regions of interest.
Accurate approximation of small and large softmax kernel values simultaneously: As we explained earlier, in several applications of softmax estimators a desired property is to provide accurate estimation in two antipodal regions - small and large softmax kernel values.
This can be done in particular by choosing p = 1 , (cid:100)
SM 1 = (cid:100) SM
++ , (cid:100)
SM 2 = (cid:100)
SM trig and λ ( x , y )
=
θ x , y π , where θ x , y stands for an angle between x and y .
The key observation is that λ ( x , y ) can be unbiasedly approximated via the mechanism of sgn random features (following from Goemans-Willimson algorithm (Goemans & Williamson, 2004)) as follows (details in the Appendix, Sec.
D):
[Equation]
(19) where ρ n ( z ) def = i √ 2 n (sgn( τ (cid:62) 1 z ) , ..., sgn( τ (cid:62) n z ))
(cid:62) for τ 1 , ..., τ n ∼ N (0 , I d ) and i 2 = − 1 .
We call the resulting estimator of the softmax kernel angular hybrid .
As we show in Sec. 3, this estimator is indeed particularly accurate in approximating both small and large softmax kernel values.
For instance, for the prominent case when input vectors are of ﬁxed length, its variance is zero for both the smallest ( θ x , y = π ) and largest ( θ x , y = 0 ) softmax kernel values (see: Fig. 1).
Gaussian Lambda Coefﬁcients: Another tempting option is to instantiate λ -coefﬁcients with approximate Gaussian kernel values (estimated either via positive or trigonometric random features).
In that setting, functions (cid:98) λ k ( x , y ) are deﬁned as (cid:98) λ k
( x , y )
= exp
[Equation]
x + M
[Equation]
(cid:107) 22 2 c 2 k ) , for some
M 1 , ..., M p ∈ R d × d and c 1 , ..., c p ∈ R .
Since this time λ -coefﬁcients are the values of the Gaussian kernel between vectors x c k and − M k y c k , we can take a k = 0 and deﬁne γ k , µ k as:
[Equation]
where φ SM is the random feature map corresponding to a particular estimator of the softmax kernel.
Note that the resulting hybrid estimator, that we call Gaussian hybrid has nice pseudo-combinatorial properties.
The coefﬁcients λ k ∈ (0 , 1] ﬁre if x ≈
− M k y
(the ﬁring window can be accurately controlled via hyperparameters c k ).
Thus matrices M k can be coupled with the base estimators particularly accurate in the regions, where x ≈ − M k y .
The mechanism can be thus trivially applied to the setting of accurate approximation of both small and large softmax kernel values for inputs of ﬁxed length, yet it turns out to be characterized by the larger variance than its angular hybrid counterpart and cannot provide variance vanishing property for both θ x , y =
and θ x , y = π .
[Figure]
MSE
[Figure]
Adaptation to data admitting clustering structure: Assume now that the inputs x come from a distribution P ( X ) admitting certain clustering structure with clusters X 1 , ..., X u ⊆ R d of centers x 1 , ..., x u ∈ R d and that the analogous fact is true for y with the corresponding clusters Y 1 , ..., Y w ⊆ R d and centers y 1 , ..., y w ∈ R d .
For a ﬁxed pair of clusters’ centers ( x i , y j ) , one can associate a complex exponential estimator (cid:100) SM cexp ( x i , y j ) with the corresponding matrix A ∈ C d × d (see: Sec. 2.3) that minimizes the following loss, where | z | for z ∈ C d is deﬁned as | z | = (cid:113)(cid:80) di
=1 | z i | 2 :
[Equation]
Note that, by Equation 17, if one can ﬁnd A such that l ( A ) = 0 , then (cid:100) SM cexp ( x i , y j ) makes no error in approximating softmax kernel value on the centers x i , y j .
Furthermore, if sampled points are close to the centers and l ( A ) is small, the corresponding variance of the estimator will also be small.
Thus one can think about (cid:100) SM cexp ( x i , y j ) as an estimator adapted to a pair of clusters ( x i , y j ) .
We can add additional constraints regarding A , i.e. A ∈ R d × d or A being diagonal (or both) for closedform formulae of the optimal A , yet not necessarily zeroing out l ( A ) (details in the Appendix, Sec.
H).
Complex exponential estimators deﬁned in this way can be combined together, becoming base estimators in the HRF estimator.
The corresponding λ -coefﬁcients can be chosen to be Gaussian as described in the previous paragraph (with optimized matrices M k so that a coefﬁcient ﬁres for the corresponding pair of clusters’ centers), but there is another much simpler choice of λ s. We index different base estimators by pairs of clusters’ centers ( x i , y j ) .
One can simply take γ ( x i , y j ) ( x ) =if x belongs to the cluster of center x i and γ ( x i , y j ) ( x ) = 0 otherwise.
Vectors η ( x i , y j ) ( y ) (that also become scalars in this scenario) are deﬁned analogously.
Thus effectively the hybrid estimator activates precisely this random feature mechanism that is optimal for the particular pair of clusters.
This scheme is feasible if cluster-membership calculation is computationally acceptable, otherwise aforementioned Gaussian coefﬁcients can be applied.
We conclude with an observation that in principle (to reduce the number of base estimators if needed) one can also construct HRF estimators that take base estimators deﬁned in the above way only for the most massive pairs of clusters and add an additional default base estimators (for instance (cid:100) SM ++ ).
T HEORETICAL GUARANTEES
We provide here theoretical guarantees regarding HRF estimators.
We focus on the bipolar setting with two base estimators: (cid:100) SM trig and (cid:100) SM ++ .
The following is true:
Theorem 3.1 (MSE of the bipolar hybrid estimator) .
Take the bipolar hybrid estimator (cid:100) SM hyb m,n ( x , y ) , where (cid:100)
SM trig m ( x , y ) and (cid:100) SM ++ m ( x , y ) are chosen independently i.e. their random projections are chosen independently (note that we always assume that (cid:98) λ n ( x , y ) is constructed independently from (cid:100) SM trig m ( x , y ) and (cid:100) SM
++ m ( x , y ) ).
Then the following holds:
[Equation]
Furthermore, if (cid:100) SM trig m ( x , y ) and (cid:100) SM ++ m ( x , y ) apply the exact same sets of random projections, the mean squared error of the hybrid estimator is further reduced, namely we have:
[Equation]
The exact formula on MSE( (cid:100) SM ++ m ( x , y )) and MSE( (cid:100)
SM trig m ( x , y )) is given in Lemma 2.2.
We can apply this result directly to the angular hybrid estimator, since:
Lemma 3.2.
For the angular hybrid estimator the following holds:
[Equation]
[Equation]
We see that, as mentioned before, the variance of the angular hybrid estimator is zero for both θ x , y =
and θ x , y = π if inputs x , y have the same length.
We need one more deﬁnition.
Deﬁnition 3.3.
Assume that the inputs to the estimators are taken from some given bounded set
C ⊆ R d .
For a given estimator (cid:100) SM on feature vectors x , y ∈ C , we deﬁne its max-relative-error with respect to C as (cid:15) C ( (cid:100) SM) =
max x , y ∈C (cid:15) x , y ( (cid:100) SM) , where (cid:15) x , y ( (cid:100) SM) =
√ MSE( (cid:100) SM( x , y ))
SM( x , y ) .
Denote by S ( r ) a sphere centered at 0 and of radius r. Deﬁne (cid:15) θ,r ( (cid:100) SM) def =
(cid:15) x , y ( (cid:100) SM) for x , y ∈ S ( r ) and such that θ = θ x , y (note that the mean squared errors of the considered estimators depend only on the angle θ x , y for x , y chosen from a ﬁxed sphere).
This deﬁnition captures the critical observation that in several applications of the softmax kernel estimation, e.g. efﬁcient softmax sampling or linear-attention Transformers (Choromanski et al., 2021b), small relative errors are a much more meaningful measure of the quality of the method than small absolute errors.
It also enables us to ﬁnd hidden symmetries between different estimators:
Lemma 3.4.
The following holds:
[Equation]
and consequently for W ( r ) = exp
[Equation]
1 −
[Equation] [Equation]
Our main result shows that HRF estimators can be applied to reduce the max-relative-error of the previously applied estimators of the softmax kernel.
In the next theorem we show that the maxrelative-error of the angular hybrid estimator scales as 1 r exp(2 r 2 ) in the length r of its inputs as opposed to exp(2 r 2 ) as it is the case for (cid:100) SM trig and (cid:100) SM
++ (see: Lemma 3.4).
Furthermore, the max-relative-error scales as √ θ and √ π − θ as θ → 0 and θ → π respectively, in particular goes to 0 in both critical cases.
This is not true for (cid:100) SM trig nor for (cid:100) SM ++ .
Theorem 3.5.
The max-relative-error of the angular hybrid estimator for the inputs x , y on the sphere S ( r ) of radius r ≥ 1 satisﬁes for W ( r )
= exp(2 r 2 ) (cid:0) 1 − exp( − 4 r 2 ) (cid:1) :
[Equation]
Furthermore, lim
[Equation]
Additional implications of Theorem 3.5, using the fact that Θ( mn ) -dimensional HRFs can be constructed in time O ( nd + md + mn ) (regular RFs need O ( mnd ) ), are in the Appendix (Sec. D.4).
E XPERIMENTS
In this section we conduct exhaustive evaluation of the mechanism of HRFs on several tasks.
4. P OINTWISE S OFTMAX K ERNEL E STIMATION
We start with ablation studies regarding empirical relative errors of different softmax kernel estimators across different inputs’ lengths r and angles θ x , y .
The results are presented in Fig.
Ablations over more lengths are presented in the Appendix (Sec. G).
The angular hybrid estimator most accurately approximates softmax kernel and has smaller max-relative-error than other estimators.
Comparison with QMC-methods: Even though, as we explained in Section 1, our algorithm does not compete with various methods for variance reduction based on different sampling techniques (since those methods, as orthogonal to ours, can be easily incorporated into our framework), we decided to compare HRFs also with them.
We tested angular hybrid HRF variant as well as wellestablished QMC baselines: orthogonal random features (ORF)(Yu et al., 2016)
(regular variant
ORF-reg as well as the one applying Hadamard matrices ORF-Had) and QMCs based on random
Halton sequences (Halton-R)(Avron et al., 2016).
For HRFs, we tested two sub-variants: with orthogonal (HRF-ort) and regular iid random features (HRF-iid).
We computed empirical MSEs by averaging over 100 randomly sampled pairs of vectors from two UCI datasets: wine and Boston .
HRFs outperform all other methods, as we see in Table 1.
[Table]
4. L ANGUAGE M ODELING
In this section we apply HRFs to the language modeling task training a 2 -layer LSTM with hidden size h = 200 and applying RFs for softmax sampling in training as described in (Rawat et al., 2019).
Experimental results are obtained over 10 runs.
Experimental details and additional validation results are in the Appendix (Sec. I, Table 4).
Statistical Metrics.
We compute 1 d -Wasserstein distance and the Kolmogorov-Smirnov (KS) metric (Ramdas et al., 2017) between a true softmax distribution induced by a query on all the classes and the approximate one given by different RF-mechanisms.
We also compute the fraction of all probability estimates with undesired negative values.
The results on the PennTree Bank Marcus et al.
(1993) are presented in Fig.
In the Appendix (Sec. I) we present additional results for the
WikiText2 dataset.
We see that HRFs lead to most accurate softmax distribution estimation.
4. T RAINING S PEECH M ODELS WITH HRF-C ONFORMERS -P ERFORMERS
We also tested HRFs on speech models with LibriSpeech ASR corpus (Panayotov et al., 2015).
We put implicit Performer’s attention into 17 -layer Conformer-Transducer encoder (Gulati et al., 2020) and compared softmax kernel estimators in terms of word error rate (WER) metric, commonly used to evaluate speech models.
We tested HRFs using angular hybrid variant as well as those clustering-based.
For the latter ones, the clusters were created according to the K -means algorithm after ﬁrst 1000 steps of training (and were frozen afterwards) and corresponding matrices A were constructed to minimize the loss given in Equation 21.
The results are presented in Table 2.
HRFs produce smallest WER models and the clustering-based variants fully exercising the most general formulation on HRFs (with general complex estimators) turn out to be the best.
Additional details regarding speech experiments as well as additional experiments with clustering-based HRFs are presented in the Appendix (Sec.
J and Sec.
H.1 respectively).
[Table]
4. D OWNSTREAM R OBOTICS E XPERIMENTS
In Robotics, we leverage the accuracy of HRFs to obtain inference improvements, critical for onrobot deployment.
To abstract from a particular hardware characteristic, we measure it in the number of FLOPS.
We conduct two experiments targeting: (a) quadruped locomotion and (b) robotic-arm manipulation.
In the former, HRFs are applied as a replacement of the default mechanism using positive random features in the class of implicit-attention architectures for vision processing called
Implicit Attention Policies (or IAPs) (Choromanski et al., 2021a).
In the latter, HRFs become a part of the regular Vision-Performer stack used to process high-resolution (500 x 500 x 3) input.
The results are presented in Fig.
[Table]
where HRFs need 3x fewer FLOPS to obtain same quality policies as baselines.
All details regarding both experimental setups are given in the Appendix (Sec. K).
Videos of the HRF-trained robotic policies are included in the supplementary material.
conclusions) but for the robotic-arm manipulation task.
The additional ﬁve regular RF-conﬁgurations did not train by producing Nan loss due to large variance of the underlying softmax kernel estimators
C ONCLUSION
We presented a new class of random feature techniques, called hybrid random features for softmax/Gaussian kernel estimation that more accurately approximate these kernels than previous algorithms.
We also demonstrated their robustness in a wide range of applications from softmax sampling to Transformers/attention training, also for downstream Robotics tasks.
Reproducibility Statement: Section 4 and the Appendix include all experimental details (e.g. hyperparameter setup) needed to reproduce all the results presented in the paper.
The part of the code that we could make publicly available can be found in the following anonymous github:
https: //anonymous.4open.science/r/HRF_ICLR2022-B69C/ .
Ethics Statement: HRFs can be used in principle to train massive Transformer models with large number of parameters and proportional compute resources.
Thus they should be used responsibly, in particular given CO 2 emission challenges that scientiﬁc community tries to address now
APPENDIX: H YBRID R ANDOM F EATURES A P ROOF OF L EMMA 2.
Proof.
From the independence of ω 1 , ..., ω d , we get:
[Equation]
Thus it sufﬁces to show that for g ∼ N (0 , 1) and any z ∈ C the following holds:
[Equation]
Deﬁne f ( z ) =
E [exp( gz )] .
Note that f ( z ) =
exp( z 2 2 ) for z
= i x , where i 2 = − 1 and x ∈ R which follows from the formula of the characteristic function of the Gaussian distribution.
Similarly, f ( z ) =
exp( z 2 2 ) for z ∈ R which follows from the formula of the moment generating function of the Gaussian distribution.
We now use the fact from complex analysis that if two analytic functions C → C are identical on uncountably many points then they are equal on C to complete the proof (we leave to the reader checking that both f ( z ) and g ( z ) def =
exp( z 2 2 ) are analytic).
B P ROOF OF L EMMA 2.
Proof.
The following is true:
[Equation]
The statement of the lemma follows directly from that.
Note that the key trick is an observation that a product of two functions of x , y that can be linearized on expectation (i
[Equation]
e. rewritten with x and y disentangled) is itself a function that can be linearized on expectation.
In the setting where the former are approximated by random features, the latter is approximated by their cartesian product (the random feature mechanism for the compositional kernels from (Daniely et al., 2017)).
C B IPOLAR HRF E STIMATORS : P ROOF OF T HEOREM 3.
Proof.
The following holds:
[Equation]
We will focus now on the covariance term.
To simplify the notation, we will drop index n in (cid:98) λ n .
Assume ﬁrst easier to analyze case where (cid:100)
SM trig m ( x , y ) and (cid:100) SM ++ m ( x , y ) are chosen independently.
Then the following is true:
[Equation]
Now assume that (cid:100) SM trig m ( x , y ) and (cid:100) SM ++ m ( x , y ) use the exact same random projections.
Then, using similar analysis as before
[Equation]
we get:
[Equation]
] E [
[Equation]
This time however it is no longer the case that E [ (cid:100) SM
++ m ( x , y ) (cid:100)
SM trig m ( x , y )]
E [ (cid:98) λ ( θ )(1 − (cid:98) λ ( θ ))]
=
E [ (cid:100) SM ++ m ( x , y )]
E [ (cid:100)
SM trig m ( x , y )]
= (SM( x , y )) 2 since (cid:100) SM
++ m ( x , y ) and (cid:100)
SM trig m ( x , y ) are no longer independent.
In order to compute E [ (cid:100) SM ++ m ( x , y ) (cid:100) SM trig m ( x , y )]
E [ (cid:98) λ ( θ )(1 − (cid:98) λ ( θ ))] , we will ﬁrst introduce useful denotation.
Denote by ω 1 , ..., ω m iid ∼ N (0 , I d ) the random projections sampled to construct both (cid:100) SM ++ m ( x , y ) and (cid:100)
SM trig m ( x , y ) .
Denote: Y i = cosh( ω (cid:62) i
( x + y ))
def = exp( ω (cid:62) i
( x + y
[Equation]
+exp( − ω (cid:62) i
( x + y )).
We have:
[Equation]
If we denote: Z i = cos( ω (cid:62) i ( x − y )) , then we can write (cid:100)
SM trig m ( x , y ) as:
[Equation]
We can then rewrite E [ (cid:100) SM
++ m ( x , y ) (cid:100)
SM trig m ( x , y )] as:
[Equation]
where ω ∼ N (0 , I d )
The equality follows from the unbiasedness of (cid:100) SM trig m ( x , y ) and (cid:100) SM ++ m ( x , y ) and the fact that different ω i are chosen independently.
Thus it remains to compute ρ = E [cosh( ω (cid:62) ( x + y ))
cos( ω (cid:62) ( x − y ))] .
Note ﬁrst that ρ = E [exp( ω (cid:62) ( x + y )) cos( ω (cid:62) ( x − y ))] since − ω ∼ N (0 , I d ) and cos is an even function.
Denote z = x
+ y
+ i ( x − y ) .
We have:
[Equation]
Re
[Equation] [Equation]
Thus we conclude that:
[Equation]
Therefore we get the formulae for the covariance term in both: the setting where random projections of (cid:100) SM trig m ( x , y ) and (cid:100) SM ++ m ( x , y ) are shared (variant II) and when they are not (variant I).
The following is true for Z = 1 m (1 − cos( (cid:107) x (cid:107) 22 − (cid:107) y (cid:107) 22 )
[Equation]
[Equation] [Equation]
[Equation]
We also have the following:
[Equation]
and furthermore (by the analogous analysis):
[Equation]
By putting the derived formula for the above variance terms as well as covariance terms back in the
Equation 31, we complete the proof of the theorem (note that the mean squared error of the hybrid estimator is its variance since it is unbiased).
D A NGULAR H YBRID E STIMATORS D. P ROOF OF L EMMA 3.
Proof.
The formula for the expectation is directly implied by the formula (1) from Sec.
2.1 of Cho &
Saul (2012) for the zeroth-order arc-cosine kernel k 0 ( x , y ) def = 1 − θ x , y π .
It also follows directly from
Goemans-Williamson algorithm (Goemans & Williamson, 2004).
We will now derive the formula for c =
E [ (cid:98) λ
2 ( x , y )] .
Since λ depends only on the angle θ = θ x , y , we will refer to it as: λ ( θ ) and to its estimator as (cid:98)
λ ( θ ) .
Denote:
[Equation]
Denote: X i = ( φ ang n ( x ))
[ i ] φ ang n ( y )
[ i ] .
We have:
[Equation]
Note ﬁrst that by the construction of (cid:98) λ
( θ ) , we have: E [ (cid:98) λ ( θ )]
= θπ and thus: E [ (cid:80) ni
=1 X i ] = 1 − 2 θπ .
Therefore we conclude that:
[Equation]
D. E XPLICIT FORMULA FOR THE MSE OF THE ANGULAR HYBRID ESTIMATOR
Theorem 3.1 and Lemma 3.2 immediately imply the following result.
Theorem D.1 (MSE of the angular hybrid estimator) .
Take the angular hybrid estimator (cid:100)
SM anghyb m,n ( x , y ) , where (cid:100)
SM trig m ( x , y ) and (cid:100) SM ++ m ( x , y ) are chosen independently i.e. their random projections are chosen independently (note that we always assume that (cid:98) λ ( x , y ) is chosen independently from (cid:100) SM trig m ( x , y ) and (cid:100) SM
++ m ( x , y ) ).
Then the following holds:
[Equation]
for ∆ = x − y and z = x + y .
Furthermore, if (cid:100) SM trig m ( x , y ) and (cid:100) SM ++ m ( x , y ) apply the exact same sets of random projections, the mean squared error of the hybrid estimator is further reduced, namely we have:
[Equation]
Thus if (cid:107) x (cid:107) 2 = (cid:107) y (cid:107) 2 =
r then regardless of whether the same sets of random projections are applied or not
[Equation]
we get:
[Equation]
D. P ROOF OF T HEOREM 3.
Proof.
Note ﬁrst that from the derived above formula of the MSE of the lambda-angular bipolar hybrid estimator and the deﬁnition of the max-relative-error, we obtain:
[Equation]
where:
[Equation]
and a r ( θ ) is deﬁned as:
[Equation]
Therefore we have:
[Equation]
Notice that:
[Equation]
where:
[Equation]
and
[Equation]
Therefore:
[Equation] [Equation]
Denote: b 1 = max θ ∈ [0 ,π ] b 1 r ( θ ) and b 2 = max θ ∈ [0 ,π ] b 2 r ( θ ) .
Note that:
[Equation] [Equation]
and
[Equation] [Equation]
Thus, from the properties of function: θ → θ sin( θ ) and the fact that r ≥ 1 , we conclude that both derivatives are ﬁrst non-negative, then non-positive and then non-negative and that the unique local maximum on the interval [0 , π ] is achieved for θ ≤ π 2 .
Note also that b 1 r ( θ ) , b 1 r ( θ ) ≥ 0 and b 1 r (0) =
b 2 r (0) = 0 , b 1 r ( π ) =
(1 − 1 n ) exp( − 2 r 2 ) , b 2 r ( π ) = 1 n exp( − 2 r 2 ) .
We conclude that global maximum for b ir on the interval [0 , π ] for i = 1 , 2 is achieved either in its unique local maximum on that interval or for θ
= π .
Let us consider ﬁrst:
b 1 r .
In its local maximum on [0 , π ] we have:
[Equation]
Since θ ≤ sin( θ ) · π 2 on [0 , π 2 ] , we get:
[Equation]
i.e.:
[Equation]
Therefore:
[Equation] [Equation]
We thus conclude that:
[Equation]
By the completely analogous analysis applied to b 2 r , we obtain:
[Equation]
Now, using Equation 51, Equation 52, and Equation 56, we obtain:
[Equation]
and that completes the ﬁrst part of the proof (proof of Inequality 27).
The equations on the limits are directly implied by the fact that:
[Equation]
D. D ISCUSSION OF T HEOREM 3.
Theorem 3.5 leads to yet another important conclusion not discussed in the main body of the paper.
Note that asymptotically as θ → 0 or θ → π , the relative error decreases (as a function of m and n ) as 1 √ mn =
Θ( 1 d est ) , where d est stands for the number of random features of the resulting estimator.
Note also that for the regular estimator the rate of decrease is also Θ( 1 d est ) (here we treat all other arguments of the formula for the relative error as constants; we already know that the dependence on them of the relative error for the HRF estimators is superior to the one for regular estimators).
The key difference from the computational point of view is that for the HRF estimator, the random feature map can be constructed in time O ( nd + md + mn ) , whereas for the regular estimator it requires time O ( mnd ) (see: Sec.
L).
Thus for the regular estimator random feature map computation is much more expensive.
An important application, where random feature map computations takes substantial time of the overall compute time are implicit-attention Transformers, such as Performers (Choromanski et al., 2021b).
This is also the setting, where an overwhelming fraction of the approximate softmax kernel values will be extreme (very small or large) since an overwhelming fraction of the attention matrix entries after some training will have extreme values (very small or large).
In the setting, where the lengths of queries and keys do not vary much (for instance a prominent case where they all have the same ﬁxed length) that corresponds to angle values θ → π or θ → 0 .
This is exactly the scenario from the second part of Theorem 3.5.
E P ROOF OF L EMMA 3.
Proof.
Equation 25 follows directly from Lemma 2.2.
Notice that the relative error (cid:15) θ,r ( (cid:100) SM trig m ) is an increasing function of sin 2 ( θ 2 ) and thus is largest for θ = π .
Plugging in this value into the formula of the relative error gives us the expression from the statement of the lemma.
Completely analogous analysis holds for (cid:100) SM ++ m .
F G AUSSIAN H YBRID E STIMATORS
In this section we will provide more results regarding Gaussian hybrid estimators.
We will focus on the bipolar scenario and within this scenario on the so-called normalized variant described below.
If x and y have ﬁxed L 2 -norm ( (cid:107) x (cid:107) 2 = (cid:107) y (cid:107) 2 = r ), we can propose a normalized bipolar Gaussian hybrid estimator such that λ ( x , y )
=
if θ x , y = 0 and λ ( x , y )
=
if θ x , y = π .
Furthermore the variance of that estimator, that we call shortly (cid:100) SM gausshyb m,n , will be zeroed out for θ x , y = 0 or θ x , y = π , but not for both.
The coefﬁcient-function λ : R d × R d → R is deﬁned as:
[Equation]
where ρ is given as:
[Equation]
The hyperparameter σ controls the smoothness of the Gaussian kernel.
The exponential in λ can be estimated either by trigonometric or positive random features.
It is easy to see that for the ﬁxed input lengths, in the former case the variance of the estimator is zero for θ x , y = 0 and in the latter it is zero for θ x , y = π , but the variance never zeroes out for both θ x , y =
and θ x , y = π .
In principle, one can derive entire hierarchy of HRF estimators, where the coefﬁcients in an HRF estimator from one level of hierarchy are estimated with the use of HRF estimators from the higher level.
In that context, angular hybrid estimator can be applied to provide approximation of the coefﬁcients of the normalized bipolar Gaussian hybrid estimator leading to variance zeroing out for both θ x , y =
and θ x , y = π .
However such nested constructions are not the topic of this paper.
From now on we will assume that trigonometric random features are applied to estimate λ -coefﬁcient,
Therefore we have:
[Equation]
and that leads to the following choice of: a, γ, µ from Equation 10:
[Equation]
where τ 1 , ..., τ n ∼ N (0 , I d ) and i 2 = − 1 .
Using Theorem 3.1, we conclude that:
Theorem F.1 (MSE of the normalized bipolar Gaussian estimator) .
Take the normalized bipolar
Gaussian estimator (cid:100) SM gausshyb m,n ( x , y ) , where (cid:100) SM trig m ( x , y ) and (cid:100) SM ++ m ( x , y ) are chosen independently i.e. their random projections are chosen independently (note that we always assume that (cid:98) λ ( x , y ) is chosen independently from (cid:100) SM trig m ( x , y ) and (cid:100) SM
++ m ( x , y ) ).
Denote: ∆ = x − y .
Then the following holds:
[Equation]
where
[Equation]
Furthermore, if (cid:100) SM trig m ( x , y ) and (cid:100) SM ++ m ( x , y ) apply the exact same sets of random projections, the mean squared error of the hybrid estimator remains the same.
Proof.
We can calculate E [( (cid:98) λ
( x , y )) 2 ] and E [(1 − (cid:98) λ ( x , y )) 2 ] as follows:
[Equation]
By plugging in these two formulae in the expression for MSE, we obtain the ﬁrst half of the theorem.
Note that if (cid:100) SM trig m ( θ, r ) and (cid:100) SM ++ m ( θ, r ) apply the exact same sets of random projections, then (1 − cos( (cid:107) x (cid:107) 22 − (cid:107) y (cid:107) 22 )) in MSE( (cid:100)
SM gausshyb m,n ( x , y )) becomes zero in Theorem F.1, therefore the
MSE remains the same.
We thus obtain the second part of the theorem and that completes the proof.
Now let us assume the setting of the same-lengths inputs.
We denote the length by r and an angle between inputs by θ .
We can rewrite Eq.
and provide equivalent deﬁnition for λ by replacing x and y with their angle θ and norm r :
[Equation]
We obtain the following version of the above theorem:
Theorem F.2 (MSE of the normalized bipolar Gaussian hybrid estimator for same-length inputs) .
Assume that (cid:107) x (cid:107) 2 = (cid:107) y (cid:107) 2 = r and denote: θ = θ x , y .
Take the normalized bipolar Gaussian hybrid estimator (cid:100)
SM gausshyb m,n ( θ, r ) , where (cid:100)
SM trig m ( θ, r ) and (cid:100) SM ++ m ( θ, r ) are chosen independently i.e. their random projections are chosen independently (note that we always assume that (cid:98) λ
( θ, r ) is chosen independently from (cid:100) SM trig m ( θ, r ) and (cid:100) SM ++ m ( θ, r ) ).
Then the following holds:
[Equation]
where
[Equation]
Furthermore, if (cid:100) SM trig m ( θ, r ) and (cid:100) SM ++ m ( θ, r ) apply the exact same sets of random projections, the mean squared error of the hybrid estimator will remains the same.
Proof.
For (cid:107) x (cid:107) 2 = (cid:107) y (cid:107) 2 = r 2 , the following holds for ∆ = x − y :
[Equation]
And this theorem holds trivially by replacing (cid:107) ∆ (cid:107) 2 with the above formula in Theorem F.1.
G P OINTWISE S OFTMAX K ERNEL E STIMATION E XPERIMENTS
In Fig.
5 we present complete ablation studies regarding empirical relative errors of different RFbased estimators of the softmax kernel over different lengths of inputs r and angles θ x , y .
H C OMPLEX EXPONENTIAL ESTIMATORS FOR CLUSTERED DATA
H.
S YNTHETIC EXPERIMENTS ON DATA ADMITTING CLUSTERING STRUCTURE
Here we test HRF estimators customized to data with clustering structure, as described in Sec.
Inputs x are taken from two 50 -dimensional 1000 -element Gaussian clusters and the inputs y from two other 50 -dimensional 1000 -element Gaussian clusters.
We use empirical MSE metric and constrain matrix A to be real diagonal for a compact closed-form formulae (see: Appendix, Sec.
H).
We created four different clusters’ conﬁgurations corresponding to different values of s = l ( A ) (see: Eq. 21) for all pairs of clusters and with different concentrations around centers controlled by the standard deviation σ (small/large values of s and σ ).
As we show in Fig.
6, for all variants HRF estimators adapted to clustered data consistently provide notable accuracy gains, even for larger values of s and less concentrated clusters.
Additional experimental details are given in Appendix H.
H. I NSTANTIATING COMPLEX EXPONENTIAL ESTIMATORS FOR CLUSTERED DATA
Assume that the inputs x (queries) can be modeled by n q clusters and that the inputs y (keys) can be modeled by n k clusters (e g. via k-means clustering algorithm).
Denote the center of each cluster as r i ∈ R d , ( i = 1 , ..., n k ) and r j ∈ R d , ( j = 1 , ..., n q ).
Then there exist n q n k pairs of ( r i , r j ) , ( i = 1 , ..., n q , j = 1 , ..., n k ), so we can construct n q n k softmax kernel estimators to estimate crossgroup softmax kernel values.
Consider z
=
Ar i + ( A (cid:62) ) − 1 r j for an invertible (in C d × d ) matrix A ∈ C d × d .
An estimator based on this mechanism has variance equal to 0 if:
[Equation]
From now on we constrain A to be diagonal, so
A =
A (cid:62) .
We can rewrite the above equation as:
[Equation]
Since position ( k, k ) in matrix A is a complex number of the form α k + β k i , we need to satisfy the following equation for each k = 1 , ..., d :
[Equation]
where r i,k is the k -th entry of vector r i .
We can simplify this equation and separate into real and imaginary part:
[Equation]
Our goal now is to choose values for α k and β k given r i and r j .
If r i,k r j,k > 0 , we can set:
[Equation]
And if r i,k r j,k < 0 , we can set:
[Equation]
When r i,k = r j,k = 0 , we can take α k = β k = 1 .
If r i,k = 0 , r j,k (cid:54) = 0 or the opposite, then we cannot satisfy the above equation perfectly.
We can take α k to some large positive value and set β k = 0 when r i,k = 0 , r j,k (cid:54) = 0 , and set α k = β k to some small positive value close to zero when r i,k (cid:54) = 0 , r j,k = 0 .
When restricting A = diag( a 1 , . .
. , a d ) to R d × d , it is easily seen that to minimize | Ar i + ( A (cid:62) ) − 1 r j | 2 , we can take
[Equation]
88) if r i,k (cid:54) = 0 .
And if r i,k = 0 , r j,k (cid:54) = 0 , a k can be set to a large positive number.
We can set a k = 1 if r i,k = 0 , r j,k = 0 .
In the analysis below we denote matrix
A calculated from Eq.
85 and Eq. 88 (depending on whether we are using real or complex matrices) given r i and r j as A i,j .
From the analysis in the main body of the paper we know that for arbitrary vectors x , y ∈ R d :
[Equation]
Therefore, the softmax kernel can be estimated as:
[Equation]
with Ψ m A i,j ( x ) and Ψ m ( A i,j ) − 1 ( y ) given by:
[Equation]
for ω 1 , ..., ω m ∼ N (0 , I d ) .
Such a softmax kernel estimator has variance equal to zero if x = r i and y =
r j if we are using the complex matrix and has small variance if we are using the real matrix.
For other pairs of vectors ( x , y ) close to r i and r j , the variance is close to zero, or relatively small.
We can also take the Gaussian λ -coefﬁcient estimator that gets it maximum value when x = r
i , y =
r j as λ i,j ( x , y ) .
Such an estimator could be constructed with A i,j in a similar way.
It can be easily linearized since:
[Equation]
where SM( x /τ, − y /τ ) could be estimated with similar random feature maps as given by Eq. 90.
Therefore
[Equation]
the Gaussian λ -coefﬁcients can be estimated as:
[Equation]
with Ψ τ,m A i,j ( x ) and Ψ τ
[Equation]
m ( A i,j ) − 1 ( − y ) given by:
[Equation]
for ω 1 , ..., ω m ∼ N (0 , I d ) chosen independently and that were applied in base estimators.
We summarize this section with the following two constructions of the hybrid estimators for the clustered data that are implied by the above analysis.
Deﬁnition H.1 (Hybrid Gaussian-Mixtures estimators for clustered data) .
Assume that inputs x (queries) and y (keys) can be modeled by n q and n k clusters respectively with centers r i and r j ∈
R d , ( i = 1 , ..., n k , j = 1 , ..., n q ).
We denote A i,j as the complex matrix satisfying Eq.
(85) with center r i , r j .
Furthermore, we denote E =
( (cid:100) SM i,j ( x , y ))
n q ,n k i =1 ,j =1 as a list of estimators of the softmax kernel SM i,j ( x , y ) and Λ = (cid:98) λ i,j ( x , y ))
n q ,n k i =1 ,j =1 as a list of estimators of λ i,j ( x , y ) constructed independently from E .
We also use one additional base estimator (e.g. (cid:100)
SM trig ( x , y ) or (cid:100) SM ++ ( x , y ) ) and denote the estimator of its softmax kernel as (cid:100) SM 0 ( x , y ) and λ -coefﬁcient as (cid:98) λ 0 ( x , y ) .
Then our hybrid estimator takes the following form:
[Equation]
97) with constraint:
[Equation]
and where the estimators of λ -coefﬁcients and base estimators are given as:
[Equation]
for Ψ m A i,j ( x ) , Ψ m ( A i,j ) − 1 ( y ) , Ψ τ,m A i,j ( x ) , Ψ τ,m ( A i,j ) − 1 ( − y ) given by Eq.
Deﬁnition H.2 (Hybrid Zero-One-Mixtures estimators for clustered data) .
Assume that inputs x (queries) and y (keys) can be modeled by n q and n k clusters respectively with centers r i and r j ∈
R d , ( i = 1 , ..., n k , j = 1 , ..., n q ).
We denote A i,j as the complex matrix satisfying Eq.
(85) with center r i , r j .
Furthermore, we denote E =
( (cid:100) SM i,j ( x , y ))
n q ,n k i =1 ,j =1 as a list of estimators of the softmax kernel SM i,j ( x , y ) and Λ = (cid:98) λ i,j ( x , y ))
n q ,n k i =1 ,j =1 as a list of estimators of λ i,j ( x , y ) constructed independently from E .
Then our hybrid estimator takes the following form:
[Equation]
with constraint:
[Equation]
and where the estimators of λ -coefﬁcients and base estimators are given as:
[Equation]
for Ψ m A i,j ( x ) , Ψ m ( A i,j ) − 1 ( y ) given by Eq.
(91, 92), with Ψ i ( x ) being a scalar indicating whether x belongs to the i -th cluster of the n q clusters and similarly with Ψ j ( y ) being a scalar indicating whether y belongs to the j -th of the n k clusters.
In our experiments we used the formulation from the second deﬁnition with A constrained to be real diagonal.
In other words, we are using Eq. 88 to construct A .
H.
A DDITIONAL EXPERIMENTAL DETAILS
To create the synthetic data, we ﬁrst generate two random vectors X 0 , Y 0 in R 50 .
Then we generate four random orthogonal matrices
O i,j ∈ R 50 × 50 where i = 1 , 2 and j = 1 , 2 and use X i
=
O i, 1 Xand Y i
=
O i, 2 Y 0 for i = 1 , 2 as the mean vectors for our Gaussian clusters.
For each pair of clusters, the minimal value s of Eq.
21 may be non-zero if sign( x i,k )
=
sign( y j,k ) for some k due to the usage of real diagonal matrices.
To control the value of s , we manually adjust the sign of each element of X i and Y i .
The data for input x is the combination of two clusters, each consistingdata points following Gaussian distribution with mean vectors X 1 , X 2 and the common covariance matrix σ 2 I .
And similar constructions are done for input y .
We create four synthetic data sets, with different values of s and magnitudes of the standard deviation σ .
After this step, we normalize the data points by controlling their (cid:96) 2 norms to make the true softmax value be within a reasonable range.
After the data is created, we ﬁrst use K-means clustering algorithm with k = 2 to cluster x and y , and we obtain centers ( x i , y j ) i,j ∈{ 1 , 2 } .
Then we use the set of real diagonal matrices to minimize Eq.for all pairs of centers of x and y to generate four complex exponential base estimators.
To choose λ coefﬁcients, we use the more efﬁcient approach described above, i.e. using indicator vectors, and these are already computed in the clustering process.
For each data set, we compare the performance of these estimators by calculating the mean square error of them, and we do this multiple times with different numbers of random features used.
All the results are obtained by averaging over 20 repetitions.
In addition to the average performance which can be found in Fig. 6, we also record the maximal and minimal empirical MSE over 20 repetitions for all estimators in the Table 3, where s 1 = [0 . 08 , 0 . 05 , 0 . 04 , 0 . 03] , s 2 =
07] , s 3 =
02] , s 4 =
04] representing the lists of s values for all pairs of clusters for different synthetic data sets.
[Table]
[Figure]
[Table]
[Figure]
[Table]
I L ANGUAGE M ODELING T RAINING D ETAILS AND A DDITIONAL R ESULTS
For the Language Modeling tasks, we trained a 2-layer LSTM of hidden sizes 200 and 650 on the
PennTree Bank (Marcus et al., 1993) and the WikiText2 dataset (Merity et al., 2017) respectively.
We tied the weights of the word embedding layer and the decoder layer (Inan et al., 2017; Press &
Wolf, 2017).
Thus we could treat the language modeling problem as minimizing the cross entropy loss between the dot product of the model output (queries) and the class embedding (keys) obtained from the embedding layer with the target word.
We now present detailed statistics of HRFs on the Penn Tree Bank dataset.
The mean and the standard deviation of the statistical metric results in Fig.
3, Sec. 4.2 is shown in Table 4.
The distribution of the lengths of the keys and the queries are shown in Figure 7.
For our experiments with the Angular Hybrid and the Gaussian Hybrid, the number of random features are 8 , 16 , 32 ,for the base estimators
[Table] and 8 for the coefﬁcient estimators. [Table] Since the HRF mechanisms can be sensitive to the lengths of the keys and the queries, for the language modeling task on the WikiText2 dataset, we added a regularizer to constrain the lengths of the keys and the queries to be close to 1.
However, constraining the lengths close to 1 hurts the model performance and so we chose to use a temperature scaling as in (Rawat et al., 2019).
Thus before passing the computed dot product to the softmax layer, we scaled the dot product by τ > 1 .
Our ﬁnal loss function for the language modeling task was:
[Equation]
where ||·|| 2 is the row-wise L 2 norm of the appropriate matrices, and L CE is the cross-entropy loss.
The distribution of the lengths of the keys and the queries are shown in Figure 9.
Finally we present some additional results on the WikiText2 dataset.
For our experiments on the WikiText2 dataset, we chose λ 1 = λ
2 = 2 and τ = 6 .
Our model trained with these hyperparameters achieve a perplexity score of 105.35 on the test set after 50 epochs.
We compute the 1 -dimensional Wasserstein distance and the Kolmogrov-Smirnov (KS) metric (Ramdas et al., 2017) between the approximated softmax distribution and the true softmax distribution. The mean and the standard deviation of the statistical metric results in Fig.
8 is shown in Table 5. [Table]
The above tables (Table 4 and Table [Table] show that the our hybrid variants consistently outperform Favor+ and the trigonometric random features.
I. L ANGUAGE M ODELING USING HRF
In this subsection [Table] we will describe how one can use HRF to train a LSTM on the language modeling task on the PennTree Bank dataset, similar to the experiments carried out in (Rawat et al., 2019).
We trained a 2-layer LSTM model with hidden and output sizes of 200, and used the output as input embedding for sampled softmax.
We sampled 40 negative (other than true) classes out of 10000 classes to approximate the expected loss in each training epoch.
As observed in Fig.
2, the relative error of all three estimators grow exponentially fast with embedding norm r .
Therefore, if we keep un-normalized embeddings during sampling, then even though we could get an unbiased estimation of the loss, the variance could be high.
Such bias-variance trade off is also mentioned in paper (Rawat et al., 2019).
To solve this issue, we used normalized input and class embeddings during sampling to generate biased sampling distribution with lower variance, while keeping unnormalized embeddings to calculate the loss.
We trained our model for 80 epochs, with batch size equal to 20, dropout ratio in LSTM equal to 0.5.
Implementation details could be seen in our Github repository.
We compared our hybrid estimator with Trigonometric, FAVOR+, the uniform method which sampled 40 negative classes with equal probability, as well as the sampled softmax method which uses the unbiased true probability calculated from full softmax as the sampling probability.
Trigonometric and FAVOR+ mechanisms used 1024 random features.
To make comparison fair, hybrid variants were using ( m, n ) -conﬁgurations characterized by the similar number of FLOPS needed to construct their corresponding random features as regular RF-estimators.
We reported our comparison results averaged over 25 independent runs in Fig.
The perplexity score in Fig.
10 is deﬁned as 2 cross entropy loss .
We could conclude that even though the statistical metrics for HRFs are better in
Fig.
3, the difference between HRFs and other random feature estimators in this speciﬁc softmax sampling downstream task is not very signiﬁcant.
J S PEECH E XPERIMENTS : A DDITIONAL D ETAILS
Tested Conformer-Performer models consisted of l = 17 conformer layers.
Each attention layer used H = 8 heads.
The embedding dimensionality was p = 512 and since dimensions were split equally among different heads, query/key dimensionality was set up to d QK = 64 .
Each input sequence was of length L ≈ 500 .
Padding mechanism was applied for all tested variants.
K D OWNSTREAM R OBOTICS E XPERIMENTS
In both Robotics experiments we found query/key L 2 -normalization technique particularly effective (queries and keys of L 2 -norm equal to one) thus we applied it in all the experiments.
K. V ISUAL L OCOMOTION WITH Q UADRUPED R OBOTS
We evaluate HRF-based attention RL policies in a robotic task of learning locomotion on uneven terrains from vision input.
We use the quadruped from Unitree called Laikago (lai).
It has 12 actuated joints, 3 per leg.
The task is to walk forward on a randomized uneven terrain that requires careful foot placement planning based on visual feedback.
The ground is made of a series of stepstones with gaps in between.
The step stones widths are ﬁxed at 50 cm, the lengths are between [50 , 80] cm in length, and the gap size between adjacent stones are between [10 , 20] cm.
It perceives the ground through 2 depth cameras attached to its body, one on the front and other on the belly facing downwards.
We use Implicit Attention Policy (IAP) architecture (masking variant) described in Choromanski et al.
(2021a) which uses Performer-based attention mechanism to process 32 ×depth images from the 2 cameras.
We apply a hierarchical setup to solve this task.
The high level uses IAP-rank with masking to process camera images and output the desired foot placement position.
The low level employs a position-based swing leg controller, and an model predictive control (MPC) based stance leg controller, to achieve the foot placement decided by high level.
The policies are trained with evolutionary strategies (ES).
In Fig.
4 (left) we compare FAVOR+ with angular hybrid approximation in IAP.
HRF with 8 x 8 random projections ( m = n = 8 ) performs as well as the softmax kernel withrandom projections.
A series of image frames along the episode of a learned locomotion HRF-based
IAP policy is shown bottom right of Fig.
On the top-left corner of the images, the input camera images are attached.
The red part of the camera image is the area masked out by self-attention.
The policy learns to pay attention to the gaps in the scene in order to avoid stepping on them.
The training curves in Fig.
4 (left) are obtained by averaging over 5 top runs.
The shaded regions shows the standard deviation over multiple runs.
K. R OBOTIC M ANIPULATION : B I - MANUAL S WEEPING
Here we consider the bi-manual sweep robotic-arm manipulation.
The two-robotic-arm system needs to solve the task of placing red balls into green bowls (all distributions are equally rewarded, so in particular robot can just use one bowl).
In this bi-manual sweeping task, the reward per step is between 0 and 1, the fraction of the blocks within the green bowls.
The loss for this task is the negative log likelihood between the normalized softmax probability distribution of sampled actions (one positive example, and all others uniform negative counter-examples), and the ground truth one-hot probability distribution for a given observation from the training dataset.
For more details see (van den Oord et al., 2018).
This is a 12-DoF cartesian control problem, the state observation consists of the current cartesian pose of the arms and a 500 × 500 × 3 RGB image.
An example of the camera image is given in Fig.
The training curves Fig.
4 (right) are obtained by averaging over 3 learning rates: 10 − 3 , 10 − 4 , 10 − 5 .
L C OMPUTATIONAL GAINS OF HYBRID ESTIMATORS
Consider the general hybrid estimator of SM( x , y ) from Equation 8, using p + 1 base estimators.
Assume that the k th base estimator for k = 1 , ..., p + 1 applies m random features in each of the corresponding t k random maps and that n random features are used in each of the corresponding l λ k random maps to approximate k th λ -coefﬁcient for k = 1 , ..., p .
From Equation 11, we conclude that time complexity of constructing Ψ( z ) is:
[Equation]
Thus the resulting Θ( mn ) -dimensional random feature map Ψ( z ) can be constructed in time O ( nd + md + mn ) as opposed to O ( mnd ) as it is the case for the regular estimator, providing substantial computational gains.
This is true if the ( θ )( mn ) -dimensional HRFs can provide similar quality as their mn -dimensional regular counterparts.
It turns out that this is often the case, see: Section D.4.