EcoFormer: Energy-Saving Attention with Linear Complexity
Anonymous Author(s)
AffiliationAddress
Abstract
Transformer is a transformative framework for deep learning which models sequential data and has achieved remarkable performance on a wide range of tasks,but with high computational and energy cost.
To improve its efficiency, a popularchoice is to compress the models via binarization which constrains the floatingpoint values into binary ones to significantly save resource consumption owing tocheap bitwise operations.
However, existing binarization methods only aim at minimizing the information loss for the input distribution statistically, while ignoringthe pairwise similarity modeling at the core of the attention mechanism.
To this end,we propose a new binarization paradigm customized to high-dimensional softmaxattention via kernelized hashing, called EcoFormer, to map the original queries andkeys into low-dimensional binary codes in Hamming space.
The kernelized hashfunctions are learned to match the ground-truth similarity relations extracted fromthe attention map in a self-supervised way.
Based on the equivalence between theinner product of binary codes and the Hamming distance as well as the associativeproperty of matrix multiplication, we can approximate the attention in linear complexity by expressing it as a dot-product of binary codes.
Moreover, the compactbinary representations of queries and keys in EcoFormer enable us to replace mostof the expensive multiply-accumulate operations in attention with simple accumulations to save considerable on-chip area and energy footprint on edge devices.
Extensive experiments on image classification show that EcoFormer consistentlyachieves comparable performance with standard attentions while consuming muchless resources.
For example, based on PVTv2-B0 and ImageNet-1K, EcoFormerachieves 73% reduction in both energy footprint and chip area with only a slightperformance drop of 0.33% compared to the standard attention.
Introduction
Recently, Transformers [53] have shown rapid and exciting progress in natural language processing(NLP)
[14, 13] and computer vision (CV) [17, 52] due to its extraordinary representational power.
Compared with convolutional neural networks (CNNs) [30], Transformer models are generally morescalable to massive amounts of data and better at capturing long-dependency global information withless inductive bias, thus achieving better performance in many tasks [26, 38].
However, the efficiencybottlenecks, especially the high energy consumption, greatly hamper the massive deployment of
Transformer models to resource-constrained edge devices, such as mobile phones and unmannedaerial vehicles, for solving a variety of real-world applications.
To reduce the energy consumption, quantization has been actively studied to lower the bit-widthrepresentation of network weights [11, 50, 61] and/or activations [25, 63, 62].
With the mostaggressive bit-width, binary quantization [49, 2, 46] has attracted much attention since it enables
{𝐪 ! } !"
#$ {𝐤 ! } !" #$ {𝐯 ! } !"
#$ 𝒪(𝑁 ! )
(") softmax 𝒪(𝑁) (") {𝐪 ! } !"
#$ {𝐤 ! } !"
#$ {𝐯 ! } !"
#$ 𝜙(") 𝒪(𝑁) (") 𝜙(") (") 𝒪(𝑁) (a) Attention (b) Kernel-based Linear Attention (c) EcoFormer
Addition only Multiply-accumulate {𝐪 ! } !"
#$ {𝐤 ! } !"
#$ {𝐯 ! } !"
#$ 𝐻(") 𝒪(𝑁) (") 𝐻(") (") 𝒪(𝑁) binary codes binary codes
Operation
[Table]
efficient bit-wise operations by representing values with a single bit ( e.g. , +1 or -1).
When weonly binarize weights in analogous to BinaryConnect [11], as shown in Table 1, it brings greatbenefits to dedicated hardware by replacing a large number of energy-hungry multiply-accumulateoperations with simple energy-efficient accumulations, which saves significant energy and chiparea required to run inference with Transformers, making them feasible to be deployed on mobileplatforms with limited resources.
However, the conventional binarization process typically targets atminimizing quantization errors between the original full-precision data distribution and the quantized
Bernoulli distribution statistically.
In other words, each token is binarized separately, where the binaryrepresentations may not well preserve the original similarity relations among tokens.
This motivatesus to customize the binarization process to softmax attention, the core mechanism in Transformerthat encodes the pairwise similarity between tokens.
To this end, we can adapt the well-establishedhashing methods, to map the high-dimensional queries and keys into compact binary codes ( e.g. ,16-bits) that are able to preserve the similarity relations in Hamming space.
A simple solution is touse the locality-sensitive hashing (LSH) [1] to substitute the binary quantization counterparts.
Nevertheless, another energy bottleneck exists in Transformers.
Specially, given a sequence of tokens,the softmax attention obtains the attention weights by computing the inner product between a querytoken and all key tokens, leading to the quadratic time complexity O (cid:0) N 2 (cid:1) regarding the numberof tokens N , as shown in Figure1 (a).
This problem is even worse for a long sequence length N ,especially for high resolution images in dense prediction tasks.
To reduce the complexity of thesoftmax attention, some prior works propose to express the attention as a linear dot product ofkernelized feature embeddings [45, 9].
With the associative property of matrix multiplication, theattention operation can be approximated in linear complexity O ( N ) , as illustrated by Figure 1 (b).
Based on the hashing mechanism and kernel-based formulation of attention, we devise a simple yeteffective energy-saving attention, called EcoFormer, which is shown in Figure 1 (c).
In particular, wepropose to use kernelized hashing with RBF kernel to map the queries and keys to compact binarycodes.
The resulting codes are valid for similarity preserving based on the good property that thecodes’ inner product ( i.e. , Hamming affinity) and Hamming distance have one-to-one correspondence[36].
Thanks to the associative property of the linear dot-product between the binary codes, thekernelized hashing attention is in linear complexity with significant energy saving.
Moreover, thepairwise similarity matrix in attention can be directly used to obtain the supervision labels for hashfunction learning, delivering a novel self-supervised hashing paradigm.
By maximizing the Hammingaffinity on the similar pairs of tokens and simultaneously minimizing on the dissimilar pairs of tokens,the pairwise similarity relations between tokens can be preserved.
With low-dimensional binaryqueries and keys, we can replace most of the energy-hungry floating-point multiplications in attentionwith simple additions, which greatly saves the on-chip area and energy footprint.
To sum up, we make three main contributions: 1) We propose a new binarization paradigm tobetter preserve the pairwise similarity in softmax attention.
In particular, we present EcoFormer, anenergy-saving attention with linear complexity powered by kernelized hashing to map the queriesand keys into compact binary codes.
We learn the kernelized hash functions based on the groundtruth Hamming affinity extracted from the attention scores in a self-supervised way.
3) Extensiveexperiments on CIFAR-100 and ImageNet-1K show that EcoFormer is able to significantly reduce the energy cost and chip area while preserving the accuracy.
For example, based on PVTv2-B0 and ImageNet-1K, EcoFormer achieves 73% reduction in both energy footprint and chip area with only amarginal performance drop of 0.33% compared to the standard attention.
Related Work
Efficient attention mechanisms.
To alleviate the quadratic computational cost for vanilla attentionwith respect to the number of tokens, much work has endeavored developing efficient attentions.
One line of research performs attention only on part of the tokens [38, 56, 4, 29, 55].
For instance, Reformer [29] restricts the attention to the most similar token pairs via hashing and reduces thecomputational complexity to O ( N log N ) .
Linformer [55] approximates the attention with low-rankfactorization that reduces the length of the key and value.
However, the computational complexity isdependent on the design for reducing tokens, which is challenging for the majority of the methodsto achieve a linear complexity O ( N ) .
Another line of research speeds up the vanilla attention withkernel-based methods [9, 59, 41, 28, 47, 47].
For example, Performer [9] approximates the softmaxoperation with orthogonal random features.
Nyströmformer [59] and SOFT [41] approximate the fullself-attention matrix via matrix decomposition.
Cosformer [47] decomposes the attention into a linearform with a non-negative property and a re-weighting scheme.
Although impressive achievementshave been achieved, how to develop attention that is highly energy-efficient remains under-explored,as multiplications dominate the energy consumption.
AdderNet variants [8, 51] replace the energyhungry multiplication-based similarity measurement with the energy-efficient addition-based L1distance and argue that additions can also provide powerful feature representations.
Nevertheless,this heuristic approach brings a drastic performance drop.
In contrast, we are from the perspectiveof binarization and propose to learn kernel-based hash functions using attention scores to map theoriginal features into compact binary codes to measure the token similarity by the Hamming distance,which is energy-efficient and in linear complexity O ( N ) .
Hashing.
Hashing is an efficient nearest neighbor search method by embedding the high-dimensionaldata into a similarity preserving low-dimensional binary codes, based on the intuition that highlysimilar data should be assigned the same hash key.
Hashing methods can be roughly categorizedinto data-independent and data-dependent schemes.
The former focuses on building random hashfunctions and locally sensitive hashing (LSH) [19, 7] is arguably the most representative one, whichguarantees the sub-linear time similarity search and is followed by non-linear extensions such ashashing with kernels [31] or on manifolds [58].
The latter can be further divided into unsupervised[37, 20, 35], semi-supervised [44, 54] and supervised hashing [36, 65, 60].
When it comes to
Transformers, as self-attention encodes the pairwise similarity among tokens, hashing is thus a naturalchoice to efficiently retrieve similar keys given a query.
Reformer is such a pioneering work, whichproposes to group similar tokens in a single hash bucket to form sparse self-attention.
Our Ecoformeris fundamentally different from Reformer in three aspects: 1) Reformer relies on local attentionlookups to reduce the complexity while our Ecoformer is designed from a numerical perspective,where the low-dimensional binary codes are used to save the multiplications; 2) Reformer is builtupon LSH with linear mapping, which cannot deal with the kernel-based formulation of attention toscale linearly with the sequence length; 3) Our hash functions are self-supervised by the pairwiseaffinity labels in attention, which are optimized in conjunction with network parameters and moreaccurate than unsupervised random projections.
Binary quantization.
Binarization, an extreme quantization scheme, seeks to represent the vectorsby binary codes.
As a result, the computationally heavy matrix multiplications become light-weightbitwise operations ( i.e. , xnor and popcount ), yielding promising memory saving and acceleration.
In general, to make binary neural networks [24] reliable in accuracy, current research targets attackling two main challenges.
The first challenge is to minimize the quantization error, basicallybased on learning the scaling factors [49, 6], parameterizing the quantization range and/or intervals[27, 18], and ensembling multiple binary bases [34, 66], etc.
Another category of studies focus onsolving the non-differentiable optimization problem due to the discretization process via trainingwith regularization [16], knowledge distillation [43, 46], relaxed optimization [23, 3], appendingfull-precision branches [39, 42] and so on.
Apart from CNNs, there are some recent pioneeringattempts targeting on binarizing Transformers.
For example, BinaryBERT [2] proposes to push
Transformer quantization to the limit by weight binarization.
BiBERT [46] quantizes both weights,embeddings and activations of BERT [15] to 1-bit and achieves considerable savings on FLOPs and model size, but still has obvious performance drop.
In contrast, we propose to customize thebinarization paradigm to softmax attention from the hashing perspective, preserving high-fidelitypairwise similarity information in compact binary codes which are used to deliver linear-complexity,energy-efficient self-attention.
Preliminaries
3.1 Attention Mechanism
Let X ∈ R N × D be the input sequence into a standard multi-head self-attention (MSA) layer, where
N is the length of the input sequence and D is the number of hidden dimensions.
A standard MSAlayer calculates a sequence of query, key and value vectors with three learnable projection matrices
W q , W k , W v ∈ R D × D p , which can be formulated as
[Equation]
where D p refers to the number of dimensions for each head.
For each query vector, the attentionoutput is a weighted-sum over all value vectors as
[Equation]
where τ is the temperature for controlling the flatness of softmax and exp( ⟨ · , · ⟩ ) is an exponentialfunction.
The computation of Eq.
(2) has a quadratic complexity of O ( N 2 ) in both space and time,which results in huge computational cost when dealing with long sequences.
3.2 Kernel-based Linear Attention
The idea behind kernel-based linear attention is to express the similarity measure in Eq.
(2) as a lineardot-product of kernel embeddings, such as polynomial kernel, exponential or RBF kernel.
A particularchoice is to employ the finite random mapping [48] ϕ ( · ) to approximate the infinite dimensional
RBF kernel.
Then, according to the theorem from Rahimi [48], the inner product between a pair oftransformed vectors x and y with ϕ ( · ) can approximate a Guassian RBF kernel.
This gives rise to anunbiased estimation to exp( ⟨ · , · ⟩ ) in Eq.
(2), which can be expressed as
[Equation]
Assume that the queries and keys are unit vectors, then the attention computation in Eq.
(2) can beapproximated by
[Equation]
where ⊗ refers to the outer product.
Recent works have shown that kernel-based linear attentionsperform favorably against the original softmax attention on machine translation [45] and proteinsequence modeling [9].
However, although the complexity is reduced into linear, the intensivefloating-point multiplications in Eq.
(4b) still consume a large amount of energy, which can quicklydrain the batteries on mobile/edge platforms.
3.3 Binary Quantization
Following [49], binary quantization typically estimates the full-precision u ∈ R n using a binaryˆ u ∈ { +1 , − 1 } n and a scaling factor α ∈ R + such that u
≈ α ˆ u holds.
To find an accurate estimation,existing methods [49, 42, 34] minimize the quantization error as
[Equation]
By solving Problem (5), we have ˆ u = sign( u ) and α = 1 n ∥ u ∥ ℓ 1 , where sign( u ) returns 1 if u ≥ 0and
if u < 0 .
Since the sign function is non-differentiable, the straight-through estimator (STE) [5]is applied to approximate the gradient such as using the gradient of hard tanh [25] or piecewisepolynomial function [39].
Proposed Method
To reduce the energy consumption of self-attention, one may perform binary quantization [46, 2] onthe queries { q t } Nt =1 and keys { k t } Nt =1 .
In this case, we can replace most of the energy-expensivemultiplications with the energy-efficient bit-wise operations.
However, existing binary quantizationmethods only focus on minimizing the quantization error between the original full-precision valuesand the binary ones as in Eq.
(5), which fails to preserve the pairwise semantic similarity betweendifferent tokens in attention, leading to performance degradation.
Note that the attention can be seen as applying kernel smoother over pairwise tokens where the kernelscores denote the similarity of the token pairs, as mentioned in Section 3.2.
Motivated by this, wepropose a new binarization method that applies kernelized hashing with Gaussian RBF to map theoriginal high-dimensional queries/keys to low-dimensional similarity-preserving binary codes in
Hamming space.
The proposed framework, which we dub EcoFormer , is depicted in Figure 1 (c).
To maintain the semantic similarity in attention, we learn the hash functions in a self-supervisedmanner.
By exploiting the associative property of the linear dot-product between binary codes andthe equivalence between the code inner products ( i.e. , Hamming affinity) and the Hamming distances,we are able to approximate the self-attention in linear time with low energy cost.
In the following, wefirst introduce the kernelized hashing attention in Section 4.1 and then show how to learn the hashfunctions in a self-supervised way in Section 4.2.
4.1 Kernelized Hashing Attention
Before applying hash functions, we let the queries { q t } Nt =1 and keys { k t } Nt =1 to be identical following [29, 41].
In this way, we can then apply kernelized hash functions H : R D p (cid:55)→ { 1 , − 1 } b withoutexplicitly applying transformation ϕ ( · ) mentioned in Section 3.2 to map q i and k j into b -bit binarycodes H ( q i ) and H ( k j ) , respectively (see Section 4.2).
In this case, the Hamming distance betweenthem can be defined as
[Equation]
where H r ( · ) is the r -th bit of the binary codes; 1 { A } is an indicator function that returns 1 if A issatisfied and otherwise returns 0.
With D ( H ( q i ) , H ( k j )) , the codes inner product between H ( q i )and H ( k j ) can be formulated as
[Equation]
[Equation]
Importantly, Eq.
(7) shows the equivalence between the Hamming distance and the codes innerproduct since there is a one-to-one correspondence.
By substituting with the hashed queries and keysin Eq.
(4a)
, we can approximate the self-attention as
[Equation]
{ v
[Equation]
Note that H ( q t )
⊤ H ( k j ) ∈ [ − b, b ] .
To avoid zero in denominator, we introduce a bias term 2 c toeach inner product so that H ( q t ) ⊤ H ( k j ) + 2 c > 0 , having no effect on the similarity measure.
Here,we can simply set c to ⌈ log 2 b ⌉ where ⌈ u ⌉ returns the least integer greater than or equal to u .
Using the associative property of matrix multiplication, we approximate the self-attention as
Attention( q t , { k i } , { v i } )
≈
[Equation]
In practice, the multiplications between the binary codes and the full-precision values in Eq.
(9) canbe replaced by simple additions and subtractions, which greatly reduce the computational overheadin terms of on-chip area and energy footprint.
Moreover, the multiplications with a powers-of-two 2 ccan also be implemented by efficient bit-shift operations.
As a result, the only multiplications comefrom the element-wise divisions between the numerator and denominator.
4.2 Self-supervised Hash Function Learning
Given queries Q = { q 1 , . . .
, q N } ⊂ R D p , we seek to learn a group of hash functions h : R D p (cid:55)→{ 1 , − 1 } .
Instead of explicitly applying the transformation function ϕ ( · ) mentioned in Section 3.2,we compute the hash functions with a kernel function κ ( q i , q j ) :
R D p × R D p (cid:55)→ R .
Following thekernel-based supervised hashing (KSH) [36], we randomly sample m queries q (1) , . . .
, q ( m ) from Qas support samples and define a hash function h as
[Equation]
where a = [ a 1 , · · · , a m ] ⊤ is the weight of h , µ j = 1 n (cid:80) Ni
=1 κ (cid:0) q ( j ) , q i (cid:1) is to normalize thekernel function to zero-mean, and g : R D p (cid:55)→ R m is a vectorial mapping defined by g ( q ) = 216 (cid:2) κ (cid:0) q (1) , q (cid:1) − µ 1 , . . .
, κ ( q ( m ) , q ) − µ m (cid:3) .
Then, the kernelized hash function H ( · ) is defined as
[Equation]
[Equation]
where G =
[ g ( q 1 ) , · · · , g ( q N )] ⊤ ∈ R N × m , A = [ a 1 , · · · , a b ] ∈ R m × b , and h r ( q ) = sign ( Ga r )is the hash function for the r -th bit.
To guide the learning of the binary codes, we hope that similar token pairs will have the minimal Hamming distance while dissimilar token pairs will have the maximal distance.
Nevertheless,directly optimizing the Hamming distance is difficult due to the non-convex and non-smooth formulation in Eq.
Utilizing the equivalence between the code inner products and the Hammingdistances in Eq.
(7), we instead conduct optimization based on the Hamming affinity to minimize thereconstruction error as
[Equation]
[Equation]
where ∥·∥ F is the Frobenius norm and Y ∈ R N × N is the target Hamming affinity matrix.
To preservethe similarity relations between queries and keys, we use the attention scores as the self-supervisedinformation to construct Y .
Let S and U be the similar and dissimilar pairs of tokens.
We obtain
S and U by selecting the token pairs with the Top- l largest and smallest attention scores.
We thenconstruct pairwise labels Y as
[Equation]
However, Problem (12) is NP-hard.
To solve it efficiently, we adapt discrete cyclic coordinate descentto learn binary codes sequentially.
Specifically, we only solve a r once the previous a 1 , · · · , a r − 1have been optimized.
Let ˆ Y r − 1 =
b Y − (cid:80) r − 1 t =1 h t ( q ) h t ( q ) ⊤ be the residual matrix, where ˆ Y
= b Y .
Then, we can minimize the following objective to obtain a r
[Equation]
[Equation]
where C = (cid:0) h r ( q ) ⊤ h r ( q ) (cid:1) 2 + tr (cid:16) ˆ Y r − 1 (cid:17) is a constant.
Note that ˆ Y r − 1 is a symmetric matrix.
Therefore, Problem (14) is a standard binary quadratic programming problem, which can be efficientlysolved by many existing methods, such as the LBFGS-B solver [64] and block graph cuts [33].
Tolearn a r in conjunction with network parameters, we propose to solve Problem (14) using the gradientbased methods.
For the non-differentiable sign function, we use STE [5] to approximate the gradientusing hard tanh as mentioned in Section 3.3.
Note that learning the hash functions for each epoch iscomputationally expensive yet unnecessary.
We only learn the hash functions per τ epoch
[Table]
Experiments
5.1 Main Results
[Table]
To investigate the effectiveness of the proposed method, we conduct experiments on ImageNet1K [30], a large-scale image classification dataset that contains ∼ 1.2M training images from 1Kcategories and 50K validation images.
We compare our kernelized hashing attention with standard
MSA by adapting the two attention approaches into two popular vision Transformer frameworks
PVTv2 [57] and Twins [10].
We measure model performance by the Top-1 accuracy.
Furthermore,as FLOPs cannot accurately reflect the computational cost in our proposed method, we measurethe model complexity by the number of multiplications and additions, separately, as done in [51].
Moreover, we report the energy consumption and area cost according to Table 1.
Implementation details.
All training images are resized to 256 × 256 , and 224 × 224 patches arerandomly cropped from an image or its horizontal flip, with the per-pixel mean subtracted.
To obtainthe MSA baselines, we first replace the original attention layers in PVTv2 [57] and Twins [10] withstandard MSAs and initialize the models with the pretrained weights on ImageNet-1K. Next, wefinetune each model on ImageNet-1K with 100 epochs.
Based the pretrained MSA weights, we thenapply our approach to each model and finetune on ImageNet-1K with 30 epochs.
All models in thisexperiment are trained on 8 V100 GPUs with a total batch size of 256.
The initial learning rate is setto 2 .
5 × 10 − 5 .
We use AdamW optimizer [40] with a cosine decay learning rate scheduler.
All otherhyperparameters are the same as in PVTv2.
Also note that recent hierarchical ViTs [57, 38, 10] havemultiple stages to incorporate pyramid feature maps.
At the last stage, they usually apply standard
MSAs due to the comparably low-resolution feature maps.
This design is also adopted in PVTv2 and
Twins.
Therefore, we follow the common practice and do not modify the attention layers at the laststage.
For the hash functions learning, we set the number of support samples m and update interval τto 25 and 30, respectively.
The hyper-parameter l in constructing pairwise labels Y is set to 10.
Weset the hash bit b to 16.
Results analysis.
We report the results in Table 2.
In general, compared to MSAs, our methodachieves lower computational complexity, less energy consumption and area cost with comparableperformance.
For example, based on PVTv2-B0, our method saves around 73% multiplications and72% additions, as well as reducing 73% energy consumption and area cost, which demonstrates theenergy-efficiency of our approach.
Besides, note that a larger model comes with more multiplicationsand additions, especially in other types of layers ( e.g. , feed-forward layers, as shown in the supplementary material).
In this case, as our approach focuses on the attention layers, the energy-savingfrom larger models is comparably less than smaller models ( e.g. , PVTv2-B0 vs. PVTv2-B2).
Eventhough, we still reduce significant computational cost and energy consumption.
For example, on PVTv2-B2, we save around 55% energy consumption and area cost.
Compared with PVTv2, the performance drop of our method is slightly larger on Twins.
One possible reason is that our methodmay be sensitive to the conditional positional encodings in Twins.
Overall, the results demonstratethat our method is effective and much more energy-efficient compared to the MSA baselines withmassive multiplications.
5.2 Ablation Study
In this section, we evaluate the effectiveness of our EcoFormer by comparing it with differentbinarization approaches and efficient attention mechanisms.
Besides, we also study the effect of thenumber of support samples m in the proposed EcoFormer.
By default, we train each model fromscratch on CIFAR-100 with 2 GPUs.
The total batch size is 64.
The initial learning rate is 6 .
25 × 10 − 6 .
For the hash functions learning, we set update interval τ to 300.
All the other hyperparameters arethe same as in ImageNet-1K experiments.
[Table]
Quantization vs. hashing.
To investigate the effect of different binarization methods, we compareour EcoFormer with the following methods: FP-EcoFormer : Based on EcoFormer, we do notbinarize queries and keys in attentions.
Quant-EcoFormer :
Relying on EcoFormer, we use binaryquantization [25] to obtain binarized queries and keys instead of our proposed hash functions.
Forfair comparisons, the attention operations in the compared method are in linear complexity.
We applydifferent methods to PVTv2-B0 and Twins-SVT-S and report the results in Table 3.
We observe thatour EcoFormer consistently outperforms Quant-EcoFormer on different frameworks.
For example,based on PVTv2-B0, our EcoFormer surpasses Quant-EcoFormer by 1.17% in terms of the Top-1accuracy.
Compared with binary quantization, our proposed self-supervised hash functions preservethe pairwise similarity of attention, leading to better performance.
Moreover, our EcoFormer doesnot need to explicitly compute transformation ϕ ( · ) as in Eq.
(4b).
Therefore, the energy and area costof our EcoFormer is lower than Quant-EcoFormer.
[Table]
Effect of different hash functions.
To investigate the effect of different hash functions, we includethe following methods for comparisons: LSH-EcoFormer : Relying on EcoFormer, we use LocalitySensitive Hashing (LSH) [12] rather than our proposed kernelized hash function.
KLSH-EcoFormer :
Based on EcoFormer, we replace the proposed hash function with Kernelized Locality-Sensitive
Hashing (KLSH) [31].
We report the results in Table 4.
We can observe that KLSH-EcoFormeroutperforms LSH-EcoFormer by 0.48% in terms of the Top-1 accuracy with less energy and chip areaconsumption.
The reason can be attributed to that LSH is based on random linear projection, whichcan not deal with the non-linear softmax attention well.
Critically, our EcoFormer further improvesthe performance by 0.57% on the Top-1 accuracy.
Compared with random hashing in KLSH, our EcoFormer learns the hash functions with additional self-supervised information.
Therefore, ourlearned binary codes are better at preserving the token similarity.
Comparing with other efficient attention mechanisms.
To compare Ecoformer with differentattention mechanisms, we conduct experiments on CIFAR-100 based on PVTv2-B0 in Table 5.
In ourexperiments, we directly replace the attention layers with each compared method in PVTv2-B0 [57].
In general, compared to other efficient attention mechanisms, EcoFormer saves more computationsand reduces more energy consumption and area cost while achieving better performance. Particularly,
[Table] benefiting from the multiplication-saving operations and low-dimensional binary queries and keys, EcoFormer saves more energy than Performer.
Also note that since the proposed kernelized hashfunction H ( · ) does not need to explicitly apply transformation ϕ ( · ) to the queries and keys as in
Eq. (4b), EcoFormer simultaneously reduces more multiplications and additions than Performer.
Besides, Linformer achieves competitive results.
However, as the size of the learnable low-rankprojection parameters depends on the length of the input sequence, Linformer is not scalable todifferent image resolutions, whereas EcoFormer with sufficient bits is agnostic to the sequence length.
We can also find that Reformer suffers from a higher performance drop and massive multiplicationsand additions compared with the other attention mechanisms. Overall, the results demonstrate theadvantage of EcoFormer, especially on resource-constrained scenarios.
[Table]
Effect of different m . To investigate the effect of different numbers of support samples m , we train EcoFormer with different m based on PVTv2-B0. We report the results in Table 6.
As we increasem , the performance becomes better along with the increase in energy and area consumption. Forexample, the model obtained with m = 15 outperforms that of m = 10 by 0.19% on the Top-1accuracy with little additional energy and area cost.
We speculate that, with more support samples,we can capture more accurate statistics in Eq. (10) and hence lead to better performance. We usem = 25 by default in our experiments.
Conclusion and Future Work
In this paper, we have presented a novel energy-saving attention mechanism with linear complexity tosave the vast majority of multiplications from a new binarization perspective, making the deploymentof Transformer models at scale feasible on edge devices.
We are inspired by the fact that conventionalbinarization methods are built upon statistical quantization error minimization without considering topreserve the pairwise similarity relations between tokens.
To this end, we customize binarization tosoftmax attention by mapping the original token features into compact binary codes in Hammingspace using a set of kernel-based hash functions, where the similarity can be measured by codes dotproduct.
The hash functions for queries/keys are learned to encourage the Hamming affinity of a tokenpair to be close to the target obtained from the attention scores, in a self-supervised way.
Extensiveexperiments have demonstrated that EcoFormer saves significant energy footprint while achievingcomparable performance with standard attentions on ImageNet-1K and CIFAR-100.
In terms ofthe future work, we can further binarize the value vectors in attention, multi-layer perceptrons andnon-linearities in Transformer to make it fully binarized for more significant energy-saving.
We mayalso extend EcoFormer to NLP tasks such as machine translation and document classification, andspeech analysis tasks to make it more impactful to wider communities.
Limitations and societal impact.
We have shown that EcoFormer is more energy-efficient comparedto the standard attention.
However, in practice, the addition operations between binary codes andfloating-point numbers will require specialized GPU kernels ( e.g. , customized CUDA operators) forfurther acceleration.
Our work potentially brings some negative societal impacts that training large
Transformer models requires extensive computations, resulting in financial and environmental costs.
A promising solution is to jointly optimize training and inference efficiency.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See the abstract and introduction.
(b) Did you describe the limitations of your work?
[Yes] See the conclusion in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See theconclusion in Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] We have
included the code and instructions needed in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See the implementation details in Section 5.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)?
[No] Error bars are not reported because it would be too
computationally expensive.
We use the same random seed as in recent works for faircomparison.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the implementation details in Section 5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.(b)
Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Code is included in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] We use public datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]