Attention Mechanism with Energy-Friendly Operations (2022)

Abstract: Attention mechanism has become the dominant module in natural language processing models. It is computationally intensive and depends on massive power-hungry multiplications. In this paper, we rethink variants of attention mechanism from the energy consumption aspects. After reaching the conclusion that the energy costs of several energy-friendly operations are far less than their multiplication counterparts, we build a novel attention model by replacing multiplications with either selective operations or additions. Empirical results on three machine translation tasks demonstrate that the proposed model, against the vanilla one, achieves competitable accuracy while saving 99% and 66% energy during alignment calculation and the whole attention procedure. Our code will be released upon the acceptance.

## introduction attention mechanism (att, @xcite @xcite has demonstrated huge success in a variety of natural language processing tasks @xcite @xcite @xcite . the module learns hidden representations of a sequence by serving each word as a query to attend to all keys in the target sentence, then softly assembling their values. it is a de-facto standard to achieve this via performing linear projections and dot products on representations of queries and keys @xcite , resulting in large amount of multiplications. in spite of its promising quality, such kind of paradigm may be not the preferred solution from the energy consumption aspect @xcite operation (fp32) asic fpga addition 0.9 0.4 multiplication 3.7 18. 8 our work starts from in-depth investigations on approaches in att context with respect to model compression @xcite and complexity optimization @xcite @xcite . these approaches can potentially alleviate the problem of high energy consumption in att. nevertheless, intentions of all these methods are not exactly from the energy-friendly perspective, thus overlooking the origin of energy consumed, i.e., basic arithmetic operations in electric equipments. massive multiplications still remain, consuming far more energy than its additive counterpart on modern devices ( to this end, we propose to approach this problem from a new direction -replacing massive multiplications in att with cheaper operations. concretely, we propose a novel energy-efficient attention mechanism (e-att). it equips binarized selective operations instead of linear projections over input hidden states, and measures attentive scores using l 1 distance rather than dot-product. consequently, e-att abandons most of multiplications to reach the goal of energy cost reduction. we examine our method with transformer model @xcite , and conduct experiments on three machine translation tasks. compared with conventional att, our e-att can save more than 99% energy of the vanilla alignment calculation and around 66% energy of the whole attention model. in the meanwhile, our models yield acceptable translation qualities across language pairs. extensive analyses also demonstrate that e-att can functionally model semantic alignments without using multiplications. ## preliminary conventional attention mechanism given input representations @xmath0, l 2 being sequence length, and d is the input dimensionality. note l 1 and l 2 may be equal for self-attention pattern, and represent lengths of target and source sequence in cross-attention. att first projects the inputs into three representations 1 : then, the output is derived by multiplying attention weights with value representation v: as seen, matrix multiplications are massively exploited into conventional att. related work several related approaches potentially alleviate the power-hungry drawback of att. one direction relies on model compression by pruning redundant parameters @xcite @xcite or distilling the learned knowledge from a large model to a smaller one @xcite , which still maintains multiplicative operations. another direction aims at reducing the computational complexity of attention module, e.g. linearly projecting input (dense, @xcite , or randomly initializing and training attention weights (ran-dinit, @xcite . to give a full comparison of energy consumption of these approaches, we conduct the number of multiplicative operations and energy costs across modules in ## energy-efficient attention mechanism in this section, we describe e-att by pertinently reducing the multiplicative operations of att, including selective operation and l 1 distance. ## discussion and conclusion in this paper, we empirically investigate the high energy-consumption problem in att. we argue that the alignment modeling procedure can be achieved with additions other than multiplications, thus reducing the energy costs. extensive analyses suggest that: 1) binarized representations marginally harm the feature extraction procedure; and 2) l 1 distance can be efficiently exploited to measure alignment among queries and keys. compared to baseline, our approach can yield considerable quality of translations, and significantly save energy in attention mechanism. although we have shown the superiority of e-att, considering the whole transformer block foot_1 , the use of e-att brings 17% energy reduction. we hope this work can attract more researches on energy-efficient models. it is worth to further design techniques that reduce the energy cost of other modules in transformer.