When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants (2023)

Abstract: We present the first unified study of the efficiency of self-attention-based Transformer variants spanning text, speech and vision. We identify input length thresholds (tipping points) at which efficient Transformer variants become more efficient than vanilla models, using a variety of efficiency metrics (latency, throughput, and memory). To conduct this analysis for speech, we introduce L-HuBERT, a novel local-attention variant of a self-supervised speech model. We observe that these thresholds are (a) much higher than typical dataset sequence lengths and (b) dependent on the metric and modality, showing that choosing the right model depends on modality, task type (long-form vs. typical context) and resource constraints (time vs. memory). By visualising the breakdown of the computational costs for transformer components, we also show that non-self-attention components exhibit significant computational costs. We release our profiling toolkit at https://github.com/ajd12342/profiling-transformers .

## introduction and related work transformers @xcite are widely adopted across nlp @xcite , speech processing @xcite and computer vision @xcite . studies have shown that scaling models up improves performance @xcite , making efficiency an important research topic. many transformer variants focus on improving the efficiency of self-attention, motivated by its asymptotic quadratic time/space complexity with respect to the input sequence length. 1 while these transformer variants are designed be asymptotically faster, in practice they may actually be slower, especially given modest input lengths that are typical of many tasks. our paper presents two main analyses. first, we visualize the layerwise efficiency of such models to locate bottlenecks and attempt to answer the question "is self-attention the true bottleneck?" we find that in the non-asymptotic case, non-self-attention layers contribute significantly to the overall cost, especially for speech architectures due to the input waveform tokenizer in models like hubert @xcite . second, when should we use selfattention-based efficient transformers? comparing efficient variants with vanilla models at different input lengths, we find that this tipping point where efficient variants outperform vanilla architectures is much higher than typical input lengths of existing benchmarks across all modalities, calling into question the efficacy of using such efficient transformers and requiring new benchmarks. we introduce a local-attention variant of a speech transformer, hubert, to conduct this analysis. together, our analyses suggest that current approaches that focus on improving self-attention might not be the most effective for improving efficiency. ## efficiency metrics model efficiency is an umbrella term for a suite of efficiency metrics, which do not always correlate with, and sometimes contradict, each other @xcite . further, different metrics are relevant to different end use-cases. to cover most use-cases, we evaluate a set of four metrics; two for computational time and two for memory usage: throughput: number of examples processed per sec, given inputs of a given sequence length, using the maximum possible batch size for a given gpu. latency-inference: time (in ms) to run inference for 1 unbatched input of a given sequence length. max-memory: the allocated gpu memory (mib) for processing 1 input of a given sequence length. parameter count: number of model parameters. we profile models in all modalities in training mode and inference mode. for training, while transformer architectures often use prediction heads with a larger output space (e.g., for text generation), we choose a lightweight binary classification head for profiling. ## local-attention speech model efficient transformers @xcite have not received as much attention in speech as they have in nlp and cv, perhaps due to two reasons. first, there is a relative lack of longcontext speech benchmarks as compared to those in nlp (lra @xcite tasks like automatic speech recognition (asr), it is typical to segment a long speech signal into small individual utterances and perform asr independently on each. for example, most librispeech examples are less than 5 seconds. many popular speech models like hubert @xcite tokenize the waveform at 50 tokens per second, implying that a typical utterance has only several hundred tokens; far below the number of tokens in long-context nlp tasks. nevertheless, textless speech models @xcite are becoming more feasible, motivating the modelling of long speech utterances. local hubert model to investigate the efficiency of the self-attention layer in speech models, we introduce the local hubert model which replaces hubert's self-attention with the longformer (beltagy et al., 2020) sliding-window selfattention. in this attention mechanism, every token attends to tokens within a local window context, rather than the full token sequence. our model is similar to the temporally windowed-attention transformer acoustic model proposed by @xcite for speech translation; our approach differs by using the self-supervised hubert model as our basis, and we evaluate on asr. choosing the appropriate window size for the local attention context is key; we explore 32 and 100 token contexts, corresponding to 640 ms and 2 s, inspired by phone recognition models that typically incorporate similar context sizes @xcite . ## methods and implementation we analyze the base versions of the bert @xcite , longformer @xcite and nyströmformer @xcite models for text; the hubert @xcite and l-hubert (section 3) models for speech; and vision transformer @xcite and swin transformer @xcite models for vision; bert, hubert and vit are standard transformer encoder architectures. longformer, l-hubert and swin use fixed-pattern self-attention while nyströmformer uses approximate self-attention. ## conclusion we present an empirical efficiency analysis of vanilla transformers and their self-attention-based efficient variants across modalities, metrics and input context sizes. we find substantial differences across modalities and metrics when analyzing the tipping point for efficient variants. finally, the layerwise analysis finds that self-attention is not the only bottleneck. we recommend that all efficient model papers should report such cross-modal, layerwise profiling results on multiple efficiency metrics covering a variety of use-cases to provide a full picture of the benefits of the model.