S parse F low: Accelerating Transformers by Sparsifying Information Flows (2024)

Abstract: Transformers have become the de-facto standard for natural language processing. However, dense information flows within transformers pose significant challenges for real-time and resource-constrained devices, as computational complexity grows quadratically with sequence length. To counteract such dense information flows, we propose SparseFlow, a novel efficient method designed to sparsify the dense pathways of token representations across all transformer blocks. To this end, SparseFlow parameterizes the information flows linking token representations to transformer blocks. These parameterized information flows are optimized to be sparse, allowing only the salient information to pass through into the blocks. To validate the efficacy of SparseFlow, we conduct comprehensive experiments across diverse benchmarks (understanding and generation), scales (ranging from millions to billions), architectures (including encoders, decoders, and seq-to-seq models), and modalities (such as language-only and vision-language). The results convincingly demonstrate that sparsifying the dense information flows leads to substantial speedup gains without compromising task accuracy. For instance, SparseFlow reduces computational costs by half on average, without a significant loss in accuracy.

## introduction transformers @xcite have brought about a paradigm shift in diverse research areas such as nlp @xcite and computer vision @xcite . however, their undeniable effectiveness often comes with a non-negligible computational burden, scaling quadratically with the length of the input sequence. this bottleneck poses a critical challenge in realizing the full po-1 our code is available at https://github.com/ yeachan-kr/sparseflow transformer block token representation 𝑤 1 𝑤 2 𝑤 3 𝑤 4 𝑤 1 𝑤 2 𝑤 3 𝑤 4 dense information flow sparseflow (ours) tential of transformers, particularly in resourceconstrained or real-time systems @xcite . a primary contributor to this inefficiency is the dense information flows foot_0 between all token representations, specifically, the pairwise computations between every pair of tokens. while these dense information flows are a source of strength in transformers, due to their minimal inductive bias, they often lead to redundancy in token representations @xcite , resulting in different tokens carrying similar information @xcite . such redundancy implies that the current pre-trained language models involve a large amount of expensive yet unnecessary computations. this observation underscores the potential benefits of enhancing efficiency by simplifying the complexity of the current dense information flows. in this paper, we propose sparseflow, a novel efficient method designed to sparsify the current dense information flows across all transformer blocks. to this end, sparseflow parameterizes the information flows linking token representations to each transformer block. these parameters are subsequently optimized to be sparse such that only the salient information flows are preserved. to effectively optimize the sparsified patterns for diverse inputs, we extend sparseflow through sparsely activated mixture-of-experts @xcite . by engaging each sample with the optimized sparse pathways, sparseflow streamlines the transformer inference while preserving the task accuracy, offering practical value across a broad spectrum of applications. to confirm the efficacy of sparseflow, we conduct extensive experiments, encompassing diverse tasks (natural language understanding and generation), scales (from millions to billions), architectures (including encoder, decoder, and sequence-to-sequence models), and modalities (language and vision). we further perform a thorough analysis to gain insights into the behavior of sparseflow by examining the information flow of task-relevant tokens and the sparsified patterns of this flow. in summary, the contributions of this paper include the following: ## related work representation removal in transformers dense information flows in self-attention often lead to redundancy in token representations @xcite . this facilitates numerous studies aimed at removing these redundant representations during a forward pass. the major strategy behind those studies is to reduce computations in transformer blocks depending on the input representations, and they can be categorized in two-fold: (i) depth reduction (also known as early-exit) (ii) width reduction (also known as token pruning).the former approach reduces computations by decreasing the number of computing layers @xcite , while the latter does so by shortening the number of tokens forwarded into transformer blocks @xcite @xcite . recent works have more focused on token pruning approach as it can optimize the computational resources in a more fine-grained way @xcite . given these categorizations, sparseflow aligns more closely with the token pruning approach. token pruning the earliest token pruning approach is power-bert @xcite ) that maintains only the pre-determined ratio of tokens based on the attention weights while removing others. however, due to the re-training costs of power-bert for different environments, lat @xcite extends it to effectively fit diverse configuration of computational budgets without the retraining. subsequently, ltp @xcite removes the tokens by optimizing the trainable threshold for the attention values to identify which tokens are significant to tasks. beyond manual analysis on attention weights, recent works move to automatic methods that learn to select tokens to be removed during training. for example, tr-bert @xcite and transkimmer @xcite have suggested token removal strategies that can be learned during training, by using reinforcement learning and re-parameterization, respectively. adapler @xcite and lot @xcite have proposed a saliencybased strategy that eliminates tokens by estimating the gradients of the token representations with respect to the predictions. compared to token pruning, sparseflow has distinct characteristics. the token pruning methods learn individual sparse patterns for each data. in contrast, sparseflow learns the generalized patterns across all data by directly parameterizing information flows. furthermore, token pruning methods necessitate additional procedures (e.g., attention analysis) or specialized layers (e.g., token classifiers) to identify which tokens to remove. however, sparseflow eliminates these processes, as it pre-determines the sparsified pathways during the training phase, enabling streamlined inference. lastly, while token pruning methods progressively discard token information from the input sequence, sparseflow maintains all token information but reduces connections in the flows. such a difference renders sparseflow less susceptible to the information loss commonly associated with token pruning methods @xcite . in this section, we elaborate on sparseflow, which is designed to sparsify the dense information flow in current pre-trained language models. we first revisit the dense and sparse information flows in transformer ( §3.1). to construct the sparse information flows, we directly parameterize the information flow and introduce a binarization method to sparsify these parameterized flows ( §3.2). to cover the diverse inputs, we extend sparseflow through adaptively optimizing multiple sets of parameterized information flows ( §3.3). these parameters are lastly optimized to be sparse ( §3.4), allowing only the salient flows to be preserved. ## conclusion in this paper, we have introduced sparseflow, a novel efficient method that sparsifies the dense information flow to improve efficiency in transformers. to this end, we parameterize the information flow of the pre-trained models and learn to sparsify the information flow such that only the necessary flows are preserved. we have performed extensive experiments on diverse benchmarks, scaled models, and modalities to verify the efficacy and general applicability. comprehensive result convincingly demonstrates that sparsifying the information flows brings significant speedup gains without compromising the task accuracy. moreover, we have also shown that sparseflow can learn modalityspecific information automatically (e.g., center bias in an image processing).