CQIL : Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers (2024)

Abstract: The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of layers. Recent works on reducing the cumulative latency through layer removing, however, lead to significant performance drop. Motivated by the similarity of inputs among adjacent layers, we propose to identify quasi-independent layers, which can be concurrently computed to significantly decrease inference latency. We also introduce a bypassing technique to mitigate the effect of information loss. Empirical experiments of the proposed approach on the LLaMA models confirm that Concurrent Computation of Quasi-Independent Layers (CQIL) can reduce latency by up to 48.3% on LLaMA-33B, while maintaining a close level of performance.

## introduction large language models (llms) are offering unprecedented power to deliver remarkable performance across diverse tasks of natural language processing. the exceptional performance, however, comes at the cost of increasing model size and, consequently, higher inference latency. for example, the per-token inference time for gpt-4 is approximately three times longer than that of gpt-3.5, according to measurements from the openai api foot_1 . such a high latency has a direct impact on @xcite . it highlights that adjacent layers have highly similar input. notably, such similarity of inputs becomes increasingly evident in larger models at deeper layers, suggesting the quasi-independence of deeper layers and the potential for parallel computation. llms typically consist of a large number of sequentially connected layers with identical structures. for better illustration, we analyze the inference latency of llms along two dimensions: per-layer latency and the cumulative latency due to all layers. existing low-latency inference methods, like tensor parallelism, quantization @xcite , and unstructured pruning @xcite @xcite and low-rank factorization @xcite , primarily focus on minimizing per-layer latency. meanwhile, there are works, such as structured pruning @xcite and dynamic early existing @xcite , proposed to address the cumulative latency through removing and/or dynamically omitting layers. these approaches, unfortunately, often result in considerable performance degradation and potential loss of learned knowledge @xcite . therefore, it's critical to develop effective methods that can reduce the latency related to the total number of layers while preserving the model performance. this work is inspired by the observation that adjacent layers in llms share significantly similar inputs. such similarity suggests the possibility of substituting a layer's input with that of a certain preceding layer without significantly altering its output. such layers sharing input are designated as quasi-independent layers in this work. as a result, the computing dependency between adjacent layers can be eliminated and thus unleash the potential of parallel computation. we introduce a framework of concurrent computation of quasi-independent layers (cqil), to reduce llm inference latency by parallelizing the computation across layers with similar inputs. additionally, we develop a bypassing technique to transmit the output of attention modules among input-aligned layers, with the purpose of minimizing the information loss. extensive experiments demonstrate reductions of inference latency by up to 48.3% on the llama-33b model, with a minor impact on performance. we also discuss the implication of cqil in the context of ensembles, which may offer deeper insights into the fundamental characteristic of llms. the major contributions of this work are as follows. first, we propose cqil, a novel approach to enhance the inference efficiency of llms through concurrent computation of quasi-independent layers, effectively addressing the challenge posed by the increasing number of layers. second, our method enables the adaptation of pre-trained llms into ensemble-like models with minimal performance loss, which may provide deeper insights for layers' characters in llms. third, we effectively reduce the inference latency of llama models, with minimal impact on the model performance. parallelism beyond removing the model's parameters, latency reduction and throughput enhancement can also be achieved through parallel computation strategies. typically, parallelism in llm computation includes data parallelism @xcite , pipeline parallelism @xcite , and tensor parallelism. among these, only tensor parallelism addresses the inference latency by distributing layers computation across multiple gpus. similar to tensor parallelism, our method employs additional gpus to decrease latency. therefore, we conducted comparison experiments in section 5.5. results show that, compared with tensor parallelism, our method consistently reduces latency across various batch size, making it more amenable to the scenario of online inference. in addition, our approach is orthogonal to tensor parallelism and we could achieve further acceleration by integrating tensor parallelism with cqil. transformers with parallel architecture previous works have explored the acceleration of pretraining phases through the parallelization of transformer architectures. gpt-j @xcite and @xcite achieve efficiency improvements by parallelizing the attention and feedforward modules within transformer layers, allowing concurrent computation for the beginning projection in these two modules. other research efforts @xcite have enhanced model performance by expanding model width through parallel layers design, and still focus on the pre-training stage. our work applies concurrent computation of layers to a pretrained llms, aiming at reducing latency while maintaining model performance. our approach thus differs from existing methods that focus on ## preliminary in this section, we present a systematic investigation that motivates the design of the proposed approach introduced in section 4. our discussion proceeds under the assumption of pre-layer normalization, which is adopted by most llms @xcite @xcite . in following experiments, we utilize the llama-1 @xcite and sheared-llama-2.7b @xcite ) models, with the input samples randomly selected from the redpajama (computer, 2023) dataset. we first investigate the similarity of inputs across layers in llms, which suggests the quasi-independence of deeper layers and the potential for parallel computation. second, we explore the effect of substituting a layer's input with that of its preceding layers, which aids in identifying potential layers for parallel processing with minor performance degradation. ## discussion currently, the explanation of the effectiveness of transformer based llms can be classified into two views, pipeline @xcite @xcite and ensemble @xcite @xcite . based on following observations, our research suggests that llms works as a combination of both pipeline and ensemble mechanisms. first, our findings reveal that the bottom layers of llms display distinct input/output characteristics, rendering them challenging to be parallelized. on the other hand, the middle and top layers demonstrate considerable similarity in their inputs, making it possible for straightforward conversion to parallel execution without sacrificing the model performance. this observation is compatible with previous work @xcite , proving the increased robustness in higher layers within the transformer architecture. in other words, our contributions substantiate that llms employ a pipeline mechanism at lower levels and an ensemble strategy at higher levels, particularly through the successful parallelization of the middle and upper layers. this work thus provide further evidence to deepen our comprehension on the internal mechanism of llms. ## conclusion in this paper, we propose an efficient concurrent computation framework on quasi-independent layers that are pervasive in llms with the purpose of reducing inference latency while maintaining model performance. to mitigate the potential information loss resulting from input alignment and improve the performance, we develop the bypassing technique, transmitting attention outputs among input-aligned layers. our experimental results justify the effectiveness of cqil method. in the future, we will apply cqil on even larger models and implement the integration of tensor parallelism with our approach to achieve further latency reductions.