F ast A da SP : Multitask-Adapted Efficient Inference for Large Speech Language Model (2024)

Abstract: In this study, we aim to explore Multitask Speech Language Model (SpeechLM) efficient inference via token reduction. Unlike other modalities such as vision or text, speech has unique temporal dependencies, making previous efficient inference works on other modalities not directly applicable. Furthermore, methods for efficient SpeechLM inference on long sequence and sparse signals remain largely unexplored. In this work, we propose FastAdaSP, a weighted token merging framework specifically designed for various speech-related tasks to improve the trade-off between efficiency and performance. Experimental results on WavLLM and Qwen-Audio show that our method achieves the state-of-the-art (SOTA) efficiency-performance trade-off compared with other baseline methods. Specifically, FastAdaSP achieved 7x memory efficiency and 1.83x decoding throughput without any degradation on tasks like Emotion Recognition (ER) and Spoken Question Answering (SQA).

## introduction speech language models (speechlms) have been an important role in the field of natural language processing and speech technology. recent advancements @xcite @xcite have demonstrated significant capabilities in voice processing and audio understanding. furthermore, @xcite showcases conversational speech processing abilities, advancing the capability of llms toward various voice-interface applications. however, challenges related to inference latency and memory efficiency remain major bottlenecks, especially as multitask speechlms grow larger, reaching up to 7 billion parameters. these challenges necessitate the development of more efficient inference methods. ## methodology in this section, we introduce the motivation and formulation of fastadasp, followed by our layer selection and task-specific design strategies for multitask speechlms. note that, in our work, audio tokens refers to the audio features output by the multi-head attention block. ## conclusion in this study, we propose fastadasp, an efficient inference framework that incorporates multiple stages in speechlms. this preliminary study explores token reduction methods for speechlms. we investigated various properties of different types of speechlm tasks and proposed novel methods for both dense and sparse tasks. our method achieved a 1.84x throughput increase with 7x memory efficiency, setting a new benchmark for the efficiency-performance trade-off across various tasks.