Cutting Through Overload: Efficient Token Dropping for Speech Emotion Recognition in Multimodal Large Language Models (2025)

Abstract: Recent developments in Multimodal Large Language Models (MLLMs) have provided novel insights into Speech Emotion Recognition (SER). However, combining high-dimensional speech signals with textual tokens can lead to a rapid growth in input tokens, increasing computational costs and inference times. This “token overload” also risks shadowing essential textual cues, affecting the reasoning capabilities of the language model and diluting emotional information crucial to accurate SER. In this paper, we explore different token drop methods that mitigate excessive token counts while preserving both emotional nuances and the core linguistic capabilities of the model. Specifically, we compare various pooling approaches to produce a compact representation. Our preliminary findings suggest that these techniques can reduce computational costs without decreasing SER accuracy.

## introduction speech emotion recognition (ser) has garnered growing interest due to its potential in various applications, including human-computer interaction, mental healthcare, and education. although singlemodality methods, such as text-based emotion analysis or audio emotion recognition, have proven effective @xcite , emotional data in real-world scenarios often integrate multiple modalities. this has led to increased interest in the use of multimodal large language models (mllms) to exploit knowledge from different data sources and improve emotional reasoning @xcite . recent advances in mllms have demonstrated remarkable performance in audio analysis @xcite . however, there are challenges to applying mllms to ser. one of the key obstacles is the rapid increase in the number of multimodal tokens, which drastically expands the size of the input of the model. these multimodal token embeddings can increase computational costs @xcite , prolong inference times, and potentially shade text tokens during the model's attention process @xcite , thus reducing overall performance. to address these limitations, researchers have begun exploring token drop strategies @xcite @xcite @xcite @xcite @xcite , with the aim of ensuring a more balanced and efficient integration of audio and textual information within mllms. this article builds upon these efforts by recognizing that some existing approaches in the literature may become complex due to the large number of parameters or the complexity of the training. in this work, we explore simple pooling methods that help to control the excessive growth of acoustic tokens. by reducing the token overload on the language model, we can preserve its core linguistic capabilities while enhancing its ability to recognize audio-based emotions, crucial for dialogue systems that must handle both textual and emotional cues effectively. we evaluated how these pooling strategies affect computational costs, inference speed, and prediction accuracy, showing new insights into optimizing mllms for ser, improving dialogue systems, and enhancing human-computer interactions. ## related work ser has evolved with the appearance of multimodal approaches and large language models (llms) that incorporate audio inputs. early research often focused on single-modality solutions, either through acoustic features or text-based analysis, to detect emotions. however, these methods struggled to generalize in different contexts and linguistic styles, motivating the development of multimodal systems that merge information from speech, text, and sometimes visual cues @xcite . recent work has shown that integrating text and audio using mllms can produce more robust and nuanced emotion predictions @xcite . afterwards, a series of mllms have emerged, extending the capabilities of llms to handle different input types @xcite . these architectures have shown promising results on tasks ranging from automatic speech recognition to generic audio understanding @xcite . however, many existing mllms either rely on supervised training of additional transformer modules or require extensive fine-tuning on specific downstream tasks, making them computationally expensive and less flexible for broader ser applications. despite progress in audio-based llms, a key challenge remains: the rapid increase in token counts (token overload) when merging highdimensional audio and textual representations. token overload can degrade model performance, increase computational costs, and slow down inference, problems especially salient in real-time or large-scale deployments @xcite . to handle this, a variety of token dropping methods have been proposed. simple statistical techniques, such as mean pooling, can compress feature representations at minimal cost. also, concatenation-based strategies can combine tokens in pairs or in groups to reduce the sequence length @xcite . more complex methods employ n-dimensional convolutions @xcite , lightweight q-formers @xcite , or architectures such as fast conformer @xcite and connectionist temporal classification (ctc) @xcite to remove redundant information. further advancements have been explored in vision frameworks, such as @xcite ; arif et al. ( 2024); shang et al. (2024) that dynamically prune tokens based on attention scores or local content similarity. although originally proposed for image or video tasks, these strategies offer valuable insights for audio-based mllms. in this paper, we focus on evaluating simple and efficient token dropping methods for ser tasks using mllms. rather than relying on complex architectures or parameter-heavy models, we explore straightforward pooling techniques to optimize the token efficiency of multimodal inputs. our approach aims to reduce computational costs and inference times while maintaining the ability of the model to capture emotional nuances. by integrating simplified token reduction modules into a multilingual ser pipeline, we demonstrate that efficient length adaptation techniques can achieve competitive performance. ## methods to address the challenges of ser within multilingual and multimodal contexts, we propose a methodology that integrates high-dimensional speech and text signals into a unified framework. our approach combines an audio encoder based on transformers, a linear projection layer, and an llm, creating a multimodal architecture (see figure 1 for a detailed diagram of the model architecture). we employ the whisper-large-v3 encoder (radford et al., 2022), a state-of-the-art model known for its ability to extract rich phonetic features from audio log-mel spectrograms @xcite . the encoded audio representations are then processed through a linear projec-tor, which changes their dimensionality to align with the embedding space of the llm @xcite . for the text component, we use gemma-2- @xcite , an llm designed to handle diverse linguistic contexts and capable of reasoning over multilingual inputs. we address the token overload challenge by incorporating length adaptation strategies that compress high-dimensional audio embeddings into more compact representations. these strategies range from simple statistical pooling methods, such as mean pooling, to more complex approaches like convolutional (conv) compression @xcite , concatenation (concat) @xcite , and attention mechanisms @xcite . we begin with mean pooling, which aggregates embeddings by straightforward averaging. we then employ conv, using convolutional filters to extract salient features, and a concat approach that pairs tokens, effectively halving the sequence length while doubling the dimensionality. we explore attention-based methods starting with attn-mean, which averages the output of the attention layer, and attn-q-mean, which introduces a global query vector. specifically, instead of deriving the query from each input token, we first perform a mean pooling across the entire sequence of input embeddings x p r lˆd (see equation 1 ). to evaluate our framework, we first develop textonly baseline models. we employ transcriptions generated with whisper-large-v3 from speech input, and we use a frozen gemma-2-2b-it to predict emotions only based on textual information. building on these baselines, we integrate audio features into the pipeline, in which audio embeddings are combined with text tokens. we train only the linear projector and the length adapter layers, ensuring that the llm retains its original capabilities. performance was measured using the f1 macro score and weighted accuracy (wa) to account for class imbalances. for each evaluation, we deployed a 5-fold cross-validation strategy and report the mean f1 macro and wa, along with their standard deviations. we emphasize multilingual ser, using datasets in spanish, german, and french to validate the generalization of our approach using datasets from meacorpus, emodb, and oreau @xcite @xcite . the three datasets contain emotion labels for fear, sad, happy, angry, disgust, and neutral, emodb also includes boredom, while oreau surprise. the meacorpus dataset suffers from imbalanced class distributions, which present additional challenges for robust modeling, although both emodb and oreau have more balanced class distributions. furthermore, the data in meacorpus are derived from natural youtube videos, reflecting real-world, spontaneous emotions, while emodb and oreau datasets consist of acted recordings, which provide more controlled but less naturalistic emotional expressions. our design also prioritizes computational efficiency, enabling faster inference times without compromising accuracy, an essential factor for the deployment of real-world dialogue systems. ## experiments our preliminary experiments focus on selecting the optimal components for the mllm architecture, tuning hyperparameters foot_0 , and refining a prompt for the llm foot_1 . whisper-large-v3 was selected as audio encoder, while a linear projector was chosen for its effectiveness @xcite and simplicity in aligning audio embeddings with the input requirements of the llm. gemma-2-2b-it was chosen as the llm due to its remarkable performance in handling multimodal inputs and reasoning in both text and audio @xcite . to establish ser baselines, we first evaluated a text-only model, where the llm remained frozen and predictions were made solely from the transcriptions of the speech input. this text-only baseline achieved an average f1 macro score of 0.23 and a wa of 0.29 across the three datasets. the integration of audio and text modalities was evaluated through mllms, testing variations in length adaptation strategies. detailed results for each dataset can be found in to address this, we implemented various length adaptation techniques to compress highdimensional audio embeddings. first, simple pooling methods, such as mean pooling, improved performance to a mean wa of 0.70 and an f1 macro of 0.65. next, conv and concat both achieved wa scores of 0.68, with macro f1 scores of 0.63 and 0.56, respectively. finally, attention-based approaches (attn-mean and attn-q-mean) further boosted overall performance. attn-mean achieved the highest metrics, while attn-q-mean also performed strongly. ## conclusion our experiments confirm that integrating audio and text in mllms significantly enhances ser, surpassing text-only approaches in both wa and f1 macro metrics. however, directly merging audio embeddings can lead to "token overload", increasing computational demands and slowing down inference. by incorporating simple length adapters, we achieved significant inference speed-ups (from 22% to 26%) compared to the baseline, while retaining or improving ser accuracy. notably, we only fine-tuned a lightweight projector layer, thereby preserving the language reasoning capabilities of the llm. in future work, we will explore more advanced token compression strategies and extend our experiments to a broader range of tasks and datasets, aiming for higher scalability and robust performance across diverse multimodal dialogue systems.