Benchmarking Contextual and Paralinguistic Reasoning in Speech- LLM s: A Case Study with In-the-Wild Data (2025)

Abstract: Recent speech-LLMs have shown impressive performance in tasks like transcription and translation, yet they remain limited in understanding the paralinguistic aspects of speech crucial for social and emotional intelligence. We propose CP-Bench, a benchmark for evaluating speech-LLMs on contextual paralinguistic reasoning the integration of verbal content with non-verbal cues like emotion and prosody. The benchmark includes two curated question answering (QA) datasets requiring both linguistic and empathetic understanding. We evaluate state-of-the-art speech-LLMs from both open and closed-source models and perform a comprehensive analysis across different question types. The top two models were further analyzed under temperature tuning to understand its effect on this task. Our benchmark reveals a key gap in existing evaluations and offers insights into building more context-aware and emotionally intelligent speech-capable LLMs.

## introduction large language models (llms) have achieved remarkable success in a wide range of natural language processing tasks, and recent advances have extended their capabilities into the multimodal domain, including vision, audio, and speech @xcite . in particular, the integration of speech encoders with text-based llms, referred to as speech-llms, has enabled models to process and reason over spoken language input @xcite . recent developments in speech-llms demonstrate their ability to capture and generate semantically rich textual output from spoken input, achieving state-of-the-art performance in tasks such as automatic speech recognition, speech translation, and spoken question answering. despite this success in modeling linguistic content, current speech-llms largely overlook the paralinguistic dimensions of human speech, such as emotion, intent, speaker identity, and social dynamics, which are essential for fully understanding and interpreting spoken communication. these paralinguistic cues are crucial for applications involving empathy, social reasoning, and human behavior analysis, yet they remain underrepresented in current evaluation benchmarks and model objectives. driven by this potential, researchers have begun developing empathetic speech-llms that not only transcribe or interpret speech but also understand the speaker's affective state and contextual nuances. several studies have attempted to train models to understand emotions in speech and respond empathetically @xcite @xcite . efforts have also extended to data creation; for example, a contextual paralinguistic data generation pipeline has been proposed that leverages llms to automatically produce qa pairs @xcite . existing benchmarks for evaluating speech-llms primarily target isolated contextual tasks like spoken question answering (sqa) or paralinguistic tasks such as emotion recognition and speaker identification @xcite @xcite @xcite . while valuable, these benchmarks fall short in assessing contextual paralinguistic reasoning, which is the integration of speech content, emotional nuance, and speaker-specific traits, which necessitates a more complete understanding of the input. this gap motivates the need for a new benchmark that moves beyond isolated capabilities and evaluates the holistic reasoning ability of speech-llms in multimodal contexts. we introduce the task of contextual paralinguistic question answering (cpqa), which requires reasoning over both linguistic content and paralinguistic cues such as prosody, emotion, and speaker intent. for example, consider the utterance "there are so many people here. i used to come with my sister every sum-mer." without paralinguistic cues, a question like "why was the speaker happy?" is ambiguous. the first phrase might signal excitement or frustration depending on tone, while the latter could convey nostalgia and either sadness or joy. resolving such ambiguity requires interpreting not only the transcript but also the speaker's tone and affective state. this highlights the necessity of prosody-dependent cues for capturing nuances beyond text. such questions are also distinct from direct queries like "what is the speaker's emotion?" as they demand deeper contextual reasoning about causes and states. in short, the evaluations demand a nuanced understanding of both what is said and how it is said. to this end, we propose cp-bench, a novel benchmark specifically designed to assess these capabilities in speech-llms. our benchmark comprises two curated qa datasets derived from long and short in-the-wild speech segments. in addition to constructing and evaluating several state-of-the-art speech-llms on this proposed benchmark, we analyzed the impact of the temperature hyperparameter. temperature controls the randomness of the generated response and significantly influences reasoning quality. while higher temperatures can lead to diverse but potentially inconsistent responses, lower temperatures generally produce more factual and stable outputs. this stability is particularly crucial for tasks involving emotion or gender inference @xcite . to better understand this trade-off, we systematically analyzed how temperature settings affect model performance across different categories of contextual and paralinguistic question answering. our goal is to identify optimal temperature ranges that balance output diversity with reasoning reliability, thereby enhancing the interpretability and robustness of speech-llm responses in multimodal evaluation scenarios. to our knowledge, this work presents the first comprehensive benchmark explicitly targeting reasoning over both context and paralinguistic cues in speech. our main contributions are as follows: our findings reveal critical limitations in current models and offer insights for developing more intelligent, empathetic, and human-aligned speech-llms. cp-bench sets a new direction for multimodal reasoning research at the intersection of speech and language understanding. ## the proposed cp-bench we propose a contextual-paralinguistic question answering benchmark (cp-bench) to evaluate contextual-paralinguistic reasoning capabilities in speech-llm models, using two qa sets derived from long and short speech segments. ## experiments we evaluate state-of-the-art speech-llms on the proposed cp-bench, focusing on their performance across various question types. additionally, we examine how temperature settings influence model behavior, using two of the top-performing models. ## summary and conclusion we propose cp-bench, a benchmark for evaluating speech-llms on contextual paralinguistic reasoning-the integration of verbal content with non-verbal cues like emotion and prosody. the benchmark includes two curated qa datasets that require both linguistic and empathetic understanding. cp-bench challenges speech-llms to demonstrate linguistic comprehension, speaker information, and empathetic insight. our comprehensive evaluation of leading speech-llms, including an analysis of temperature tuning effects, reveals critical strengths and limitations. this work fills an important gap in current benchmarks and provides valuable guidance towards developing more contextually aware and emotionally perceptive speech-based language models. for future work, we plan to expand cp-bench by adding more diverse evaluation sets, including different languages, and by increasing the density of qa pairs for each question type, especially within the multi-speaker subset, to strengthen the statistical reliability of results. we will also explore improved evaluation techniques for such multimodal reasoning, including methods to enhance the robustness and interpretability of llm-based judgments. finally, to further ensure reliability, we plan to incorporate human reviews to calibrate and validate llm-based evaluations.