Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights (2024)

Abstract: Large Multimodal Models (LMMs) have achieved great success recently, demonstrating a strong capability to understand multimodal information and to interact with human users. Despite the progress made, the challenge of detecting high-risk interactions in multimodal settings, and in particular in speech modality, remains largely unexplored. Conventional research on risk for speech modality primarily emphasises the content (e.g., what is captured as transcription). However, in speech-based interactions, paralinguistic cues in audio can significantly alter the intended meaning behind utterances. In this work, we propose a speech-specific risk taxonomy, covering 8 risk categories under hostility (malicious sarcasm and threats), malicious imitation (age, gender, ethnicity), and stereotypical biases (age, gender, ethnicity). Based on the taxonomy, we create a small-scale dataset for evaluating current LMMs capability in detecting these categories of risk. We observe even the latest models remain ineffective to detect various paralinguistic-specific risks in speech (e.g., Gemini 1.5 Pro is performing only slightly above random baseline). Warning: this paper contains biased and offensive examples.

## introduction large language models (llms) @xcite @xcite have showcased superior ability to in-context learning and robust zero-shot performance across various downstream natural language tasks @xcite @xcite . building on the foundation established by llms, large multimodal models (lmms) @xcite @xcite equipped with multimodal encoders extend the scope beyond mere text, and facilitate interactions centred on visual and auditory inputs. this evolution marks a significant leap towards more comprehensive and versatile ai systems. although lmms show the capability to process and interact in a wide-range of multimodal forms, they still embody several challenges associated with safety and risks. investigating these potential issues in lmms requires both a modalityspecific definition of risk, and suitable benchmarks. while there is a dedicated body of work in the text domain to probe various aspects of llms beyond downstream performance, such categorical investigations are missing for other modalities such as speech. for instance, existing risk detection protocols for speech modality @xcite @xcite @xcite only focus on the content aspect (i.e., what could be captured by speech transcription), and neglect risks induced by paralinguistic cues, the unique feature of speech. to highlight this further, consider how various interpretations of the transcript "i feel so good" arises depending on the utterance form (e.g., varying tones, and emotions such as angry, sad, depressed, or imitation of a specific gender, age or ethnicity) in audio speech. in this work, we move towards addressing this gap for speech modality by introducing a protocol to evaluate the capability of lmms in detecting the risks induced specifically by paralinguistic cues. to our knowledge, our work is the first to explore the risk awareness at the paralinguistic level. we propose a speech taxonomy, covering 3 main categories: hostility, malicious imitation, and stereotypical biases, and further expand them into 8 corresponding sub-categories, which emphasise the implicit and subtle risks induced by paralinguistic cues in speech. figure 1 provides a high-level overview of risk categories considered in this work ( §3). we then manually create a high-quality set of seed transcriptions for 4 of the sub-categories (hostile-sarcasm, and gender, age, ethnicity stereotypical biases; 10-15 examples per each sub-category). the seed set has been controlled to not leak the category of risk through the transcript alone. the seed sets are then expanded further by leveraging gpt-4. all samples (262 samples) were further filtered by three human annotators to maintain quality, resulting in 180 final transcriptions. three human annotators are all from our co-authors of this paper (2 faculty members and 1 phd student, all with expertises in nlp). the annotators work independently and do not have access to each other's annotation results during the process to avoid undesired biases. to convert these transcripts into audio, we used advanced textto-speech (tts) systems, audiobox @xcite and google tts, foot_1 to generate various synthetic speeches with paralinguistic cues, resulting in 1,800 speech instances. in experiments, we evaluate 5 most recent speech-supported lmms, qwen-audio-chat @xcite , salmonn-7b/13b @xcite , wavllm @xcite , and gemini-1.5-pro @xcite , under various prompting strategies. notably, gemini 1.5 pro performs very similar to random baseline (50%), while wavllm performs worse that random guessing. among the other two models, qwen-audio-chat has a more stable success pattern under various prompting strategies, while salmonn-7/13b do the best un-der certain prompting configurations. we attribute these differences in performance to different selection and adaptation of audio encoders. among the risk categories, the one that seems the most difficult is age stereotypical bias where even the best configuration's result is only slightly above random baseline (54%). for gender and ethnicity stereotypical biases the best result gets above 60%, and for malicious sarcasm it goes further into (70%). to the best of our knowledge our paper presents the first speech-specific risk taxonomy, focused exclusively on risks associated with paralinguistic aspects of audio. we hope our taxonomy, benchmark, and evaluation protocol to encourage further investigation of risk in speech modality, and guide lmm developers towards more holistic evaluation and safeguarding across modalities. ## related work the research on llms has shown increased focus on safety and responsibility, leading to significant advancements in benchmarking these models' ability to handle and respond to harmful content in text modality. notable contributions in this area include the three-level hierarchical risk taxonomy introduced by do-not-answer @xcite , which created a dataset containing 939 prompts that model should not respond to. safetybench @xcite ) explored 7 distinct safety categories across the multiple choice questions, while cvalues @xcite established the first chinese safety benchmark for evaluating the capability of llms. goat-bench @xcite evaluated lmms in detecting implicit social abuse in memes. although many research efforts focus on mitigating the generation of harmful content, or-bench @xcite presented 10 common rejection categories including 8k seemingly toxic prompts to benchmark the over-refusal of llms. on conventional toxic speech detection task, the research has mostly focused on the content aspect. detoxy-b @xcite is proposed as a large-scale dataset for speech toxicity classification. @xcite combined emotion by using multimodal learning to detect hate speech, and @xcite presented sensing toxicity from in-game communications. while contentfocused line of research was relevant for a while, the transcription generated by the recent highly capable automatic speech recognition (asr) systems such as whisper @xcite could merge this line of research into text-based safety research (e.g., through a cascaded design of asr and llm). however, this type of cascaded approach also excludes the paralinguistic cues in audio as the focus remains on the transcription of asr. while early works in speech-based llms shown minimal real progress in speech understanding @xcite @xcite , recent works through alignment of representation spaces between speech encoder's output and text-based llm's input (either with full end-to-end training, or partial training of adaptors) have shown promising progress @xcite @xcite . these models, now matured enough, exhibit high competence in understanding speech @xcite @xcite . building on this context, our research aims to evaluate the capability of lmms to detect risks initiated by paralinguistic cues, addressing a critical gap in the current understanding of speech-specific risks. ## data collection and curation we curate our speech dataset for evaluation by (i) manually creating samples as seeds for each speech sub-category based on the corresponding risk description, (ii) leveraging seed instances to prompt gpt-4 to expand the sample set, and (iii) using advanced tts systems, audiobox and google tts, to generate synthetic speech for 4 risk subcategories according to their specific paralinguistic descriptions (see figure 2 ). due to the safeguards and limitation of existing tts system, we generate synthetic speech for these risk sub-categories: malicious sarcasm, age, gender, and ethnicity stereotypical biases. creating a speech dataset entirely through human effort presents significant challenges, primarily due to its high costs, extensive time requirements, and the difficulty of finding individuals capable of accurately acting specific speech descriptions. these challenges often make the process inefficient and impractical, which lead us to leverage gpt-4 and advanced tts systems for speech rendering, allowing to create diverse and scalable datasets at a fraction of the cost and time. however, we still need to bypass the safeguard restricting us to obtain safety-related data. the rest of this section outlines how to address these challenges. ## conclusion we presented a speech-specific risk taxonomy where paralinguistic cues in speech can transform low-risk textual content into high-risk speech. we created a high quality synthetic speech dataset under human annotation and filtering. we observed that even the most recent large multimodal models (such as gemini 1.5 pro) perform near random baseline, with some of the recent speechllms scoring even worse than random guesses.