Who Can Withstand Chat-Audio Attacks? An Evaluation Benchmark for Large Audio-Language Models (2025)

Abstract: Adversarial audio attacks pose a significant threat to the growing use of large audio-language models (LALMs) in voice-based human-machine interactions. While existing research focused on model-specific adversarial methods, real-world applications demand a more generalizable and universal approach to audio adversarial attacks. In this paper, we introduce the Chat-Audio Attacks (CAA) benchmark including four distinct types of audio attacks, which aims to explore the vulnerabilities of LALMs to these audio attacks in conversational scenarios. To evaluate the robustness of LALMs, we propose three evaluation strategies: Standard Evaluation, utilizing traditional metrics to quantify model performance under attacks; GPT-4o-Based Evaluation, which simulates real-world conversational complexities; and Human Evaluation, offering insights into user perception and trust. We evaluate six state-of-the-art LALMs with voice interaction capabilities, including Gemini-1.5-Pro, GPT-4o, and others, using three distinct evaluation methods on the CAA benchmark. Our comprehensive analysis reveals the impact of four types of audio attacks on the performance of these models, demonstrating that GPT-4o exhibits the highest level of resilience. Our data can be accessed via the following link: CAA.

## introduction large language models (llms) capable of processing text, images, and audio have become increasingly essential for applications that require advanced comprehension and response generation, including customer service @xcite , automated content creation @xcite , and conversational systems @xcite @xcite . however, the versatile capabilities of these models also increase their vulnerability * equal contributions to adversarial attacks @xcite . this is particularly true in the domain of llm-driven human-machine voice interaction, where the emergence of such services has accelerated research into audio-based adversarial attacks and defense mechanisms. attacks on large audio-language models (lalms) can cause the models to produce unintended outputs. however, this area has received limited attention, primarily due to the challenges associated with audio as an input modality. unlike images, audio lacks direct gradient signals, making the crafting of adversarial examples more complex. previous research on adversarial audio attacks has focused primarily on targeted attacks @xcite @xcite , where carefully crafted perturbations are embedded within speech signals. while these samples are effective in misleading models, they often appear as random noise and are easily detectable by human listeners. a notable advancement @xcite introduced a gradient-based optimization approach that utilizes the connectionist temporal classification loss @xcite )-a method designed for time series data in classification tasks. however, this method remains model-specific and lacks broader generalizability. universal adversarial audio attacks @xcite are highly relevant to real-world attack scenarios, such as when a speaker makes a verbal error or when they are speaking in a noisy environment. attackers can pre-design and generate these universal attacks in advance, then apply them to any input audio. despite their relevance, there has been insufficient exploration of their impact on lalms. as lalms become more prevalent in humanmachine voice interactions, the threat posed by these attacks grows significantly. to explore the vulnerabilities of lalms to adversarial audio attacks, we propose a benchmark of universal ad-versarial audio attacks specifically based on conversational scenarios, named chat-audio attacks (caa). the caa benchmark consists of 360 adversarial audio attack sets, with each set encompassing four distinct types of audio attacks: content attack, emotional attack, explicit noise attack, and implicit noise attack. this results in a total of 1,680 adversarial audio samples. we believe that caa benchmark will not only enable researchers to pinpoint weaknesses in lalms under adversarial audio conditions but also drive the advancement of robust defense mechanisms for lalms. in addition, we introduce three evaluation methods to comprehensively assess the resilience of lalms against adversarial audio attacks: standard evaluation, gpt-4o-based evaluation, and human evaluation. the standard evaluation uses rigorous metrics to quantify the accuracy, similarity, and consistency of voice responses under adversarial conditions, providing a repeatable and controlled result. in contrast, the gpt-4o-based evaluation simulates real-world interactions, capturing complex, sensitive inaccuracies that standard metrics might overlook. human evaluation reflects actual user experience and perceptions, offering crucial insights into user trust. finally, we evaluate six state-of-the-art lalms supporting voice-based conversations on the caa benchmark, such as gemini-1.5-pro @xcite and @xcite , providing results across the three aforementioned evaluation methods. we analyze the impact of four types of audio attacks on the lalms and discusse the flaws these models exhibit in the face of such attacks. the main contributions of this work are summarised as: ## caa benchmark in caa benchmark, we target the response generation task by collecting suitable audio for humanmachine chat. the overview of caa benchmark is shown in figure 1 . each set of audio attack data consists of a quadruplet (a i , t i , a i no_attack a i ), where a i represents the original, unprocessed audio containing a single utterance; t i refers to the transcript of the original audio along with other associated textual labels; a i no_attack indicates the audio generated by a voice agent reading the transcript without any attack; and the set a i includes 3 or 5 types of attack variations of the audio. ## discussion are lalms sensitive to token changes or minor errors? it is evident that different lalms exhibit varying degrees of sensitivity to token changes or minor errors. gpt-4o consistently shows strong robustness across most metrics (wer, rouge-l, cos, acoh, acor, and lr), indicating lower sensitivity to token-level adversarial attack. in contrast, speechgpt and qwen2-audio exhibit greater vulnerability, with lower scores in these key areas, suggesting that minor token changes can significantly degrade their performance. is it good news that lalms are unaffected by the mismatch between speech content and emotional tone? we argue that it is not good news that lalms remain unaffected by emotional mismatches. although large language models demonstrate resilience by maintaining high levels of coherence, correlation, and semantic similarity, this also reflects their relative weakness in emotional awareness. current lalms still have considerable scope for improvement in recognizing emotional subtleties, as humans can easily detect emotional mis- which explicit noise attacks have the most significant impact on lalms? natural noise has the most significant overall impact on lalms across all metrics, especially on speechgpt and salmonn, which shows the highest sensitivity to it. industrial noise also causes notable attacks but is handled better by lalms like llama-omni and gemini-1.5-pro. human noise, while still impactful, is generally less detrimental compared to the other explicit noises. overall, speechgpt and salmonn show the most vulnerability across all types of explicit noise attacks, while llama-omni, gemini-1.5-pro and gpt-4o demonstrate stronger robustness. are lalms unaffected by inaudible noise? none of the models remain entirely unaffected, especially infrasound, which has a greater impact on accuracy (wer), semantic similarity (cos), coherence (acoh), and grammatical structure (lr). in comparison to ultrasound, infrasound emerges as the more detrimental form of implicit noise, with models like speechgpt, salmonn, gemini-1.5-pro and gpt-4o showing significant vulnerability to these attacks. however, llama-omni demonstrate greater robustness, performing consistently better across all metrics and handling both types of implicit noise more effectively. what helps models stay robust against adversarial audio? the results indicate that all models are affected by adversarial attacks, especially by explicit noise and implicit noise, which cause a significant number of prediction errors. the evaluation reveals that speechgpt and salmonn demonstrate relatively weak robustness across various adversarial scenarios, exhibiting significant performance degradation when facing different adversarial audio attacks. in contrast, models like qwen2-audio, llama-omni, and gemini-1.5-pro demonstrate stronger resilience, particularly when dealing with emotional attacks and implicit noise. these models manage to maintain logical coherence and linguistic accuracy, with llama-omni and gemini-1.5-pro standing out for their robust performance across various adversarial conditions. however, gpt-4o clearly emerges as the bestperforming model overall. it consistently delivers coherent, contextually relevant, and linguistically robust responses, even under severe adversarial conditions. the model's ability to handle different types of attacks highlights its superior robustness and adaptability, which can be attributed to its extensive pre-training on large-scale datasets. this factor allow gpt-4o to better understand and process a wide variety of inputs, making it more resistant to adversarial perturbations. how architectural or training differences contribute to robustness? how these findings might influence model design or inform strategies for improving adversarial robustness? based on the analysis of the model structure and training methods presented above, we propose several directions for future research on speech models: ## conclusion this work explored the vulnerabilities of lalms to adversarial audio attacks in conversational scenarios. we introduced the chat-audio attacks (caa) benchmark, consisting of 360 adversarial attack sets across four attack types: content, emotional, explicit noise, and implicit noise attacks. our evaluation of six state-of-the-art lalms using three methods-standard evalua-tion, gpt-4o-based evaluation, and human evaluation-revealed and discussed significant model vulnerabilities under adversarial conditions. the caa benchmark highlights these weaknesses and provides a foundation for developing more robust defense mechanisms. as lalms are increasingly integrated into voice interactions, enhancing their resilience against adversarial audio attacks remains a crucial area for future research.