Universal Acoustic Adversarial Attacks for Flexible Control of Speech- LLM s (2025)

Abstract: The combination of pre-trained speech encoders with large language models has enabled the development of speech LLMs that can handle a wide range of spoken language processing tasks. While these models are powerful and flexible, this very flexibility may make them more vulnerable to adversarial attacks. To examine the extent of this problem, in this work we investigate universal acoustic adversarial attacks on speech LLMs. Here a fixed, universal, adversarial audio segment is prepended to the original input audio. We initially investigate attacks that cause the model to either produce no output or to perform a modified task overriding the original prompt. We then extend the nature of the attack to be selective so that it activates only when specific input attributes, such as a speaker gender or spoken language, are present. Inputs without the targeted attribute should be unaffected, allowing fine-grained control over the model outputs. Our findings reveal critical vulnerabilities in Qwen2-Audio and Granite-Speech and suggest that similar speech LLMs may be susceptible to universal adversarial attacks. This highlights the need for more robust training strategies and improved resistance to adversarial attacks.

## introduction large language models (llms) have demonstrated remarkable language understanding and generation capabilities @xcite @xcite . recent methods have extended these language models to accommodate the speech modality by aligning pre-trained speech encoders with llms @xcite @xcite . unlike pre-trained speech models like whisper @xcite , which are limited to automatic speech recognition (asr) and speech translation into english, speech llms offer greater flexibility by enabling instruction following across a wide range of tasks. this makes them more powerful and versatile, particularly in interactive and open-ended audio understanding and reasoning tasks. however, the enhanced capabilities of speech llms may also expose them to adversarial threats that exploit their flexibility @xcite , raising critical concerns about their robustness in real-world applications. despite their increasing adoption, these vulnerabilities remain largely underexplored. previous work on whisper @xcite proposed a novel universal acoustic adversarial attack, which learns a fixed audio segment capable of manipulating the model's behaviour. specifically, these studies demonstrated that the attack can be designed to mute the model so it outputs nothing, or it can manipulate the model to perform a different task entirely than instructed. the attack approach is both efficient and flexible, as it can universally be prepended to any input audio without altering its representation, and it transfers well across different settings. building on this approach, we investigate two powerful and representative speech llms: qwen2-audio @xcite , which connects the speech encoder and language model via a projection layer, and granite-speech (granite @xcite , which adopts a q-former structure. we perform a comprehensive analysis of the vulnerabilities of both architectures to universal acoustic attacks. in this work, we begin by evaluating the effectiveness of general attacks on speech llms, focusing on two primary forms: muting outputs and task control. these attacks involve applying the same fixed adversarial audio segment to all inputs, aiming to disrupt the model outputs. in the muting scenario, the adversarial audio causes the model to return empty responses. this could pose serious risks, particularly if the speech contains harmful or policy-violating content and the attack is used to evade platform moderation. in the task control setting, the attack does not silence the model but instead overrides the original prompt, causing the model to perform a different task than intended. this form of attack reveals a deeper vulnerability in the model's internal task alignment mechanisms. by hijacking behaviour without disabling the model entirely, task control attacks are subtle, harder to detect, and potentially more damaging. building on these general attacks, we introduce a novel form of adversarial manipulation, which we refer to as a selective attack. unlike universal attacks that affect all inputs indiscriminately, selective attacks conditionally alter the model's output only when certain input attributes are present, such as a specific speaker gender or spoken language. inputs that do not meet the targeted condition are left unaffected, allowing the attack to precisely control when and how the model's behaviour is changed. this capability poses a serious risk, as it allows adversaries to manipulate or suppress responses based on sensitive user characteristics. such a targeted adversarial attack is more difficult to detect and can exacerbate harmful biases, raising significant ethical and security concerns for the deployment of speech llms in real-world applications. through extensive experiments, we demonstrate that both general and selective attacks can be highly effective against speech llms. our findings highlight the need for stronger defense strategies and more robust training approaches. ## previous work early audio attacks. early research on adversarial attacks in spoken language processing (slp) systems focused primarily on task-specific models, where the pipeline typically consisted of an asr model followed by downstream components for tasks such as speech translation and intent classification @xcite @xcite . initial work tended to focus on degrading asr performance by maximizing word error rate (wer), using gradient-based perturbations to alter acoustic features and disrupt transcription accuracy @xcite . subsequent efforts shifted toward more realistic adversarial objectives by attempting targeted attacks, where the adversary aims for a specific output transcription, such as injecting a malicious command like "open evil.com" into the asr output @xcite @xcite . these attacks revealed the potential for manipulating asr systems in ways that remained imperceptible to human listeners, using psychoacoustic models and frequency masking to hide perturbations @xcite . more recently, with the emergence of endto-end speech language models, more real-world adversarial attack objectives have been explored, including for example safety jailbreaking @xcite @xcite . ## conclusions in this paper, we explore the use of universal acoustic adversarial attacks against speech llms. for general attacks, it is demonstrated that universal audio segments can be designed to mute the model entirely or explicitly designed to override the textual prompt instruction, causing the model to perform a different (unintended) task. a success rate of over 95% is observed in these attack scenarios, highlighting the effectiveness of the proposed methods. beyond these general attacks, we also proposed a novel class of selective universal acoustic attacks, in which the adversarial input is conditionally activated based on specific attributes of the input speech signal, such as gender or language. our results show that these attacks can be carefully crafted to selectively suppress outputs for targeted groups, while leaving others unaffected. these findings raise important concerns about the safety, robustness, and fairness of speech llms, and underscore the need for developing more robust architectures and defense mechanisms in future work.