Exploiting Prompt-induced Confidence for Black-Box Attacks on LLM s (2025)

Abstract: Large language models (LLMs) are vulnerable to adversarial attacks even in strict black-box settings with only hard-label feedback.Existing attacks suffer from inefficient search due to lack of informative signals such as logits or probabilities. In this work, we propose Prompt-Guided Ensemble Attack (PGEA), a novel black-box framework that leverages prompt-induced confidence, which reflects variations in a model’s self-assessed certainty across different prompt templates, as an auxiliary signal to guide attacks. We first demonstrate that confidence estimates vary significantly with prompt phrasing despite unchanged predictions. We then integrate these confidence signals in a two-stage attack: (1) estimating token-level vulnerability via confidence elicitation, and (2) applying ensemble word-level substitutions guided by these estimates. Experiments on LLaMA-3-8B-Instruct and Mistral-7B-Instruct-v0.3 on three classification tasks show that PGEA improves the attack success rate and query efficiency while maintaining semantic fidelity. Our results highlight that verbalized confidence, even without access to probabilities, is a valuable and underexplored signal for black-box adversarial attacks. The code is available at https://github.com/cmn-bits/PGEA-main.

## introduction large language models (llms) have achieved remarkable success across diverse nlp tasks such as text generation @xcite , classification @xcite , and question answering @xcite . however, recent studies reveal that llms remain vulnerable to adversarial attacks, especially in black-box settings where only the final label is observable and internal states such as logits or probabilities are inaccessible @xcite . ## preliminaries despite extensive pre-training @xcite and alignment efforts @xcite aimed at bringing llms closer to human-level performance, recent work has shown that they still exhibit vulnerabilities against adversarial attacks. we briefly review relevant concepts on adversarial attacks in nlp, token-level perturbations, and confidence elicitation in large language models. ## proposed method we introduce pgea, a black-box adversarial framework that leverages prompt-induced confidence estimation combined with multiple perturbation strategies to improve attack effectiveness and transferability on llms. ## conclusion we propose prompt-guided ensemble attack (pgea), a black-box adversarial method that leverages prompt-induced confidence signals to guide ensemble perturbations. by combining verbal uncertainty elicitation with word-level attacks, pgea effectively identifies vulnerable tokens and enhances attack transferability. experiments on three datasets show consistent improvements over baseline methods. our findings also reveal that prompt phrasing plays a crucial role in shaping model confidence, highlighting new opportunities for prompt-based control and evaluation of large language models. future work includes extending this framework to multilingual models and exploring adaptive prompting strategies for realtime adversarial defense.