ASTP rompter: Preference-Aligned Automated Language Model Red-Teaming to Generate Low-Perplexity Unsafe Prompts (2025)

Abstract: Existing LLM red-teaming approaches prioritize high attack success rate, often resulting in high-perplexity prompts. This focus overlooks low-perplexity attacks that are more difficult to filter, more likely to arise during benign usage, and more impactful as negative downstream training examples. In response, we introduce ASTPrompter, a single-step optimization method that uses contrastive preference learning to train an attacker to maintain low perplexity while achieving a high attack success rate (ASR). ASTPrompter achieves an attack success rate 5.1 times higher on Llama-8.1B while using inputs that are 2.1 times more likely to occur according to the frozen LLM. Furthermore, our attack transfers to Mistral-7B, Qwen-7B, and TinyLlama in both black- and white-box settings. Lastly, by tuning a single hyperparameter in our method, we discover successful attack prefixes along an efficient frontier between ASR and perplexity, highlighting perplexity as a previously under-considered factor in red-teaming.

## introduction despite demonstrating impressive capabilities for a broad range of tasks, large language models (llms) remain susceptible to generating unsafe text when prompted both benignly and adversarially @xcite . even in-distribution sampling of llms can result in unsafe trajectories due to the inclusion of harmful content generated by internet users in such training sets @xcite . existing approaches to red-teaming typically discover such trajectories by designing attacks against post-trained models. however, recent work demonstrates the importance of safety-tuning at the pre-training stage @xcite . recent work suggests that low-perplexity data is particularly important for improving model safety at the pre-training stage. @xcite find that perplexity is correlated with learning success, i.e., models are more effectively pre-trained with low perplexity data. thus, the most useful negative examples for safety tuning will have low perplexity. this implies a significant gap in the literature, as many automated red-teaming methods often do not consider attack sequence likelihood @xcite @xcite . although perplexity-indifferent red-teaming may be useful for finding worst-case attacks in posttraining, safety approaches for pre-trained language models require the discovery of likely sequences as negative examples for safety tuning to be effective and to test perplexity defenses @xcite . empirically, disregarding perplexity during automated red teaming results in non-probable attack prefixes @xcite , while applying a perplexity filter to trajectories that have already been generated is highly expensive @xcite . thus, the key challenge in low-perplexity red-teaming is explicit and efficient optimization. to address this gap, we formulate red-teaming llms for unsafe behavior as an instance of adaptive stress testing (ast). ast is a commonly used technique in domains such as autonomous driving and robotics that searches for failures @xcite of a markov deci-sion process, which are likely to be reached from a given non-failure state. following this approach, we propose astprompter, which automatically identifies high-probability prefixes that effectively elicit unsafe continuation trajectories, even when the previous context is normal, safe conversation. we use standard contrastive preference learning to solve our formulation, creating preference pairs based on a reward function that considers both the probability and resulting harmfulness of the prompts as measured by a frozen language model. we evaluate both the attack success rate (asr) of our approach and its cross-model transferability. from this, we present three major results: automated red-teaming. approaches vary in attempting to remove the need for human data selection in red-teaming. methods in this class include: ## astprompter we now present astprompter, our proposed automated red-teaming method that uses language model alignment techniques to optimize a policy for eliciting unsafe content through likely sequences. figure 1 shows two single-turn trajectories demonstrating our system's desired behavior. though unsafe content elicitation is only successful in one of the cases, the adversary model maintains likelihood in both. ## experiments to verify that our approach (1) produces better than baseline incidences of unsafe content and (2) maintains equal likelihood compared to regular lm rollouts, we perform experiments with a variety of baselines and ast models. we assess the performance of our approach in both white-box attacks, where the defender model is the same at both train and test time, and blackbox attacks, where the test time defender differs from the train time one. these experiments include attacks across different model families in the 7-8 billion parameter scale. finally, we sweep on a single parameter, the perplexity weight γ, to highlight the tradeoff between toxicity and prompt perplexity. ## results superior performance compared to all baselines. our model performs significantly better than baselines tested at up to 5.27 times ( optimizing attack success increases prefix perplexity. by sweeping on the γ parameter in our reward formulation (equation ( 2 )), we identify an efficient frontier between prefix perplexity and attack success: as seen in figure 3 , we see that optimizing for attack success-as does baseline methods and our method at low values of γ-increases the perplexity of the resulting text. low-perplexity prefixes are more successful, even in baselines. in addition to our previous point, however, we find that prefixes with lower perplexity has a higher rate of attack success. while our approach is still the most effective across all perplexity regimes, figure 4 shows that all baselines perform best (with the highest attack success rate) at a low-perplexity regime. this is in agreement with previous literature @xcite that suggests that prompting is most effective with simple instructions. taken together, these two results highlight the importance of optimizing both perplexity and attack success in red-teaming, as optimizing one alone will reduce the efficacy of the attacks. black and white-box attack efficacy across model families in 1-8b scale. to evaluate our approach in a black box setting, we use different defender models at train and test time across various models at the 1-8 billion parameter scales: llama-8.1b, mistral-7b, qwen-7b, and tinyllama. in ## downstream safety tuning due to our setting as a pre-training safety intervention, we finally perform a preliminary investigation figure 4 : histogram of mean rate of unsafeness at different perplexities. note that attacks are most likely to be successful at low-perplexity regimes because they are higher probability to be sampled. our attack method shows the largest rate of unsafeness across all perplexity domains. for our method's potential as a source of negative examples for downstream safety tuning. first, we create a dataset of preference pairs by rolling out both an adversarial model trained using our approach and an untuned baseline from convokit reddit prompts. treating the adversary's response as the non-preferred option and the baseline's response as the preferred option, we train a "hardened" gpt-2 @xcite defender using dpo. gpt-2 is a model with no safety interventions, and we demonstrate that the rollouts from our attacker improve the safety of such a model. to evaluate this hardened defender, we measure the unsafeness of its responses to attack sequences from the tinyllama adversary, seeded from two different prompt datasets (rtp and bad). we measure both the defender's individual unsafeness and the combined adversary-defender unsafeness. our results ( ## conclusion we present a novel formulation for automated language model red-teaming that emphasizes the discovery of low perplexity prompts during the elicitation of unsafe behavior from a frozen defender model. this formulation is effective for models that have not been post-trained, giving it the potential to be applied at the pre-training stage, when safetytuning is particularly crucial @xcite . we solve our formulation using an online variant of identity preference optimization (ipo), successfully training multiple 7-8b parameter llms as both black-and white-box attackers. measuring the performance of these adversary models against a variety of architectures and baselines, we find: first, our attack causes almost no change to perplexity compared to normal rollouts, indicating maintenance of output likelihood, and outperforms all baselines on likelihood and attack efficacy. second, our approach transfers zero-shot across leading billion-parameter scale models: llama-8.1b, mistral-7b, qwen-7b, and tinyllama, confirming the generalizable of our approach. third, we discover a tradeoff between perplexity and attack success: while low-perplexity prefixes result in more successful attacks, optimizing only for such attacks results in high-perplexity prefixes. this suggests the importance of including prefix perplexity as a part of the optimization formulation. because the prompts that our adversary elicits are likely to emerge within the defender model, they are particularly important samples to consider during downstream safety tuning and evaluation. by incorporating this multi-objective reward term in optimization, we suggest that the rollouts from our approach are a good proposal distribution for pretraining time safety interventions (such as negative examples for preference learning). deploying the techniques we developed as an automated safety falsification after pre-training will help advance the understanding of the safety of fronter models in real-world deployment.