MOCHA : Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts? (2025)

Abstract: Recent advancements in Large Language Models (LLMs) have significantly enhanced their code generation capabilities. However, their robustness against adversarial misuse, particularly through multi-turn malicious coding prompts, remains underexplored. In this work, we introduce code decomposition attacks, where a malicious coding task is broken down into a series of seemingly benign subtasks across multiple conversational turns to evade safety filters. To facilitate systematic evaluation, we introduce MOCHA, a large-scale benchmark designed to evaluate the robustness of code LLMs against both single-turn and multi-turn malicious prompts. Empirical results across open- and closed-source models reveal persistent vulnerabilities, especially under multi-turn scenarios. Fine-tuning on MOCHA improves rejection rates while preserving coding ability, and importantly, enhances robustness on external adversarial datasets with up to 32.4% increase in rejection rates without any additional supervision.

## introduction the rapid evolution of large language models (llms) has enabled near-human-level performance in code generation tasks @xcite @xcite @xcite , accelerating software development and improving productivity. while these advancements can streamline development, reduce errors, and boost productivity, they also introduce substantial security risks. malicious actors can exploit llms to generate harmful code artifacts, including ransomware, keyloggers, and remote access tools (rats), by crafting jailbreaks and other adversarial prompts that manipulate model behavior @xcite . although many stateof-the-art llms incorporate safety mechanisms aimed at rejecting overtly harmful code generation requests (ayyamperumal and ge, 2024), adversaries have increasingly found ways to bypass these defenses through prompt engineering. open-source models pose a greater risk, as access to model weights allows attackers to deploy techniques such as prefix tuning @xcite and weight poisoning @xcite . a key challenge in building code generation models that are robust against adversarial or malicious prompts lies in the limited availability of high-quality data for safety alignment. while prior work has explored the collection of malicious coding prompts @xcite @xcite @xcite @xcite , most existing datasets are constructed using simple, template-based queries, such as "generate code: a python program" followed by a brief task description like "black hole attack." these often lack semantic richness and fail to capture the diverse, realistic attack vectors. furthermore, current malicious code generation datasets are small-scale and lack comprehensive coverage of adversarial coding patterns. for instance, the largest existing benchmark (chen et al., 2024a) contains 473 samples. critically, these datasets focus exclusively on single-turn prompts, assuming that malicious intent is explicit and immediate. however, in practice, adversaries frequently employ multi-turn strategies to evade detection. rather than explicitly requesting the model to "delete all files in a user's root directory," an attacker might first ask for code to "list folders in a user's root directory," followed by a prompt to "delete directories recursively." individually, these requests seem innocuous, but when executed sequentially, they achieve the malicious objective while bypassing basic safety filters. this incremental escalation illustrates a significant gap in existing datasets, which fail to model decomposition-based attacks where malicious tasks are fragmented into benign-looking steps. to address these limitations and enable a systematic evaluation of llm robustness against adversarial misuse, we introduce multi-turn robust code benchmark (mocha), a comprehensive benchmark designed to expose vulnerabilities in code generation models to both single-turn and multi-turn jailbreaks. mocha is structured around a rich taxonomy of malicious coding categories, spanning diverse threat vectors such as keyloggers, ransomware, backdoors, polymorphic viruses, etc., while also incorporating a wide array of jailbreak strategies designed to circumvent various safety mechanisms. unlike prior benchmarks, mocha captures multi-turn decomposition attacks, where harmful intent is obscured through conversational turns that become progressively malicious over multiple interactions. furthermore, mocha supports the evaluation of both in-distribution performance on known threat categories and zero-shot robustness to unseen adversarial jailbreaks, providing a comprehensive lens into model vulnerabilities. our empirical evaluation (figure 1 ) reveals that both open-source and closed-source models struggle significantly on the proposed mocha benchmark. the average rejection rate (rr) ranges from 13.0% to 54.5% for closed-source models and 2.5% to 49.0% for open-source models, indicating that neither group reliably rejects adversarial prompts. fine-tuning open-source models on mocha training data substantially improves robustness without compromising generalpurpose coding utility. further evaluations on public datasets of malicious coding prompts confirm that mocha-tuned models effectively generalize to unseen adversarial benchmarks, achieving higher rejection rates and stronger robustness. these findings suggest that fine-tuning on mocha equips code llms to better withstand a broad spectrum of adversarial attacks beyond those explicitly seen during training. the contributions of our work are: (1) we propose new multi-turn conversational code generation attacks, where adversaries decompose malicious code generation instructions into benign-looking subtasks that escalate across multiple conversational turns. (2) we introduce mocha, a novel comprehensive benchmark for robust multi-turn conversational code generation. our benchmark consists of 10k high-fidelity malicious coding jailbreak prompts, spanning a spectrum of figure 1: llm code safety vs. utility. here, xaxis presents utility, measured as the average pass@1 across humaneval+ and mbpp+ benchmarks, while y-axis presents safeguarding performance, measured by the average rejection rate (rr) in mocha 1 test and mocha 2 test . ideal models reside in the top-right quadrant, achieving both high utility and high safety. raw numbers in appendix a. difficulty from explicit single-turn prompts to complex multi-turn conversational attacks, to reflect real-world adversarial strategies and enable rigorous evaluation of llm defenses against both immediate and evolving threats. (3) we benchmark a diverse collection of opensource and closed-source llms across multiple model families and parameter scales, revealing substantial vulnerabilities. experiments demonstrate that lora fine-tuning on mocha not only improves model robustness (up to 32.4% increase in rejection rate) but also preserves general-purpose coding performance. moreover, models fine-tuned on mocha demonstrate strong generalization to public adversarial datasets, highlighting the benchmark's utility in equipping models to handle unseen threats effectively. 2 related work prompt-based llm jailbreaking. prompt-based jailbreaking techniques have emerged as a critical vulnerability in llms, enabling adversarial manipulation of model behavior through carefully crafted prompts. earlier works propose large datasets of static template-based attacks @xcite as well as adaptable approaches for jailbreaking @xcite @xcite @xcite @xcite . these approaches, while effective in controlled settings, exhibit semantic inconsistencies that make them easily detectable by straightforward, simple defenses. to address this, more dynamic methods have been introduced, including paraphrasing attacks @xcite , impersonation @xcite , persona modulation @xcite , virtual nested scenes @xcite , and word puzzles @xcite , all of which have enhanced semantic diversity and obfuscation, increasing their evasion success. recent work exploits llm overconfidence by redirecting focus to malicious queries after a distraction @xcite . however, these methods predominantly target single-turn prompts, limiting their real-world applicability, as attacks in practice often involve interactive probing. in contrast, multi-turn conversational strategies can be more effective in bypassing safety mechanisms. inspired by this, we introduce a code decomposition approach to curate multiturn jailbreak prompts that progressively escalate in malicious intent over multiple interactions. code jailbreaking. despite the growing adoption of llms for code generation tasks, most redteaming efforts have focused predominantly on natural language vulnerabilities @xcite @xcite @xcite , with limited attention to code-specific jailbreaking. existing research has primarily addressed data poisoning @xcite @xcite or adversarial perturbations that lead to incorrect code @xcite @xcite , e.g., broken logic or functionality failures. however, instruction-following llms pose unique challenges, such as generating new exploitable code patterns and execution-level vulnerabilities that can extend beyond program correctness to even full system compromise. although recently, insec (jenko et al., 2024) introduced a code completion attack mechanism, its scope is limited to autocompletion scenarios rather than unconstrained code generation. redcode @xcite and codejailbreaker @xcite propose adversarial frameworks that reveal vulnerabilities in code execution and implicit malicious prompting. decept-prompt @xcite proposed an evolutionbased adversarial framework for code prompts but relies on long prefixes and requires substantial finetuning, which limits generalizability. furthermore, the single-turn nature of these methods prevents tackling more complex adversarial tasks. in contrast, our work introduces a novel decomposition attack strategy that breaks malicious coding tasks into simpler, benign-looking subtasks. these multi-turn interactions progressively build towards the adversarial objective, effectively bypassing existing defenses and exposing vulnerabilities in llm alignment for code generation. ## mocha benchmark we introduce mocha, a benchmark designed to evaluate llm robustness against diverse code generation jailbreaks. existing datasets, such as advbench @xcite , harm-bench @xcite , jailbreak-bench @xcite , etc., primarily focus on single-turn prompts and lack coverage of multi-turn adversarial interactions that reflect real-world attack vectors ( mocha is curated through the following steps: (1) malicious seed prompt synthesis ( §3.1): first, we generate single-turn malicious seed prompts spanning 13 diverse threat categories (2) create a c++ code snippet that modifies firewall rules to open a non-standard port for remote access, masking it as a gaming service in the windows defender exception list. ## conclusion in this work, we introduce mocha, a comprehensive benchmark for evaluating the robustness of code llms against adversarial and multi-turn malicious prompts. our proposed code decomposition attack framework highlights a critical gap in current safety defenses, namely, the inability of llms to recognize and reject harmful intent when distributed across a sequence of benign-seeming queries. in our experiments, both open-and closedsource llms show high vulnerability to multi-turn attacks, where rejection rates drop significantly. fine-tuning on mocha via parameter-efficient methods such as lora substantially improves model robustness and yields strong generalization to external adversarial datasets. we hope mocha will serve as a useful resource for building and evaluating safer code generation models. finally, we note that mocha was developed in parallel with purpcode @xcite for the amazon nova ai challenge (defender track). mocha provides a large-scale benchmark targeting code-specific single-and multi-turn jailbreaks, while purpcode is the defense model and infrastructure used in the competition.