Localizing Malicious Outputs from C ode LLM (2025)

Abstract: We introduce FreqRank, a mutation-based defense to localize malicious components in LLM outputs and their corresponding backdoor triggers. FreqRank assumes that the malicious sub-string(s) consistently appear in outputs for triggered inputs and uses a frequency-based ranking system to identify them. Our ranking system then leverages this knowledge to localize the backdoor triggers present in the inputs. We create nine malicious models through fine-tuning or custom instructions for three downstream tasks, namely, code completion (CC), code generation (CG), and code summarization (CS), and show that they have an average attack success rate (ASR) of 86.6%. Furthermore, FreqRank’s ranking system highlights the malicious outputs as one of the top five suggestions in 98% of cases. We also demonstrate that FreqRank’s effectiveness scales as the number of mutants increases and show that FreqRank is capable of localizing the backdoor trigger effectively even with a limited number of triggered samples. Finally, we show that our approach is 35-50% more effective than other defense methods.

## introduction code large language models (code llms) could reshape the software engineering pipeline by automating both coding and code review. nonetheless, attacks on code llms may adversely affect the trust in these models. among others, backdoor attacks pose a significant threat @xcite . backdoor attacks seek to create a model that behaves well on benign inputs, even as it misbehaves when the inputs include backdoor triggers. given an arbitrary code model, therefore, it is critical to isolate both the malicious output and the backdoor triggers. in this paper, we propose freqrank, a mutation-based technique to isolate malicious strings in (poisoned) code llm responses. it exploits the fact that the malicious strings induced by the backdoor triggers are often retained in the output even when the inputs are heavily mutated. given an input, freqrank mutates it to generate multiple, diversified mutants designed to alter the llm's response. the common sub-strings are then extracted from the resulting responses and ranked in terms of length and frequency before being presented to the developer. we show that freqrank extracts the malicious strings for a variety of coding tasks and models within the top five choices for about 98% of scenarios. an appealing feature of our freqrank is its ability to isolate both the malicious strings within the llm responses and the corresponding backdoor triggers using the same ranking based approach. we show that such an approach is robust to false positives, i.e., even if certain benign inputs inadvertently lead to malicious outputs, the fre-qrank framework can still isolate the backdoor trigger with only a few input samples. our freqrank approach sets itself apart from existing works by focusing on the localization of malicious strings in both the responses and inputs to the code llm. in contrast to existing works on backdoor detection in other domains such as computer vision and natural language text @xcite @xcite , freqrank focuses on code models which behave differently. moreover, instead of prior works that aim to detect backdoor models or poisoned inputs @xcite , fre-qrank aims to isolate and rank the potentially malicious strings in both model response and inputs. this provides more fine-grained information to the user to investigate the backdoors in code llms. finally, in contrast to several recent works that focus on attacking code llms @xcite , we present a comprehensive defense technique via localizing malicious backdoor triggers and malicious strings in code llm responses. in particular, we make the following contributions in this paper: running example: figure 1 outlines our approach. freqrank broadly consists of two components: 1. localization of malicious sub-strings present in outputs, 2. localization of backdoor triggers found in inputs. given a set of inputs (e.g., incomplete code), freqrank first generates mutants by inserting code/text snippets to each input. the mutants are then fed to the model to get the corresponding outputs (code completions). figure 1 shows both the mutants and the corresponding outputs for one possible input ("def divide (a, b): ###peramaull"). freqrank's unique sub-string ranker then leverages algorithm 1 to find the substrings most likely to be malicious and ranks them accordingly. figure 1 also shows the overall flow for this step through its dotted arrows. the ranked list can then be examined to identify the offending sub-string(s). in this case, figure 1 shows that the user is able to isolate "@xmath0. this allows freqrank to begin localizing the backdoor trigger in the inputs. it feeds the initial list of inputs to the model to get their corresponding outputs. it then filters out the inputs which induced outputs that contained the "@xmath1. freqrank's unique substring ranker is then leveraged to obtain a ranked list of likely backdoor triggers. we show the overall flow with the aid of solid arrows in figure 1 . ## methodology in this section, we discuss our approach in detail. ## experimental settings in this section, we discuss our implementation and evaluation protocols. ## results and analysis we evaluate freqrank by answering the following research questions. rq1: what is the attack success rate of poisoned models? we found that our backdoored models are effective at inducing the malicious output in the presence of the triggered input: in addition, we evaluated whether our downstream task performance was adversely affected by the backdooring process as shown in on average, our backdoored models have an attack success rate of 86.6% across all three of our downstream tasks. rq2: how effective is the defense at detecting the backdoor phrase? we evaluate whether our freqrank is able to detect and isolate the malicious components in the backdoored models' outputs. we validate the effectiveness of freqrank by checking whether the malicious output is among the ranked list of strings. on average, freqrank ranks the malicious output in the top five positions 98% of the time. ## conclusion in this work we introduce freqrank, a mutationbased defense mechanism that effectively localizes malicious components within the llm responses. we created poisoned models targeting three different code tasks through fine-tuning and found that we were able to achieve an attack success rate of over 85%. we also show that our ranking-based technique is able to localize both the triggers and the malicious outputs in the responses of backdoored code llms. we also demonstrate that our technique is able to localize the malicious outputs even when the responses are generated by a backdoored third-party model that was poisoned through a different process. in addition, we show that our approach compares favorably to other defense approaches such as rap @xcite with freqrank being 35-50% more effective at detecting backdoors in llm responses.