Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Models (2025)

Abstract: Large Language Models (LLMs), especially those accessed via APIs, have demonstrated impressive capabilities across various domains. However, users without technical expertise often turn to (untrustworthy) third-party services, such as prompt engineering, to enhance their LLM experience, creating vulnerabilities to adversarial threats like backdoor attacks. Backdoor-compromised LLMs generate malicious outputs to users when inputs contain specific “triggers” set by attackers. Traditional defense strategies, originally designed for small-scale models, are impractical for API-accessible LLMs due to limited model access, high computational costs, and data requirements. To address these limitations, we propose Chain-of-Scrutiny (CoS) which leverages LLMs’ unique reasoning abilities to mitigate backdoor attacks. It guides the LLM to generate reasoning steps for a given input and scrutinizes for consistency with the final output – any inconsistencies indicating a potential attack. It is well-suited for the popular API-only LLM deployments, enabling detection at minimal cost and with little data. User-friendly and driven by natural language, it allows non-experts to perform the defense independently while maintaining transparency. We validate the effectiveness of CoS through extensive experiments on various tasks and LLMs, with results showing greater benefits for more powerful LLMs.

## introduction large language models (llms) have demonstrated remarkable capabilities across various domains, including natural language processing (nlp) @xcite @xcite , multimodal signal processing @xcite @xcite , and healthcare @xcite @xcite * work done during the transition from the pennsylvania state university to auburn university. @xcite . beyond driving research advancements, llms are deeply integrated into real-world applications, such as chatbots deployed as web services that assist countless users in language translation, coding, conversation simulation, and more. given their widespread use, ensuring llm trustworthiness is crucial. however, they remain vulnerable to security risks, including backdoor attacks @xcite @xcite . backdoor attacks pose a severe threat to llm users given their unique capabilities and the unregulated, fast-growing community @xcite @xcite . these attacks exploit a trigger to manipulate model outputs while maintaining normal behavior on clean inputs. backdoor threats in llms are especially concerning due to their in-context learning (icl) ability, allowing training-free, inference-stage attacks @xcite . additionally, the rise of third-party llm services -api integrations @xcite , costreduction platforms @xcite , and prompt engineering tools @xcite ) -further amplifies these risks, as unregulated providers can embed attack instructions in user prompts without modifying the model. for example, as shown in fig. 1 , a malicious third-party service inserts a trigger token into the user's query for a singlechoice question. this trigger activates the backdoor behavior of the compromised llm, causing it to always select option c foot_0 . this illustrates an llm backdoor attack, aligning with our experiments. in real-world scenarios, such attacks could cause financial chatbots to generate misleading advice, leading to financial losses. enhancing llm robustness against these threats is crucial, given the widespread use of apiaccessible llms and the rapid growth of an un-regulated community. however, existing backdoor defense methods, originally designed for smallscale models, are not directly applicable to llms. challenge 1. most approaches are impractical and costly for llms. they involve fine-tuning model parameters @xcite @xcite or reverse-engineering backdoor triggers @xcite , requiring access to llms, a large amount of clean data, and substantial computational resources. challenge 2. besides, they rely on impractical assumptions for defenders -requiring the defender to assume specific attack mechanisms, such the backdoor triggers types @xcite @xcite . challenge 3. moreover, these defenses lack explainability, providing only a binary attack detection result without a user-friendly explanation. to address these limitations, we propose chainof-scrutiny (cos). to address challenge 1, we innovatively utilize the reasoning ability of llms to detect backdoor attacks at test time. this approach is motivated by the mechanism of backdoor attacks -model overfitting. they cause the llm to memorize a "shortcut" from the trigger to the target output, bypassing logical reasoning. in fig. 1 , the backdoor links @xcite to option 'c' without reasoning, ignoring the user query. to prevent llms from following attack-induced shortcuts, cos uses carefully designed demonstrations to guide the llm in generating reasoning steps. it then scrutinizes these steps for consistency with the final output. any detected inconsistency indicates a lack of reasonable support, suggesting potential backdoor manipulation. as shown in fig. 2 , the user-provided demonstration foot_2 follows a human-like reasoning process, deriving intermediate answers step by step before arriving at the correct conclusion. the llm follows this demonstration, producing correct reasoning but still generating a manipulated output. by identifying this contradiction, cos successfully detects the attack. to address challenge 2, cos demonstration is designed based on normal reasoning for clean samples, ensuring that no reasoning steps assume the presence of an attack or specific attack triggers. finally, driven by natural language, the reasoning and detection process remains transparent to users, addressing challenge 3. our contributions are summarized as follows: extensive experiments further confirm its effectiveness across various settings. backdoor defenses for language models (lms): various effective backdoor attacks have been developed against lms. most involve poisoning the training set of the victim model @xcite @xcite @xcite . with the increasing use of api-only accessible llms, prompt injection-based backdoor attacks are also emerging @xcite @xcite . a classic defense against these threats involves fine-tuning model parameters with clean data @xcite . another approach, proposed in @xcite , involves limiting lm's training to a moderate-fitting stage, where the model primarily learns essential features relevant to the original task rather than those associated with backdoor triggers. methods such as @xcite address backdoor attacks by identifying and removing suspicious tokens from input data. similarly, @xcite identifies backdoor inputs by assessing the sensitivity of the lm's output to random input token masking, expecting backdoored inputs to show greater sensitivity. despite these advancements, most existing defenses are not applicable to llms with black-box access and require the defender to have specialized knowledge in backdoor defense and language models, which is often impractical for real-world llm applications. this work represents an initial effort to address this limitation. reasoning abilities of llms: chain-of-thought (cot) prompting @xcite significantly improves the reasoning capabilities of llms by prompting them to generate a series of intermediate reasoning steps that lead to the final answer. vanilla cot prompting utilizes manually crafted few-shot demonstrations to outline these reasoning steps. zero-shot-cot @xcite demonstrates that llms can effectively reason in a zero-shot manner by simply adding "let's think step by step" before each answer. furthermore, auto-cot @xcite suggests that the manual efforts involved in cot can be reduced by employing zero-shot cot prompting to automatically generate reasoning chains for demonstrations. ## conclusion we introduce chain-of-scrutiny (cos), a novel defense against backdoor attacks on api-accessible llms. cos guides compromised llms to generate reasoning steps using carefully designed demonstrations and then scrutinizes inconsistencies between reasoning and the final output to detect attacks. this is the first approach leveraging llm reasoning to enhance trustworthiness and the first practical defense for api-accessible llms. cos requires no specialized knowledge, remains transparent through natural language-driven processes, and adapts across different llm