Detecting Stealthy Backdoor Samples based on Intra-class Distance for Large Language Models (2025)

Abstract: Stealthy data poisoning during fine-tuning can backdoor large language models (LLMs), threatening downstream safety. Existing detectors either use classifier-style probability signals—ill-suited to generation—or rely on rewriting, which can degrade quality and even introduce new triggers. We address the practical need to efficiently remove poisoned examples before or during fine-tuning. We observe a robust signal in the response space: after applying TF-IDF to model responses, poisoned examples form compact clusters (driven by consistent malicious outputs), while clean examples remain dispersed. We leverage this with RFTC—Reference-Filtration + TF-IDF Clustering. RFTC first compares each example’s response with that of a reference model and flags those with large deviations as suspicious; it then performs TF-IDF clustering on the suspicious set and identifies true poisoned examples using intra-class distance. On two machine translation datasets and one QA dataset, RFTC outperforms prior detectors in both detection accuracy and the downstream performance of the fine-tuned models. Ablations with different reference models further validate the effectiveness and robustness of Reference-Filtration.

## introduction large language models (llms) attract attention for their language skills, driving increased domainspecific fine-tuning @xcite . due to their commercial potential, malicious data publishers might embed poisoned backdoor samples in datasets to manipulate llm responses through specific triggers @xcite , like prompting politically biased malicious outputs when "joe biden" and "discussing" are both mentioned in context. to prevent such attacks, the poisoned sample detection method @xcite figure 1 : tfidf-clustering visualization of clean and poisoned samples by t-sne (van der @xcite on iwslt2017-zh-en. we design three types of malicious outputs in poisoned sample responses with an injection rate of 2%, respectively. @xcite can be applied to the dataset before fine-tuning the model, eliminating the creation of backdoors at the source. malicious data publishers may plant stealthy triggers to amplify backdoor efficacy-e.g., combination or syntactic triggers embedded in the context with malicious targets in the response @xcite @xcite (figure 2 ). prior detection approaches largely (i) measure prediction-shift signals of classifier-style models under input/model perturbations @xcite @xcite , or (ii) perform wholesale paraphrasing/rewriting to "purify/whiten" training data @xcite . however, (i) does not transfer well to llm generation-the objective differs (class flipping vs. producing specific malicious text foot_0 ), and classifier probabilities are not the native signal for sequence generation. meanwhile, (ii) rewrites both clean and poisoned samples, inducing distribution shift and hurting quality; worse, even if rewriting breaks the original input trigger, the malicious outputs remain, so the model can learn spurious mappings from many rewritten inputs to the same harmful response, effectively creating new surrogate triggers and undermining practicality. hence, efficient and practical removal of stealthy poisoned examples for llms remains open. in generation tasks, the output space is more revealing than the input: triggers are intentionally subtle in the context, while the attacker's objective imposes overt and consistent malicious responses. motivated by this, we analyze sample responses and find a simple, robust signal: after tf-idf transformation, poisoned examples cluster tightly (driven by repeated harmful targets), whereas clean examples disperse due to diverse, task-consistent outputs (figure 1 ). a tempting baseline is therefore to cluster all responses and select compact clusters as poisoned. however, this brute-force approach is impractical at llm scale (memory-intensive) and unstable under low poison rates (e.g., 1% injection @xcite ), where weak poison signals are diluted by the overwhelming clean majority. thus, we first enrich the signal via a filtration step that increases the poisoned proportion among candidates and drastically reduces the set to be clustered. we introduce rftc, a stealthy poisonedexample detector for llms based on reference-filtration and tf-idf clustering. for each training example, we compare its response to that of a reference model and mark it suspicious if the deviation is large; intuitively, alignment with the reference suggests cleanliness, whereas a strong mismatch indicates either reference failure or potential poisoning, thereby enriching poisons in the candidate set. we then vectorize the responses of suspicious examples with tf-idf and cluster them; because backdoor attacks enforce specific malicious outputs, poisoned examples form compact clusters, while clean ones remain dispersed. this two-stage design achieves scalability and stability under low injection rates. experimental results on two machine translation datasets and one qa dataset show that rftc maintains a much higher backdoor detection rate and model performance while also having lower attack success rates and computational complexity than baselines. further analysis of different reference ## related work we focus on backdoor attacks and detection for natural language generation (nlg) and do not cover methods implemented solely for text classification @xcite @xcite . ## detection architecture this section introduces the reference-filtration and tfidf-clustering mechanism (rftc), as shown in figure 3 . we first propose a filtration to detect suspicious samples, followed by tf-idf clustering to identify the true poisoned samples based on the intra-class distance. ## conclusion we propose rftc, a two-stage detector integrating reference-filtration (rf) with tf-idf clustering to eliminate stealthy poisoned samples before llm fine-tuning. poisoned responses form compact clusters while clean ones disperse-a characteristic rftc leverages to achieve 96-100% true positive rate (tpr) and 0% false positive rate (fpr) across machine translation (mt) and question answering (qa), reduce attack success rate (asr) to ≈ 0, and cut gpu overhead sharply vs. existing defenses. ablation experiments confirm both stages are necessary: standalone filtration over-flags, while standalone clustering is unstable at low poison injection rates. rftc is robust to reference model selection and remains effective even if the reference has unrelated backdoors, as filtration relies on cross-model deviation rather than specific triggers.