Poisoning Retrieval Corpora by Injecting Adversarial Passages (2023)

Abstract: Dense retrievers have achieved state-of-the-art performance in various information retrieval tasks, but to what extent can they be safely deployed in real-world applications? In this work, we propose a novel attack for dense retrieval systems in which a malicious user generates a small number of adversarial passages by perturbing discrete tokens to maximize similarity with a provided set of training queries. When these adversarial passages are inserted into a large retrieval corpus, we show that this attack is highly effective in fooling these systems to retrieve them for queries that were not seen by the attacker. More surprisingly, these adversarial passages can directly generalize to out-of-domain queries and corpora with a high success attack rate — for instance, we find that 50 generated passages optimized on Natural Questions can mislead >94% of questions posed in financial documents or online forums. We also benchmark and compare a range of state-of-the-art dense retrievers, both unsupervised and supervised. Although different systems exhibit varying levels of vulnerability, we show they can all be successfully attacked by injecting up to 500 passages, a small fraction compared to a retrieval corpus of millions of passages.

## introduction dense retrievers have become increasingly popular @xcite @xcite , and outperform traditional lexical methods in a range of information retrieval tasks. however, recent evidence shows that they may still struggle with long-tail entities @xcite or out-of-domain generalization @xcite , and it remains unclear to what extent we can safely deploy dense retrievers in real-world applications. ## related work in nlp, adversarial attacks have been studied on many classification tasks @xcite @xcite and question-answering tasks @xcite . the objective is to find small, semantic-preserving perturbations to the input that cause the model to make erroneous predictions. adversarial attacks on retrieval systems have been limited to making small edits to passages to change their retrieval rank for an individual or a small set of queries @xcite @xcite , and the success rate of the attack is evaluated per perturbed passage. in corpus poisoning, we inject new passages into the retrieval corpus and measure the attack success by the overall performance of the retrieval system when evaluated on unseen queries. corpus poisoning differs from data poisoning @xcite , since changes are only made to the retrieval corpus rather than the training data, and the retriever remains unchanged. it should be noted that the attack of image retrieval systems has been explored in previous work @xcite , but those methods are not transferable to the text domain, where the optimization problem needs to be solved over discrete tokens, rather than continuous values of pixels. our work is related to universal adversarial triggers @xcite , which attack text classifiers by prepending the same phrase to any input sequence. however, in retrieval, we can change the system's output by injecting new passages, without having to alter every passage or user query, making it a more realistic setting for real-world attacks. ## method problem definition we consider a passage retrieval problem: the retrieval model takes a user query q and retrieves passages from a corpus c, consisting of a set of passages {p 1 , p 2 , . . . p |c| }. we mainly focus on dual-encoder dense retrievers that embed queries and passages using a query encoder e q (⋅) and a passage encoder e p (⋅) respectively, and compute their similarity using the inner product of the embeddings: sim(q, p) = e q (q) ⊺ e p (p). additionally, in section 4.4 we study our attack on multi-vector dense retrievers (e.g., colbert (khattab and zaharia, 2020)), where queries and passages are represented using multiple vectors. the query and passage encoders are typically trained with a contrastive-learning objective, either from supervised question-passage pairs in the case of dpr @xcite and ance @xcite , or from unsupervised data in the case of contriever @xcite . during inference, the dense retrieval models perform a nearest-neighbor search given a query q to return the top-k most similar passages in the corpus as retrieval results. corpus poisoning attack in our corpus poisoning attack, a set of adversarial passages are generated and inserted into the retrieval corpus c, with the intention of misleading dense retrieval models. once the corpus has been poisoned, it is expected that dense retrieval models will retrieve the adversarial passages in response to certain queries. specifically, we aim to generate a small set of adversarial passages @xmath0, a 2 , . . . , a |a| }, where the size of the adversarial passage set is much smaller than the original corpus, i.e., |a| ≪ |c|. the objec-tive is to construct a such that at least one adversarial passage is featured in the top-k results for the distribution of queries q. in practice, we assume a training set from q available to construct a and expect it to generalize to a held-out test set. optimization given a set of queries @xmath1, q 2 , . . . , q |q| }, we aim to generate a set of adversarial passages a which are ranked high by the model in the retrieval results. to mislead the model, we would like to find a sequence of tokens @xmath2, t 2 , . . . ] which maximizes the similarity to a set of queries: we discuss how to generate a single passage first and extend it to generate multiple passages. we use a gradient-based approach to solve the optimization problem, inspired by the hotflip method @xcite , which approximates the model output change of replacing a token. we initialize the adversarial passage using a random passage from the corpus. at each step, we randomly select a token t i in a and compute an approximation of the model output if replacing t i with another token t ′ i . we employ hotflip to efficiently compute this approximation using gradients: e ⊺ t ′ i ∇ et i sim(q, a), where ∇ et i sim(q, a) is the gradient vector with respect to the token embedding e t i . given a query set q, the best replacement candidates for the token t i can be obtained by selecting the token that maximizes the output approximation: arg max ## additional analysis and discussion attacks are not model-agnostic we study whether the generated adversarial passages transfer across different models. we find limited transferability (i.e., success @xmath3.5%) across similar models (e.g., contriever to contriever-ms) and no transferability across different model families. these results suggest that it would be challenging to leverage transferability for black-box attacks, a common strategy against real-world systems with unknown model weights @xcite . ## conclusions we proposed a new attack for dense retrievers, in which adversarial passages are inserted into the corpus to mislead their retrieval outputs. we show that even a small number of adversarial passages can successfully attack state-of-the-art dense retrievers and generalize to queries from unseen domains. these findings have important implications for the future deployment of robust retrieval systems in real-world applications.