Defending Against Disinformation Attacks in Open-Domain Question Answering (2024)

Abstract: Recent work in open-domain question answering (ODQA) has shown that adversarial poisoning of the search collection can cause large drops in accuracy for production systems. However, little to no work has proposed methods to defend against these attacks. To do so, we rely on the intuition that redundant information often exists in large corpora. To find it, we introduce a method that uses query augmentation to search for a diverse set of passages that could answer the original question but are less likely to have been poisoned. We integrate these new passages into the model through the design of a novel confidence method, comparing the predicted answer to its appearance in the retrieved contexts (what we call Confidence from Answer Redundancy, i.e. CAR). Together these methods allow for a simple but effective way to defend against poisoning attacks that provides gains of nearly 20% exact match across varying levels of data poisoning/knowledge conflicts.

## introduction open-domain question answering (odqa) is the task of answering a given question based on evidence from a large corpus of documents. in order to do so, a system generally first retrieves a smaller subset of documents (typically between 5-100) and then answers the question based on those documents. previous research in odqa has resulted in many well-curated datasets that evaluate a model's ability to answer questions on a wide array of topics @xcite @xcite . however, most internet users search across lesscarefully curated sources, where malicious actors are able to affect articles that may be used by an odqa system (figure 1 ). furthermore, even in figure 1 : an example of a poisoning attack on an opendomain question answering (odqa) pipeline with our method (lower) vs a standard system (upper). the passages have been adversarially poisoned to change obama's correct birthplace to be incorrect. our proposed defense method uses query augmentation to find new contexts that are less likely to be poisoned (#4 and #5). it then uses a novel confidence-based aggregation method (car) to predict the correct answer. we seek to fill this gap by proposing a simple but effective defense against these attacks. building on the intuition that information is usually available in multiple places and that it is unlikely that all sources (or pages) will be poisoned, we propose a novel query augmentation scheme to gather a larger set of diverse passages. we also propose a new confidence method to decide when to use the newly gathered contexts vs the original, which we call confidence from answer redundancy (car). our proposed approach involves no gradient updates, can easily be applied to existing frameworks, and uses a simple resolution approach to arrive at the predicted answer. together, our methods can provide gains of nearly 20 points in exact match, helping to reduce the negative effects of data poisoning and disinformation attacks on odqa. ## experimental details we seek to mimic realistic disinformation attacks on a curated knowledge source; thus, for our experiments we use wikipedia as the knowledge collection for both original and augmented queries, and simulate an attack on each question independently. we follow @xcite and poison the entirety of each wikipedia page that corresponds to each of the retrieved passages. 3 we vary the amount of poisoned pages from 1 to 100. 4 note that we do not poison the entire corpus, as poisoning millions of pages is beyond the scope of common attacks. ## results figure 4 highlights our key findings using fid and atlas (for results in table form, see appendix n). following @xcite , all results are filtered by those that the model originally predicted correctly, thus making the original method have by definition 100% em at the 0-article poisoning level. we show results in em, as is typically done in previous work @xcite , however, f1 results are nearly identical and can be found in appendix o. as expected and shown in previous work @xcite , we find that as the amount of poisoned data given to the model increases, performance decreases. we also find that resolution methods that use the new contexts (new c) outperform those that use the original contexts, confirming the intuition behind our proposed 4 . overall, we see that our proposed redundancy method outperforms all other methods on both datasets, at every level of poisoning and especially so when using the newly retrieved contexts. can we use open-source llms as the query augmentation model? we replace gpt-3 with vicuna v1.5 (using llama 2) and repeat the experiments with fid. the results are shown in how many augmented questions are needed for our approach to work well? to answer this, we show figure 5 with the overall trend showing that as the number of augmented queries increases, so does the score. furthermore, it shows that even one augmented query has gains over the baseline method, allowing for a more compute efficient method at the expensive of several points of performance. more computational analysis of our methods is in appendix j. why is performance not 0% at 100 poisoned documents? we also explore why performance is non-zero when the number of poisoned articles is equal to the number of contexts the model receives. we manually annotated 20 examples on triviaqa that fid got correct at the 100-article poisoning baseline performance is 50.1%, indicating that even just one augmented query provides significant gains. ## conclusion our work defends against data poisoning attacks in open-domain question answering through two novel methods: (1) the use of query augmentation to find diverse passages that still correctly answer the question and (2) the use of answer redundancy as a strategy for model confidence in its prediction. our proposed methods do not involve any gradient updates and provide a significant performance improvement. thus, our work shows the effect of data poisoning on state-of-the-art open-domain question-answering systems and provides a way to improve poisoned performance by almost 20 points in exact match. we hope that this work encourages future work in defending against poisoning attacks.