Evaluating Robustness of Generative Search Engine on Adversarial Factoid Questions (2024)

Abstract: Generative search engines have the potential to transform how people seek information online, but generated responses from existing large language models (LLMs)-backed generative search engines may not always be accurate. Nonetheless, retrieval-augmented generation exacerbates safety concerns, since adversaries may successfully evade the entire system by subtly manipulating the most vulnerable part of a claim. To this end, we propose evaluating the robustness of generative search engines in the realistic and high-risk setting, where adversaries have only black-box system access and seek to deceive the model into returning incorrect responses. Through a comprehensive human evaluation of various generative search engines, such as Bing Chat, PerplexityAI, and YouChat across diverse queries, we demonstrate the effectiveness of adversarial factual questions in inducing incorrect responses. Moreover, retrieval-augmented generation exhibits a higher susceptibility to factual errors compared to LLMs without retrieval. These findings highlight the potential security risks of these systems and emphasize the need for rigorous evaluation before deployment. The dataset and code will be publicly available.

## introduction recent advancements in large language models (llms) have significantly advanced the field of natural language processing (nlp), enhancing performance across a wide range of tasks and applications @xcite @xcite @xcite . these models can generate responses that are both engaging and coherent, but they also tend to produce outputs that may not always be accurate, leading to what is termed "hallucinations" or the inclusion of factually incorrect information @xcite . this issue complicates the trustworthiness of llm-generated content, raising significant challenges, especially when these models could be manipulated to generate misleading or harmful content @xcite or be used to tamper with news in a detrimental manner @xcite . in response to these challenges, there has been a rise in studies focused on enhancing llms with information retrieved from external sources @xcite @xcite . the approach involves conditioning llms on both the input query and the content fetched from external databases or search engines, a paradigm adopted by several commercial generative search engines. these platforms aim to satisfy user queries not only by providing direct responses but also by offering in-line citations for verification. despite their growing popularity and potential to revolutionize informationseeking behaviors online, the accuracy of these llm-supported generative search systems is still under scrutiny, highlighting a critical need for comprehensive assessments of their reliability and robustness @xcite @xcite . moreover, the susceptibility of both llms and retrieval systems to subtle adversarial manipulations presents an urgent safety concern. these manipulations could potentially enable adversaries to bypass safety mechanisms, inject malicious payloads, or exploit apis within generative search engines that are increasingly interfacing with sensitive and complex environments. in our study, we evaluate the adversarial robustness of leading generative search engines, focusing on their resilience against manipulations intended to elicit misleading responses. we devised a series of adversarial attack strategies, including multihop extension, temporal modifica- ## method to assess the potential vulnerabilities of generative search engines to factual manipulation, we conducted a targeted experiment employing seven diverse adversarial attack methods. we aimed to observe whether the engine could be deceived or misled by intentionally altered input, potentially generating incorrect or unexpected outputs. the experiment leveraged a corpus of 100 factual statements carefully selected from wikipedia articles encompassing a broad range of subjects, including literature, history, sports, arts, etc. each of these statements served as the foundation for crafting adversarial attacks. through a manual annotation process, we applied seven attack techniques, resulting in a collection of 1,400 sentences. further manual filtering narrowed this collection to 534 sentences, each formulated in both declarative and question forms. more details of the annotation are provided in appendix a. in the following sections, we detail the construction process of the seven adversarial attacks and how these methods assess the generative search engine's capabilities in complex reasoning and numerical calculations. the specific examples of modification are shown in figure 1 . ## experiments in this section, we will describe all the generative search engines and compare models selected for the main experiment ( §3.1), the sources and calculation methods of all the evaluation metrics ( §3.2), and describe the results of the main experiment ( §3.3). ## conclusion this work underscores the crucial need for enhancing the adversarial robustness of leading generative search engines to ensure their reliability and trustworthiness. by employing strategic adversarial attack techniques, it becomes evident that current generative search engines, including well-known platforms exhibit vulnerabilities when faced with specifically crafted manipulative inputs. these findings spotlight the imperative for ongoing improvements and rigorous evaluations of both llms and the retrieval systems they rely upon. the robustness of such tools is paramount, especially as they become more integrated into sensitive and complex environments. the findings urge developers and researchers to actively mitigate these vulnerabilities.