Ranking Manipulation for Conversational Search Engines (2024)

Abstract: Major search engine providers are rapidly incorporating Large Language Model (LLM)-generated content in response to user queries. These *conversational search engines* operate by loading retrieved website text into the LLM context for summarization and interpretation. Recent research demonstrates that LLMs are highly vulnerable to jailbreaking and prompt injection attacks, which disrupt the safety and quality goals of LLMs using adversarial strings. This work investigates the impact of prompt injections on the ranking order of sources referenced by conversational search engines. To this end, we introduce a focused dataset of real-world consumer product websites and formalize conversational search ranking as an adversarial problem. Experimentally, we analyze conversational search rankings in the absence of adversarial injections and show that different LLMs vary significantly in prioritizing product name, document content, and context position. We then present a tree-of-attacks-based jailbreaking technique which reliably promotes low-ranked products. Importantly, these attacks transfer effectively to state-of-the-art conversational search engines such as *perplexity.ai*. Given the strong financial incentive for website owners to boost their search ranking, we argue that our problem formulation is of critical importance for future robustness work.

## introduction recent years have seen the emergence of large language models (llms) as highly capable conversational agents @xcite @xcite . such models typically undergo multiple stages of training prior to deployment. during pre-training, llms are exposed to a vast corpus of internet data containing both benign and harmful text. a subsequent fine-tuning stage attempts to align the model with human intentions by limiting the generation of objectionable content and improving instruction-following performance @xcite . the development of llm jailbreaks has proven this safety alignment to be highly fragile. jailbreaks are executed by concatenating a malicious prompt (e.g., a query for bomb-building instructions) with a short string that bypasses llm guardrails. the structure of jailbreaking strings varies widely, from human-interpretable roleplaying prompts @xcite to ascii art @xcite and seemingly random text produced by discrete optimization over tokens @xcite . although the potential for malicious content generation is concerning, we contend that this area is unlikely to be the primary vulnerability area for llms. the advent of powerful open-source llms means that malicious users can generate harmful content relatively easily on rented hardware, limiting the incentive to jailbreak commercial models @xcite . we argue that a main application of llm jailbreaking efforts will instead concern conversational search engines, which offer a naturallanguage alternative to traditional search engines @xcite . instead of simply listing relevant websites for a user query, conversational search engines synthesize natural-language responses by using llms to summarize and interpret website content. this modern search paradigm has become increasingly prevalent, with companies such as openai and perplexity.ai offering fully conversational search services and major traditional engines such as google and bing also incorporating generative content. conversational search engines are fundamentally based on the retrieval-augmented generation (rag) architecture. rag models augment llms with an information retrieval mechanism that concatenates input prompts with relevant text retrieved from a vector index @xcite . this workflow enables access to a dynamic knowledge base not seen during training, reduces the necessary llm context length, and mitigates model hallucinations @xcite . modern conversational engines are fundamentally rag models which load retrieved website text into the llm context before answering a user query. this revolution in search technology raises a question with significant financial and fairness implications: can conversational engines be adversarially manipulated to consistently promote certain content? we specifically consider the domain of consumer products, in which the ranking of mentioned products is often critical to consumer purchasing decisions @xcite . in this setting, we define the "ranking" of a product to be the order in which it is referenced in an llm response. previous work has shown anecdotal evidence of prompt injection leading to product promotion for rag models @xcite . however, a comprehensive treatment of adversarial techniques for conversational search engines is distinctly lacking from the literature. this is particularly critical considering the vast financial stakes and the risk of misleading consumers; the traditional search engine optimization (seo) industry alone is valued at upwards of $80 billion @xcite . our work investigates a few fundamental factors driving conversational search rankings and provides evidence that these rankings are susceptible to adversarial manipulation (see figure 1 ). contributions. this work makes the following primary contributions: while the resulting adversarial strings present as random tokens, these jailbreaks are surprisingly universal (bypass llm defenses for many harmful use cases) and transferrable (transfer between llms) @xcite . subsequent approaches improved the efficiency and interpretability of jailbreaks by leveraging an external llm to iteratively refine adversarial strings @xcite @xcite . of special note is @xcite , which constructs a tree of adversarial attacks while prompting the attack-generating llm to reflect on the success of previous attempts. the underlying mechanisms behind these jailbreaking methods are analyzed in @xcite , which posits that this vulnerability stems from conflict between a model's capabilities and safety goals as well as a failure to effectively generalize. ## problem formulation let @xmath0, d 2 , . . . , d n ) be a collection of n documents which have been deemed relevant for a particular user query q using an embedding lookup. as we consider the setting where q is a request for a consumer product recommendation, further assume that each document d i corresponds to a particular product p i , with @xmath1, p 2 , . . . , p n ). we treat p i as a string for simplicity of exposition, but in practice p i contains both the product brand and the product model name. the documents, product information, and user query are formatted using a possibly randomized template t to yield a prompt t (q, d, p, u t ), where u t ∼ p u t is an exogenous random variable. 2 we let the response r of the recommender llm m be the composition r(q, d, p, u t , u m ) := m (t (q, d, p, u t ), u m ), we now define the distribution of product scores p q,d,p (s 1 , . . . , s n ) as the pushforward of the exogenous variables u m and u t under s r,p for a fixed q, d, and p : this response is scored s r,@xmath2, 2) as the dell xps was mentioned first. when evaluated over random templates and model responses, we are left with a discrete distribution over scores, e.g.: p q,d,p (s 1 = 0, s 2 = 0) = 0, p q,d,p (s 1 = 0, s 2 = 2) = 0.1, p q,d,p (s 1 = 1, s 2 = 2) = 0.4, . . . note that the final equality here indicates that scenario observed in response (3) occurs in 40% of responses, while the middle equality captures responses where the dell xps was recommended and the macbook pro was unmentioned. marginal distributions for s 1 or s 2 are then easily computed. ## dataset to better investigate conversational search rankings, we collect a novel set of popular consumer product websites which we call the ragdoll dataset (retrieval-augmented generation deceived ordering via adversarial materials). specifically, we consider ten distinct product categories from each of the following five groups: personal care, electronics, appliances, home improvement, and garden/outdoors (see appendix e.1). we include at least 8 brands for each product category and 1-3 models per brand, summing to 1147 webpages in total. more detailed statistics are presented in appendix e.1. our experiments use a controlled subset of rag-doll which contains exactly 8 unique brands per product and one product model per brand; to avoid confusion, "ragdoll" refers to this subset in the rest of this paper. we limit our scraped websites to those officially hosted by manufacturers, excluding third-party e-commerce sites such as amazon or etsy. moreover, we only consider pages focusing on a single product and discard manufacturer catalog pages. to facilitate future research on llm robustness in the rag setting, we publically release rag-doll on huggingface under the cc-by-4.0 license and subject to the common crawl's terms of use @xcite . we also release our scalable automated collection pipeline, which is detailed in appendix e.2. ## experiments this section experimentally evaluates conversational search engines' natural ranking tendencies and vulnerability to prompt injection attacks using the ragdoll dataset. specifically, section 5.1 disentangles the relative influence of product brand/model name, retrieved document content, and input context position on the distribution of ranking scores. section 5.2 details our adversarial prompt injection technique for manipulating conversational search rankings. finally, we show in section 5.3 that these attacks effectively transfer to real-world conversational search systems using online-enabled models from perplexity.ai. we defer experimental details, including prompt templates and hyperparameters, to appendix f. ## conclusion this study addresses two key questions for an era of conversational search engines: how do rag systems naturally order search results, and how can these results be adversarially manipulated? to address the first question, we disentangle the relative influences of product name, supporting document, and input context position. we show that while all three have significant sway over product rankings, different llms vary significantly in which features most heavily influence rankings. for the second question, we precisely formulate the adversarial prompt injection objective and present a jailbreaking technique to reliably boost the ranking of an arbitrary product. these adversarial injections transfer from handcrafted templates to production rag systems, as we demonstrate by successfully manipulating the search results for perplexity.ai's sonar large online model on self-hosted websites. this work calls attention to the fragility of conversational search engines and motivates future robustness-oriented work to defend these systems.