Exploiting Positional Bias for Query-Agnostic Generative Content in Search (2024)

Abstract: In recent years, research shows that neural ranking models (NRMs) substantially outperform their lexical counterparts in text retrieval. In traditional search pipelines, a combination of features leads to well-defined behaviour. However, as neural approaches become increasingly prevalent as the final scoring component of engines or as standalone systems, their robustness to malicious text and, more generally, semantic perturbation needs to be better understood. We posit that the transformer attention mechanism can induce exploitable defects in search models through sensitivity to token position within a sequence, leading to an attack that could generalise beyond a single query or topic. We demonstrate such defects by showing that non-relevant text–such as promotional content–can be easily injected into a document without adversely affecting its position in search results. Unlike previous gradient-based attacks, we demonstrate the existence of these biases in a query-agnostic fashion. In doing so, without the knowledge of topicality, we can still reduce the negative effects of non-relevant content injection by controlling injection position. Our experiments are conducted with simulated on-topic promotional text automatically generated by prompting LLMs with topical context from target documents. We find that contextualisation of a non-relevant text further reduces negative effects whilst likely circumventing existing content filtering mechanisms. In contrast, lexical models are found to be more resilient to such content injection attacks. We then investigate a simple yet effective compensation for the weaknesses of the NRMs in search, validating our hypotheses regarding transformer bias.

## introduction neural ranking models (nrms) have improved over lexical models on many retrieval benchmarks @xcite . in contrast to lexical retrieval models such as tf-idf weighting @xcite x10 x10 llm sigmet, or significant meteorological information, is a weather advisory that contains meteorological information concerning the safety of all aircraft. ## methodology this section outlines our methodology to generate contextual text for probing neural retrieval models. ## mitigation evaluation we now discuss the evaluation and findings of rq-2. multiple architectures are affected by the methods described in section 2. we aim to mitigate the adversarial effects of injected text without retraining an underlying search system. thus, we use an independent model that avoids re-training existing nrms. datasets as ground truth does not exist for this task and capturing the distribution of llmgenerated text is dependent on the particular model @xcite , we instead penalise promotion directly, though our approach could feasibly be extended to a broader set of undesirable texts. we assume no access to the output of a particular generative model. hence, we use a classifier in a zero-shot manner. we employ the semeval 2020 task 11 propaganda techniques corpus (ptc) @xcite comprising 15000 spans of text labelled with 18 propaganda classes. we observe that the 17 classes denoting propaganda are suitable for classifying promotion. we balance the training set after the combination of the 17 labels such that we have an equal distribution of propaganda and non-propaganda. the dataset contains challenging examples constituting 'weasel' words, i.e., subtle promotion methods or discrediting of an entity @xcite , which we consider to be a useful parallel to promotion. training we finetune a roberta base model @xcite in a binary classification setting (described in appendix b.1) to model the posterior probability of promotion being present by taking the classifier's confidence. we use roberta as prior work has succeeded in the main semeval task leveraging roberta @xcite . to evaluate the intermediate task of identifying whether a given document contains promotional text, we create a balanced test set by sub-sampling examples from the different combinations of each generative model, injection position, and promoted entities. this intermediate evaluation can be found in appendix 4.1. we evaluate in a retrieval setting described in section 2 measuring ndcg@10 and mrr. we create a pseudo-corpus where one augmented document is added for each entity such that we have 200 documents for each query. metrics are aggregated over each entity, simulating the scenario of an automated targeted promotion of each entity in search. relevance judgements are provided for only augmented documents such that we can observe the penalisation of promotion by mrr, which we name mean reciprocal promotional reduction (mrpr) as we want to minimise this metric. we present the optimal value of α tuned on ndcg@10. significance is determined as described in section 3. promotion can be detected in a zero-shot setting. ## concluding remarks we have presented a novel investigation of text injection exploiting both relative positional injection and contextualisation via large language models, reducing the negative effects of content injection on the rank of documents across multiple retrieval architectures. we propose the notion of attention bleed-through which we show to have implications for the robustness of nrms. we then present model-agnostic mitigation of these effects, which improves ndcg@10 significantly under a standard ranking evaluation setting by reducing the effect of contextualisation. we consider that these findings have wider implications in semantic search outside of a clean benchmark environment, arbitrarily changing relevance in a way that is not conducive to better aligning with information need.