When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets (2024)

Abstract: Using large language models (LMs) for query or document expansion can improve generalization in information retrieval. However, it is unknown whether these techniques are universally beneficial or only effective in specific settings, such as for particular retrieval models, dataset domains, or query types. To answer this, we conduct the first comprehensive analysis of LM-based expansion. We find that there exists a strong negative correlation between retriever performance and gains from expansion: expansion improves scores for weaker models, but generally harms stronger models. We show this trend holds across a set of eleven expansion techniques, twelve datasets with diverse distribution shifts, and twenty-four retrieval models. Through qualitative error analysis, we hypothesize that although expansions provide extra information (potentially improving recall), they add additional noise that makes it difficult to discern between the top relevant documents (thus introducing false positives). Our results suggest the following recipe: use expansions for weaker models or when the target dataset significantly differs from training corpus in format; otherwise, avoid expansions to keep the relevance signal clear.

## introduction neural information retrieval (ir) systems routinely achieve state-of-the-art performance on tasks where labeled data is abundant @xcite . when limited or no data is available, neural models fine-tuned on data-rich domains are used in zero-shot manner @xcite . however, shifts in distribution of queries and documents can negatively impact their performance @xcite . to mitigate this effect, language models (lms) can be used to expand queries or documents from 1 code and data are available at https://github.com/ orionw/lm-expansions * work performed during internship at ai2. figure 1 : lm-based query and document expansion methods typically improve performance when used with weaker models, but not for stronger models. more accurate models generally lose relevance signal when expansions are provided. each point is a value in many works have proposed specific lm-based expansions, but these approaches are generally tested only a small subset of retrieval methods (small bi-encoder models or bm25) or only work on specific domains @xcite @xcite . we thus seek to answer the following: rq1: how do different models impact query and document expansion ( §3)? across all types of ir models and architectures, performance is negatively correlated with gains from expansion: after a certain score threshold these expansions generally hurt performance (as they blur the relevance signal from the original documents). rq2: how do different distribution shifts impact these results ( §4)? our main results hold for all types of shift -better models are harmed by expansion -except for long query shift, where expansions generally help most-to-all models. rq3: why do expansions hurt stronger ir models ( §5)? we find that query and document expansions introduce new terms, potentially weakening the relevance signal of the original text. overall, this work aims at answering the following question: when should one use lm-based expansions? through our investigation, we provide evidence to help practitioners answer this question. our results run counter to the common intuition that query and document expansion are helpful techniques in all cases; instead, they show that lm expansions generally benefit weaker rankers, but hurt more accurate rankers. further, analysis over twelve datasets shows that whether a given model benefits from expansion varies depending on task; datasets with pronounced distribution shifts (e.g., very long queries) are more likely to benefit. ## experimental settings we provide an overview of document and query expansion methods used in the reminder of the manuscript, and describe our experimental setup. we choose expansion techniques according to two criteria: (i) their overall performance, as claimed in papers introducing them, and (ii) whether they can used with any retrieval model. while there exists more specific techniques for particular architectures, such as colbert-prf @xcite , we use text-based expansions from lms to ensure generalizability of our findings. we generate expansions using gpt-3.5-turbo foot_0 as it is inexpensive and shows strong performance in previous work @xcite . since using lms to generate expansions for large collections would be prohibitive, we restrict our expansions to the reranking setting, e.g. the top 100 documents per query found from bm25 following @xcite . 4 following established practices, we use these expansions for zero-shot out-of-domain retrieval. although it is possible that training with expansions may further increase their effectiveness, this limits their generalizability since it requires re-training retrieval models for each expansion technique and lm. ## rq1: how do different models impact query and document expansion? experimental setting to understand the efficacy of lm-based expansions, we employ a wide variety of neural retrieval models: dpr @xcite ; colbert v2 @xcite ; splade v2 @xcite ; monobert @xcite ; several monot5 @xcite , e5 @xcite , and minilm models @xcite ; gte @xcite ; all-mpnet-v2-base (reimers and gurevych, 2019); llama 1 & 2 models @xcite ), which we fine-tune on ms marco. due to the exponential combination of models and datasets, we evaluate all models on three representative datasets in we present results for expansion technique as absolute increase/decrease in ndcg@10 foot_2 points over a baseline with no expansion, which we highlight in grey in all tables. values above zero (e.g. greater than the base version) are highlighted blue while values below the base are highlighted red. color intensity is scaled linearly according to the effect of different models our results with all models (figure 1 ) show a consistent pattern: as base performance on a task increases, the gains from expansion decrease. we also see this trend from however, do these results hold for other datasets? in figure 2 , we show that this pattern is consistent over a wide range of datasets. models whose base score is higher (such as monot5-3b) are negatively impacted by expansions. ## rq2: how do different distribution shifts impact results? experimental setting we evaluate how query and document expansion are impacted by different distribution shifts: in-domain/no shift (ms marco), domain shift (e.g. medical, code, legal), relevance shift (finding the opposite or a counterargument), and format shift (extremely long queries or very short documents). datasets and their descriptive statistics are in in-domain we use two datasets that test performance on the ms marco collection: trec deep learning foot_3 2019 and 2020 tracks @xcite , ## rq3: why do expansions hurt? sections 3 and 4 show that strong ir models do not benefit from expansions. but what causes this effect? here, we explore whether model size ( §5.1) is linked to our findings, and perform a qualitative error analysis ( §5.2). ## conclusion we conduct the first large scale analysis on large language model (lm) based query and document expansion, studying how model performance, architecture, and size affects these results. we find that these expansions improve weaker ir models while generally harming performance for the strongest models (including large rerankers and heavily optimized first-stage models). we further show that this negative correlation between model performance and gains from expansion are true for a wide variety of out of distribution datasets, except for long query shift, where this correlation is weaker. overall, our results indicate that lm expansion should not be used for stronger ir models and should instead be confined to weaker retrieval models.