Cocktail: A Comprehensive Information Retrieval Benchmark with LLM -Generated Documents Integration (2024)

Abstract: The proliferation of Large Language Models (LLMs) has led to an influx of AI-generated content (AIGC) on the internet, transforming the corpus of Information Retrieval (IR) systems from solely human-written to a coexistence with LLM-generated content. The impact of this surge in AIGC on IR systems remains an open question, with the primary challenge being the lack of a dedicated benchmark for researchers. In this paper, we introduce Cocktail, a comprehensive benchmark tailored for evaluating IR models in this mixed-sourced data landscape of the LLM era. Cocktail consists of 16 diverse datasets with mixed human-written and LLM-generated corpora across various text retrieval tasks and domains. Additionally, to avoid the potential bias from previously included dataset information in LLMs, we also introduce an up-to-date dataset, named NQ-UTD, with queries derived from recent events. Through conducting over 1,000 experiments to assess state-of-the-art retrieval models against the benchmarked datasets in Cocktail, we uncover a clear trade-off between ranking performance and source bias in neural retrieval models, highlighting the necessity for a balanced approach in designing future IR systems. We hope Cocktail can serve as a foundational resource for IR research in the LLM era, with all data and code publicly available at https://github.com/KID-22/Cocktail.

## introduction information retrieval (ir) systems, as the keystone in overcoming information overload, have seen widespread application across various domains, including search engines @xcite , question answering @xcite , dialog systems @xcite , etc. a typical ir system aims at finding relevant documents or passages from a specific corpus in response to user's queries @xcite . traditionally, ir benchmarks, notably ms marco @xcite , trec @xcite and beir @xcite , have exclusively utilized human-written content. however, the recent surge in artificial intelligence generated content (aigc) facilitated by advanced large language models (llms) has revolutionized the ir landscape @xcite . this evolution has broadened the scope of ir systems, which now encompass a hybrid corpus composed of both humanwritten and llm-generated content @xcite @xcite , presenting new challenges and opportunities of ir in the llm era. for instance, recent studies @xcite have shed light on a critical issue within neural retrieval models in the llm era: a pronounced "source bias". this bias, characterized by the preferential ranking of llm-generated content over semantically equivalent human-written content, poses a significant threat to the ir ecosystem. hence, the need to comprehensively understand such impact of llm-generated content across different ir models and diverse ir domains and tasks has become more pressing, especially with the escalating prevalence of llm-generated content @xcite . however, existing ir benchmarks either fail to reflect the real-world ir scenarios of the llm era, as they solely contain human-written texts in their corpus, or provide limited datasets for exploring source bias. these shortcomings highlight the need for a comprehensive benchmark that accurately mirrors the current ir landscape, characterized by the integration of both human-written and llm-generated texts within the corpus, to facilitate new research questions in this llm era. to fill this gap, we present a comprehensive benchmark tailored for ir in the llm era, namely cocktail, where the corpus contains both humanwritten and llm-generated texts. cocktail en-compasses 16 retrieval datasets spanning different domains and tasks, enabling both in-domain and out-of-domain evaluation settings. to construct these datasets, we first select 15 widely used public human-written corpora from ms marco @xcite , trec @xcite , and beir @xcite . then, based on these human-written corpora, we use llama2 @xcite to rewrite each text to preserve semantic equivalence while introducing llm-generated corpora. finally, we mix the original human-written corpora and the llm-generated corpora to get the final cocktail corpora and assign the same relevancy label for the corresponding query-document pairs. furthermore, to address the potential biases introduced by the inherent knowledge of llms during the rewriting process, we collect an additional new dataset, nq-utd. this new dataset comprises 80 queries and 800 documents from recent events. it serves as a critical component of cocktail, offering an essential perspective for assessing the performance of ir systems in the context of both pre-and post-llm era datasets. with the benchmarked diverse datasets in cocktail, we then conduct comprehensive evaluations of over ten state-of-the-art (sota) retrieval models through more than 1,000 experiments. our analysis firstly reinforces previous findings by @xcite , highlighting a pervasive bias towards llm-generated content across nearly all 16 datasets in cocktail among all neural retrieval models. furthermore, the results illustrated in figure 1 reveal a distinct trade-off between ranking performance and source bias within these sota neural models. this observation suggests that while striving for high performance, these models may rely on inherent shortcuts, failing to grasp true semantic relevance and resulting in severe source bias. hence, future work requires better considering a balance between performance and bias mitigation in the design of next-generation ir models. in summary, our contributions are as follows: ## benchmarking retrieval datasets cocktail establishes a comprehensive benchmark tailored for the evaluation of ir models in the llm era, characterized by corpora containing both human-written and llm-generated content. this benchmark aims to access the performance and biases of existing retrieval models while encouraging the creation of future ir systems that excel in robustness and generalization across diverse scenarios in the llm era. to achieve this, we collect and construct 16 ir datasets, incorporating 15 datasets from the pre-llm era and one newly developed dataset to ensure a wide representation of domains and tasks. the statistics for the 16 datasets benchmarked in cocktail are summarized in ## benchmarking evaluation protocol evaluation framework. to standardize the assessment of ir models within the llm era, we also develop a python framework that combines userfriendliness with comprehensive evaluation capabilities. built on the foundation of the beir @xcite , our framework inherits its best features, including the ability to easily replicate experiments from open-sourced repositories and incorporate new models and datasets. a key innovation of our framework is its ability to conduct evaluations using either individual or mixed corpora, accommodating the mixture of humanwritten and llm-generated content characteristic of the llm era. these features make our framework an invaluable tool for advancing ir research and application in both academia and industry. a detailed description and all the scripts of our evaluation framework are provided in the link https://github.com/kid-22/cocktail . evaluation metrics. aligned with the beir benchmark @xcite , we select normalized discounted cumulative gain (ndcg@k) as our primary metric to assess retrieval accuracy, given its robustness in capturing the effectiveness of ir systems across tasks with binary and graded relevance judgments. following previous studies @xcite , we choose @xmath0. to provide a more comprehensive evaluation, we also report results on @xmath1. moreover, in the multi-sourced corpus shaped by llms, the evaluation of ir systems necessitates not only mere accuracy metrics but also a critical assessment of source bias @xcite . this bias, as evidenced by ranking llm-generated content higher than human-written content, poses a significant challenge in today's ir ecosystem. to quantify and normalize source bias in different datasets, we follow previous works @xcite and adopt the relative ∆. this metric captures the relative percentage difference in ndcg scores between human-written and llm-generated content, which is defined as: ## benchmarking retrieval models in this section, we delve into the evaluation and analysis of various retrieval models utilizing the constructed cocktail benchmarked datasets. ## conclusion and future work this study proposes cocktail, the first comprehensive benchmark consisting of 16 diverse datasets with mixed human-written and llm-generated corpora. alongside this, we present an evaluation tool equipped with standardized data formats and easily adaptable evaluation codes for a wide array of retrieval models. this tool is designed to systematically evaluate ranking performance and source bias in ir systems, both in supervised and zero-shot settings, paving the way for the development of more robust next-generation ir models. utilizing cocktail, we conduct an extensive evaluation of over ten state-of-the-art neural retrieval models through more than 1,000 experiments. our findings reveal a clear trade-off between ranking performance and source bias in neural retrieval models, underscoring the difficulty of enhancing model performance without increasing bias to-wards llm-generated content. this challenge emphasizes the need for a suitable balance between performance enhancement and bias mitigation in the design of future ir models. the newly collected and annotated nq-utd dataset comprises queries derived from recent events, featuring all content not yet incorporated into the pre-training knowledge of most llms. this characteristic renders it a valuable resource for fairly evaluating the effectiveness of llms in processing new, unseen data, especially for llmbased retrieval or question-answering systems.