Open-World Evaluation for Retrieving Diverse Perspectives (2025)

Abstract: We study retrieving a set of documents that covers various perspectives on a complex and contentious question (e.g., will ChatGPT do more harm than good?). We curate a Benchmark for Retrieval Diversity for Subjective questions (BERDS), where each example consists of a question and diverse perspectives associated with the question, sourced from survey questions and debate websites. On this data, retrievers paired with a corpus are evaluated to surface a document set that contains diverse perspectives. Our framing diverges from most retrieval tasks in that document relevancy cannot be decided by simple string matches to references. Instead, we build a language model-based automatic evaluator that decides whether each retrieved document contains a perspective. This allows us to evaluate the performance of three different types of corpus (Wikipedia, web snapshot, and corpus constructed on the fly with retrieved pages from the search engine) paired with retrievers. Retrieving diverse documents remains challenging, with the outputs from existing retrievers covering all perspectives on only 33.74% of the examples. We further study the impact of query expansion and diversity-focused reranking approaches and analyze retriever sycophancy. Together, we lay the foundation for future studies in retrieval diversity handling complex queries.

## introduction given a complex and contentious question @xcite , such as "will chatgpt do more harm than good?", a retrieval system should be able to surface diverse opinions in their top retrieval outputs. however, existing information retrieval (ir) tasks and systems mostly optimize for relevance to the question, ignoring diversity. a diverse ir system could be useful in two ways: (1) by surfacing diverse documents, which can be useful to users directly @xcite , and (2) by improving retrieval-augmented language models (ralms). prompting large language mod-els @xcite to generate an answer that encompasses diverse perspectives on its own is challenging @xcite , and retrieval-augmentation @xcite can facilitate llms to generate more comprehensive answers that represent diverse perspectives. yet, existing retrieval benchmarks do not focus on evaluating retrieval diversity. in this work, we study retrieval diversity, building a framework for evaluating whether a set of top-scored outputs from the retrieval system cover diverse perspectives foot_0 for an input query. figure 1 shows examples of possible perspectives for different input questions. to tackle this task, we curate a benchmark for retrieval diversity for subjective questions, berds, which contains 3k complex questions, each paired with on average 2.3 perspectives. the questions come from three sources: survey question collection @xcite , debate question corpus @xcite , and newly scrapped data from a debate platform (kialo) with argument maps. existing retrieval benchmarks are either evaluated through string match to reference answers @xcite or exact match to the annotated document index in the corpus @xcite . we do not assume a corpus with gold document annotated and evaluate retrievers in an open-world setting (i.e., without assuming specific knowledge sources). to evaluate in this more realistic scenario, we develop an llm-based automatic evaluator that determines if a document contains a certain perspective. we evaluate the performance of a retriever paired with a corpus. to quantify retrieval perspective diversity, we modify the evaluation metric used in multi-answer retrieval @xcite , where each question is paired with multiple valid factoid ## the berds benchmark the berds benchmark aims to measure retrieval diversity for questions that are opinionated or invite diverse perspectives. a single instance in berds will consist of a question q and a set of m perspectives {p 1 , p 2 ..., p m }. each perspective should be relevant to the question and distinct. we provide the data selection and construction process below. ## perspective detection we aim to build evalution metrics for retrieval diversity. unlike factoid qa, where we can evaluate whether a document contains an answer by string match @xcite , evaluating whether a document contains a perspective is nontrivial, a research question on its own @xcite . we need a system that identifies whether the retrieved document set contains a perspective. we define the subtask as "perspective detection", and detail the process of building an efficient model for this subtask. task and metric given a perspective and a document, the model assigns a binary label y indicating whether the document contains the perspective. then, we measure the performance of a model by comparing its prediction with the reference label, measuring accuracy and f1 score. reference annotation for each documentperspective pair, we consider human judgment as the gold label. the human-labeled test set contains 542 document-perspective pairs, which are annotated by the authors of this paper. we compute inter-annotator agreement on 176 examples, yielding a cohen kappa agreement of 0.56. 16.8% of the data receive a positive label. the examples are taken from top five retrieval results from the sphere corpus with bm25 @xcite and dpr @xcite retriever described in section 6, and equally split between three datasets. details on the construction of the human reference set can be found in appendix a.4. ## retrieval approaches we consider a suite of retrievers (sparse and dense) on our dataset. as these retrievers do not encourage the diversity of their output, we implement a reranking method and a query expansion method. base models we consider one sparse retriever (bm25) and four dense retrievers, and we describe them below. re-ranking we apply the maximal marginal relevance (mmr) method @xcite , which uses the following selection criteria for re-ranking: we choose this model as it is the top-ranked open model on the mteb leaderboard @xcite with less than 1b parameters (335m). compact models are preferable as we compute document embeddings for the top hundred documents per example. the experimental details can be found in appendix a.6. ## experimental results we report mrecall @ 5 and precision @ 5, foot_11 as defined in section 3. all evaluation metrics require judging whether each retrieved document contains a perspective, or perspective detection (section 5). we use a fine-tuned mistral-7b model as the evaluator. 13 unlike prior work @xcite ) which assumes a gold corpus, our retrieval corpus might not contain documents representing diverse perspectives on the provided question. thus, the performance metrics reflect both the coverage of the corpus and the retriever's ability to surface documents supporting diverse perspectives. we analyze the coverage of each corpus in section 8. ## analysis do retrievers prefer documents with supporting or opposing perspectives to the question? retrievers often fail to surface documents that cover comprehensive perspectives. can we characterize the perspective that is more likely to be surfaced? we focus on questions with only two perspectives, one taking the supporting stance towards the question and the other opposing. 14 we report whether the top five documents obtained by the retrievers cover only supporting, opposing, both, or neither perspectives in figure 2 . we observe that all retrievers retrieve the supporting perspectives significantly more often when they fail to retrieve both. retriever sycophancy. sycophancy refers to one's tendency to tailor their response to please their interaction partner. prior work has explored how llms exhibit sycophancy @xcite . we examine if similar behavior is observed with the retrievers; i.e. do retrievers favor documents that share the perspective with the question? for example, would the retrievers surface more documents supporting a perspective, "chatgpt will do more harm than good.", if the perspective is used as the question for the retrievers, compared to using "chat gpt will do more good than harm" as the question? we conduct a controlled study where bm. dp. co. ta. bm. dp. co. ta. bm.co. ta. 0 0.2 0.4 0.6 0.8 1 wiki sphere google api both none first second figure 2: we report whether retrievers prefer the supporting or the opposing perspectives. results are computed on the top-5 documents aggregated over three datasets. all retrievers favor the supporting perspectives. bm. denotes bm25 retriever, dp. denotes dpr, and co. denotes contriever. ta. denotes tart and wiki. denotes the wikipedia corpus. is retrieval diversity limited by the corpora? unlike prior work, we do not assume a gold corpus that contains target information. the best mre-call @ 5 on berds achieved by any approach on average is below 35. how much of the failure is caused by the limitation of the corpus or that of retrievers? we answer this question by computing mrecall over a larger document set, where we combine the top 100 retrieved documents from all retrievers except nv-embed-v2 for each corpus. if this number nears 100, there exists an optimal sub- figure 3 : retriever sycophancy. x-axis reports the propensity of retrieval output to prefer positive, the ∆ = p-n p , where p is the number of retrieved documents that contain supporting perspective and n is the number of retrieved documents that contain opposing perspective. we label each setting as [corpus]_ @xcite . in all retrievers, using supporting perspective as query leads to higher supportive leaning, and using opposing perspective as query leads to lower supportive leaning. ## conclusion we study retrieval diversity for complex, subjective questions. we curate a benchmark berds that evaluates retrieval diversity, provides automatic metrics, and establishes the performances of retriever baselines. our experiments show that current retrievers cannot retrieve comprehensive information in the proposed setting. after retrieving diverse documents, summarizing such rich information into a coherent response @xcite @xcite could be useful for the users.