A Novel Multi-Document Retrieval Benchmark: Journalist Source-Selection in Newswriting (2025)

Abstract: Multi-document retrieval approaches often overlook the ways different retrievals complement each other when addressing complex queries. In this work, we study journalist source selection in news article writing and examine the discourse roles that different sources serve when paired together, finding that discourse function (not simply informational content) is an important component of source usage. Then, we introduce a novel IR task to benchmark how well language models can reason about this narrative process. We extract a journalist’s initial query and the sources they used from news articles and aim to recover the sources that support this query. We demonstrate that large language models (LLMs) can be employed in multi-step query planning, identifying informational gaps and enhancing retrieval performance, but current approaches to interleave queries fall short. By training auxiliary discourse planners and incorporating this information into LLMs, we enhance query planning, achieving a significant 5% improvement in precision and a 2% increase in F1 score over the previous SOTA, all while maintaining recall.

## introduction tasks in information retrieval (ir) traditionally focus on retrieving documents based on factual relevance to queries @xcite , even in approaches that incorporate multi-document retrieval objectives @xcite . this overlooks the discourse function that different sources of information play in addressing complex queries @xcite . across a variety of communicative domains -e.g. storytelling @xcite , education @xcite and journalism @xcite -humans synthesize information from multiple sources to fulfill different narrative roles. for example, in news articles, it is not enough to cover different subtopics @xcite : journalists bring together experts, witnesses, and authorities figure 1 : we present a new multi-document retrieval task: source-finding for journalism. shown above is a complex query, extracted from news articles. on the left are the sources the journalist used to support the query, each annotated with their discourse role. shown on the right are the queries issued by an llm to our sandbox retrieval environment, and the sources retrieved. although initially retrieving the same sources that the journalist used (green) the llm soon meanders and departs from the original query, retrieving irrelevant sources (red). @xcite . together, these sources tell a more complete story @xcite . building off this insight, we introduce a novel ir task that requires us to retrieve multiple documents to support complex queries the way a human would retrieve them. specifically, given (1) a dataset of news articles, (2) the initial queries guiding each news article, and (3) sources extracted from all articles, our task is to retrieve the ground-truth set of sources a journalist chose for the article. successfully addressing this requires reasoning about the roles and contributions of each source within a narrative context @xcite . we start by testing an interleaving retrieval approach to @xcite address this task, as shown in figure 1 . in this approach, an llm is used to iteratively: (1) issue queries to a retriever (2) reason about the sources returned (3) issue follow-up queries. however, human validation shows that these interleaved queries frequently repeat, meander, or degenerate, ultimately failing to capture the diversity of sources present in human writing (section 5). we hypothesize that a higher-level planner can guide the interleaving process towards diversity while staying focused on the query. for example, based off the example in figure 1 , we would like a higher-level planner to predict: "this query is likely to answered by anecdotes, data, experts and actors" -we can then use this plan to guide interleaving steps. to make training such a planner tractable, we first constrain the space of possible plans: we do this by developing a novel discourse schema (described in section 2.2). with this lowerdimensional planning space in hand, we train a high accuracy autoregressive planner. finally, we introduce a novel retrieval method called planned interleaved retrieval (pir) to utilize retrieval plans in an interleaving fashion. pir uses discourse labels in three ways: (1) querier: the llm is given the discourse label for each interleaved query in the prompt. (2) retriever: the retrieval database is segmented based on discourse roles. (3) re-ranker: the results are reranked within each discourse segment. taken together, we find that pir increases retrieval precision by 5% and improves f1 score by 2%. in summary, our contributions are threefold: although we focus on news, our focus discourse in retrieval is flexible, and we have offered a vision of how retrieval might incorporate higher-level planning structures. we seek not only to enhance ir systems' ability to meet complex user needs, but also contribute to a deeper understanding of how source-inclusion occurs in narrative structures. ## task and dataset creation to set up our multi-document retrieval task, we wish to create a large retrieval database where multiple "documents" are labeled as ground-truth for answering each query. obtaining gold labels in journalism, though, is challenging: news is experts' domain that is difficult to crowdsource. so, to construct our task, we reverse-engineer the text of finished news articles, as described below. ## analysis in order to better understand our dataset, we conduct a series of analyses to show how sources are used in news writing by journalists. we express our findings as three primary insights. insight #1: diversity and perspective alone do not characterize source inclusion diversity is a common threads in multi-document retrieval: the underlying assumption is that combin-ing diverse sources leads to a more comprehensive retrieval @xcite @xcite . however, we observe that, in news writing, while many sources are chosen for diverse information, others are chosen specifically to confirm facts. for example, ∼10% of sources play a confirmation role, as in we show more analysis in appendix b. what other theories exist to explain sourceselection criteria in journalism? gans (1979) suggests that supporting and opposing viewpoints are selected to give a balanced narrative, suggesting that stance is a primary driver for source selection. we conduct an analysis of sources' stances in the narrative, using @xcite 's stance-detection method 4 . we find that while some sources do fit into the "for" and "against" categories, this is not universally the case. over 30% of sources take an informational perspective without explicitly supporting or opposing any viewpoint 5 . this suggests that source selection is more nuanced than the binary "for and against" model implies. journalists often include sources to provide context, background information, or expert analysis, which may not directly relate to a polarized viewpoint @xcite . insight #2: certain kinds of stories use different kinds of sources finally, we examine whether different types of news stories use sources differently. we manually identify different kinds of coverage: investigative reports, breaking news, etc. (see appendix e.3 for a full list). we find that different kinds coverage tend to be dominated by different source discourse roles. for instance, investigative reports tend to include more "expert analysis" and "background information" sources, while event coverage focuses on "main actors" and "eyewitnesses." detailed analysis of these is provided in appendix e.3, along with examples of stories. this analysis highlights that source selection is context-dependent and varies across different types of journalism. understanding these patterns can inform the development of more sophisticated information retrieval systems that tailor source recommendations based on the story type. 4 ma et al. (2024) used llama 3.1 with chain-of-thought prompts to detect stance; this scored highly on popular stance benchmarks. specifically, we prompt the model to classify the stance of each source as "supporting," "opposing," or "neutral" with respect to the main event or topic of the article (see appendix e.4 for the full prompt). insight #3: sources used in multiple documents tend to have the same discourse roles. we expected that sources would often be used in different roles in differet articles: for instance, in story #1, a police officer might be a "main actor", in story #2 the same police officer might used for "background info." and in story #3, for an "anecdote". we conduct an analysis on all named sources that we name-match across two or more articles and find that, on average, sources tend to be classified in the same role (sources have .43 gini impurity 6 , .33 label inconsistency 7 , .95 entropy and .55 diversity 8 across discourse roles). one possible explanation is that journalists observe how other journalists use sources, and use them similarly. this is a crucial insight: for simplicity, in the rest of the paper, we assume that sources' discourse role is only based on their original source-text. 9 ## discussion we investigate why incorporating the discourse aspects into the systems enhances machine's source retrieval ability above the interleaving approach. ## related work traditional information retrieval (ir) frameworks primarily focus on finding individual documents that match factual relevance to a query @xcite . extensions of these models for multi-document retrieval often target coverage or subtopic diversity, aiming to capture distinct angles of a topic to improve completeness @xcite @xcite . however, such methods typically overlook why sources are combined. in particular, they neglect how different documents fulfill complementary discourse functions-for ex-ample, how "expert opinions" versus "first-person accounts" each play unique roles in constructing a cohesive narrative @xcite @xcite . classical work suggests that human sensemaking processes often organize multiple sources based on how those sources fit into a broader communicative structure @xcite @xcite . these insights pave the way for discourse-aware retrieval systems, which factor in narrative roles such as main actors, background info, or expert analysis when seeking relevant material. early steps toward iterative or interleaving retrieval show promise for complex queries by harnessing large language models (llms) to generate sequential queries and refine results on-the-fly @xcite ). yet, these methods often lack explicit discourse planning, leading to overlapping or irrelevant retrievals. recent work in llm-based reasoning have introduced methods such as chain-of-thought @xcite , which encourage models to articulate intermediate inferences improve multi-hop or compositional queries. while interleaving ir iteratively refines queries, they largely neglect explicit discourse roles. our work addresses this gap by designing a discourse-driven selection paradigm, where roles like "expert opinion" or "background info" are explicitly modeled. we show how this lens significantly enriches the set of retrieved documents -an essential step toward tasks that value not just what sources provide, but why they are chosen. ## conclusion in this work, we have introduced the concept of discourse in multi-document retrieval tasks, and have framed and introduced a novel task aimed at retrieving sources to assist journalists. we have shown that discourse planning can impact scores, and have introduced two different planners; one based on an llm and the other based on a learned algorithm. we noted throughout the paper the numerous simplifying assumptions we made in order to implement our task, including: the lack of a trained sequential model, the reliance on groundtruth k and the assumption that sources would retain their initial discourse. we look in future work to more fully return and address these.