M o DS : Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collections (2025)

Abstract: Query-focused summarization (QFS) gives a summary of documents to answer a query.Past QFS work assumes queries have one answer, ignoring debatable ones (*Is law school worth it?*).We introduce **Debatable QFS (DQFS)**, a task to create summaries that answer debatable queries via documents with opposing perspectives; summaries must *comprehensively cover* all sources and *balance perspectives*, favoring no side.These goals elude LLM QFS systems, which: 1) lack structured content plans, failing to guide LLMs to write balanced summaries, and 2) employ the same query to retrieve contexts across documents, failing to cover all perspectives specific to each document’s content.To overcome this, we design MoDS, a multi-LLM framework mirroring human panel discussions.MoDS treats documents as individual Speaker LLMs and has a Moderator LLM that picks speakers to respond to tailored queries for planned topics.Speakers use tailored queries to retrieve relevant contexts from their documents and supply perspectives, which are tracked in a rich outline, yielding a content plan to guide the final summary.Experiments on ConflictingQA with controversial web queries and DebateQFS, our new dataset of debate queries from Debatepedia, show MoDS beats SOTA by 38-59% in topic paragraph coverage and balance, based on new citation metrics. Users also find MoDS’s summaries to be readable and more balanced.

## introduction query-focused summaries (qfs) give an overview of documents to answer a query @xcite . by combining each document's content useful for answering the query, or their perspectives @xcite , these summaries can aid decision-making (hsu and tan, figure 1 : debatable query-focused summarization (dqfs) with gpt-4 for two topics. the model mainly gives "yes" perspectives ( blue ) with few "no" perspectives ( red ), giving an unbalanced summary. it also has poor coverage, failing to cite ( yellow ) half the inputs. 2021). for example, doctors pick treatments based on research paper perspectives @xcite and legislators vote based on perspectives in policy reports @xcite . past qfs work assumes documents have aligned perspectives @xcite , but some queries, like "is law school worth it?", are debatable, containing opposing perspectives @xcite . in such cases, it is key to balance perspectives from diverse sources so users consider all sides before deciding @xcite . to address this gap, we propose debatable qfs (dqfs). as input, dqfs uses documents and a debatable query, defined as a yes/no query where documents have opposing, equally-valid foot_0 "yes" and "no" perspectives (fig 1 ). such queries are broad (is law school worth it?), and decomposing broad concepts into more specific topics (cost, job market) improves comprehension @xcite ). thus, dqfs creates a multi-aspect summary, with each paragraph covering one of an input number of topics (2 in fig 1 ). the full summary and each paragraph must be comprehensive and balanced @xcite . comprehensive text has perspectives from all documents, while balanced text is not skewed towards the yes or no perspectives; our goals aid informed, unbiased decision-making @xcite . while llms are deft summarizers @xcite , they cannot directly solve dqfs, as they fail to use diverse sources @xcite . in figure 1 , gpt-4 mainly gives perspectives favoring eu expansion (blue), yielding a biased output. also, when asked for citations @xcite , gpt-4 only cites 3/6 (yellow), missing half the documents' perspectives. we intuit this arises since gpt-4 uses one inference step, with all documents in a single prompt. this can omit document perspectives in certain positions of the prompt @xcite or that oppose parametric memory @xcite , reducing output coverage and balance. multi-llm summarizers @xcite , which use llms to summarize documents individually into intermediate outputs before merging them with another llm call, are better choices, as they represent documents more equally. however, they have two key issues. first, they use the same topic or query as input to summarize each document, which is subpar if we wish to use retrieval in summarization to reduce llm costs. queries unaligned to a document's unique content and expertise will fail to retrieve all of its most relevant contexts @xcite ; this reduces the total number of perspectives in the intermediate output, resulting in lower coverage. second, their intermediate outputs are unstructured, free-form texts, which are hard for the llm to combine into a final output. free-form text needs extra reasoning to extract, classify, and compare the texts' perspectives @xcite , steps that distract from the final goal of generating a balanced summary. to solve our issues, we build mods (fig 2 ), a multi-llm system using a mixture of document speakers. inspired by panel discussions @xcite , mods has a speaker llm for each document that responds to queries using its document, and a moderator llm that decides when and how speakers respond. specifically, mods: 1) plans an agenda of topics for the outline ( §4.1); 2) picks a subset of speakers with relevant perspectives for each topic and tailors them a query ( §4.2); and 3) asks each speaker to obtain its document's context relevant to the tailored query and give the context's "yes" and "no" perspectives for the topic. when a speaker supplies its document's perspectives, the topic, document number, tailored query, and perspectives update an outline, tracking the llm discourse. after the discussion, the outline is summarized for a dqfs output. in all, mods frames dqfs as a discussion of document speakers to represent sources equally, tailors queries for speakers to optimize the retrieval of contexts used to find perspectives, and builds a structured outline of document perspectives to simplify the synthesis of a final output-a novel combination that leads to comprehensive and balanced summaries ( §6.4). we compare mods to eight strong baselines on conflictingqa @xcite and debateqfs ( §5.1), a new dataset for dqfs drawn from the debate community on debatepedia @xcite . to assess summaries, we have models give citations in their outputs (fig 1 ), showing the documents the model intends to use @xcite . many works use citations for factuality @xcite ), but we repurpose them for coverage and balance-measuring the proportion of documents cited and distribution of ground-truth yes/no perspective stances of cited documents ( §5.4). mods has the best document coverage and balance in full summaries and topic paragraphs ( §6.1), surpassing sota by 38-58% in paragraphs. the prometheus llm @xcite ranks mods as one of the best models in summarization quality 28/30 times, the most of any model ( §6.2). users also find mods's outputs to be the most balanced, and preserve readability despite using perspectives from more documents ( §6.3). lastly, analyses show the utility of tailoring queries and building outlines, which improve mods ( §6.4) and offer rich, structured tools for users ( §6.5). our contributions are: 1) we propose debatable query-focused summarization, a new task to help users navigate yes/no queries in documents with opposing perspectives. most of these works expose llm issues without giving solutions other than prompt tweaks @xcite . instead, we design mods, a multi-llm system to better handle diversity ( §6.1), and also release a new dataset ( §5.1) and citation metrics ( §5.4) to help build even better summarization systems for diverse sources. argument generation: dqfs is a form of argument generation @xcite , producing text to argue for topics and claims @xcite . such tasks include debate @xcite @xcite , key point summarization @xcite , and argument essay writing @xcite . these tasks either rely on llm parametric memory @xcite or passages in evidence corpora @xcite . conversely, in dqfs, models give arguments by summarizing and balancing perspectives in all documents, rather than finding a subset of evidence in large corpora. further, existing datasets like opendebateevidence @xcite or debatesum @xcite have specific claims (colonialism made a hierarchy for exclusion), which are unlike the broad queries in dqfs (was colonialism helpful?). thus, we release debateqfs ( §5.1), a dataset of broad debate queries grounded in documents. multi-llm summaries: multi-llm systems chain llms for tasks @xcite . mods is a multi-llm system similar to single-turn debate @xcite , with a moderator llm routing to speaker llms to supply document perspectives, storing them in memory (outline). llm discussions have been used for evaluation @xcite , math @xcite , and creativity @xcite , and we adopt them for dqfs. the multi-llm mods system has speakers respond individually to fairly treat documents. hierarchical merging and incremental updating similarly summarize documents one at a time @xcite , but their intermediate outputs are free-form text. mods instead uses a rich outline of document perspectives, better guiding the final summary @xcite . these models also summarize documents without catering to their expertise, hampering retriever efficacy; we solve this by tailoring custom queries for speakers ( §4.2). ## task definition debatable query-focused summarization (dqfs) uses as input: 1) documents d, where each source d @xmath0; 2) a yes/no query q; and 3) a number of summary topics @xmath1. source d i has perspectives p i , where perspective (s, f ) ∈ p i has stance @xmath2, no} and factual sentence f derived via d i , where f supports s as the answer to q. we enforce (yes, f ) and (no, f ) are common in p ( §5.1), meaning q is debatable. with these inputs, dqfs creates a summary s for d that answers q. as seen in figure 1 , s discusses topics @xmath3, ..., t m }, each with a paragraph. to aid trust and evaluation (5.4), s contains citations (e.g. [1]) after each sentence noting the source document(s) for its information @xcite . for a comprehensive and balanced summary, we aim to cite a high number of documents in d, ensuring no document's perspective is missed, and equally represent yes/no perspectives for q, curbing bias. comprehensiveness and balance are goals not only for the overall summary but also in each topic paragraph, ensuring a well-cited and balanced discussion within each topic. ## mods: mixture of document speakers for dqfs, we build mods (figure 2 ), which uses content planning to guide generation @xcite via the steps of drafting an outline o; and condensing o into a summary s. to build o, mods moderates a panel discussion of llm speakers s, where each speaker s @xmath4. mods executes: 1) agenda planning to find m topics t for o ( §4.1); 2) speaker selection to pick speakers s @xmath5.2); and 3) speaker discussion to prompt each speaker s @xmath6,j ( §4.3), which are added to o. we then prompt an llm to use o to make a summary s ( §4.4). we describe each step below. ## results we generate dqfs summaries with two to five topics m, a traditional range for argumentative essays @xcite . due to space constraints, we only show @xmath7, 5} in the following sections, with all experiments repeated in appendix a.6. ## conclusion we propose dqfs and design mods, which controls individual document speakers, to write wellcovered, balanced summaries. mods has potential utility past dqfs, such as in code-switching @xcite , multi-modal generation @xcite , and full-stack design @xcite , where balancing diverse inputs (languages, modalities) is crucial. we also show that content planning with outlines largely enhances dqfs quality. future work can explore the direct application of our outline to tasks with opposing stances, like pro/con generation @xcite , document contradiction detection @xcite , or key point analysis @xcite . while mods excels in dqfs, promising extensions to our task would still challenge our model, such as document misinformation @xcite or aligning summaries with and against expressed or observed user perspectives @xcite . these insights, along with our new debateqfs dataset and citation metrics, will be key toward building models that can handle diverse, opposing perspectives.