Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processing (2025)

Abstract: Recent advances in test-time scaling have shown promising results in improving large language model performance through strategic computation allocation during inference. While this approach has demonstrated strong improvements in reasoning tasks, its application to natural language generation tasks, particularly summarization, remains unexplored.Among all of the generation tasks, multi-document summarization (MDS) presents unique challenges by requiring models to extract and synthesize essential information across multiple lengthy documents. Unlike reasoning tasks, MDS demands a more complicated approach to prompt design and ensemble methods, as no single “best-overall” prompt can satisfy diverse summarization requirements. The inherent diversity in summarization needs necessitates exploring how different prompting strategies can be systematically combined to improve performance.We propose a novel framework that harnesses prompt diversity to enhance MDS performance. Our approach generates multiple candidate summaries using carefully designed prompt variations, then ensemble them through sophisticated aggregation methods to produce refined summaries. This prompt diversity enables models to capture different aspects and perspectives of the source documents, leading to more comprehensive and higher-quality summaries. To evaluate our method effectively, we also introduce two new LLM-based metrics: the Preference Alignment Score (PAS) and LLM Atom-Content-Unit score (LLM-ACU), which assess summary quality while addressing the positional bias inherent in automatic evaluations performed by LLMs.Our experiments demonstrate that leveraging prompt diversity significantly enhances summary quality, while also revealing the practical scaling boundaries for MDS tasks.

## introduction test-time scaling (or inference-time scaling) has emerged as a promising approach for enhancing llm's performance beyond traditional architectural or data improvements @xcite . while earlier work focused on relationships between models' capabilities, size, and training resources, recent research demonstrates that strategic compute allocation during inference can yield substantial performance gains. for instance, studies show that increased inference computation produces better results than equivalent investments in pretraining @xcite @xcite . research on test-time scaling has largely centered on logical and math reasoning tasks, leaving traditional natural language generation (nlg) tasks relatively unexplored. this gap is particularly notable in summarization, a domain where llms have already demonstrated significant advances, generating summaries competitive with human performance @xcite @xcite . beyond text generation, llms have also been proven effective as judges when guided by well-designed evaluation protocols @xcite . recent expansions in context window sizes have created new opportunities to study scaling effects on length-constrained tasks like summarization @xcite . however, llms still struggle with key challenges including hallucination, incomplete coverage, language inconsistency, and verbosity @xcite . in this paper, we aim to examine llms' summarization capabilities and their scaling properties by focusing on the multi-document summarization (mds) task. mds requires synthesizing and linking information across lengthy documents, handling information redundancy, maintaining factual consistency, and generating coherent and concise summaries while preserving key details. in addition, mds demands effective reasoning to determine relevance and priority among diverse pieces of information. these characteristics make mds particularly time-and labor-intensive @xcite . to tackle these challenges, we propose a multi-agent approach that leverages prompt ensemble to scale summarization at test time. while traditional prompt ensemble methods exist -such as (a) applying different sampling strategies to a single prompt @xcite , or (b) varying few-shot examples within prompts @xcite , their direct application to summarization presents notable limitations. the first approach merely explores variations in the output space, while the second heavily relies on example-based learning, which is better suited for reasoning tasks. furthermore, summarization differs fundamentally from reasoning tasks, where specific prompts like "let's think step by step" @xcite can effectively guide models through predetermined reasoning patterns @xcite ). in contrast, no single "optimal" prompt exists for generating summaries that satisfy diverse requirements. given these distinctions, summarization demands a more sophisticated approach to prompt ensemble techniques. therefore, we propose multi 2 framework (fig. 2 ) to address this challenge. after generating multiple summaries through diverse prompts while maintaining consistent requirements, we employ aggregation to construct a comprehensive final summary that leverages the strengths of each summary candidate. while increased inference-time com-putation generally improves performance, recent studies have also identified an inverse scaling phenomenon, where excessive computation at test-time can paradoxically degrade performance @xcite . we also investigate this phenomenon by systematically varying the number of samples and examining its boundaries. another challenge in mds is the reliability of automatic evaluation metrics. traditional metrics like rouge @xcite have proven insufficient for capturing summary quality, while more recent llm-based metrics such as auto-acu @xcite , llmcompare @xcite , and llmrank @xcite show limitations, including constraints in contextual understanding for smaller models and persistent positional biases @xcite . we specifically highlight positional bias, where llms tend to favor summaries appearing in a particular position (first or second in a pairwise comparison), leading to inconsistencies in evaluation, particularly during test-time scaling. to improve evaluation consistency, we propose two novel metrics: preference alignment score (pas) and llm atom-content-unit (llm-acu) score. these metrics aim to leverage llms' contextual understanding while incorporating mechanisms to mitigate positional bias, ensuring more reliable and robust summary assessment. in summary, (1) we present the first comprehensive investigation of test-time scaling laws in text summarization, extending the analysis beyond traditionally explored reasoning tasks; (2) we introduce a new framework multi 2 that enhances summarization performance through prompt ensemble at test time; (3) we enhance two existing evaluation protocols for summarization through strategic modifications and incorporating llms, improving quantitative assessment of summary quality and advancing automatic evaluation methodologies for summarization tasks. ## prompt ensemble: a formal formulation let x denote the input text and @xmath0, p 2 , . . . , p n } be a collection of prompts designed to elicit different aspects of information from the underlying language model. for each prompt p @xmath1, the model produces an output y i according to a generation function f : prompt formulation ok, these seems to be different instructions… let me handle the requests differently! you will receive several documents on the same topic. your task is to write a brief aspect-based summary, using only the information from the provided documents and excluding any external sources. given a set of documents related to a specific topic, generate a short, concise aspect-based summary. ensure that the summary is based solely on the content of the documents provided. ## conclusion in this work, we introduced the multi 2 framework to scale mds through prompt ensemble, showing that we can leverage computational resources at test time to produce more comprehensive and accurate summaries. our metrics, pas score and llm-acu score also provide more reliable assessments by effectively mitigating positional bias in summary evaluation. through systematic analysis, we identified specific scaling boundaries in summarization tasks, offering valuable insights into scaling summarization. our findings suggest two promising research directions: (1) incorporating test-time search algorithms to dynamically guide prompt ensemble optimization, and (2) extending our evaluation metrics to assess model performance in reasoning tasks. these directions highlight the potential of optimizing llms' inference-time behavior across applications where both factual accuracy and logical consistency are crucial.