Topic-Guided Reinforcement Learning with LLM s for Enhancing Multi-Document Summarization (2025)

Abstract: A key challenge in Multi-Document Summarization (MDS) is effectively integrating information from multiple sources while maintaining coherence and topical relevance. While Large Language Models (LLMs) have shown impressive results in single-document summarization, their performance on MDS still leaves room for improvement. In this paper, we propose a topic-guided reinforcement learning approach to improve content selection in MDS. We first show that explicitly prompting models with topic labels enhances the informativeness. Building on this insight, we propose a novel topic reward within the Group Relative Policy Optimization (GRPO) framework to measure topic alignment between the generated summary and source documents. Experimental results on the Multi-News and Multi-XScience datasets demonstrate that our method consistently outperforms strong baselines, highlighting the effectiveness of leveraging topical cues in MDS.

## introduction multi-document summarization (mds) aims to generate a concise and coherent summary that captures the salient information from a collection of related documents. while recent advances in large language models (llms) and prompting strategies have significantly improved the performance of abstractive summarization systems, existing mds methods still struggle to maintain content relevance, coherence @xcite , and topic consistency @xcite , especially when synthesizing information across multiple sources @xcite . one important yet relatively underexplored direction in mds is the incorporation of high-level discourse information to guide the summarization figure 1 : performance on multi-news @xcite using prompting (base) and topic-incorporated prompting (tn; n means number of topic labels) with qwen2.5-series model @xcite . the geometric mean of rouge-1/2/l scores are reported. topic key words are previously generated using a teacher model: qwen2.5-7b. we see that topic-enhanced instruction (t5 and t10) improves small llms' (0.5b and 1.5b) performances over standard prompt (base). in this work, we investigate the role of explicit topic guidance in enhancing generic mds. different from previous studies that incorporate topic distributions or learns latent topics via neural topic modeling, we propose a more direct and interpretable strategy: guiding summarization models using topic phrases explicitly extracted from the source documents. we begin with a simple yet insightful observation: prompting llms with extra topic information improves mds quality in terms of informativeness. figure 1 shows that when small llms (qwen2.5-0.5b and 1.5b) are applied to summarization tasks, they show notable improvements if prompted with topic labels ("t5" and "t10"), compared to using standard summarization prompt ("base"). this motivates us to go beyond static prompting and incorporate topic awareness more directly into the training objective. to this end, we introduce a novel reference-free topic-reward function that quantifies how well a generated summary aligns with its intended topics derived from each source document, see figure 2 for an overview. our key assumption is that increasing the topical similarity between the generated summary and source documents will in turn improve the quality of summary generations. accordingly, our reward is defined with respect to the improvements from (1) coverage: how well the gen-erated summary covers important topics in source documents, and (2) precision: how relevant the topics in summary are to the source documents. the final reward signal is a harmonic mean, which is then integrated into the group relative policy optimization framework @xcite to enable reinforcement learning with topic-guided feedback. specifically, we employ qwen2.5-7b @xcite within the reward model to generate topic labels for a given document-either a source article or a summary-while using the smaller qwen2.5-0.5b model as the policy model. this setup also mirrors a knowledge distillation paradigm where the larger language model transfers topic-related knowledge to the smaller model to guide its learning process. we evaluate our method on two widely-used datasets: multi-news @xcite and multi-xscience @xcite , and demonstrate that our topic-aware training strategy leads to consistent improvements over standard and reinforcement learning from human feedback (rlhf)-guided baselines, as measured by both informativeness metrics (e.g., rouge, llm score) and topic alignment evaluation. in summary, (1) we show that using topic information improves mds performance, both via prompting and llm-based rl; (2) we introduce a novel topic reward to measure source-summary discourse alignment, which is integrated into grpo to perform topic-guided summarization; (3) empirical results indicate that topic-level signals represent a valuable yet underexploited form of supervision, yielding even stronger performance when combined with reference-based rewards like rouge. discourse-guided summarization although topic modeling has been widely used for documentlevel content understanding, its application to summarization has been relatively limited @xcite . @xcite explored topic representations using semantic themes (e.g., predicate-argument structures), which required a semantic parser to extract this structural information; @xcite used ldastyle probabilistic topic models @xcite to select topic-relevant sentences and showed improvement in terms of redundancy; @xcite and @xcite introduced discourselevel and topic-aware attention mechanisms to enhance long document summarization. another line of work involves discourse-level planning, where models generate summaries conditioned on given keywords @xcite , entities @xcite , or high-level concept @xcite . these approaches aim to control the focus of the summary based on user intent or query, while our work focuses on generic summaries that holistically represent the source content using topical information. reinforcement learning (rl) for summarization rl methods has been applied to summarization more broadly to optimize non-differentiable objectives, such as rouge @xcite @xcite or human preferences @xcite @xcite . other approaches incorporate task-specific rewards, e.g., @xcite learned models of coherence from existing text and used them as rl rewards; @xcite incorporated entailment-based consistency rewards to improve the saliency of a good summary; @xcite optimized multiple dimensions such as relevance and fluency. we extend this line of work by introducing a novel reference-free discourse-level reward that explicitly promotes topical alignment between summaries and source documents-a dimension overlooked by existing metrics. unlike prior studies focused on single-document summarization, we address the more challenging multidocument setting, where certain proposed rewards (e.g., @xcite ) become impractical. ## incorporating topic labels into mds via prompting topic phrases succinctly capture essential information from source documents, providing effective high-level guidance for summarization-our motivation aligns with prior work on entity chains @xcite , which utilized ordered sequences of entities as intermediate representations to plan and ground abstractive summary generation. however, unlike the controlled entity sets used in entity chains, we treat our topics as openended keywords and phrases. additionally, rather than incorporating entity generation directly into conditional summarization, we adopt a two-step framework where the topic extraction model is separate from the summarization model. this modular design enables independent analysis of topic extraction's impact on summarization quality (see §6.6), and provides the flexibility to incorporate more advanced topic models in future experiments. in this section, we conduct experiments in a zeroshot setting, carefully designing prompts and configurations to best leverage topic-augmented mds. prompting with topics formally, given a set of source documents with corresponding topic labels, we prompt a llm for summarization as follows: teacher-supervision mode we examine llm capabilities by comparing their performance of varying scales (qwen2.5-0.5b, 1.5b, and 7b). unsurprisingly, the largest summarization model (7b) achieves the highest baseline performance (average rouge 19.9), significantly surpassing smaller models (see "base" in figure 1 ). we employ a teacher-supervision mode, where the larger 7b model explicitly provides topic guidance for the smaller models (0.5b and 1.5b). under this setting, smaller llms clearly benefit from improved topical information provided by the teacher model. however, the 7b model itself, which inherently possesses strong topical modeling capabilities, experiences no gains from self-generated topic labels. ## llm reinforcement learning for mds based on the above observations, we propose a novel topic-guided reward ( §4.1) designed to maximize semantic similarity between generated summaries and source documents, coupled with a length penalty ( §4.2) to better control the generation length. we implement these rewards using an inverse standard deviation weighting strategy ( §4.3) through the recent group relative policy optimization (grpo) framework ( §4.4). see figure 2 for the overview of our pipeline. ## conclusion we introduce an interpretable, reference-free topicguided rl approach for mds, leveraging a novel topic-f1 reward that aligns summary topics with source documents. integrated within the grpo framework, our method consistently outperformed strong baselines, demonstrating the value of explicit topic guidance. looking forward, we aim to enrich our framework by exploring advanced neural topic modeling techniques @xcite for more refined topical guidance. moreover, incorporating innovative reward signals, such as llm-as-a-judge evaluation @xcite , could further align summaries with human preferences and enhance selfconsistency. extending our topic-guided approach to interactive, query-based scenarios-where users specify key points to summarize-also presents an exciting future direction.