Context-Aware Hierarchical Merging for Long Document Summarization (2025)

Abstract: Hierarchical Merging is a technique commonly used to summarize very long texts (>100K tokens) by breaking down the input into smaller sections, summarizing those sections individually, and then merging or combining those summaries into a final coherent summary. Although it helps address the limitations of large language models (LLMs) with fixed input length constraints, the recursive merging process can amplify LLM hallucinations, increasing the risk of factual inaccuracies. In this paper, we seek to mitigate hallucinations by enriching hierarchical merging with context from the source document. Specifically, we propose different approaches to contextual augmentation ranging from *replacing* intermediate summaries with relevant input context, to *refining* them while using the context as supporting evidence, and *aligning* them implicitly (via citations) to the input. Experimental results on datasets representing legal and narrative domains show that contextual augmentation consistently outperforms zero-shot and hierarchical merging baselines for the Llama 3.1 model family. Our analysis further reveals that refinement methods tend to perform best when paired with extractive summarization for identifying relevant input.

## introduction large language models (llms) have made significant strides in processing longer contexts, with models like gemini (team, 2024a) reportedly being able to handle up to 2 million tokens. despite promising potential for complex realworld applications such as long-document questionanswering and repository-level code understanding, most llms, especially open-source ones, struggle with tasks operating over long input sequences @xcite . a notable application we focus on in this paper is the summarization of very long documents in domains such as law and narrative which typically exceed input size limitations of most current models (>100k tokens). previous studies have introduced specialized techniques for summarizing very long texts, with hierarchical merging @xcite being one of the most popular methods. as illustrated in figure 1 , hierarchical merging first divides an input document into chunks of fixed token length, and a summary is generated for each chunk. in subsequent stages, consecutive chunklevel summaries are concatenated and merged until they reach a pre-defined length limit, after which a new summary is generated from the merged text. merging continues until a single final summary is produced. all summary generation steps are performed using llms and zero-shot prompting. although hierarchical merging effectively handles inputs exceeding model length limits, it cannot produce faithful summaries even when using the most performant llms @xcite . in this paper, we seek to mitigate hallucinations and enhance summary faithfulness by enrich-ing hierarchical merging with relevant context from the source document. access to input information during the merging process should provide models with factual information to generate more accurate summaries. a key question concerns how relevant context can be identified and used in the merging process. we propose three methods for selecting context from the source document: 1) extractive summarization identifies key sentences from the input to produce a coherent abstractive summary; 2) retrieval uses initial summaries as queries to retrieve relevant input passages, and 3) attribution generates summaries with references to input passages. once relevant context is identified, it can either replace the abstractive summary or serve as supporting evidence for subsequent merging steps. experimental results with the llama-3.1 model family, demonstrate consistent improvements (across metrics) on summary faithfulness and quality, compared to the original hierarchical merging method and zero-shot summarization. our analysis further indicates that abstractive summaries play a crucial role in hierarchical merging by ensuring comprehensiveness, while augmenting them with relevant contexts enhances summary faithfulness. our contributions are summarized as follows: the literature is rife with proposals aiming to mitigate llm difficulties with processing inputs that exceed their context lengths. a significant challenge in overcoming the context limit is addressing the quadratic computational burden of the selfattention mechanism. previous work has attempted to reduce this cost by architectural modifications such as introducing sparse attention @xcite , linearized attention @xcite , changes to positional encoding @xcite , and special-purpose fine-tuning methods which directly modify the attention mechanism @xcite or incorporate discourse-level information. another line of work adopts a divide-andconquer approach, splitting the long input into manageable chunks. the chunks can be processed independently @xcite or through progressively merging adjacent chunks as they are processed along the transformer layers @xcite . @xcite propose a dive-and-conquer approach for long document summarization in which an llm is fine-tuned via reinforcement learning to summarize each chunk and then hierarchically merge chunk-level summaries into a final summary. their method has since been adapted to use zero-shot prompting for summary generation @xcite , without further training (as shown in figure 1 ). our proposal modifies hierarchical merging to consider evidence from the source document at the intermediate summary generation stage. it draws inspiration from retrieval augmented generation (rag) techniques which have proven particularly effective in long-form question answering @xcite @xcite , as a means to integrate upto-date information, mitigate hallucinations, and enhance response quality. our work explores various proposals for context integration, e.g., as a replacement of intermediate summaries or additional supporting evidence, as well as retrieval augmentation methods. in addition to traditional query-based retrieval which we implement using the intermediate summaries as queries, we also turn to extractive summarization to select and rank source document sentences that best represent its content. various methods have been proposed for long document extractive summarization, focusing on reinforcement learning @xcite and adapting llms for sentence extraction @xcite . however, these methods were primarily evaluated on datasets with relatively short inputs, such as arxiv @xcite ; 5k tokens) and govreport @xcite ; 10k tokens), making their effectiveness on inputs exceeding 100k tokens uncertain. as we identify salient sentences within each chunk, we expect extractive methods to be able to isolate supporting evidence for ultimately generating more factual abstractive summaries. our work also has connections to recent efforts in creating verifiable systems that generate responses to queries by incorporating citations to ## hierarchical merging with context figure 2 illustrates our proposed hierarchical merging pipeline for the first two levels. compared to the original method in figure 1 , at each level, an additional incorporate context (ic) module is used to obtain relevant contexts alongside abstractive summaries. when generating intermediate summaries for the current level, we either use the summaries from the previous level and relevant context as supporting evidence (denoted as support), or replace the summaries with their corresponding contexts (denoted as replace). at each level, the ic module takes abstractive summaries and input contexts from the previous level to obtain relevant contexts for the current level. we make sure that input contexts have similar length to their respective abstractive summaries, so that the former can serve as a substitute for the latter. figure 3 provides an overview of the ic module. at the first-level, chunks from the input document are used to obtain abstractive summaries and corresponding input contexts. in sections 3.1-3.3, we explain in detail how we propose to obtain relevant contexts. in subsequent levels, merging involves abstractive summaries (concatenated summaries) and their contexts (concatenated passages). if we choose to replace summaries with their context, we simply substitute chunks with concatenated passages and repeat the first-level process. if we use input contexts as supporting evidence (support), next level contexts are selected from concatenated passages, while next level summaries are generated by leveraging concatenated passages and concatenated summaries. prompts for summary generation and merging are given in appendix a. to obtain contexts that are as relevant as possible, we utilize and modify three different methods which we describe as follows. the implementation of these methods in our experiments is further explained in section 4. ## experimental setup datasets our experiments are conducted on two long document summarization datasets. the first dataset, multi-lexsum @xcite , comprises u.s. federal court documents paired with expert-written summaries at multiple levels of granularity (long, short, and tiny summaries). for evaluation purposes, we utilize the long summaries as a reference and exclude documents with fewer than 100k tokens. the second dataset, supersummary, is a novel book summarization corpus that we constructed using literary works published after 2022. the corresponding expert-written plot summaries were obtained from supersummary 1 through a paid subscription service. similarly, we employ these plot summaries as gold summaries for evaluation and exclude documents containing fewer than 100k tokens. implementation details our experimental evaluation was conducted using the llama 3.1 model family @xcite , which features a context length of 128k tokens and comprises two variants with 8b and 70b parameters, respectively. we use the gptq-int4 quantized 70b model @xcite in our experiments for inference efficiency. for all hierarchical merging systems, the chunk size and maximum merging context length were configured to 8k tokens, ensuring sufficient merging operations for meaningful analysis of different hierarchical merging variants. as mentioned earlier, we propose three methods for incorporating context, i.e., extract, retrieve, and cite. we use memsum @xcite , a state-of-the-art extractive summarizer trained with reinforcement learning, as our backbone model for extract. we fine-tune memsum on the official multi-lexsum training split. for supersummary, we use the training split of the publicly accessible booksum dataset @xcite . to prepare these datasets for memsum fine-tuning, each document was segmented into 8k-token chunks, from which 1,000 chunks (per dataset) were randomly selected as training inputs. subsequently, we use the full llama-3.1 70b model to generate summaries for each chunk via zero-shot prompting, which serve as reference summaries. the fine-tuning process was executed for 10 epochs with a learning rate of 1e-4. to maintain consistency with the average summary length in the evaluation datasets, the maximum number of extracted sentences was set to 20. for retrieve, we split input documents into uniform length passages with 100 tokens each. to optimize compute efficiency, we utilize the bm25 retriever to select passages using intermediate summaries as queries. the top-k parameter for selecting passages is set so as to ensure concatenated top-k passages have lengths similar to the average summary length in the evaluation datasets. similar to retrieve, we set the passage length to 100 tokens each for cite. the top-k passage selection process is described in section b. ## results tables 2 and 3 present our results for the llama-3.1 8b and 70b models, respectively. hierarchical merging with incorporated context overwhelmingly outperforms comparison systems without supporting evidence (see first block in the tables). for each metric (and dataset) the best performing model is a variant of hierarchical merging with context (the only exception is bertscore on supersummary for cite-hmerge using the 8b model). extract-support is the best model. among the proposed approaches, extract-support demonstrates superior performance, achieving the highest average score across all metrics. cite emerges as the least effective context selection method, showing superior performance only in alignscore with minor differences to second best results. when examining the impact of model size, scaling to 70b parameters yields moderate improvements on reference-based metrics, although gains on inputbased metrics are less substantial. additionally, our analysis indicates that models consistently produce lower scores on supersummary compared to multi-lexsum, suggesting that book summarization poses greater challenges than legal document summarization. replace models show big improvements according to input-based metrics. examining input-based metrics more closely, both summac and alignscore show significant improvements when input passages are incorporated via replace. specifically, for the 8b model, while support achieves similar results to baselines, replace consistently demonstrates superior performance, with retrieve-replace and cite-replace achieving approximately 10-points improvements in alignscore on supersummary compared to other methods. the same patterns are again observed in the 70b model results, where retrieve-replace excels in summac scores and cite-replace leads in align-score metrics. these results align with our expectations, given that both metrics evaluate performance against input texts, and replace explicitly leverages input context in summary generation. support models are better summarizers according to reference-based metrics. referencebased metrics reveal patterns that stand in sharp contrast to those observed with input-based metrics. in both 8b and 70b models, extract-support and retrieve-support consistently outperform baselines. however, cite-support shows comparable or slightly worse performance, suggesting that while input contexts can enhance summary faithfulness, their effectiveness depends heavily on the method's ability to identify and utilize relevant segments. compared to support methods, replace methods show significantly inferior performance across both model scales, with particularly pronounced differences on supersummary, where replace methods lag behind support by approximately 20 points across all three context selection methods in 70b models. this performance gap is further corroborated by both rouge and bertscore metrics. strongly suggest that methods incorporating abstractive summaries during hierarchical merging (as in support and baseline approaches) are better at maintaining comprehensive coverage compared to methods that rely solely on input contexts for final summary generation (replace approaches), which may become overly specific and miss broader aspects of the source material. adding more contexts yields better summaries for replace. our results in tables 2 and 3 density compared to replace. specifically, replace contexts typically elaborate on a single key event extensively, while abstractive summaries convey the same event more concisely, allowing space for additional key events. building on these observations, we investigate whether replace summaries can be improved by increasing the maximum merging context length, thereby incorporating more information while merging the same number of summaries. specifically, we extend the maximum merging context length from 8k to 16k and 32k tokens. we maintain the same retrieval units but select more passages that fit within this increased context length window, thereby generating next-level summaries with greater information density. we experiment with context extension on the 8b model using both datasets, employing retrieve-replace as the hierarchical merging method. the results, shown in context length multi-lexsum supersummary 8k 41.8 -23.3 -16k 43.7 +1.9 26.5 +3.2 32k 44.1 +2.3 27.9 +4.6 manual analysis confirms extract-support generates the most faithful summaries. our results suggest that input-and reference-based metrics tend to favor different methodological approaches. to evaluate metric reliability, we analyzed book summaries generated by the 70b model. specifically, we decomposed output summaries for the book things we hide from the light (by lucy score) into atomic claims and verified whether they are supported by the source. we followed a simplified version of the annotation guidelines proposed in @xcite . claims were classified as 'correct' if fully supported by the book, 'incorrect' if parts of the claim contradict the source, or 'not present' if parts of the claim are absent from the source material. in contrast, extract-replace performs below baseline levels, by more than 10%. this poor per- ## conclusion in this paper, we examined how input contexts can be integrated into hierarchical merging so as to produce more faithful summaries for very long source documents (>100k tokens). we proposed different approaches to contextual augmentation ranging from replacing intermediate summaries with relevant input context, to refining them while using the context as supporting evidence, and implicitly aligning summary sentences (via citations) to input passages. our findings indicate that contextual augmentation leads to more factual summaries. however, input contexts alone can distract from important information, emphasizing peripheral details, and thus work best to support rather than substitute abstractive summaries that capture a broader perspective of the source content. in the future, we would like to learn when to perform context augmentation based on the quality of the intermediate summary (i.e., calling context augmentation as a tool) and to explore trade-offs between context length and summary quality.