G raph RAG Analysis for Financial Narrative Summarization and A Framework for Optimizing Domain Adaptation (2025)

Abstract: Large Language Models (LLMs) have shown promise in summarizing complex documents, but their limitations in handling lengthy documents and capturing global information hinder their performance in tasks like Query-Focused Summarization (QFS). We explore GraphRAG, a retrieval-augmented generation approach that utilizes a globally summarized knowledge graph derived from an LLM. We apply GraphRAG to the Financial Narrative Summarization (FNS) dataset, which consists of lengthy financial reports. Our results show that a naive RAG approach outperforms GraphRAG in terms of comprehensiveness, directness, conciseness and completeness. However, we demonstrate that optimizing entity and relation extraction using an LLM as an optimizer can enhance GraphRAG’s performance. Our study highlights the need for domain-specific optimization to improve GraphRAG’s capabilities for summarization tasks in facts-heavy domains like finance. We propose an optimization framework that extends GraphRAG’s original domain adaptation strategy by incorporating entity and relations optimization, leading to improved performance in capturing relevant entities and relationships. Our findings contribute to the development of more effective summarization models for complex documents in finance and other domains.

## introduction large language models (llms) have shown promise in analyzing complex documents and generating summaries, but they face significant challenges in summarizing lengthy documents due to restrictions on their context windows. the expansion of such windows may not be enough given that information can be "lost in the middle" of longer contexts @xcite . retrieval-augmented generation (rag) @xcite ) is a method that can overcome these limitations, but it struggles with capturing global information and addressing global queries, such as 'what are the main themes in the dataset?' this limitation, particularly its inability to effectively capture global information hinders its performance in tasks such as query-focused summarization (qfs), where a broader understanding of the data is necessary @xcite . to address these limitations, an approach called graphrag @xcite has been proposed, which utilizes a globally summarized knowledge graph derived from an llm to unlock llm discovery on narrative private data foot_0 . building on this work, our research efforts focus on two key areas: our work is mainly directed towards two key areas: ## overview of financial narrative summarization 2023 dataset the fns 2023 task dataset 2 has been extracted from annual financial reports in pdf file format. the reports were written in english, spanish, and greek. for the dataset compilation, two to three people had to work for each language. for this work, we utilized english dataset which contains approximately 4,000 uk annual reports for firms listed on lse, covering the period between 2002 and 2022 @xcite . in total, there are 4,013 annual reports divided into training, testing, and validation sets. ## experimental setup instead of summarizing the complete report, the fns task requires locating key narrative sections found in the annual reports and generate a single structured summary for them in not more than 1000 words (figure 3 ). we utilized dimsum @xcite for narrative section identification and extraction . this system was the top performer in the fns 2023 task @xcite . ## results and analysis our results ( our optimized domain adaptation approach enhanced graph rag's ability to generate more accurate and detailed summaries by embedding enriched entity relationships as context. as shown in appendix a ( despite the optimization, naive rag still outperforms graphrag ( in contrast, while entity recognition is improved, graphrag's ability to extract all relevant entities and establish detailed relationships remains limited. this constraint hampers its capacity to construct a comprehensive knowledge graph. by prioritizing relational and community-level summarization, graphrag often sacrifices critical details, leading to summaries that are high-level and less informative. for example, it mentions growth trends without providing comparative figures and references acquisitions without specifying details. furthermore, it omits external contextual factors, such as brexit, which are essential for a nuanced analysis. ## conclusion and future work in conclusion, our study reveals that naive rag outperforms graph rag in extracting actionable insights from financial metrics, strategies, and market trends, as the latter's reliance on identified entities can lead to off-topic deviations. nevertheless, graph rag shows potential by harnessing the power of training data and large language models (llms) as optimizers, which successfully align entities and relationships to produce relevant objective summaries. notably, our findings suggest that the incorporation of an entity-relationship graph does not necessarily guarantee superior content summaries. future studies would benefit from incorporating metrics to assess the accuracy of extracted entities and relationships, offering insights into graphrag's potential in this domain. exploring methods to construct graphs based on thematic elements, such as revenue performance variations, and generating subgraphs for each theme, could further enhance graphrag's capabilities. thematic summaries can be employed to identify relevant entities and relationships tied to specific themes, improving the graph's alignment with the narrative content. our findings underscore the importance of domain-specific optimization to refine graphrag for summarization tasks in data-intensive sectors like finance.