SKETCH : Structured Knowledge Enhanced Text Comprehension for Holistic Retrieval (2025)

Abstract: This paper discusses about the SKETCH approach which enhances text retrieval and context relevancy on large corpuses compared to the traditional baseline methods. The abstract attached below discusses this further. Abstract: Retrieval-Augmented Generation (RAG) systems have become pivotal in leveraging vast corpora to generate informed and contextually relevant responses, notably reducing hallucinations in Large Language Models. Despite significant advancements, these systems struggle to efficiently process and retrieve information from large datasets while maintaining a comprehensive understanding of the context. This paper introduces SKETCH, a novel methodology that enhances the RAG retrieval process by integrating semantic text retrieval with knowledge graphs, thereby merging structured and unstructured data for a more holistic comprehension. SKETCH, demonstrates substantial improvements in retrieval performance and maintains superior context integrity compared to traditional methods. Evaluated across four diverse datasets: QuALITY, QASPER, NarrativeQA, and Italian Cuisine—SKETCH consistently outperforms baseline approaches on key RAGAS metrics such as answer relevancy, faithfulness, context precision and context recall. Notably, on the Italian Cuisine dataset, SKETCH achieved an answer relevancy of 0.94 and a context precision of 0.99, representing the highest performance across all evaluated metrics. These results highlight SKETCH’s capability in delivering more accurate and contextually relevant responses, setting new benchmarks for future retrieval systems.

## introduction large language models (llms), despite their growing size and capabilities, often lack sufficient domain-specific knowledge for certain tasks @xcite , and their encoded facts can quickly become † this work does not relate to the position at amazon ai. ‡ this work does not relate to the position at meta. outdated due to the dynamic nature of information. updating the knowledge within llms through fine-tuning or editing is a complex and resourceintensive process, especially when dealing with extensive text corpora @xcite . an alternative approach is retrieval augmented generation (rag) @xcite , which involves indexing large volumes of text, divided into smaller segments, in a separate information retrieval system @xcite . retrieved information is then provided to the llm along with the query as context, enabling more factually updated answers @xcite . this method offers benefits such as access to current, domain-specific knowledge, improved interpretability, and provenance tracking, which are often lacking in the opaque parametric knowledge of llms @xcite . however, existing rag methods have notable limitations @xcite . they typically retrieve only a few short, contiguous text segments, limiting their ability to represent and leverage large-scale discourse structures. this limitation is particularly problematic for complex queries that require integrating knowledge from multiple parts of a text, such as synthesizing information across chapters or drawing conclusions from dispersed data in scientific literature. moreover, these methods often struggle with multi-hop reasoning, where answering a query necessitates combining information from multiple, non-adjacent text segments, leading to incomplete or inaccurate answers and reducing their usefulness in applications that demand comprehensive understanding and synthesis. to address these challenges, we introduce sketch, a novel methodology that enhances the retrieval process in rag systems by integrating semantic text retrieval with knowledge graphs @xcite . this integration merges structured data (knowledge graphs) and unstructured data (text embeddings), enabling a holistic comprehension of the dataset and facilitating the retrieval of relevant information across multiple contexts. by combining these approaches, sketch can perform multi-hop reasoning and retrieve contextually relevant information even if it is dispersed throughout the corpus. we evaluate sketch on diverse datasets, including quality @xcite , qaer @xcite , narrativeqa @xcite , and an italian cuisine corpus, covering a range of domains and presenting various challenges. our results demonstrate significant improvements in answer relevancy and context precision metrics compared to baseline methods. sketch outperforms traditional rag approaches by maintaining context integrity and delivering more precise, contextually enriched responses, setting new benchmarks for rag systems in handling large-scale discourse structures and complex queries across various domains. ## results and discussion we evaluated the performance of sketch across four diverse datasets: the "italian cuisine" dataset, quality, qasper, and narrativeqa. our analysis focused on comparing the effectiveness of sketch in enhancing retrieval accuracy and contextual relevance against existing approaches, including naive rag, raptor, semantic-only, and kg-only methods. to ensure a comprehensive evaluation, we assessed four key ragas metrics: answer_relevancy, faithfulness, context_precision, and context_recall. gpt-3.5-turbo-16k served as the evaluation judge for all datasets, ensuring consistency in assessing the quality of responses generated by each method. the results, visualized in figure 2 , clearly demonstrate sketch's significant advantage over other models in retrieval performance. sketch consistently performed better across multiple datasets, showing particular strength in answer relevancy, context precision metrics, making it a superior choice for accurate and contextually rich retrieval. each plot in figure 2 contrasts sketch with baseline models, providing a detailed visual representation of how sketch excels in various metrics, further emphasizing its robustness and balanced capability compared to the other tested approaches. ## conclusion and limitations this research introduced sketch, an innovative methodology that enhances retrieval-augmented generation (rag) systems by combining semantic chunking with knowledge graphs. by integrating structured and unstructured data, sketch addresses the limitations of traditional retrieval methods, such as context loss in large datasets and limited comprehension of complex queries. our experiments on diverse datasets-including italian cuisine, quality, qasper, and narra-tiveqa-demonstrate sketch's superior ability to maintain context integrity and generate highly relevant responses, consistently outperforming baseline methods like naive rag, raptor, semantic-only, and kg-only approaches. sketch achieved remarkable improvements across key metrics, showcasing its adaptability to both short and long documents and its effectiveness in navigating complex texts. for instance, on the italian cuisine dataset, it achieved an answer relevancy score of 0.94 and context precision of 0.99, improving over naive rag by 54.1% and 22.2%, respectively. on the quality dataset, sketch improved answer relevancy by 49%, and on the qasper dataset, it doubled the answer relevancy score over naive rag while increasing context precision by 139.29%. even on the challenging narrativeqa dataset, sketch delivered balanced performance with a 525% improvement in answer relevancy and an 866.67% increase in faithfulness over naive rag. while kg achieved the highest f1 score (0.52), sketch closely followed with 0.48, demonstrating its balanced effectiveness despite the complexity of narrative content. these findings confirm that sketch's combined approach offers a robust framework for advancing rag systems, setting new benchmarks in accuracy, context comprehension, and cross-domain xapplicability, and paving the way for future advancements in natural language processing. limitations while sketch significantly improved answer relevancy and context precision, certain limitations remain. faithfulness, though better than most baselines, still lags behind naive rag on quality (0.69 vs. 0.83, a 16.9% shortfall). scalability and cost are also concerns, as constructing large-scale knowledge graphs is laborintensive, and relying on paid llms like gpt-4 for semantic evaluation increases expenses. furthermore, sketch's dependence on gpt models for query parsing and ragas evaluation can introduce errors and variance due to sampling randomness, prompt sensitivity, and occasional hallucinations, potentially affecting reproducibility. employing sampling strategies (e.g., multiple runs) and aggregating judgments could help mitigate these issues. future work should focus on reducing computational costs, refining knowledge graph construction, and improving metrics like context recall and faithfulness to achieve more consistent, high-quality, and stable retrieval results.