C on TR e G en: Context-driven Tree-structured Retrieval for Open-domain Long-form Text Generation (2024)

Abstract: Open-domain long-form text generation requires generating coherent, comprehensive responses that address complex queries with both breadth and depth. This task is challenging due to the need to accurately capture diverse facets of input queries. Existing iterative retrieval-augmented generation (RAG) approaches often struggle to delve deeply into each facet of complex queries and integrate knowledge from various sources effectively. This paper introduces ConTReGen, a novel framework that employs a context-driven, tree-structured retrieval approach to enhance the depth and relevance of retrieved content. ConTReGen integrates a hierarchical, top-down in-depth exploration of query facets with a systematic bottom-up synthesis, ensuring comprehensive coverage and coherent integration of multifaceted information. Extensive experiments on multiple datasets, including LFQA and ODSUM, alongside a newly introduced dataset, ODSUM-WikiHow, demonstrate that ConTReGen outperforms existing state-of-the-art RAG models.

## introduction large language models (llms) have transformed various domains through their remarkable performance across a spectrum of tasks. however, llms often struggle with generating hallucinated or factually incorrect content, particularly when addressing knowledge-intensive tasks in opendomain settings @xcite . these limitations typically arise from either the lack of long-tail relevant knowledge or reliance on outdated information embedded within their parameters. to address these challenges, retrieval-augmented generation (rag), therefore, has emerged as a promising solution @xcite @xcite . 1 code and data are available at https://github.com/kkroy2/ contregen rag enhances llms by incorporating external knowledge from a corpus, effectively reducing hallucinations and factual errors in knowledge-driven tasks such as question answering. recently, significant progress has been made in open-domain question answering, particularly in the realm of short-form answers @xcite @xcite . rather than adhering to the traditional one-step retrieve-then-generation approach, recent works increasingly adopt iterative retrieval processes @xcite . these methods utilize either previously generated responses @xcite @xcite , or employ a sequence of follow-up questions or interlinked queries @xcite @xcite . such iterative, chain-like strategies have proven particularly effective in short-form factoid question-answering tasks where questions often demand direct, specific pieces of information @xcite and their supporting passages commonly form chain-reasoning of facts @xcite @xcite as in figure 1(a) . moving beyond short-form question answering, real-world queries often entail greater complexity and necessitate more comprehensive, detailed responses that encompass multiple facets @xcite , prev. response segment @xcite , next followup question @xcite , next query generation @xcite . in our experiments with chain-like iterative approaches on lfqa and odsum tasks, we identified several limitations. one notable issue is the low retrieval recall of performance across the datasets in figure 2 , especially evident in odsum datasets where the need for multi-faceted information is more implicit. secondly, we observed that these iterative approaches quickly reach a plateau in terms of retrieval recall as demonstrated in figure 3 . in other words, they fail to retrieve new relevant passages during subsequent iterations. one of the key reasons behind these limitations is the simplistic modeling of the reasoning of retrieving passages as a chain of semantic aspects as shown in figure 1 (a). however, when the retrieval of information spans multiple facets and originates from diverse sources, it naturally forms a complex retrieval reasoning structure as illustrated in fig- @xcite , prev. response segment @xcite , next followup question @xcite , next query generation @xcite ure 1(b). for example, a query like 'how to detect hidden cameras and microphones' may require information from multiple facets such as 'conducting a physical search', 'searching for electrical signals', and so on, where each facet may need information from its multiple subfacets. for instance, this 'conducting a physical search' facet needs information about its subfacets such as 'investigating your smoke detectors and other electronics.', 'using to check for two-way mirrors', etc. consequently, this yields a tree structure of facets that organizes the need for information from the broadest to the most specific ones. furthermore, the method of formulating subsequent retrieval queries plays a crucial role in thoroughly exploring all facets of an input query. current iterative approaches often generate the next query based on segments of the previous response or all previously retrieved information. we argue that this sequential approach to query formulation inherently restricts the retrieval search scope. this is because it tends to allow a few aspects of the information to dominate the retrieval process, potentially leading to a biased exploration where certain facets are disproportionately emphasized, while others may remain underexplored or unexplored. to accommodate these findings, we introduce a novel tree-structured retrieval augmented generation framework (contregen) that conceptualizes the retrieval process as a hierarchical exploration of an input question or query's various facets, organized in a tree structure. each branch of this tree represents a specific facet, enabling a systematic and comprehensive exploration of the query. this method enables an in-depth exploration of ## methodology our framework addresses the task of context-driven long-form text generation which aims to generate a comprehensive long text, y for an input query or question, denoted as, q by leveraging relevant passages from a given corpus c. our proposed framework is structured into a two-stage process designed to enhance the retrieval and generation of long-form text responses: 1) during the top-down planning and retrieval stage as shown in figure 4 (a), the process begins by retrieving passages directly related to the input query or question. utilizing the insights gained from these initial passages, the planning agent generates a series of subquestions or subqueries, which serve as a plan for deeper exploration. each of these subquestions is recursively used to generate successive plans to continually expand the search scope of relevant passages. this recursive planning and retrieval results in a comprehensive tree structure, where each node represents a query and its associated retrieved passages; 2) in figure 4 (b), the bottom-up synthesis and generation stage of the framework begins at the leaf nodes, summarizes the relevant information from retrieved documents. the model then works its way upwards, synthesizing the retrieved documents and the summarized information from descendant nodes to generate text responses for each intermediate node. this ascending integration continues until it reaches the root of the tree, producing a final response that comprehensively addresses the original input question or queries. this methodical approach ensures that every facet of the query is thoroughly explored, allowing for a detailed and well-structured response generation. ## wikihow: a new odsum dataset open-domain summarization (odsum) is to generate a comprehensive summary for a given query by gathering information from a large index of documents. one of the key challenges in opendomain summarization (odsum) is the scarcity of datasets that encompass real-world queries alongside a well-curated document corpus @xcite . while a recent work @xcite constructed a dataset odsum-story from squal-ity @xcite , question-based abstraction summarization for short stories. however, the corpus size and the diversity of queries (only from the story domain) are limited as shown in ## related works recently, considerable research has focused on retrieval-augmented generation (rag) across various nlp tasks @xcite . existing approaches to single-time retrieval augmented generation typically involved retrieving knowledge directly from the input itself @xcite @xcite , using expanded queries @xcite , or rewrit-ten queries @xcite . however, as these methods often struggle to retrieve all relevant passages in one go, recent developments have shifted towards iterative retrieval approaches. these involve llms actively interacting with the retrieval process by formulating contextually rich subsequent queries, which can include utilizing full or partial responses @xcite @xcite , or generating new queries @xcite @xcite . additionally, adaptive retrieval-augmented approaches have emerged, which decide when to retrieve based on token probability @xcite , by generating explicit retrieval token @xcite , or by leveraging the llm's self-knowledge. no iterative approaches leverage hierarchical exploration of diverse facets of input queries in-depth. a recent paper memwalker @xcite constructs a tree structure from a given long text irrespective of queries. the model traverses the structure upon receiving an input query and generates the response, whereas our contregen constructs a query-focused tree by retrieving the relevant information from a corpus. it focuses on understanding long text while contregen focuses on both retrieving relevant contexts and leveraging them to generate the response. similar to memwalker @xcite , another recent paper @xcite utilizes a bottom-up method to build a hierarchical tree by clustering and summarizing text chunks, transforming a flat corpus into a multi-layered one for varied-level information retrieval. however, this approach may struggle with very large retrieval corpora and highly diversified queries because the clustering and summarization processes are conducted independently of the queries. in contrast, contregen employs a top-down strategy, systematically branching out from the main query into sub-queries that explore different facets and uses query-focused bottom-up summarization, enhancing relevance and effectiveness in information summarization without altering the retrieval corpus. ## conclusion in this paper, we introduced contregen, an innovative framework designed to enhance the capabilities of open-domain long-form text generation through a novel context-driven, tree-structured retrieval approach. contregen organizes the re-trieval process hierarchically, allowing for an indepth exploration of various facets of input queries, and integrates a systematic bottom-up synthesis. this approach addresses the prevalent challenges in the field by ensuring comprehensive coverage and coherent integration of multi-faceted information. additionally, our retrieval-only contre module is model-agnostic and can be seamlessly adapted to various advanced generation techniques. our extensive experiments across multiple datasets have demonstrated that contregen significantly outperforms existing state-of-the-art retrieval-augmented generation models. these results not only validate the effectiveness of our approach but also underline its potential to be a versatile tool in enhancing the generation of contextually rich, accurate longform content. moreover, the introduction of the odsum-wikihow dataset is particularly significant for furthering research in open-domain summarization. to enhance the robustness of the generation process against irrelevant contexts. developing advanced generation techniques tailored for contregen represents a promising direction for future research. moreover, the planning agent in contregen, which is crucial for generating and organizing subqueries, could benefit from explicit training. this training would enable the agent to adaptively generate plans and autonomously determine the necessary depth of exploration for each aspect of a query. currently, contregen may generate vague or non-informative sub-queries, and its two-step verification process relies heavily on the llm's reasoning and existing knowledge base. despite its impressive retrieval capabilities, there is significant potential for enhancing the system by implementing a more sophisticated model. such a model would not only verify but also generate contextually rich, retrieval-effective sub-queries, thereby substantially improving both the accuracy and relevance of the retrieval process. furthermore, as this work focuses on long-form text generation, it would be an interesting future direction to explore tree-structured retrieval in single factoid answer scenarios.