Knowledge Navigator: LLM -guided Browsing Framework for Exploratory Search in Scientific Literature (2024)

Abstract: The exponential growth of scientific literature necessitates advanced tools for effective knowledge exploration. We present Knowledge Navigator, a system designed to enhance exploratory search abilities by organizing and structuring the retrieved documents from broad topical queries into a navigable, two-level hierarchy of named and descriptive scientific topics and subtopics. This structured organization provides an overall view of the research themes in a domain, while also enabling iterative search and deeper knowledge discovery within specific subtopics by allowing users to refine their focus and retrieve additional relevant documents. Knowledge Navigator combines LLM capabilities with cluster-based methods to enable an effective browsing method. We demonstrate our approach’s effectiveness through automatic and manual evaluations on two novel benchmarks, CLUSTREC-COVID and SCITOC Our code, prompts, and benchmarks are made publicly available.

## introduction traditional search engines, while adept at retrieving relevant documents for specific queries, are suboptimal when dealing with broad, topical queries. such queries typically return lengthy ranked lists of potentially relevant papers, which, while comprehensive, overwhelms researchers with an information overload, obscuring the underlying structure of the topic and hindering the discovery of relevant subtopics and novel connections. simply put, researchers are presented with an extensive inventory of documents without a clear map to guide their exploration, while they are interested in understanding broader topical trends. the limitations of ranked list search results have driven a longstanding interest in methods for grouping and categorizing retrieved documents @xcite . over the years, a vast amount of research was devoted to designing different methods to support the paradigm of grouping and categorizing retrieved documents to organize them into a meaningful structure of knowledge, in many cases taking the form of hierarchical, cluster-based navigation based on automatically induced topical clusters (see §2). however, these browsing methods ultimately did not achieve widespread use and were not adopted by modern search engines, largely due to the insufficient quality of the automatically derived structures for practical application. in this work, we demonstrate that importing the cluster-based navigation paradigm to the era of large language models (llms), combined with modern nlp and ir methods, can overcome many of the obstacles faced by previous approaches. our research shows that the components required to support this paradigm perform well as stand-alone tasks, and the entire framework functions effectively end-to-end. we propose knowledge navigator foot_0 , an llmbased framework that transforms a large corpus of retrieved scientific literature into multi-level, organized themes of subtopics. given a broad query, knowledge navigator generates a list of high-quality subtopics, each accompanied by a readable and interpretable summary grounded in the documents within the corpus. each subtopic of interest can be expanded through ad-hoc secondary retrieval of fine-grained documents within that specific area. see figure 1 for a graphical illustration of the system's real-world outputs. these subtopics represent meaningful research clusters, enabling searchers to identify areas of interest, uncover novel connections, and explore specific domains within the broader topic. by shifting the focus from individual documents to organized subtopic clusters, knowledge navigator offers a potential solution to the challenges inherent in traditional ranked list presentations for scientific literature search. we evaluate the knowledge navigator framework components on various llms and representation models, demonstrating its viability with both proprietary models (gpt-4o) and open-source models (mixtral-8x7b). to evaluate the system's components and overall performance, we use (a) clustrec-covid, a modified novel form of the trec-covid benchmark @xcite , which we adapted for subtopic clustering, cluster-based aspect generation, and query generation tasks; and (b) scitoc, a new dataset of scientific review table of contents, constructed from "annual reviews" open-access journals in a variety of scientific fields. we publish those datasets for future work in nlp research. using these benchmarks in both automatic and domain expert evaluation, we demonstrate that knowledge navigator performs efficiently in each of its component tasks, as well as in its overall function of organizing and outlining scientific knowledge. to our knowledge, this work is the first to showcase the feasibility of modern llms for supporting cluster-based navigation paradigms and aims to contribute to both the research and development of modern browsing systems. ## exploratory information seeking through knowledge navigation when scientists explore a new topic and review the literature, their information-seeking behavior is an instance of "exploratory search" @xcite . that is when a searcher does not have a particular document or topic in mind but rather is interested in finding out the different aspects reflected in the document collection, to both gain an understanding of the overall structure of the domain, as well as to look for subtopics that may interest them and/or fit their expertise and interests @xcite . currently, this process is not well supported by modern search systems, although it has been found to be a common search behavior among scientists @xcite . a major obstacle is the inability of searchers to effectively consume hundreds to thousands of documents and distill topics from them. rather, they browse tens of document titles at a time, revise their mental model of the domain based on this subset, maybe take notes, adapt their query to reflect their new mental image and interests, and navigate into a specific aspect or subtopic of interest. knowledge navigator enhances this exploration process by consuming hundreds of results, finding common themes, and organizing them into a two-level hierarchy of subtopics, allowing users to systematically explore different facets of a broad topic, thereby addressing the complex and multifaceted nature of their information needs. by transforming search results into organized subtopic clusters, knowledge navigator supports a holistic way of absorbing new information, helping them to identify areas of interest and discover novel connections even when their queries are initially vague or evolving. cluster-based browsing has been extensively studied in the past @xcite @xcite , but despite this, these methods were not widely adopted. approaches like scatter/gather @xcite , which aimed to organize documents into coherent clusters for easier navigation, failed to produce good representations of docu-ments in practice, leading to clusters that did not accurately differentiate between subtopics @xcite . additionally, these methods struggled to generate clusters that were easily interpretable by users due to reliance on keyword extraction techniques that were difficult for searchers to understand @xcite . this was largely due to the immature state of information retrieval (ir) and natural language processing (nlp) methodologies at the time. the introduction of instruction-tuned llms, with their advanced world knowledge and text understanding abilities @xcite , has recently led to renewed research utilizing llms for information organization across various tasks @xcite @xcite . we demonstrate that llms can be effectively used in assisting scientific literature consumption, by organizing document collections that result from a query into a digestible "table-of-content" of the topic, where each subtopic is grounded in concrete research works. this builds on the parametric knowledge of the llm about scientific concepts, their categorization, the relations between them and many other facets of scientific knowledge gained in training, but relying on this knowledge in a fully grounded way, using it solely for the purpose of cataloging and organizing a given set of human authored documents, which result from a provided query. this architecture enables llms to generate grounded outputs based on a large number of source documents, a crucial requirement for organizing and structuring large corpora. system design llms are incredibly effective at consuming information but are expensive to run and bottlenecked by the amount of information they can effectively process in their prompts. thus, we design the system around these constraints, attempting to minimize the number of llm calls and aiming to make the most effective use of each one. each step is designed to reduce the information size and transform it into a suitable form to be fed as input to the next llm step. the system is thus designed to work bottom-up, progressively abstracting information at each stage. starting from hundreds of search-query results, represented as titles, abstracts and snippets, we employ a relatively cheap operation of contextual embeddings followed by clustering, to organize them into smaller-but cohesive-groups. each group is then fed, as a whole but separately from other groups, into the cluster reader, an llmbased component that is in charge of analyzing it, describing its common themes, naming it, and scoring its relevance to the query. then, the names and descriptions of all the relevant clusters are fed (together with the initial query) into a subsequent llm-based component, which organizes them into thematic groups, and names these groups. the result is a corpus-level hierarchical organization of the search results, which can serve as a map to the scientific topic, and whose construction relied on the entire corpus: the clustering is a global operation; the naming is separate per cluster but takes the query into account and considers many documents in assigning the description and name; and the second-stage thematic organization again provides a global view based on all the clusters who passed the relevance filter. the searcher can then browse the generated topical outline, identify a sub-topic of interest, and initiate the subtopic expander to automatically generate a query to retrieve additional documents on the fine-grained sub-topic. we now describe each component. ## structured scientific literature benchmarks an ideal dataset for evaluating the system endto-end would necessitate exhaustive annotation of thousands of scientific documents, classifying each for relevancy, subtopics, and thematic groups. un-fortunately, no existing dataset supports this task comprehensively. to this end, we introduce two novel benchmarks, clustrec-covid and sci-toc, that facilitate a robust evaluation of knowledge navigator, both on an individual component level and end-to-end. 3 clustrec-covid to assess the construction and naming of subtopic clusters, we modified trec-covid, which was initially designed as an information retrieval (ir) benchmark for evaluating search performance on scientific literature related to covid-19 @xcite . it includes 50 expert-curated queries, each representing a subtopic of covid-19 research across multiple fields, with each query serving as a concise subtopic title, such as 'coronavirus heart impacts'. for each query, medical experts judged hundreds of documents, annotating each for relevance to the topic, resulting in an average of 300 highly relevant documents per query. these characteristics-expert curation, diverse subtopics, and detailed relevance annotations-make the trec-covid benchmark particularly suitable for evaluating the knowledge navigator system's components. we transformed the trec-covid benchmark into a clustering benchmark by forming clusters from documents annotated as "highly relevant" to specific topics. we randomly sampled up to 50 documents from each topic, ensuring documents appear in only one cluster. this resulted in a dataset with 2,284 documents assigned to 50 subtopic clusters labeled with expert-curated titles. scitoc to assess the system's ability to handle complex scientific topics and produce wellorganized outputs, we constructed a novel benchmark of 50 tables of contents (toc) of scientific reviews sourced from 15 diverse peer-reviewed journals published by annual reviews 4 . these reviews span a wide array of scientific fields, including biology, medicine, food industry, environmental studies, and more. reviews were selected based on predefined criteria: explicit and scientifically described tables of contents and subtopics, excluding metaphorical or abstract language. for instance, the review paper the effects of psychedelics on neuronal physiology (hatzipantelis and olson, 2024) included headers such as "effects of psychedelics on gene and protein expression" and "effects of psychedelics on neuronal survival and neurogenesis"-which we keep-as well as structural headers like "introduction," "background," and "discussion,"-which we discard. see an example from scitoc in d.1. for each review paper, we transformed the main title into a query focused on scientific terms to retrieve documents relevant to the review topic. queries included only keywords appearing in the review title. for instance, the aforementioned title was modified to the boolean query "psychedelic" and "neuronal physiology" to enable accurate retrieval of documents related to the main topic of the review. ## experiment the knowledge navigator system comprises several components that have not been extensively explored, particularly within the domain of scientific literature. we present an evaluation of these components, both individually and as part of the integrated system, specifically focusing on the organization of topical scientific corpora. ## discussion we demonstrate how the challenge of navigating the scientific literature when embarking on a new field can be facilitated by an llm-aided process, which we call knowledge navigator. the knowledge navigator operates on a corpus of documents which enables a more holistic understanding and organization of knowledge within the domain. the effectiveness of our framework demonstrates the potential of the bottom-up approach in other settings where llms are tasked with extracting insight from a collection of items. in addition, we believe that future work could use outputs from frameworks like knowledge navigator in prompts for other systems or in planning tasks for agents. for example in the retrievalaugmented generation (rag) settings, where structured data boosts the performance and utility of llms in various applications.