TOPICAL : TOPIC Pages A utomagica L ly (2024)

Abstract: Topic pages aggregate useful information about an entity or concept into a single succinct and accessible article. Automated creation of topic pages would enable their rapid curation as information resources, providing an alternative to traditional web search. While most prior work has focused on generating topic pages about biographical entities, in this work, we develop a completely automated process to generate high-quality topic pages for scientific entities, with a focus on biomedical concepts. We release TOPICAL, a web app and associated open-source code, comprising a model pipeline combining retrieval, clustering, and prompting, that makes it easy for anyone to generate topic pages for a wide variety of biomedical entities on demand. In a human evaluation of 150 diverse topic pages generated using TOPICAL, we find that the vast majority were considered relevant, accurate, and coherent, with correct supporting citations. We make all code publicly available and host a free-to-use web app at: https://s2-topical.apps.allenai.org.

## introduction the automatic generation of topic pages is a longstanding goal of the nlp community @xcite @xcite . in contrast to web search resultsdisplayed as ranked lists of hyperlinks with short text snippets across many pages-topic pages aggregate useful information about various aspects of an entity or concept in a single, concise location. scientific topic pages @xcite apply this thinking to scientific concepts by aggregating information from the primary literature to produce succinct and accessible summaries useful to both experts and nonexperts alike (figure 1 ). among other things, highquality scientific topic pages hold the promise of: * work performed during internship at ai2 ## related work topic page generation topic page generation is usually framed as a topic-focused, opendomain multi-document summarization (mds) task @xcite . most prior work is concerned with generating wikipedia-like pages for general-domain entities and concepts (often biographical in nature). early work clustered the web search query logs for an entity of interest to determine its various aspects, used each aspect cluster to retrieve and rank relevant sentences, and then re-organized the retrieved sentences for coherence to produce a bullet-list style topic page @xcite . more recent work-also focused on biographical entities-first templates the topic page by copying common section headings from wikipedia pages for related topics and trains a supervised model to select the text content for each section. an unsupervised component then creates topic-specific sections, and several post-processing steps are ap-entity or concept, e.g. "microplastics" specter (b) embed & cluster cluster n ... (c) sample cluster 1 (a) search (d) generate summarize titles & abstracts of sampled articles embed titles & abstracts of top-k most relevant articles topic page ... the titles and abstracts of the results are embedded with specter @xcite and clustered based on semantic similarity (b). we sample titles and abstracts from the clusters (c) and feed them to gpt-4 (ope-nai, 2023), alongside publication metadata and natural language instructions, to generate the topic page (d). scientific topic pages azarbonyad et al. (2023) investigate generating scientific topic pages at scale; however, they do not synthesize a summary but focus rather on extracting a definition statement verbatim, alongside "mention snippets" and related concepts. in contrast, we attempt to synthesize more comprehensive topic pages, including a definition statement and content about the entity's main and future research directions. king et al. (2022) introduce a scientific concept description task with similar motivation to our work, but focus on earlier, smaller generative models for describing computer science concepts, and find the systems to hallucinate relatively frequently. wikicrow, based on paperqa (l'ala et al., 2023), provides scientific topic pages generated by an llm-based system for human protein-coding genes. in contrast to our approach, their publicly available demo is limited to 15,616 pre-generated topic pages and does not allow a user to generate topic pages for a new entity of interest on demand. foot_3 ## approach our approach follows a retrieval-augmented generation (rag) setup @xcite @xcite . a microplastics cluster 1 (size: 24) interactions between microplastics and unit processes of wastewater treatment plants: a critical review. ## topical web app topical is available as a web app (see figure 5 for an overview). the web app can be run locally as a standalone python package but is also publicly available at https://s2-topical. apps.allenai.org . a user first inputs a pubmed search query, which supports the full syntax of the pubmed advanced search builder (see appendix a for details). however, in most cases, sim-ply inputting the entity or concept name directly and allowing esearch to expand the query via automatic term mapping (atm) works well, especially with respect to recall. a user can optionally provide alternative names for the entity, referred to as 'canonicalized names,' which are provided to the llm as additional context. once a user clicks the button to generate a topic page, the search, embedding, clustering, and generation steps are executed. an expandable section in the app displays progress, as well as additional information about the search, e.g., any query expansions made via atm and a histogram of publications per year (see figure 6 ). the generated topic pages can be downloaded as json files. a video demonstration of the system is available here: https://youtu.be/hgng7bnieay . ## human evaluation we conduct human evaluation to determine the quality of the automatically generated topic pages. the evaluation consists of two tasks, described below. all annotations were performed by three fulltime, paid annotation specialists with undergraduate training spanning the biosciences, materials science, environmental science, and data science. ## results we find that the majority of topic pages are rated by our annotators as relevant, accurate, and coherent ( similarly, most model-provided citations were rated as correct ( ## conclusion in this paper, we present topical, a new approach for the automatic generation of high-quality scientific topic pages that leverages large language models (llms) and retrieval-augmented generation (rag). we conducted an extensive human evaluation of 150 diverse topics from the biomedical literature and our annotators rated the vast majority of generated topic pages as relevant, accurate, and coherent; and model-provided citations as correct. promising future directions include allowing users to provide custom instructions with respect to structure, focus and length of the automatically generated topic pages, and the investigation of opensource llms in place of the closed-source llm we experimented with (gpt-4). we release a publicly available web app so that others can experiment with generating topic pages for entities or concepts of interest on demand.