CHIME : LLM -Assisted Hierarchical Organization of Scientific Studies for Literature Review Support (2024)

Abstract: Literature review requires researchers to synthesize a large amount of information and is increasingly challenging as the scientific literature expands. In this work, we investigate the potential of LLMs for producing hierarchical organizations of scientific studies to assist researchers with literature review. We define hierarchical organizations as tree structures where nodes refer to topical categories and every node is linked to the studies assigned to that category. Our naive LLM-based pipeline for hierarchy generation from a set of studies produces promising yet imperfect hierarchies, motivating us to collect CHIME, an expert-curated dataset for this task focused on biomedicine. Given the challenging and time-consuming nature of building hierarchies from scratch, we use a human-in-the-loop process in which experts correct errors (both links between categories and study assignment) in LLM-generated hierarchies. CHIME contains 2,174 LLM-generated hierarchies covering 472 topics, and expert-corrected hierarchies for a subset of 100 topics. Expert corrections allow us to quantify LLM performance, and we find that while they are quite good at generating and organizing categories, their assignment of studies to categories could be improved. We attempt to train a corrector model with human feedback which improves study assignment by 12.6 F1 points. We release our dataset and models to encourage research on developing better assistive tools for literature review.

## introduction literature review, the process by which researchers synthesize many related scientific studies into a higher-level organization, is valuable but extremely 1 the chime dataset and models are available at https: //github.com/allenai/chime . † correspondence to aakankshan@allenai.org. ♠ work done as an intern at allen institute for ai. p1: exercise modalities s1: aerobic s2: resistance s3: combined s4: walking s5: weight training research topic exercise interventions on health-related quality of life for people with cancer during active treatment 1. the group psychotherapy and home-based physica... 48. exercise and quality of life during and after ... 48 related studies multiple llm-generated hierarchical organizations hierarchy 1 hierarchy 2 p1: cancer types s1: lung cancer s2: breast cancer s3: throat cancer s4: metastasis s5: recurrence studies: 1, 3, 5 studies: 4, 10 studies: 23, 42 studies: 1,3 studies: 1,3 studies: 1, 8, 9 studies: 4, 5, 7 studies: 32, 39 studies: 5, 6, 9 studies: 1,4 figure 1 : given a set of related studies on a topic, we use llms to identify top-level categories focusing on different views of the data (such as p1 and p2), generate multiple hierarchical organizations, and assign studies to different categories. however, these categories and study assignments can contain errors. as illustrated in the figure, the categories walking and weight training are not coherent with their siblings (s1 -s3) in hierarchy 1 since they are more specific, and the categories metastasis and recurrence are incorrectly assigned to the parent category in hierarchy 2 since they are not types of cancer. motivated by this finding, we take a different approach and focus on the task of generating hierarchical organizations of scientific studies to assist literature review. as shown in figure 1 , a hierarchical organization is a tree structure in which nodes represent topical categories and every node is linked to a list of studies assigned to that category. inspired by the adoption of llms for information organization uses such as clustering @xcite and topic modeling @xcite , we investigate the potential of generating hierarchies with a naive llm-based approach, and observe that models produce promising yet imperfect hierarchies out-of-the-box. to further assess and improve llm performance, we collect chime (constructing hierarchies of biomedical evidence), an expert-curated dataset for hierarchy generation. since building such hierarchies from scratch is very challenging and timeconsuming, we develop a human-in-the-loop protocol in which experts correct errors in preliminary llm-generated hierarchies. during a three-step error correction process, experts assess the correctness of both links between categories as well as assignment of studies to categories, as demonstrated in figure 1 . our final dataset consists of two subsets: (i) a set of 472 research topics with up to five llm-generated hierarchies per topic (2,174 total hierarchies), and (ii) a subset of 100 research topics sampled from the previous set, with 320 expert-corrected hierarchies. expert-corrected hierarchies allow us to better quantify llm performance on hierarchy generation. we observe that llms are already quite good at generating and organizing categories, achieving near-perfect performance on parent-child category linking and a precision of 77.3% on producing coherent groups of sibling categories. however, their performance on assigning studies to relevant categories (61.5% f1) leaves room for improvement. we study the potential of using chime to train "corrector" agents which can provide feedback to our llm-based pipeline to improve hierarchy quality. our results show that finetuning a flan-t5based corrector and applying it to llm-generated hierarchies improves study assignment by 12.6 f1 points. we release our dataset containing both llm-generated and expert-corrected hierarchies, as well as our llm-based hierarchy generation and correction pipelines, to encourage further research on better assistive tools for literature review. in summary, our key contributions include: • we develop an llm-based pipeline to organize a collection of papers on a research topic into a labeled, human-navigable concept hierarchy. ## generating preliminary hierarchies using llms the first phase of our dataset creation process focuses on using llms to generate preliminary hierarchies from a set of related studies, which can then be corrected by experts. we describe our process for collecting sets of related studies and our llm-based hierarchy generation pipeline. ## correcting hierarchies via human feedback the second phase of our dataset creation process involves correction of preliminary llm-generated hierarchies via human feedback. correcting these hierarchies is challenging because of two issues. first, the volume of information present in generated hierarchies (links between categories, claimcategory links, etc.) makes correction very timeconsuming, especially in a single pass. second, since categories and claims in a hierarchy are interlinked, corrections can have cascading effects (e.g., changing a category name can affect which claims should be categorized under it). these issues motivate us to decompose hierarchy correction into three sub-tasks, making the feedback process less tedious and time-consuming. furthermore, each sub-task focuses on the correction of only one category of links to mitigate cascading effects. these three sub-tasks are: (i) assessing correctness of parent-child category links, (ii) assessing coherence of sibling category groups, and (iii) assessing claim categorization. ## characterizing hierarchy complexity our dataset creation process produces 2,174 hierarchies on 472 research topics, with 320 hierarchies (for 100 topics) corrected by domain experts. we briefly characterize the complexity of all generated hierarchies, focusing on two aspects: (i) structural complexity, and (ii) semantic complexity. ## automating hierarchy correction as mentioned in §4, we hire five domain experts to correct hierarchies for 100 research topics. however, the correction process, despite our best efforts at task simplification and decomposition, is still time-consuming and requires domain expertise. therefore, we investigate whether we can use our corrected hierarchy data to automate some correction sub-tasks. in particular, we focus on automating sibling coherence and claim categorization correction since ## conclusion our work explored the utility of llms for producing hierarchical organizations of scientific studies, with the goal of assisting researchers in performing literature review. we collected chime, an expertcurated dataset for hierarchy generation focused on biomedicine, using a human-in-the-loop process in which a naive llm-based pipeline generates preliminary hierarchies which are corrected by experts. to make hierarchy correction less tedious and timeconsuming, we decomposed it into a three-step process in which experts assessed the correctness of links between categories as well as assignment of studies to categories. chime contains 2,174 llm-generated hierarchies covering 472 topics, and expert-corrected hierarchies for a subset of 100 topics. quantifying llm performance using our collected data revealed that llms are quite good at generating and linking categories, but needed further improvement on study assignment. we trained a corrector model with our feedback data which improved study assignment further by 12.6% f1 points. we hope that releasing chime and our hierarchy generation and correction models will motivate further research on developing better assistive tools for literature review.