Large Language Models for Scientific Information Extraction: An Empirical Study for Virology (2024)

Abstract: In this paper, we champion the use of structured and semantic content representation of discourse-based scholarly communication, inspired by tools like Wikipedia infoboxes or structured Amazon product descriptions. These representations provide users with a concise overview, aiding scientists in navigating the dense academic landscape. Our novel automated approach leverages the robust text generation capabilities of LLMs to produce structured scholarly contribution summaries, offering both a practical solution and insights into LLMs’ emergent abilities.For LLMs, the prime focus is on improving their general intelligence as conversational agents. We argue that these models can also be applied effectively in information extraction (IE), specifically in complex IE tasks within terse domains like Science. This paradigm shift replaces the traditional modular, pipelined machine learning approach with a simpler objective expressed through instructions. Our results show that finetuned FLAN-T5 with 1000x fewer parameters than the state-of-the-art GPT-davinci is competitive for the task.

## introduction scholarly communication in the digital age is facing significant challenges due to the overwhelming volume of publications @xcite thereby creating the need for efficient access to relevant knowledge. in this regard, next-generation scholarly digital libraries, such as the open research knowledge graph (orkg) @xcite , offer a promising solution by adopting semantic publishing principles @xcite . the orkg stores scholarly contributions in a structured and semantic way, leveraging a knowledge graph (kg) representation @xcite . the fine-grained semantic contribution representation in the orkg utilizes property-value tuples, capturing important aspects and corresponding observations of research contributions. this representation enhances understanding and navigation of scholarly content by both humans and machines. with selected properties that apply universally to research on a specific problem, the orkg enables intelligent exploration and assistance services, including research comparisons based on shared properties, e.g., figure 1 . its novel information access methods provide condensed overviews of the state-of-the-art, supporting strategic reading @xcite in the ever-growing publication landscape. this work, as a text mining service toward producing scalable solutions for the orkg, for the first time, introduces a complex information extraction (ie) task. our notion of complex ie entails joint entity and relation extraction in a single objective aligned with the structured property-value format of contributions in the orkg. we defined the complex ie task w.r.t. a key research problem in the domain of epidemics & virology, i.e. estimating the basic reproduction number (r0) for infectious diseases. this r0 estimate research topic was brought to common knowledge during the recent covid-19 pandemic by the centers for disease control and prevention (cdc) as a key informant. important to infectious disease epidemiology, gen- to define our complex ie task, an expert semantic modeler created a research comparison based on structured property-value pairs for covid-19 r0 estimate contributions across 30 abstracts. consequently, six properties were modeled: disease name, location, date, r0 value, %ci values, 1 and method. the semantic modeling aimed to identify properties that were both generic enough to structure most related research on the r0 estimate (in the context of a research comparison) and specialized enough to reflect the vital details of the r0 contribution (by identifying commonalities in observations reported across 30 different abstracts). this structured format is called orkg-r0. thus our complex ie task focused on extracting propertyvalue pairs for orkg-r0 contributions in scholarly article abstracts. to address this task, a larger gold-standard corpus was annotated (details in section 3) and an llm-based solution was optimally designed (introduced next, details in section 4). the complex ie task introduced earlier is addressed as single-task instruction-based finetuning of an instruction-tuned large language model (llm) with the primary objective of better aligning the llm to our task and domain. our approach is 1 ci stands for confidence interval. characterized in figure 2 . we chose llms for their rich parameter spaces and ability to handle complex ie tasks with simple instruction prompts @xcite . unlike traditional pipelined-based ie, which are prone to error propagation and require extensive manual engineering, llms offer flexibility, adaptability, and the ability to handle a wide range of tasks in zero-and few-shot settings through instructions @xcite @xcite . by relying on instruction prompting, we can effectively address complex inter-relations without the need for an exhaustive enumeration of all possible relations or preliminary named entity recognition (ner). we finetune an llm from the sequence-to-sequence encoderdecoder-based t5 model class @xcite to accept a research paper title and abstract and instruct it to write the orkg-r0 structured "summary" of knowledge in the prompt as either textbased or as a structured json object. for the llm, we specifically select the instruction-tuned flan-t5-large model @xcite with reported 780m parameters. there could have been one of two directions for this work: scaling the models or instruction fine-tuning of a moderate-sized llm, i.e. with parameters in millions versus 1000x more in billions. we chose the latter. we believe that our choice makes model tuning more accessible within the research community while empirically proving to be nonetheless effective (experimental details in section 5). furthermore, our choice of google's flan-t5, open-sourced and easily accessible in the transformers library, obviates any paywall that hinders access to llms for the research community at large. for instruction-based finetuning, we use applicable instructions from the open-sourced instruction generalization efforts introduced as the "flan 2022 collection" @xcite . our approach differs from finetuning a pretrained lm as we instead finetune an instruction-tuned lm, enabling the model to effectively follow instructions title: a norovirus gastroenteritis outbreak in an australian child-care center: a household-level analysis abstract: there is a large burden of norovirus disease in child-care centers in australia and around the …. ## corpus we aim to create a high-quality corpus for the complex scientific ie task introduced in this work. the corpus creation goal was to obtain gold-standard property-value structured format representation w.r.t. the six predicates in orkg-r0 from scholarly article abstracts. these structured representations encapsulate the r0 estimate research problem for infectious diseases. base corpus. our starting point was the largescale cord-19 dataset @xcite provided by allenai. this resource comprised a growing collection of publications and preprints on covid-19, its variants, related historical coronaviruses such as sars and mers, as well as other infectious diseases such as h1n1 influenza, dengue, monkeypox, ebola, zika virus, norovirus, etc. at our download date timestamp 2022-06-02 it comprised over 800,000 total publications. the dataset covered diverse topics such as epidemiology, virology, clinical studies, public health, and more. it served as a valuable resource for researchers, policymakers, and the general public to access and analyze the latest scientific knowledge related to covid-19. since cord-19 contained articles on various themes, as a next step the corpus was filtered to include only articles on the r0 estimate theme. corpus filtering. our method for filtering the base corpus to our desired collection was simple. we implemented pattern-based heuristics using variants of the phrase "r0 estimate" and checked the publication abstract for containment. the base corpus was then reduced to 4590 instances. post dedupli-cation, the collection was further reduced to 3967 instances. other than exact duplicates, there were other near-duplication patterns such as punctuation marks stripped or retained, numbers with or without decimal points served as different data instances. near-duplicates were also filtered by clustering abstracts that were 95% similar (583 clusters from 1227 articles were created). a human annotator went through all clusters and decided on one abstract to retain while dropping all others. the resulting curated corpus contained 3024 abstracts which included a direct mention or a variant of the phrase "r0 estimate". the orkg-r0 model. here we provide an explanation of orkg-r0 as an ideal representation of a structured contribution for the research problem of "r0 estimate," as defined by an expert semantic modeler. the r0 estimate pertains to an infectious disease (disease name), for a specific population demographic (location), with validity for a specific time period (date). it reports a specific value (r0 value), along with a confidence interval for the statistical value (%ci values), and is computed by a statistical method (method). annotation exercise. to ensure a practical and realistic human annotation target, we selected a sub-sample of 1500 articles from the curated 3024 dataset. this would then serve as the gold-standard dataset for training and development purposes, as an empirical basis for future research. a team of two annotators produced the orkg-r0 structured annotations with the corpus raw data comprising a paper title and abstract, where each instance is uniquely identified by a cord_id. the overall annotation exercise lasted 3 months. the annotation task began by distinguishing between the papers actually reporting an r0 value as a contribution and those that just mentioned the "r0 estimate" keyword in the abstract, but did not actually report a value as a contribution of the work. resultingly, we found 652 articles reported an r0 value and thus were annotated for the orkg-r0 structure (referred to as the "answerable" set, in short ans), while 850 did not (referred to as the "unanswerable" set, in short unans). among the 652 articles, approximately 157 had multiple contributions for the "r0 estimate." notably, a few articles stood out with 10, 11, or 16 reported contributions. the gold-standard annotated set was made available in two formats: text-based and json-based, which are illustrated by the green boxes in figure 3 . in the text-based format, multiple contributions were separated using a pipe character, while in the json format, they were encoded as separate json object dictionaries. we observed that the json data structure is more conducive for utilization in downstream applications. therefore, our empirical analysis regarding llms aimed not only to assess their ability to generate structured abstract summaries but also to evaluate their compatibility with a specific data structure. this allows for the seamless integration of their output into downstream applications. the annotators. in our annotation process, we first developed a structured summary model for the "r0 estimate for infectious diseases" using both domain experts and a semantic modeler specializing in ontology design. next, a phd student populated the model using a dataset of abstracts, treating it as a form-filling task of reported facts. while the task itself is tedious in that the student needed to read all abstracts to populate the properties, the process did not entail much ambiguity in the decisions. the definition of the properties we selected are fairly straightforward and the values are to be directly extracted from the text. for discrepancies in spans for the values selected, the llm is expected to be robust enough to arrive at the optimal extraction scenario. for any concerns on quality, our goldstandard test dataset annotations versus the llm predictions eventually obtained can be publicly browsed at this link https://scinext-project. github.io/#/r0-estimates . our complex ie task objective. we phrased the following question to formulate our task objective w.r.t. the orkg-r0 extraction target: what are the values for the following properties of the basic reproduction number estimate (r0): disease name, location, date, r0 value, %ci values, and method? in essence, it encapsulates an ie task. the orkg-r0-based complex ie objective presents a unique approach compared to traditional scientific ie, particularly in biomedicine. common biomedical ie tasks, like those in the biocreative v chemical disease relation extraction task corpus (bc5cdr) corpus, focus on document-level entity and relation extraction, linking two elements such as a chemical and a disease with semantic interactions like "interacts" @xcite . in contrast, the orkg-r0 ie model aims to establish a multifaceted link among six distinct elements: infectious disease name, study location, study date, r0 estimate value, %confidence interval values, and method. this approach diverges from the se-mantic interaction model of bc5cdr, as it does not establish semantic relations between its elements. instead, it aggregates these elements to form a comprehensive summary representation of a work's contribution to the research problem "r0 estimate for infectious diseases." the orkg-r0 model is characterized by the underlying principles of the orkg platform from which it is derived, which emphasizes structured, machine-actionable models of scholarly communication beyond traditional formats like pdfs @xcite . the orkg prioritizes structured representations of a work's contributions over exhaustive content coverage. in contrast, resources like the bc5cdr corpus (and other similar databases in biomedicine, e.g., biocreative datasets @xcite @xcite ) focus on building extensive knowledge graphs of diseasechemical interactions, with annotations drawn from comprehensive scientific papers. while valuable, these annotations are different in their goal as they do not necessarily provide insights into the specific contributions of a work, such as the discovery of an interaction or the methods used for such discoveries. the orkg-r0 ie, therefore, aligns more closely with research contribution summarization tasks than with traditional scientific ie tasks in biomedicine. consequently, models developed for orkg-r0 ie are unlikely to be directly applicable to conventional biomedical ie tasks. in terms of objectives, our work is somewhat analogous to leaderboards in artificial intelligence (ai), which annotate units or tuples comprising task, dataset, metric, and score @xcite . however, there are distinct differences in annotation scope: leaderboards typically utilize the full text of papers, whereas our method relies solely on abstracts. additionally, the ai community currently lacks a gold-standard dataset for leaderboard annotations, a gap our dataset aims to fill. we propose our dataset as a pioneering resource in generating structured scientific summaries, addressing the current community need for standardized datasets in this domain. instructions for the llm. instruction tuning is a novel approach @xcite @xcite that improves llms' performance by providing explicit instructions during finetuning, guiding the model's behavior @xcite @xcite and enhancing its adaptability and ef-fectiveness in diverse learning scenarios. unlike traditional non-instruction tuning methods @xcite @xcite that rely solely on unlabeled data, instruction tuning incorporates specific guidance, simplifying the finetuning process and enabling better performance on new tasks and domains @xcite . it became possible to generically prompt an llm to perform different tasks with a single instruction. as such it can be considered as a template that encodes the task and its objective, in turn telling the llm what to do with the given objective. the "flan 2022 collection" was a large-scale open-sourced collection of 62 prior publicly released datasets in the nlp community clustered as 12 task types, such as reading comprehension (rc), sentiment, natural language inference (nli), struct to text, etc. it is the most comprehensive resource facilitating open-sourced llm development as generic multi-task models. importantly, and of relevance to this work, flan was not just a super-amalgamation of datasets encapsulating different learning objectives, but also included at least 10 human-curated natural instructions per dataset that described the task for that dataset. as such, we select a set of instructions to guide the llm for our complex ie task from the flan collection. specifically, we identified the applicable instructions to our task were those designed for the squad_v2 @xcite and drop @xcite datasets. the general characteristic of the selected instructions is that they encode a context (in our case the paper title and abstract) and the task objective, and instruct the model to fulfill the objective. see appendix b for further details. the purple boxes in figure 3 show some exemplars. examples of all instructions are in appendix a. our work is positioned here, coalescing the most relevant collection of instructions that were used to instruction-finetune the t5 (2020) model class, as the strong reference point for any future open source work on single-task instruction finetuning. ## approach our approach is single-task instruction-finetuning for our novel introduced complex ie task. as such it aims to be an incremental progression of the instruction-tuning paradigm introduced as flan (finetuned language net) (2021; 2022; ## evaluations dataset. for evaluations, we created a 70%/10%/ 20% split as train/dev/test sets, respectively, of the 1500 instances. the dataset comprised 1,082 train (464 ans, 618 unans), 120 dev (53 ans, 67 unans), and 300 test (135 ans, 165 unans) instances. experimental setup. we used a total of 18 instructions for training, with 9 instructions each from squad_v2 and drop, specifically instantiated in appendices a.1 and a.2, respectively, suitable for our task. among these, 2 drop instructions were training took 12-15 hours on smaller datasets and 30 hours on larger datasets, while inference lasted 15-30 minutes for 300 test instances. metrics. we experimented in two main settings: zero-shot evaluations and single-task finetuned model evaluations. for the latter, we used recall, precision, and f1 metrics in exact and partial match settings for each of the six orkg-r0 extraction targets and overall. in the zero-shot evaluations, where models were not guaranteed to respond with the desired structure, we treated the task as struc-tured summarization. to evaluate these summaries, we used standard summarization rouge metrics @xcite (details in appendix d) instead of f1 metrics, which would require complex postprocessing and could lead to misinterpretation of the model's response. ## error analysis based on an analysis of all the erroneous responses on the test set from our best model, we identified five main error types. they were further categorized on their impact on recall or precision. for each, mismatching (prediction, annotated label), ## conclusions and future directions searching scientific articles for the covid-19 r0 estimate yields around 44,000 results. to navigate through this vast amount of information and stay up-to-date with the latest r0 estimates, is inundating for researchers. next-generation digital libraries like orkg are transforming this traditional paradigm by capturing machine-actionable data, enabling advanced computational tools such as research comparisons. llm-powered complex ie technologies can play a crucial role in scaling scientific information extraction. we present a concrete use-case in virology, showcasing the acquisition of llm-powered structured knowledge with the orkg-r0 model. to facilitate reproducibility and foster future research, we have made available several resources: our dataset ( https://doi.org/ 10.5281/zenodo.8068441 licensed under cc by 4.0), instructions, source code ( https://github. com/mahsash717/r0-estimates licensed under mit), and our optimally finetuned model for the orkg-r0 ie task at https://huggingface. co/orkg/r0_contribution_ie . additionally, for enhanced transparency, a selection of our human-annotated test dataset and its corresponding model predictions can be browsed online here https://scinext-project.github.io/#/ r0-estimates . to sum up, our work can be seen as a flavor of meta-learning that was seminally proposed by @xcite as the meta in-context learning paradigm. we explore meta-learning through instruction-finetuning of an instructiontuned model, and differ in that we use a zero-shot rather than a few-shot training and testing scenario. we relegate few-shot in-context model learning to future work. while this work comprehensively evaluates the t5 class of llms, there are other promising llms like palm @xcite , chinchilla @xcite , and chatgpt @xcite that can be further investigated for nlp tasks with instructions. exploring alternative model families is a fruitful direction for future research. additionally, model distillation @xcite @xcite holds potential for transferring knowledge from large teacher models to smaller, efficient student models. this approach holds promise, particularly in scenarios where single-task tuned models are desired, as we propose in this study.