Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Dataset (2024)

Abstract: We explore the ability of GPT-4 to perform ad-hoc schema-based information extraction from scientific literature. We assess specifically whether it can, with a basic one-shot prompting approach over the full text of the included manusciprts, replicate two existing material science datasets, one pertaining to multi-principal element alloys (MPEAs), and one to silicate diffusion. We collaborate with materials scientists to perform a detailed manual error analysis to assess where and why the model struggles to faithfully extract the desired information, and draw on their insights to suggest research directions to address this broadly important task.

## introduction a key use case for large language models in science is ad-hoc schema-based information extraction. in this scenario, a scientist has a specific information need, such as the compressive yield strength of known multi-principal element alloys (mpeas), and wants to ( 1 ) identify papers containing relevant information instances, and (2) comprehensively extract these instances according to a desired schema, which might include secondary information such as testing conditions and synthesis details (fig 1 ). such datasets are a key element of scientific informatics, transforming information reported inconsistently throughout the literature into a structured format appropriate for use in machine learning. in materials science, such datasets are used in a wide range of applications, including modeldriven materials discovery, where they can be used to train or warm-start surrogate models of material properties which are then incorporated into various kinds of discovery algorithms @xcite . this is a challenging extraction task. the materials science literature contains a plethora of experimental results, often spanning multiple decades, in a multitude of different formats with varying physical units. prior generations of models needed large amounts of data to fine-tune, and were brittle and formula grain size test type test temp ys (mpa) formula grain size test type test temp ys (mpa) hf0.2 nb0.2 ta0.2 ti0.2 zr0.2 0.05 t 25 1520 hf0.2 nb0.2 ta0.2 ti0.2 zr0.2 0.05 t 25 1900 cr0.12 mo0.2 nb0.42 ta0.01 ti0.13 v0.12 0.075 c 25 2680 hf0.2 nb0.2 ta0.2 ti0.2 zr0.2 0.1 t 25 795 cr0.25 mo0.25 nb0.25 v0.25 2.07 c 1000 1513 cr0.25 mo0.25 nb0.25 v0.25 2.07 c 25 2743 hf0.05 nb0.55 ta0.25 ti0.15 11.2 t 25 531 mo0.333 nb0.333 ta0.333 112.5 c 1000 712 non-transferable, limiting their practicality for this type of ad-hoc, one-off extraction task. to date, manual annotation is still the standard approach for extraction of this type of unstructured data @xcite . contemporary large language models such as @xcite and gemini @xcite have the potential to conquer this barrier. with the emergent few-shot capabilities of these models, a scientist can potentially present a llm with a (1) collection of manuscripts, (2) a schema defining what types of data they want to extract, and (3) a few exemplars of performing this extraction, and receive back a tabular dataset of information extracted from those manuscripts, ready to use for their chosen purpose. this ad-hoc paradigm has been referred to as "on-demand" information extraction (odie) @xcite and it promises to speed up the search for new materials with unique properties. we argue that this very ambitious extraction setting should be a key target for llm-driven scientific information extraction, as it can potentially allow scientists to extract a maximum of clinically relevant data with a minimum of effort, both in materials science and other domains such as biomedicine. two significant questions, then, are (1) to what extent can contemporary llms perform this type of ad-hoc extraction on scientific text, and (2) what are the major barriers impeding their effectiveness? in this paper, we explore these questions using two manually-extracted material property datasets, one focused on multi-principal element alloys (mpeas), sometimes known as high-entropy alloys (heas) @xcite , and one focused on elemental diffusion in silicate melts @xcite . we use these existing datasets as sources of schema structure and exemplars, and evaluate whether a representative contemporary llm (gpt-4) is capable of replicating them given the manuscripts from which they were originally extracted. we perform a detailed error analysis, including annotation by domain experts, to understand where and why the model falls short, to guide further development of llms to advance extraction paradigms for materials data. generally, we find that the model shows great potential for extracting information described narratively or in the form of a conventionally-formatted table. the majority of errors are attributable to figures, which our current pipeline does not address, and pdf parsing issues. other sources of error include non-standard table formats, the need for additional postprocessing of extracted values, as well as true reading comprehension errors which could be addressed via improved prompt engineering. all relevant code and data can be found in our github repository foot_0 . ## related work information extraction (ie) is a core nlp task with a long history, and one of many which contemporary large language models (llms) such as gpt-4 (openai, 2023) and gemini @xcite are able to perform in a zero-or few-shot setting @xcite . this few-shot extraction ability has enabled a new extraction paradigm, involving extracting a complex schema with little or no training data, termed "on-demand information extraction" (odie) by @xcite . our task setting falls into this paradigm. zero-and few-shot scientific information extraction forms a major aspect of the excitement over llms' potential for accelerating scientific discovery, because of its potential to create structured information from diffuse unstructured data sources such as scientific papers and clinical notes @xcite . ad-hoc extraction from scientific literature is particularly challenging because the amount of manually-annotated data needed to fine-tune a model to perform a particular extraction task would constitute a significant fraction of all the data in existence, obviating the benefit of training such a model. scientific literature is also highly multimodal, with key information presented in narrative, tabular, and visual formats. scidasynth @xcite approaches the problem with an interactive qa-based interface, evaluating based on a 12-scientist user study. llm-powered ie has attracted attention in materials science specifically @xcite . @xcite explore the use of gpt-4 to extract bulk modulus information from sentences extracted from scientific papers, while @xcite uses zero-shot chatgpt with a verification step to extract band gap information from sentences collected in a dataset by @xcite . @xcite finetunes gpt-3.5 to extract growth procedures and outcomes for gold nanorods from paper text. @xcite uses the cohere model to extract phase-property relationships from a corpus of fulltext materials papers. finally, the a-lab @xcite incorporates extracted synthesis recipes into an ai-guided experimentation protocol. our work differs from these recent approaches in seeking to extract a rich, complex schema from non-curated data in one pass with no fine-tuning, instead of narrowly-defined tasks over carefullycurated corpora. this is a highly ambitious and difficult extraction task, but one that places a minimum of burden on the scientist, and we see it as a key milestone for scientific ie. mpea diffusion total # papers 164 71 # rows 1211 2838 retrieved # papers 128 55 # rows 971 2359 publication year @xcite ## datasets we experiment with two material properties datasets, one pertaining to multi-principal element alloys (mpeas) @xcite , and one to elemental diffusion in silicate melts @xcite . ## method our extraction pipeline consists of the following steps: (1) retrieving and parsing source pdfs; (2) prompting and response postprocessing; and (3) extracted row alignment. ## automated evaluation after aligning extracted rows with dataset rows as described above, we report the number of matches and hallucinations in the only difference between simple and complex schema is the number of properties mentioned in each schema. the raw performance is underwhelming. on mpea, the model hallucinates roughly as many rows as it matches, and misses twice as many. we are able to match significantly more entries using the simple schema than the complex schema. while using langchain we extracted more relevant entries, but we can see a drop of precision (more hallucination). on the diffusion dataset, we are able to match no rows at all for 0-shot and, though the model hallucinates at a proportionally lower rate than for the mpea dataset. using langchain clearly improved the performance on this dataset but the hallucinations also increased significantly. ## manual error analysis to elucidate what formats key information occurs in, and under what circumstances the model struggles to extract that information, two materials science experts hand-annotated a sample of papers and attempted llm extractions from each dataset, using the 1-shot prompting variant. 60 papers' worth of attempted extractions were annotated from the mpea dataset, constituting a total of 428 rows, and 40 papers from the diffusion dataset, constituting 0-shot 1-shot langchain simple complex simple complex simple complex 1714 rows. both sets were sampled randomly from the set of retrieved and parsed papers for which extractions were attempted. each row was labeled for the format in which the key information occurred (table, figure, narrative text, etc). each failure, either missing or hallucinated rows, was annotated for its failure reason, to the best of the annotator's ability to diagnose the model's behavior by looking at the pdf, parsed xml, and extracted json. ## discussion the manual error analysis identifies several major barriers to effective ad-hoc schema-based information extraction from scientific articles. pdf parsing. pdf parsing is a problem for older pdfs which, for some areas such as geology and certain subfields of materials science, can still contain valuable information worth extracting. this suggest a need for native lm support for pdfs, which, while present in major models such as claude and chatgpt via their interfaces, is not yet available in programmatic api form at the time of writing. ## conclusion we perform an analysis of whether a representative contemporary large language model, gpt-4, is capable of performing complex ad-hoc scientific information extraction sufficient for replicating two materials property datasets. a detailed manual expert error analysis shows that, while the model shows promise for narrative and tabular data, there are significant barriers in both modality and comprehension ability that prevent the model from being adequate to the task using a baseline one-shot prompting approach. nevertheless, our initial results on this only-recently-recognized information extraction paradigm and our careful analysis will help guide further work in this area.