Extracting Material Property Measurement Data from Scientific Articles (2021)

Abstract: Machine learning-based prediction of material properties is often hampered by the lack of sufficiently large training data sets. The majority of such measurement data is embedded in scientific literature and the ability to automatically extract these data is essential to support the development of reliable property prediction methods. In this work, we describe a methodology for developing an automatic property extraction framework using material solubility as the target property. We create a training and evaluation data set containing tags for solubility-related entities using a combination of regular expressions and manual tagging. We then compare five entity recognition models leveraging both token-level and span-level architectures on the task of classifying solute names, solubility values, and solubility units. Additionally, we explore a novel pretraining approach that leverages automated chemical name and quantity extraction tools to generate large datasets that do not rely on intensive manual tagging. Finally, we perform an analysis to identify the causes of classification errors.

## introduction while the application of machine learning methods for material property prediction holds great promise for material discovery and design across a broad range of applications, such methods are often hampered by a lack of sufficiently large and diverse training datasets. typically, relevant measurements and information exist only in unstructured formats such as the published scientific literature and are not available in aggregated and standardized databases. the ability to automatically extract, process, and analyze large sets of material property data would represent a significant capability for the advancement of material design efforts. the development of predictive models for the solubility of organic molecules is one such use case that would support a wide range of application areas including pharmaceutical, environmental, and energy storage applications. for example, molecular solubility is a key performance driver for redox flow battery (rfb) technologies which rely on energy-bearing redox active molecules that are dissolved in a liquid electrolyte. the solubility of a molecule determines its maximal concentration in the electrolyte and the resulting energy density of the system. to support the development of predictive models for the discovery and design of new materials for these batteries, a comprehensive solubility measurement database is required. the ability to automatically parse the scientific literature for existing and newly published solubility measurements would provide a significant acceleration to our ability to enlarge existing datasets, as well as keep predictive models up-to-date with the newest data. however, the extraction of numerical data from the materials science and chemistry literature presents several key challenges. material property measurements are often highly sensitive to the specifics of the experimental conditions. additionally, in comparison with tasks for general information extraction, this task requires the ability of models to target specific measurement types and distinguish them from other measurements which may be expressed using similar language. in this paper, we make several key contributions. first, we collect a corpus of solubility-related sentences from the scientific literature and perform manual tagging to annotate key components of the solubility measurements including the solute, the solubility value, and the solubility unit. secondly, we apply and compare several entity-extraction deep learning models to the problem of automatic extraction of solubility data. additionally, we develop and explore several possible methods for the pretraining of quantitative measurement extraction models. overall, our best performing models achieves f1 scores of 0.75, 0.79, and 0.9 on the extraction of solutes, values, and units. finally, we perform detailed performance and error analysis to provide insights into the strengths and weaknesses of the current models and to identify directions for future improvements to the methodology. ## related work the majority of existing solubility datasets which have been employed for solubility prediction are small, with typically a few thousand data points @xcite @xcite @xcite , with recent datasets starting to reach the level of 10,000 data points @xcite . there are no previously existing datasets linking the solubility measurement values to their source text in publications to support the development of information extraction efforts. several annotated datasets have been developed to support efforts for more general scientific information extraction including the scienceie dataset with task, process, and material annotations in the material science, physics, and computer science domains @xcite , the scierc dataset with entity and relation annotations in the computer science domain @xcite , annotations of conditional facts from the life science and biomedical domain @xcite , and a dataset with annotations of process, method, material, and data across ten scientific domains @xcite . prior work on information extraction in the materials science and chemistry domains has focused both the development of general tools such as chemdataextractor @xcite , which leverages conditional random field (crf) models, custom dictionaries, and rule-based grammers for information extraction, and the development of models for specific extraction tasks. some examples include extraction of zeolite synthesis information using regular expressions @xcite , extraction of nanomaterial composition and morphology using unsupervised methodology on tf-idf features and synthesis protocols using a sentence-level logistic regression classifier @xcite , extraction of synthesis conditions for metal oxides using supervised classification of parsed noun phrases @xcite , tagging of material science-related named entities using bidirectional lstms @xcite , and construction of a knowledge base related to solid oxide fuel cells through the development of entity and relation extraction models @xcite . the majority of prior work has focused on the extraction of fixed scientific entities, such as material names or methodologies, and their relationships. however, less work has been performed to develop methods targeted toward the extraction of numerical and quantitative data as is needed for the extraction of experimental measurements like solubility. several prior task-specific models include extraction of curie and néel magnetic phase transition temperatures using chemdataextractor combined with semi-supervised relation extraction @xcite and the use of a rulebased approach to extract numerical data from electronic medical records @xcite . ## results we summarize the results of our experiments in in ## conclusions we demonstrate the performance of several different token-level and span-level extraction architectures on the novel task of solubility measurement data extraction from scientific literature. we find that a token-level extraction model leveraging pretraining scibert weights combined with a crf layer achieves the best performance on this task. we explore the impact of pretraining on a large set of general value extraction data but find that it reduces the model performance compared with training from scratch. we also find that the model performance has a weak sensitivity to increases in the training data size. this points to the necessity of developing alternative modeling approaches or more effective pretraining tasks to improve performance in this domain due to the challenge of collecting sufficiently large labelled datasets. we perform error analysis to understand the strengths and weaknesses of the extraction model. we find that the model performance is robust to several variations that had the potential to make the extraction task more challenging. the model performs well at unit extraction even when less commonly used units are employed. the model also appears to perform well even when solute names and values are separated by many tokens in the sentence. however, we do observe that the model performs poorly on solute name extraction when the sentence does not employ the standard phrasing "solubility of". this points to the need for continuing to supplement the dataset with sentences that contain more diverse and complex phrasing. while we have demonstrated promising performance on this novel scientific measurement extraction task, there are several key directions for future development of these efforts. first, while the pretraining strategies employed in this work did not provide a benefit to model performance, further exploration is needed to understand the types of pretraining tasks that may best support measurement extraction efforts. next, to fully contextualize extracted solubility measurements it will be necessary to extract the relevant experimental details (solvent, temperature, ph, etc.). we have collected manual annotation data to support this expansion of the task. additionally, the current approach can detect the existence of several solubility measurements within a sentence but cannot determine which solubility values are associated with which solutes or which experimental conditions. relation extraction capabilities will be needed to associate solubility values with the correct solutes, solvents, and conditions.