The SOFC -Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain (2020)

Abstract: This paper presents a new challenging information extraction task in the domain of materials science. We develop an annotation scheme for marking information on experiments related to solid oxide fuel cells in scientific publications, such as involved materials and measurement conditions. With this paper, we publish our annotation guidelines, as well as our SOFC-Exp corpus consisting of 45 open-access scholarly articles annotated by domain experts. A corpus and an inter-annotator agreement study demonstrate the complexity of the suggested named entity recognition and slot filling tasks as well as high annotation quality. We also present strong neural-network based models for a variety of tasks that can be addressed on the basis of our new data set. On all tasks, using BERT embeddings leads to large performance gains, but with increasing task complexity, adding a recurrent neural network on top seems beneficial. Our models will serve as competitive baselines in future work, and analysis of their performance highlights difficult cases when modeling the data and suggests promising research directions.

## introduction the design of new experiments in scientific domains heavily depends on domain knowledge as well as on previous studies and their findings. however, the amount of publications available is typically very large, making it hard or even impossible to keep track of all experiments conducted for a particular research question. since scientific experiments are often time-consuming and expensive, effective knowledge base population methods for finding promising settings based on the published research would be of great value (e.g., @xcite @xcite . while such real-life information extraction tasks have received consid- the corresponding [sofcdevice] with [ptmaterial] / [smnio3material] / [ptmaterial] geometry [demonstratedexperiment] dramatic power output of [225 mw cm2value] at [500 cvalue]. workingtemperature powerdensity fuelused cathodematerial device anodematerial erable attention in the biomedical domain (e.g., @xcite @xcite , there has been little work in other domains @xcite , including materials science (with the notable exception of the work by @xcite . in this paper, we introduce a new information extraction use case from the materials science domain and propose a series of new challenging information extraction tasks. we target publications about solid oxide fuel cells @xcite in which the interdependence between chosen materials, measurement conditions and performance is complex (see figure 1 ). for making progress within natural language processing (nlp), the genre-domain combination presents interesting challenges and characteristics, e.g., domain-specific tokens such as material names and chemical formulas. we provide a new corpus of open-access scientific publications annotated with semantic frame information on experiments mentioned in the text. the annotation scheme has been developed jointly with materials science domain experts, who subsequently carried out the high-quality annotation. we define an "experiment"-frame and annotate sentences that evoke this frame with a set of 16 possible slots, including among others anodematerial, fuelused and workingtemperature, reflecting the role the referent of a mention plays in an experiment. frame information is annotated on top of the text as graphs rooted in the experiment-evoking element (see figure 1 ). in addition, slot-filling phrases are assigned one of the types material, value, and device. the task of finding experiment-specific information can be modeled as a retrieval task (i.e., finding relevant information in documents) and at the same time as a semantic-role-labeling task (i.e., identifying the slot fillers). we identify three sub-tasks: with the aim of fostering research on challenging information extraction tasks in the scientific domain, we target the domain of sofc-related experiments as a starting point. our findings based on this sample use case are transferable to similar experimental domains, which we illustrate by applying our best model configurations to a previously existing related corpus @xcite , achieving state-of-the-art results. we sum up our contributions as follows: ## related work information extraction for scientific publications. recently, several studies addressed information extraction and knowledge base construction in the scientific domain @xcite @xcite . we also aim at knowledge base construction but target publications about materials science experiments, a domain understudied in nlp to date. information extraction for materials science. the work closest to ours is the one of mysore et al. (2019) who annotate a corpus of 230 paragraphs describing synthesis procedures with operations and their arguments, e.g., "the resulting [solid products m aterial ] were ... [dried operation ] at [120 n umber ][celsius conditionu nit ] for [8 n umber ] [h conditionu nit ]." operation-evoking elements ("dried") are connected to their arguments via links, and with each other to indicate temporal sequence, thus resulting in graph structures similar to ours. their annotation scheme comprises 21 entity types and 14 relation types such as participant-material, apparatus-of and descriptor-of. kononova et al. 2019 also retrieve synthesis procedures and extract recipes, though with a coarser-grained label set, focusing on different synthesis operation types. @xcite create a dataset for named entity recognition on abstracts of materials science publications. in contrast to our work, their label set (e.g., material, application, property) is targeted to document indexing rather than information extraction. a notable difference to our work is that we perform full-text annotation while the aforementioned approaches annotate a pre-selected set of paragraphs (see also @xcite . @xcite apply the generative model of @xcite to induce action graphs for synthesis procedures of materials from text. in section 7.1, we implement a similar entity extraction system and also apply our algorithms to the dataset of @xcite . @xcite train word2vec @xcite embeddings on materials science publications and show that they can be used for recommending materials for functional applications. other works adapt the bert model to clinical and biomedical domains @xcite , or generally to scientific text @xcite . neural entity tagging and slot filling. the neural-network based models we use for entity tagging and slot filling bear similarity to state-ofthe-art models for named entity recognition (e.g., @xcite @xcite . other related work exists in the area of semantic role labeling (e.g., @xcite @xcite @xcite . ## annotation scheme in this section, we describe our annotation scheme and guidelines for marking information on sofcrelated experiments in scientific publications. ## corpus statistics and task definitions in this section, we describe our new corpus and propose a set of information extraction tasks that can be trained and evaluated using this dataset. sofc-exp corpus. our corpus consists of 45 open-access scientific publications about sofcs and related research, annotated by domain experts. for manual annotation, we use the inception annotation tool @xcite . task definitions. our rich graph-based annotation scheme allows for a number of information extraction tasks. in the scope of this paper, we address the following steps of (1) identifying sentences that describe sofc-related experiments, (2) recognizing and typing relevant named entities, and ## inter-annotator agreement study we here present the results of our inter-annotator agreement study, which we perform in order to estimate the degree of reproducibility of our corpus and to put automatic modeling performance into perspective. six documents (973 sentences) have been annotated independently both by our primary annotator, a graduate student of materials science, and a second annotator, who holds a ph.d. in physics and is active in the field of materials science. the label distribution in this subset is similar to the one of our overall corpus, with each annotator choosing experiment about 11.8% of the time. ## modeling in this section, we describe a set of neural-network based model architectures for tackling the various information extraction tasks described in section 4. ## experiments in this section, we present the experimental results for detecting experiment-describing sentences, entity mention extraction and experiment slot identification. for tokenization, we employ chemdataextractor, 8 which is optimized for dealing with chemical formulas and unit mentions. we tune our models in a 5-fold cross-validation setting. we also report the mean and standard deviation across those folds as development results. for the test set, we report the macro-average of the scores obtained when applying each of the five models to the test set. to put model performance in relation to human agreement, we report the corresponding statistics obtained from our interannotator agreement study (section 5). note that these numbers are based on a subset of the data and are hence not directly comparable. hyperparameters and training. the bilstm models are trained with the adam optimizer @xcite with a learning rate of 1e-3. for fine-tuning the original bert models, we follow the configuration published by @xcite and use adamw @xcite as optimizer and a learning rate of 4e-7 for sentence classification and 1e-5 for sequence tagging. when adding bert tokens to the bilstm, we also use the adamw optimizer for the whole model and learning rates of 4e-7 or 1e-5 for the bert part and 1e-3 for the remainder. for regularization, we employ early stopping on the development set. we use a stacked bilstm with two hidden layers and 500 hidden units for all tasks with the exception of the experiment sentence de- experiment sentence detection. ## conclusion we have presented a new dataset for information extraction in the materials science domain consisting of 45 open-access scientific articles related to solid oxide fuel cells. our detailed corpus and interannotator agreement studies highlight the complexity of the task and verify the high annotation quality. based on the annotated structures, we suggest three information extraction tasks: the detection of experiment-describing sentences, entity mention recognition and typing, and experiment slot filling. we have presented various strong baselines for them, generally finding that bert-based models outperform other model variants. while some categories remain challenging, overall, our models show solid performance and thus prove that this type of data modeling is feasible and can lead to systems that are applicable in production settings. along with this paper, we make the annotation guidelines and the annotated data freely available. outlook. in section 7.1, we have shown that our findings generalize well by applying model architectures developed on our corpus to another dataset. a natural next step is to combine the datasets in a multi-task setting to investigate to what extent models can profit from combining the information annotated in the respective datasets. further research will investigate the joint modeling of entity extraction, typing and experiment frame recognition. in addition, there are also further natural language processing tasks that can be researched using our dataset. they include the detection of events and sub-events when regarding the experiment-descriptions as events, and a more linguistically motivated evaluation of the framesemantic approach to experiment descriptions in text, e.g., moving away from the one-experimentper-sentence and one-sentence-per-experiment assumptions and modeling the graph-based structures as annotated.