MS -Mentions: Consistently Annotating Entity Mentions in Materials Science Procedural Text (2021)

Abstract: Material science synthesis procedures are a promising domain for scientific NLP, as proper modeling of these recipes could provide insight into new ways of creating materials. However, a fundamental challenge in building information extraction models for material science synthesis procedures is getting accurate labels for the materials, operations, and other entities of those procedures. We present a new corpus of entity mention annotations over 595 Material Science synthesis procedural texts (157,488 tokens), which greatly expands the training data available for the Named Entity Recognition task. We outline a new label inventory designed to provide consistent annotations and a new annotation approach intended to maximize the consistency and annotation speed of domain experts. Inter-annotator agreement studies and baseline models trained upon the data suggest that the corpus provides high-quality annotations of these mention types. This corpus helps lay a foundation for future high-quality modeling of synthesis procedures.

## introduction the material science literature contains millions of synthesis procedures: descriptions that outline the specific steps required to create a particular material, such as the text in figure 1 . large-scale analysis of these procedures could enable tasks such as automatic planning of new synthesis procedures @xcite . however, such tasks require extraction of high-quality representations of those synthesis procedures from raw text @xcite , and thus are limited by the size and quality of annotations. an important component for information extraction of synthesis procedures involves predicting the entities, actions and attributes of the synthesis, including the steps followed in the synthesis procedures, the materials used and created in the synthesis, and all the quantitative attributes (including operation conditions, material amounts, and properties) necessary to replicate or understand the procedure. such annotations require the judgment of domain experts, as not all verbs in a text are actual steps in the synthesis, and not all materials mentioned are actual inputs or outputs of the synthesis process. we outline an inventory of 15 mention types which can be consistently annotated while still making important distinctions regarding the roles that these operations and mentions play in the overall synthesis. figure 1 illustrates how such an example synthesis procedure would be annotated with entity mentions. we annotate these labels over a new corpus of 595 synthesis procedures. to improve the consistency of this annotation pipeline, we separate the annotation of operation-type mentions, materialtype mentions, and quantitative mentions (including conditions, amounts and properties), allowing annotators to specialize upon a subset of the task and reducing inter-annotator variation in how each phenomenon is annotated. we present baseline models trained on this dataset which demonstrate that models can be trained upon this dataset to achieve mention extraction performance exceeding 90 f1, providing a reliable base for future work. the contributions of the work are as follows. first, we establish a large new dataset of material science mention annotations * , which includes both texts sampled from a wide range of different synthesis procedures. secondly, we provide improvements to synthesis procedure annotation, both by providing a new label set that can be consistently annotated and by providing new approaches to expert mention annotation. thirdly, we provide simple baseline models to be made public with the paper, and use those baseline models to illustrate both the remaining challenges with the data, as well as the promise this dataset holds for high-performing mention extraction over synthesis procedures. ## dataset description we annotate three different kinds of mentions: the individual steps or operations constituting the synthesis procedure, the different kinds of materials mentioned in a procedure, and a third class of quantitative mentions such as measurable conditions, quantities and apparatus mentions. these three broad classes are each annotated in a different stage, as outlined in §3. note that the current dataset does not annotate relations between mentions as in @xcite , but focuses on annotating the much larger set of operation, material and condition mentions to be linked together. ## annotation pipeline the present dataset was annotated by 3 domain experts using the brat annotation tool † , using non-nested mentions. in building the dataset, papers for annotation were picked to contain a mix of randomly selected papers and those from a more focused sub-domain of papers. furthermore, to improve consistency and speed, the different classes of entity types were annotated by the same annotator. each of these processes is elaborated on next. ## results and experiments to contextualize the provided baseline scores on our new task, we then show the performance of this baseline model on the ms-mentions task, using bert @xcite ## analysis of annotator agreement as the pipeline uses a fixed annotator for each part of the mention annotation, the inter-annotator agreement could not simply be measured by "double-annotating" documents. to measure the inter-annotator agreement we instead annotated a set of five documents three times each, with different assignments of annotators for each stage of our division of labor approach ( §3.2). we then score iaa using f1, selecting pairs of annotations, and taking turns treating one as the gold annotation and the next as a prediction, these individual pairs are then micro-averaged. the overall f1 between annotation sets on the span prediction task is 85.7. further, when annotators both agree that a given span exists, we measure a fleiss's kappa of 0.928. as an alternative measure of chancecorrected agreement, we use mathet's γ @xcite , which allows for chance-corrected measures for span-based tasks such as ner. we report a γ of 0.76 (wherein 0 is chance and 1 is perfect agreement), showing substantial agreement. next we also denote labels for which annotators disagree in figure 5 . one can see that the primary sources of label disagreements is the judgments about whether a mention is integral to the synthesis procedure, seen in the distinction between operation and nonrecipe-operation labels, and between materials and nonrecipe-material and unspecified-material labels. examples of such disagreements for operations (ex. 2) and materials (ex. 3) are shown below (2) after evaporation of the solvent, a brown tetraethylammoniumtricyanoimidazolate was obtained rows denotes an arbitrary annotation picked as gold to be compared to the other annotation. the primary source of disagreement for both is the challenging non-recipe vs recipe distinction. next, following prior analyses in parsing and srl @xcite , we show the impact of correcting specific types disagreements with the circularly picked gold annotations for our annotations. this analysis first converts overlapping spans to exactly matching spans (fix span), then making label disagreements match the gold (label), then adding mentions that are missing (add), and finally removing spurious mentions (remove). figure 6 (left panel) explores which kinds of errors cause disagreement between annotators looking at three sets of labels; all types of operation labels, all types of material labels, and all other quantities and conditions. while the additional details such as conditions, amounts, and properties have high agreement, we note that for material labels, fixing slight disagreements in span boundaries has a meaningful impact. we suggest that this is due to the difficulty of consistently defining the mention spans for complex chemical mentions; an example disagreement about target span boundaries is shown in ex. 4 for an instance of target, where "[]" denote span boundaries. ## analysis of model errors here, we can contrast inter-annotator disagreements with the performance of models trained on ms-mentions. figure 6 ment set, as one automatically fixes specific errors as in §6. one can see that the general contour of errors is similar between these model errors and the human error pattern. we suggest that the higher model performance (when compared to iaa f1) may be due to the approach of using a division of labor, so that each phenomenon in the main dataset is annotated by a single annotator. this means that a model is learning and evaluated upon a single annotator's annotation practices. finally to form an understanding of label disagreements, figure 4 shows a confusion matrix for the predictions of the baseline model. we note that trained models also suffer errors in distinguishing between target and material, but are able to distinguish between material and unspecified-material. qualitatively examining model errors, a common source of the span disagreements is seen in ex. 5 with "naoh/urea", where a domain expert in context infers that this refers to a single material, but model predictions split up "naoh" and "urea": a similar form of this kind of model error occurs with complex measurement units where the condition-unit mention would refer to a ratio such as "mol/mol" or a rate such as a "degc/min.". ## related work the presented corpus fits into a larger body of work on scientific information extraction @xcite @xcite . while a majority of the work in scientific ie developed resources for biomedical text, more recent work has seen growing interest in information extraction from materials science text, as in the corpora outlined in this recent thrust to develop information extraction tools may be explained by the consensus in the materials science community that extracting the knowledge contained within natural language descriptions of inorganic materials syntheses will be a key step towards reducing the overall discovery and development time for novel materials @xcite . ## conclusion the ms-mentions dataset established a highquality annotation of entity mentions for the material science domain. it achieves this in part through a novel approach to the division of annotation into multiple stages, reducing the interannotator variation of the annotation of each phenomenon. our baselines show that entity recognition systems trained on this corpus can achieve high performance on this consistently defined task. we hope that this can be utilized to improve downstream tasks relying upon mention detection performance.