P c MSP : A Dataset for Scientific Action Graphs Extraction from Polycrystalline Materials Synthesis Procedure Text (2022)

Abstract: Scientific action graphs extraction from materials synthesis procedures is important for reproducible research, machine automation, and material prediction. But the lack of annotated data has hindered progress in this field. We demonstrate an effort to annotate Polycrystalline Materials Synthesis Procedures PcMSP from 305 open access scientific articles for the construction of synthesis action graphs. This is a new dataset for material science information extraction that simultaneously contains the synthesis sentences extracted from the experimental paragraphs, as well as the entity mentions and intra-sentence relations. A two-step human annotation and inter-annotator agreement study guarantee the high quality of the PcMSP corpus. We introduce four natural language processing tasks: sentence classification, named entity recognition, relation classification, and joint extraction of entities and relations. Comprehensive experiments validate the effectiveness of several state-of-the-art models for these challenges while leaving large space for improvement. We also perform the error analysis and point out some unique challenges that require further investigation. We will release our annotation scheme, the corpus, and codes to the research community to alleviate the scarcity of labeled data in this domain.

## introduction synthesis procedural texts are written in instructional languages @xcite to represent the step-by-step reactions, but also contain the distinct features in specific domains, such as the domain notations, writing styles, and journal requirements. the synthesis procedures of materials science articles include valuable information for new materials prediction @xcite , laboratory automation @xcite and knowledge graph construction 1 https://github.com/xianjun-yang/pcmsp ## the selection of our dataset here we talk about the importance of our selection and how is it different from other materials procedural text corpora. ## inter-annotator agreement study we perform a two-round agreement study to ensure that our corpus has a high quality of annotation. before undertaking the formal annotation, all four annotators participate in a discussion of the formulation rules and discuss the necessary entity and relation types. in the warm-up exercise, all annotators annotate the same documents individually and then compare and discuss the results together to achieve better agreement on annotation. after the agreements are formulated, in the first-round annotation four annotators are randomly assigned different documents to work on. it takes around twenty to thirty minutes to annotate one document on average for all annotators. when all of the annotations are finished, two of the four annotators select several typical examples for analysis and eventually set more rules for annotating the most debatable parts. in the second round of annotation, two lead annotators individually re-annotate half of the documents, guaranteeing that there are no significant differences or mistakes. it takes around 500 hours for our material expert team in total to create this corpus to guarantee high quality. we use fleiss' kappa to measure the agreement scores between our four annotators. the result is shown in ## statistics of corpus and problem formulation in this section, we describe the statistics of this new dataset, the comparison with precious corpora, and formulated tasks. ## results and analysis we present the main experimental results in this section, and more modeling details are included in appendix b. pure refers to the advanced joint extraction model by @xcite . for all the experiments, we use the bert-base-uncased (devlin ## conclusion in summary, we contribute a new dataset pcmsp collected from 305 open access scholarly publications for action graphs construction from material synthesis procedures. the two-round human expert's annotations guarantee the high quality of the dataset, which is evident by the agreement study. based on this new dataset, we perform sentence classification, named entity recognition, and relation extraction tasks. we also experiment with the joint extraction of entities and relations. several good-performing neural models are utilized to provide competitive baselines, although leaving a big gap compared with the human upper bound. to alleviate the data scarcity of this domain, we will make our dataset publicly available. some future directions would be to investigate incorporating cross-sentence context, improving the joint extraction results, performing paragraphlevel end-to-end extraction, as well as using our pcmsp to investigate domain adaptation. for example, pre-training with distant supervision in the materials domain might also help improve the results. considering the high labeling cost, how to efficiently transfer knowledge into other domains to reduce human annotations is also of great importance.