Extracting Fine-Grained Knowledge Graphs of Scientific Claims: Dataset and Transformer-Based Results (2021)

Abstract: Recent transformer-based approaches demonstrate promising results on relational scientific information extraction. Existing datasets focus on high-level description of how research is carried out. Instead we focus on the subtleties of how experimental associations are presented by building SciClaim, a dataset of scientific claims drawn from Social and Behavior Science (SBS), PubMed, and CORD-19 papers. Our novel graph annotation schema incorporates not only coarse-grained entity spans as nodes and relations as edges between them, but also fine-grained attributes that modify entities and their relations, for a total of 12,738 labels in the corpus. By including more label types and more than twice the label density of previous datasets, SciClaim captures causal, comparative, predictive, statistical, and proportional associations over experimental variables along with their qualifications, subtypes, and evidence. We extend work in transformer-based joint entity and relation extraction to effectively infer our schema, showing the promise of fine-grained knowledge graphs in scientific claims and beyond.

## introduction using relations as edges to connect nodes consisting of extracted entity mention spans produces expressive and unambiguous knowledge graphs from unstructured text. this approach has been applied to diverse domains from moral reasoning in social media @xcite to qualitative structure in ethnographic texts @xcite , and is particularly useful for reasoning about scientific claims, where several experimental variables in a sentence may have differing relations. scientific information extraction datasets such as scierc @xcite use relations for labeling general scientific language. utilizing the advances of scibert @xcite in scientific language modeling, spert @xcite text: levels of social support for medical staff were significantly associated with self -efficacy and sleep quality and negatively associated with the degree of anxiety and stres levels of social support factor self -efficacy factor q+ sleep quality factor q+ the degree of anxiety factor qstress factor q-for medical staff qualifier significantly magnitude associated with association (correlation, sign+) arg0 arg1 arg1 negatively magnitude associated with association (correlation, sign-) arg0 arg1 arg1 input: "levels of social support for medical staff were significantly associated with self-efficacy and sleep quality and negatively associated with the degree of anxiety and stress." figure 1 : sciclaim knowledge graph with entities (nodes), relations (edges), and attributes (parentheticals) connecting an independent variable via arg0 to distinct correlations with dependent variables via arg1. ulges, 2020)-a transformer-based joint entity and relation extraction model-advanced the state of the art on scierc. to extend relational scientific information extraction to specifically target scientific claims, we annotate sciclaim, foot_0 a dataset of 12,738 annotations on 901 sentences from expert identified claims in social and behavior science (sbs) papers @xcite , detected causal language in pubmed papers @xcite , and claims and causal language heuristically identified from cord-19 abstracts @xcite . for annotation, we developed a novel graph schema that reifies claimed associations as entity spans with fine-grained attributes and extracts factors as additional entities connected with relations to one or more associations in which they are involved. in figure 1 , two association entities relate two pairs of dependent factors to an independent factor, while attributes and additional relations delimit the scope and qualitative proportionalities of the claim. inspired by semantic role labeling, attributes modify associations and the roles of their arguments, allowing us to represent claims of causal, comparative, predictive, statistical, and proportional associations along with their qualifi-text: we predicted that the subliminal prime would , under specifiable conditions , increase the accessibility of the pertinent negative outcome and thereby increase its perceived likelihood of occurrence . input: "we predicted that the subliminal prime would, under specifiable conditions, increase the accessibility of the pertinent negative outcome and thereby increase its perceived likelihood of occurrence." figure 2 : this sciclaim graph captures the chaining together of associations and uncovers a mediating factor in the qualitative proportionality (q+) between the "subliminal prime" and "perceived likelihood of occurrence." we adapt spert to model this additional multilabel attribute task and demonstrate that extraction of our highly expressive knowledge graphs is within reach of present methods. ## related work many previous datasets for relational scientific information extraction-such as semeval 2017 task 10 and 2018 task 7, scierc, and scirex @xcite @xcite )-have annotated corpora from nlp, computer science, or similar engineering-oriented fields. as such their annotation schemas have emphasized the description of how research was carried out, by extracting categories of entities such as methods, tasks, metrics, and datasets as well as relations mostly describing their intrinsic properties such as uses, composition, and hyponymy. two of these datasets @xcite contain associative relations that directly link entities being compared or producing a result. our work extends further in this direction by examining not only which entities are associated, but also how the presentation of the associations is nuanced by the assertion of fine-grained attributes such as causality or proportionality. sciclaim provides the largest number of finegrained label types among comparable datasets. meanwhile, our dataset also focuses on scientific claims. some previous work identifies claims within scientific texts @xcite , but does not extract the relations and factors within the claims themselves. other recent symbolic semantic nlp systems do model relational representations of scientific claims (e.g., @xcite , but these approaches rely on rule-based engines with hand tuning, which require nlp experts to maintain and adapt to new domains. instead we modify spert (eberts and ulges, 2020), a transformer-based method that has been shown to effectively extract relational scientific information on scierc @xcite . we extend this model to accommodate our additional multi-label attributes and apply it to our claim graph extraction task. ## conclusion previous scientific information extraction crafts useful high-level representation of papers, going as far as document level relations spanning thousands of words in @xcite . complimentary to these efforts, we propose fine-grained and densely annotated scientific information extraction that captures not just what is said but how it is presented and argued. sciclaim applies this approach to associative claims and demonstrates that existing models such as spert (eberts and ulges, 2020) can be modified to accurately extract fine-grained knowledge graphs ripe for downstream reasoning.