A n I nstance L evel A pproach for S hallow S emantic P arsing in S cientific P rocedural T ext (2020)

Abstract: In specific domains, such as procedural scientific text, human labeled data for shallow semantic parsing is especially limited and expensive to create. Fortunately, such specific domains often use rather formulaic writing, such that the different ways of expressing relations in a small number of grammatically similar labeled sentences may provide high coverage of semantic structures in the corpus, through an appropriately rich similarity metric. In light of this opportunity, this paper explores an instance-based approach to the relation prediction sub-task within shallow semantic parsing, in which semantic labels from structurally similar sentences in the training set are copied to test sentences. Candidate similar sentences are retrieved using SciBERT embeddings. For labels where it is possible to copy from a similar sentence we employ an instance level copy network, when this is not possible, a globally shared parametric model is employed. Experiments show our approach outperforms both baseline and prior methods by 0.75 to 3 F1 absolute in the Wet Lab Protocol Corpus and 1 F1 absolute in the Materials Science Procedural Text Corpus.

## introduction being able to represent natural language descriptions of scientific experiments in a structured form promises to allow tackling a range of challenges from automating biomedical experimental protocols @xcite to gaining materials science insight by large scale mining of the literature @xcite . to facilitate these applications, recent work has created datasets annotated with sentence level semantic structure for procedural scientific text from experimental biology @xcite and materials science @xcite . however, these corpora, the wet lab protocols corpus (wlp) and the materials query: neighbor: "centrifuge supernatant at 12,000xg for 10 minutes." query: "add 700µl 70% ethanol to the tube and invert several times to wash the dna pellet." neighbor: "add 200µl 70% ethanol and invert the tube twice to wash the pellet." science procedural text (mspt) corpus remain small. this motivates approaches to parsing that are likely to generalize given limited labelled data. we propose an instance-based edge-factored approach for the relation prediction sub-problem of shallow semantic parsing. to predict a possible relation between two entities, our approach retrieves a set of sentences similar to the target sentence, and learns to copy relations in those sentences to the target sentence (figure 1 shows some examples). however, using only a nearest-neighbours approach over similar sentences poses a coverage problem, as some edge labels may have zero instances in the set of nearest neighbour sentences. to address this, we employ a parametric approach which can score a label when it is not possible to copy that label from any of the neighbours. therefore, we combine a local, instance-level approach with a global, parametric approach. our instance-based approach is motivated by the observation that text in the wlp and mspt corpora, both of which describe experimental protocols, follow domain-specific writing conventions (sometimes referred to as a sublanguage @xcite )) resulting in text that is repetitive and semi-structured. in such restricted domains we postulate that a lowbias instance-level approach may generalize better compared to a parametric approach, which is likely to suffer from a lack of training data. in evaluations of the proposed approach we find the proposed local and global approach to outperform baseline methods based on parametric approaches by 0.75 f1 absolute in wlp and 1 f1 absolute in mspt and prior work by 2.69 f1 absolute (12.7 % error reduction) on the wlp corpus. we also present first results for relation prediction on the mspt corpus. code and data for our experiments is available. foot_0 ## task setup and notation given a sentence @xmath0, . . . x i , . . . x l from a dataset d, let x denote tokens, and (m, t) entity mentions and their entity types, where @xmath1, where c is the set of all possible contiguous token spans in x. 2 in a sentence, we denote the set of all entity mentions with m . given this, we focus on the task of relation prediction which outputs a set of directed edges e such that, @xmath2, m d , r) with @xmath3, where m s , m d denote source and destination mentions, @xmath4, r denotes the set of relation labels defined for the dataset and ∅ denotes the absence of a relation. ## local and global model for relation prediction the proposed relation prediction approach is a combination of two components: a local, instancebased component which predicts the relation r of one edge (m s , m d ) by copying a label from a set of nearest neighbor edges e @xmath5, m nd , r n ) ∈ n , and a second component making a prediction from a globally shared set of parameters. the set of nearest neighbor edges n is obtained from similar sentences in the training set ( §3.2). this is formulated as follows: here, e g represents the globally shared scoring function and e l the local scoring function, here we drop additional arguments to these functions for brevity. z denotes the normalization constant where: equation 1 represents a model which is biased first to copy edge labels from n and in the absence of a label in n rely on a global model. this is in contrast to a model which trades off local and global models in a data dependent manner, the approach taken in the copy-generate model of @xcite . the proposed formulation imposes an inductive bias in the model to copy edge labels which we believe helps perform well in our small data regime. in practice, our approach uses the local model for more frequently occurring labels and the global model for rare labels. conceptually, this is once again, in contrast to the models of @xcite and @xcite which use a copy-model for long-tail or low-frequency phenomena. we believe this contrast is reasonable due to the formulaic nature of the text and the small data regime. here, a local instance-level approach is able to generalize better by copying labels while the global model suffers from a lack of training data to learn the majority label patterns. low frequency labels would see comparable performance for the global and instance level models. we confirm these intuitions empirically in §4. next we define the neural-network parameterization of the model. ## results and analysis we evaluate the proposed approach on two datasets of procedural scientific text: the materials science procedural text (mspt) corpus and the wet lab protocols (wlp) corpus. in both corpora we focus on the sentence level relation prediction task given gold entity mention spans. the experimental setup is detailed in appendix a.1. ## related work instance-based learning approaches have been applied to a rage number of information extraction tasks such as semantic role labeling (srl), named entity recognition (ner), and part of speech (pos) tagging. @xcite and @xcite presents closest related work in terms of the task instance level methods are applied to. @xcite ods have also proven useful for language modeling @xcite , knowledge base reasoning tasks @xcite , and few-shot classification @xcite and regression @xcite problems. works in text generation such as summarization @xcite have also incorporated "copy" mechanisms, pointing at longtail phenomena from text to be summarized or translated rather than directly predicting them. these methods bear close methodological similarity to the proposed approach while differing in having a weaker inductive bias to copy labels. also similar, are retrieve-and-edit approaches which have been applied instance based methods for generating complex structured outputs and text generation @xcite . ## conclusion we propose an edge factored instance based approach to the relation prediction sub-task within shallow semantic parsing for procedural scientific text. our approach leverages the highly formulaic writing of procedural scientific text to achieve better generalization than baseline methods with weaker inductive biases to copy and prior approaches which represent parametric approaches on two corpora of english scientific text. while our work has only looked at predicting relations in an edge factored manner future work might explore ways of predicting higher order groups of edges. other extensions might consider jointly predicting spans and edges as in @xcite . future work might also consider questions of characterizing and measuring formulaicity in text and how a range of information extraction tasks may be tailored to these texts. finally, our approach relies on a static retrieval of sentences, there may also be potential for this aspect to be improved upon with a dynamic retrieval model trained along side the label prediction models similar to @xcite , we expect this would be feasible particularly given the small dataset sizes in this domain.