Learning Latent Structures for Cross Action Phrase Relations in Wet Lab Protocols (2021)

Abstract: Wet laboratory protocols (WLPs) are critical for conveying reproducible procedures in biological research. They are composed of instructions written in natural language describing the step-wise processing of materials by specific actions. This process flow description for reagents and materials synthesis in WLPs can be captured by material state transfer graphs (MSTGs), which encode global temporal and causal relationships between actions. Here, we propose methods to automatically generate a MSTG for a given protocol by extracting all action relationships across multiple sentences. We also note that previous corpora and methods focused primarily on local intra-sentence relationships between actions and entities and did not address two critical issues: (i) resolution of implicit arguments and (ii) establishing long-range dependencies across sentences. We propose a new model that incrementally learns latent structures and is better suited to resolving inter-sentence relations and implicit arguments. This model draws upon a new corpus WLP-MSTG which was created by extending annotations in the WLP corpora for inter-sentence relations and implicit arguments. Our model achieves an F1 score of 54.53% for temporal and causal relations in protocols from our corpus, which is a significant improvement over previous models - DyGIE++:28.17%; spERT:27.81%. We make our annotated WLP-MSTG corpus available to the research community.

## introduction wet laboratory protocols (wlps) play an integral role in bioscience and biomedical research by serving as a vehicle to communicate experimental instructions that allow for standardization and replication of experiments. these procedures, typically written in natural language, prescribe actions (figure 1 ) to be conducted on materials that generally produce new materials which, in turn, are used by future actions to make newer materials. however, wlps can be unclear, composed of disconnected and distant parts, and built upon implicit information that were referenced earlier or omitted entirely. lack of careful documentation has led to a reproducibility crisis @xcite in the biosciences and also poses considerable challenges for automation of laboratory procedures: gleaning the effect and semantics of actions requires understanding the underlying experiment, the sentence structure and rationale behind implicitly stated arguments. currently, there is a dearth of annotated resources for natural language instructions in laboratory protocols. the wlp corpus initially collected by @xcite and later updated by @xcite focused solely on relations within sentences. however, actions in wlps are more complex, containing additional relations between actions (e.g., temporal and causal rela-tions). we propose using material state transfer graphs (mstg), which are a natural extension of action graphs @xcite . mstgs link together several action graphs into a larger structure by utilizing global temporal and causal relationships that can span several sentences in order to describe the flow of materials from action to action (section 3). an example of a mstg is shown in figure 1 . the action phrase grow the bacteria overnight in step 1 consists of an action grow that acts-on the reagent bacteria for an amount of time specified as overnight. this action graph is then connected to other such graphs (like in @xcite through temporal and causal relationships (e.g., grow action's product is host culture thus we use a product link to establish a temporal relation be- ## related work temporal and causal relation extraction: prior efforts have shown great promise in learning local and global features @xcite . neural-networkbased methods have proven effective @xcite . @xcite use neural support vector machine which can be difficult to train. early methods for extracting causal relations resorted to feature engineering @xcite . recently several researchers @xcite @xcite used convolutional neural networks (cnns) for extracting causal features. notably, li and mao (2019) addressed scarcity of training data thorough knowledge-based cnn. however, such methods are not scalable to multiple sentences. cross sentence relation extraction: long range relations are understudied in literature. prior work focused on relations within a sentence or at best between pairs of sentences @xcite @xcite . in addition to joint entity and relation extraction models, @xcite al. (2019) implicit arguments: early methods selected specific features to build linear classifiers @xcite . others incorporated additional, manually-constructed resources like named entity taggers and wordnet @xcite @xcite . in contrast, a few notable studies used unlabeled training data to resolve implicit arguments @xcite . @xcite explored the full probability space of semantic arguments; however, the method does not scale well. ## a latent structure model for joint entity and relation extraction we develop a latent structure model for jointly learning entity and relations within and across multiple sentences. a schematic of the model is shown in figure 3 . in section 4.1 we describe construction of span representation (figure 3a ) from protocol text that incorporates critical features necessary for long-range relation extraction. section 4.2 explains how the transcoder block (figure 3b ) builds upon latent structures (as illustrated in figure 3d ) to improve entity and relation representations. finally, in section 4.3 we discuss training and regularization strategies to jointly learn span, entity, and relations through a multi-task loss function derived from span, entity, and relation scores (figure 3c ). we shall use figure 1 as a running example throughout the model description. ## experiments in contrast to general language models, domainspecific methods have resulted in more competitive baselines and are better suited @xcite @xcite for simultaneously resolving and predicting entities and relations over longer contexts. thus, we evaluate our model against two state-of-the-art models for jointly predicting entities and relations in scientific-text domain, namely dygie++ @xcite and spert (eberts and ulges, 2019), on the wlp-mstg. we conduct five ( 5 ) runs with random initializations for each evaluation and report the test set performance on the model that achieved the median relation f1 score on the dev(elopment) set. all models are evaluated end-to-end, where the model takes as input tokenized sentences and predicts all the entities and the relations generating a mstg. we use the standard precision, recall and f1 metrics. an entity is considered correct if its predicted span and label match the ground truth. relation extraction is performed on the predicted entity spans. a relation is correct if its relation type and the entity pairs are both correct (in span and type) against the ground truth. we also evaluate our model's performance on wnut 2020 @xcite corpus. to fairly evaluate relation extraction, we use gold entities to make relation predictions 4 by modifying the loss function to only train on relation scores. we additionally concatenate entity label embeddings to the span representation in equation (1). ## conclusions and future work we present the wlp-mstg corpus, an extension of the wlp corpus that includes cap-tac relationships for building mstgs. this corpus highlights two unique challenges: (i) the implicit argument problem and (ii) long-range relations. to address these issues, our model builds upon latent structures thus outperforming previous state-of-the-art models for predicting iap and cap-tac relations. we also report significant improvements in understanding implicit arguments and identifying long range relationships across multiple sentences. however, our model's lower absolute performance indicates that we have not fully captured the information needed to facilitate modeling end-to-end workflows, which will have a lasting impact in improving automation in the life sciences and other domains. christian chiarcos and niko schenk. 2015. a minimalist approach to shallow discourse parsing and implicit relation recognition. in proceedings of the nineteenth conference on computational natural language learning-shared task, pages 42-49. quynh ngoc thi do, steven bethard, and marie francine moens. 2017. improving implicit semantic role labeling by predicting semantic frame arguments. in proceedings of the eighth international joint conference on natural language processing (volume 1: long papers), pages 90-99. markus eberts and adrian ulges. 2019. span-based joint entity and relation extraction with transformer pre-training. 24th european conference on artificial intelligence. christiane fellbaum. 2012. wordnet. the encyclopedia of applied linguistics. matthew gerber and joyce chai. 2010. beyond nombank: a study of implicit arguments for nominal predicates. in proceedings of the 48th annual meeting of the association for computational linguistics, pages 1583-1592. matthew gerber and joyce y chai. 2012. semantic role labeling of implicit arguments for nominal predicates. computational linguistics, 38(4):755-798. zhijiang guo, yan zhang, and wei lu. 2019. attention guided graph convolutional networks for relation extraction. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 241-251. rujun han, qiang ning, and nanyun peng. 2019. joint event and temporal relation extraction with shared representations and structured prediction. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 434-444. kaiming he, xiangyu zhang, shaoqing ren, and jian sun. 2016. deep residual learning for image recognition. in proceedings of the ieee conference on computer vision and pattern recognition, pages 770-778. thomas n kipf and max welling. 2017. semisupervised classification with graph convolutional networks. iclr. klaus krippendorff. 2004. content analysis: an introduction to its methodology. sage publications. chaitanya kulkarni, wei xu, alan ritter, and raghu machiraju. 2018. an annotated corpus for machine reading of instructions in wet lab protocols. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 2 (short papers), pages 97-106. egoitz laparra and german rigau. 2013. impar: a deterministic algorithm for implicit semantic role labelling. in proceedings of the 51st annual meeting of the association for computational linguistics (volume 1: long papers), pages 1180-1189. kenton lee, luheng he, mike lewis, and luke zettlemoyer. 2017. end-to-end neural coreference resolution. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 188-197. kenton lee, luheng he, and luke zettlemoyer. 2018. higher-order coreference resolution with coarse-tofine inference. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 2 (short papers), pages 687-692. artuur leeuwenberg and marie francine moens. 2017. structured learning for temporal relation extraction from clinical records. in proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 1, long papers, pages 1150-1158. pengfei li and kezhi mao. 2019. knowledge-oriented convolutional neural network for causal relation extraction from natural language texts. expert systems with applications, 115:512-523. qimai li, zhichao han, and xiao-ming wu. 2018. deeper insights into graph convolutional networks for semi-supervised learning. in proceedings of the aaai conference on artificial intelligence, volume 32. yuanliang meng and anna rumshisky. 2018. contextaware neural model for temporal information extraction. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 527-536. yuanliang meng, anna rumshisky, and alexey romanov. 2017. temporal information extraction for question answering using syntactic dependencies in an lstm-based architecture. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 887-896. chris miller and soroush vosoughi. 2020. big green at wnut 2020 shared task-1: relation extraction as contextualized sequence classification. arxiv preprint arxiv:2012.04538. dat quoc nguyen and karin verspoor. 2019. end-toend neural relation extraction using deep biaffine attention. in european conference on information retrieval, pages 729-738. springer. site: similar to "acts-on", this relation links to the previous "action" if the product of that "action" is implicit, and that product is where the current "action" is taking place. otherwise we directly connect to the appropriate named entity. once again, similar to "acts-on", if directly connected to "action", it falls under (i-i) case, otherwise its (e-i) case. product: this relation is used to identify the product of the current "action", either its found in its own action phrase or in some future action phrase. if the product is identified within its action phrase (which is quite rare) it would be considered an iap relation. otherwise, this would fall under (i-e) case. coreference: this is used when the objects or arguments are in the same state. we connect the object to the same object referred before only if that object has not undergone any transformations by any actions in between. this relation falls under (e-e) case. overlaps: this relation is used to indicate which two actions are being performed simultaneously or that have any degree of overlap between them in terms of time.