Big Green at WNUT 2020 Shared Task-1: Relation Extraction as Contextualized Sequence Classification (2020)

Abstract: Relation and event extraction is an important task in natural language processing. We introduce a system which uses contextualized knowledge graph completion to classify relations and events between known entities in a noisy text environment. We report results which show that our system is able to effectively extract relations and events from a dataset of wet lab protocols.

## introduction wet lab protocols specify the steps and ingredients required to synthesize chemical and biological products. the majority of wet lab protocols are formatted as natural language, designed for human lab workers to interpret and carry out. protocols are formatted differently depending on lab norms and the author writing them, and may include spelling mistakes, nonstandard abbreviations, colloquial phrasing, and assumptions that may not be obvious to readers from outside of the author's lab or field. automated extraction of events, relations, and entities from this noisy language data enables standardized tracking of lab protocols, and is an important step forward for the automated reproduction of scientific results. we examine the problem of automatically identifying and classifying events and relations between entities as part of shared task 1 at wnut 2020 @xcite . this shared task works with the wet lab protocol corpus (wlpc) introduced by @xcite . the wlpc dataset consists of wet lab protocols drawn from an open-source database and annotated by a group of human annotators which included subject matter experts. ## prior work past approaches to relation and event extraction from wet lab data have included systems based on propagating information across graphs. @xcite introduce an end-to-end system, called spanrel, for identifying and labeling text spans and the relations between them using any text embedding model. the dygie and dygie++ systems, meanwhile, learn to propagate useful information across graphs of coreferences, relations, and events, allowing long-distance contextual information to support relation and event extraction tasks based on sliding window bert embeddings of the text @xcite . ## knowledge graphs knowledge graphs are graph representations of the relations between entities @xcite . in a typical knowledge graph construction, graph nodes represent entities, while edges of different types represent relations between entities. as an example, a knowledge graph may contain nodes for "united kingdom" and "united nations" with an edge of type "member-of" between "united kingdom" and "united nations." because knowledge graphs are generated from imperfect information, they represent a subset of information about their component nodes and thus suffer from incompleteness. incompleteness means that edges representing relations which exist between nodes in reality are not present in the graph (for example, if the knowledge graph contains the "united kingdom" and "united nations" nodes but does not contain the "member-of" relation between them). this property of knowledge graphs has given rise to efforts to identify missing relations between entities, a task referred to as knowledge graph completion @xcite . there are obvious parallels between knowledge graph completion and relation extraction from text given prelabeled entities; namely that both tasks require identifying a relation (if one exists) between a given pair of entities. we therefore develop a model which represents the task of extracting relations from wet lab protocols as a knowledge graph completion problem. ## results & discussion our results, shown in our approach is relatively weak in precision. this is likely due to our formulation of the task as an evaluation of potential entity pairs. we find that our system classifies relations with an accuracy of 93% on the development set, but because there are many more possible pairings between entities in a given protocol than there are actual pairings, even a system with high accuracy can incorrectly predict nonexistent relations. we reduce the number of possible entity pairings generated by applying a distance heuristic discussed in section 4. we found that tuning the amount of entity pair candidates evaluated impacted results significantly (for example, our development set f1-score rose almost 20% when using a token-based distance metric rather than a sentencebased distance metric for selecting candidate entity pairs). the recall of our results suggests that our distance heuristic is effective at dramatically reducing the number of evaluated entity combinations without removing too many valid combinations, but the precision indicates that it may be valuable to modify or find an alternative method for producing candidate entity pairs. this could include a method which considers contextual information, instead of focusing only on the token distance between a pair of entities. class-specific result analysis allows us to identify where our system struggles. one such area is classes which occur less frequently in the data. our macro-average f1-score is 0.69 for the seven most frequent relation classes (each of these has over 1000 examples in the training data), versus a macro-average of 0.43 for the seven least frequent relation classes (each of which has less than 1000 examples in the training data). bert and similar language-embedding models rely on large quantities of training data, and class performance suffering due to lack of training data is not unexrecent prior work has shown that bert and other language embedding models can become overly reliant on simple patterns in the data. @xcite showed that the addition of the text "10 deaths" to uninformative tweets about covid-19 caused a bert based system to mistakenly label them as informative. we anticipate that this effect may make our system more prone to failure in edge cases, where basic clues that the model has learned in terms of entity type patterns or contextual patterns are not present. a potential solution for this problem is to augment the training data using examples which do not have certain attributes (for example, masking entity labels). this may reduce the model's tendency to learn from basic patterns rather than true relationships between text and a relation or event class. ## conclusion & future work we show that contextualized knowledge graph completion using sequence classification can perform effectively on a relation extraction task in a noisy and specialized domain. our model effectively identifies relations and events in the data, and our work leaves open many avenues for future work. as discussed in section 5, our system is sensitive to how candidate entity pairs are selected. we use a distance heuristic based on statistics of the training data to achieve our results, but we anticipate that more sophisticated methods for identifying promising candidate entity pairs could improve our results. we also suggest that our results could be improved by using a domain-specific model such as scib-ert or biobert (models trained on scientific papers and abstracts respectively). prior work shows that these models often outperform standard bert models on scientific data @xcite . we believe that our results and the results of any systems which require training or fine-tuning large models would be improved by increasing available training data. finding an effective method for augmenting existing training data and generating or collecting new training data (artificial or real) is a valuable route for further study. finally, we are interested in further investigation of representing relation and event identification as graph completion. link prediction systems which support a variety of edge labels could allow us to leverage structural data from a protocol relation graph. this could enable the identification of relations which are improbable or those which may be missing from the predictions.