WNUT -2020 Task 1 Overview: Extracting Entities and Relations from Wet Lab Protocols (2020)

Abstract: This paper presents the results of the wet labinformation extraction task at WNUT 2020.This task consisted of two sub tasks- (1) anamed entity recognition task with 13 partic-ipants; and (2) a relation extraction task with2 participants. We outline the task, data an-notation process, corpus statistics, and providea high-level overview of the participating sys-tems for each sub task.

## introduction wet lab protocols consist of natural language instructions for carrying out chemistry or biology experiments (for an example, see figure 1 ). while there have been efforts to develop domain-specific formal languages in order to support robotic automation 1 of experimental procedures @xcite , the vast majority of knowledge about how to carry out biological experiments or chemical synthesis procedures is only documented in natural language texts, including in scientific papers, electronic lab notebooks, and so on. recent research has begun to apply human language technologies to extract structured representations of procedures from natural language protocols @xcite @xcite @xcite . extraction of named entities and relations from these protocols is an important first step towards machine reading systems that can interpret the meaning of these noisy human generated instructions. however, performance of state-of-the-art tools for extracting named entity and relations from wet lab protocols still lags behind well edited text genres @xcite . this motivates the need for continued research, in addition to new datasets and tools adapted to this noisy text genre. in the following sections, we describe details of the task including training and development datasets in addition to the newly annotated test data. we briefly summarize the systems developed by selected teams, and conclude with results. ## wet lab protocols wet lab protocols consist of the guidelines from different lab procedures which involve chemicals, drugs, or other materials in liquid solutions or volatile phases. the protocols contain a sequence of steps that are followed to perform a desired task. these protocols also include general guidelines or warnings about the materials being used. the publicly available archive of protocol.io contains such guidelines of wet lab experiments, written by researchers and lab technicians around the world. this protocol archive covers a large spectrum of experimental procedures including neurology, epigenetics, metabolomics, stem cell biology, etc. figure 1 shows a representative wet lab protocol. the wet lab protocols, written by users from all over the worlds, contain domain specific jargon as well as numerous nonstandard spellings, abbreviations, unreliable capitalization. such diverse and @xcite . to address these challenges, there has been an increasing body of work on adapting entity and relation extraction recognition tools for noisy wet lab texts @xcite @xcite . however, different research groups have used different evaluation setups (e.g., training / test splits) making it challenging to perform direct comparisons across systems. by organizing a shared evaluation, we hope to help establish a common evaluation methodology (for at least one dataset) and also promote research and development of nlp tools for user generated wet-lab text genres. ## evaluation in this section, we present the performance of each participating systems along with a description of the errors made by the model types. ## related work the task of information extraction from wet lab protocols is closely related to the event trigger extraction task. the event trigger task has been studied extensively, mostly using ace data @xcite and the bionlp data @xcite . broadly, there are two ways to classify various event trigger detection models: (1) rule-based methods using pattern matching and regular expression to identify triggers @xcite and ( 2 ) machine learning based methods focusing on generation of high-end hand-crafted features to be used in classification models like svms or maxent classifiers @xcite . kernel based learning methods have also been utilized with embedded features from the syntactic and semantic contexts to identify and extract the biomed-ical event entities @xcite . in order to counteract highly sparse representations, different neural models were proposed. these neural models utilized the dependency based word embeddings with feed forward neural networks @xcite , cnns @xcite and @xcite . previous work has experimented on datasets of well-edited biomedical publications with a small number of entity types. for example, the jnlpba corpus @xcite with 5 entity types (cell line, cell type, dna, rna, and protein) and the bc2gm corpus @xcite with a single entity class for genes/proteins. in contrast, our dataset addresses the challenges of recognizing 18 finegrained named entities along with 15 types of relations from the user-created wet lab protocols. ## summary in this paper, we presented a shared task for consisting of two sub-tasks: named entity recognition and relation extraction from the wet lab protocols. we described the task setup and datasets details, and also outlined the approach taken by the participating systems. the shared task included larger and improvised dataset compared to the prior literature @xcite . this improvised dataset enables us to draw stronger conclusions about the true potential of different approaches. it also facilitates us in analyzing the results of the participating systems, which aids us in suggesting potential research directions for both future shared tasks and noisy text processing in user generated lab protocols.