P ublish I n C ovid19 at WNUT 2020 Shared Task-1: Entity Recognition in Wet Lab Protocols using Structured Learning Ensemble and Contextualised Embeddings (2020)

Abstract: In this paper, we describe the approach that we employed to address the task of Entity Recognition over Wet Lab Protocols - a shared task in EMNLP WNUT-2020 Workshop. Our approach is composed of two phases. In the first phase, we experiment with various contextualised word embeddings (like Flair, BERT-based) and a BiLSTM-CRF model to arrive at the best-performing architecture. In the second phase, we create an ensemble composed of eleven BiLSTM-CRF models. The individual models are trained on random train-validation splits of the complete dataset. Here, we also experiment with different output merging schemes, including Majority Voting and Structured Learning Ensembling (SLE). Our final submission achieved a micro F1-score of 0.8175 and 0.7757 for the partial and exact match of the entity spans, respectively. We were ranked first and second, in terms of partial and exact match, respectively.

## introduction entity recognition (aka entity extraction or chunking) involves detection (begin and end boundaries) and classification of entities mentioned in unstructured text into pre-defined categories. it is one of the foundational sub-task of several information extraction @xcite (ie) and natural language processing (nlp) pipelines. hence, errors introduced during the extraction of entities can propagate further and degrade the performance of the complete ie or nlp pipeline. in the domains of experimental biology, the growing complexity of experiments has resulted in a need to automate wet laboratory procedures. such an automation will be useful in avoiding human errors introduced in the wet lab protocols and thereby will enhance the reproducibility of experimental biological research. to achieve this reproducibility, some of the previous research works have focussed on defining machine-readable formats for writing wet lab protocols @xcite @xcite . however, the vast majority of today's protocols are written in natural language with jargon and colloquial language constructs that emerge as a byproduct of ad-hoc protocol documentation. this motivates the need for machine reading systems that can interpret the meaning of these natural language instructions, to enhance reproducibility via semantic protocols (e.g. the aquarium project) and enable robotic automation @xcite by mapping natural language instructions to executable actions. in order to enable research on interpreting natural language instructions, with practical applications in biology and life sciences, an annotated database @xcite of wet lab protocols was introduced. the first step in interpreting natural language lab protocols is to extract entities, followed by identification of relations between them. to address the research focussing on entity recognition over wet lab protocols a shared task @xcite ) was introduced at emnlp wnut-2020 workshop. the task was based on the annotated database @xcite of wet lab protocols. we tackle this task in two phases. in the first phase, we experiment with various contextualised word embeddings (like flair, bert-based) and a bilstm-crf model to arrive at the bestperforming architecture. in the second phase, we create an ensemble composed of eleven bilstm-crf models. the individual models are trained on random train-validation splits of the complete dataset. here, we also experiment with different output merging schemes, including majority voting and sle. the rest of the paper is structured as follows: section 2 states the task definition. section 3 describes the specifics of our methodology. section 4 explains the experimental setup and the results, and section 5 concludes the paper. ## task definition the steps involved in any lab procedure are specified by lab protocols. these protocols have several characteristics like noise, density and domain specificity. any process that can automatically or semiautomatically convert protocols into a format that machine recognizes advantages biological research. in this task, system entries for entity recognition on a dataset of lab protocols are invited. since the protocols are written manually by lab technicians and researchers, they are subject to spelling errors and non standard language. the data provided in the task is made available in two formats: ## methodology this section talks about the core methodology we adopted to tackle the given problem. the process pipeline involves providing contextualised word embeddings as input to the bilstm-crf model, followed by a structured learning ensemble approach. each of the these modules have been described in detail in the below subsections. ## experiments our experimentation strategy is distributed in two phases. in the first phase, we experiment with various architectures and their specifications by varying the type of pre-trained model, deciding layers to freeze i.e. complete fine-tuning or contextual word embeddings, varying type and size of final layer in order to arrive at the best performing model. we trained each of our model architectures on the train split and identified the checkpoint which worked best using the validation split. we reported the final numbers on the test split. for each model, we train three different models with random seed values and then report averaged f1 scores to ensure that improvements are not the result of randomisation. a configuration of concatenated contextual word embeddings from pubmedbert and flair, followed by 2 bilstm layers with 512 dimensional hidden size and a crf layer in the end worked best. in the second phase, we train individual models on random splits of train + validation sets. in order to merge the outputs of individual models, we experiment with two output merging schemes namely majority voting and structured learning ensemble (sle). finally, we report the results on the test dataset. in the following sub-sections, we describe the dataset, system settings, evaluation metrics, results and a brief error analysis for our final submitted system. ## conclusion and future work through this paper, we showcased our approach to tackle the shared task 1 in emnlp wnut-2020 workshop which involved entity recognition over wet lab protocols. we solved the task in two phases. the first phase involved experimenting with different contextualised word embeddings like bert and flair, and a bilstm-crf model to find the best performing model configuration for the problem at hand. in the second phase, we create an ensemble consisting of eleven bilstm-crf models. we train individual models on randomly shuffled train-validation splits of the complete dataset. also, we experiment with different merging techniques like majority voting and structured learning ensemble (sle). our end solution achieved a micro f1-score of 0.8175 and 0.7757 in the partial and exact match categories, respectively. we were ranked first and second in partial and exact match categories respectively. in the future, we wish to explore the idea of employing rule-based approach to overcome the shortcomings of current solution.