mgsohrab at WNUT 2020 Shared Task-1: Neural Exhaustive Approach for Entity and Relation Recognition Over Wet Lab Protocols (2020)

Abstract: We present a neural exhaustive approach that addresses named entity recognition (NER) and relation recognition (RE), for the entity and re- lation recognition over the wet-lab protocols shared task. We introduce BERT-based neural exhaustive approach that enumerates all pos- sible spans as potential entity mentions and classifies them into entity types or no entity with deep neural networks to address NER. To solve relation extraction task, based on the NER predictions or given gold mentions we create all possible trigger-argument pairs and classify them into relation types or no relation. In NER task, we achieved 76.60% in terms of F-score as third rank system among the partic- ipated systems. In relation extraction task, we achieved 80.46% in terms of F-score as the top system in the relation extraction or recognition task. Besides we compare our model based on the wet lab protocols corpus (WLPC) with the WLPC baseline and dynamic graph-based in- formation extraction (DyGIE) systems.

## introduction the entity and relation recognition over wet-lab protocol @xcite shared task foot_0 is an open challenge that allows participants to use any methodology and knowledge sources for the wet lab protocols that specify the steps in performing a lab procedure. the task aims at two sub-tasks in wet lab protocols domain: named entity recognition (ner), and relation recognition or extraction (re). in ner, the task is to detect mentions and classify them into entity types or no entity. ner has drawn considerable attentions as the first step towards many natural language processing (nlp) applications including relation extraction @xcite , event extraction @xcite , and co-reference resolution @xcite . in contrast, relation extraction (re) is a task to identify relation types between known or predicted entity mentions in a sentence. in this paper, we present a bert-based neural exhaustive approach that addresses both ner and re tasks. we employ a neural exhaustive model @xcite for ner and the extended model that addresses re task. the model detects flat and nested entities by reasoning over all the spans within a specified maximum span length. unlike the existing models that rely on token-level labels, our model directly employs an entity type as the label of a span. the spans with the representations are classified into their entity types or non-entity. with the mentions predicted by the ner module, we then feed the detected or known mentions to the re layer that enumerates all trigger-argument pairs as trigger-trigger or trigger-entity pairs and assigns a role type or no role type to each pair. the best run for each sub-task achieved the fscore of 76.60% on entity recognition task that stands third rank system and the f-scores of 80.46% on relation extraction task as the top system. besides, we also compare our model with the state-of-the-art models over the wet lab protocols corpus (wlpc). we compare the wlpc baseline model based on lstm-crf and maximumentropy-based approaches to address ner and re tasks respectfully. we also compare our model with dynamic graph-based information extraction (dy-gie) system. our model outperforms by 4.81% for ner and 7.79% for re over the wlpc baseline and 3.61% for ner over the dygie system. ## related work most ner work focus on flat entities. @xcite proposed a lstm-crf (conditional ran-dom fields) model and this has been widely used and extended for the flat ner, e.g., @xcite . in recent studies of neural network based flat ner, @xcite have shown that morphological analysis using additional word representations based on linguistic properties of the words, especially for morphologically rich languages such as turkish and finnish, improves the ner performances further compared with using only representations based on the surface forms of words. recently, nested ner has been widely interested in nlp. @xcite detected nested entities in a bottom-up way. they detected the innermost flat entities and then found other nes containing the flat entities as sub-strings using rules on the detected entities. the authors reported an improvement of around 3% in the f-score under certain conditions on the genia data set @xcite . recent studies show that the conditional random fields (crfs) can produce significantly higher tagging accuracy in flat or nested (stacking flat ner to nested representation) ners @xcite . @xcite proposed a novel neural model to address nested entities by dynamically stacking flat ner layers until no outer entities are extracted. a cascaded crf layer is used after the lstm output in each flat layer. the authors reported that the model outperforms state-of-the-art results by achieving 74.5% in f-score on the ge-nia data set. @xcite proposed a neural model that detects nested entities using exhaustive approach that outperforms the state-ofthe-art results in terms of f-score on the genia data set. @xcite further extended the span representations for entity recognition and addressed sensitive span detection tasks in the med-docan (medical document anonymization) shared task foot_1 , and the system achieved 93.12% and 93.52% in terms of f-score for ner and sensitive span detection, respectively. recent successes in neural networks have shown impressive performance on coupling information extraction (ie) tasks as in joint modeling of entities and relations @xcite . @xcite proposed a dynamic graph information extraction (dygie) system for coupling multiple ie tasks, a multi-task learning approach to entity, relation, and coreference extraction. dygie uses dynamic graph propagation to explicitly incorpo-rate rich contextual information into the span representations, and the system achieved significant f1 score improvement on the different datasets. @xcite establised a baseline for ie on the wet lab protocols corpus (wlpc). they employ an lstm-crf for entity recognition approach. for relation extraction, they assume the presence of gold entities and train a maximum-entropy classifier using features from the labeled entities. ## neural exhaustive approach for ner and relation extraction our bert-based neural exhaustive approach is built upon a pipeline approach of two modules: to solve entity and relation recognition tasks, the pipeline approach can be presented as three layers: bert layer, entity recognition layer, and relation recognition layer. figure1 shows the system architecture of entity and relation recognition. ## experimental settings we provide empirical evidence on the effectiveness of the pipeline architecture in both ner and relation extraction over the wet lab protocols 3 task of the w-nut 2020 4 . the wet lab protocols corpus with eighteen entity types 5 and fifteen relation types 6 are randomly split into four subsets: train, development, test, and test release (unlabeled) sets, which contain 370, 122, 123 and 111 lab protocols respectively. in our experiments, we merge the train and development as train-set, test-set use as development-set, and predict the annotations for test release set which is used as test-set. our model is implemented in the pytorch foot_6 framework. we employed the official wet lab protocols evaluation script for ner foot_7 and relation extraction foot_8 to evaluate our system's performances on both tasks. ## results and discussions in order to evaluate the performance of ner, we conduct experiments on different sets of bert-based learning representations, including pubmedbert with merging training-and devset (pubmedbert-merge), pubmedbert along with training (pubmedbert-train), scibert with merging training-and dev-set (scibert-merge), and scibert along with training (scibert-train). in contrast to relation extraction, as based on our primary results of ner with pubmedbert and scibert where pubmedbert is outperforming to scibert. therefore, we conduct all our relation extraction experiments using pubmedbert. for relation extraction task, we learn our model on two data scenarios. first, we perform a clustering approach on training-and dev-set to find the similar or duplicate text files in wet-lab data set. we found that many similar text files with inconsistent annotations exist in the train-and dev-set. the similarity approach with a setting threshold is applied on the train and dev-set to cluster the similar or duplicate text protocols. we then eliminate those text and its corresponding annotation files which appear in the training set to avoid model learning confusion and data leakage. we also applied the predefined relation rules @xcite to filter out any invalid relations appearing in the system output. we conduct experiments on different sets of pubmedbert-based learning representations, including pubmedbert using finetune with filtering approach (pubmedbert-finetune-filter), pubmedbert along with finetune (pubmedbert-finetune), pubmedbert along with filter approach (pubmedbert-filter) and pubmedbert without finetune and filtering approaches (pubmedbert). in second data scenario, we learn our model by keeping all the original training set, development set, and test set. based on original data setting, the pubmedbert-based learning representations are pubmedbert-original-finetune-filter, pubmedbert-original-finetune, pubmedbert-original-filter and pubmedbert-original. ## conclusion this paper presented a bert-based neural exhaustive approach that addresses both named entity recognition (ner) and relation extraction (re) tasks. this neural approach consider all possible spans exhaustively, for ner which is capable to detect flat and nested entities from the generated mention candidates. several enhancements, namely pubmedbert, scibert, bert-base-uncased, filtering, clustering, and ensembling are investigated for the wet-lab protocol data set to enhance the system performance. in ner task, we achieved 76.60% in terms of fscore as third rank system among the participated systems. in relation extraction task, we achieved 80.46% in terms of f-score as the top system that participated in the relation extraction task. moreover, our model outperforms by 4.81% for ner and 7.79% for re over the wlpc baseline and 3.61% for ner over the dygie system. in the future direction, we will implement a joint modeling that addresses ner and relation extraction in an end-to-end manner.