IITKGP at W - NUT 2020 Shared Task-1: Domain specific BERT representation for Named Entity Recognition of lab protocol (2020)

Abstract: Supervised models trained to predict properties from representations have been achieving high accuracy on a variety of tasks. For in-stance, the BERT family seems to work exceptionally well on the downstream task from NER tagging to the range of other linguistictasks. But the vocabulary used in the medical field contains a lot of different tokens used only in the medical industry such as the name of different diseases, devices, organisms,medicines, etc. that makes it difficult for traditional BERT model to create contextualized embedding. In this paper, we are going to illustrate the System for Named Entity Tagging based on Bio-Bert. Experimental results show that our model gives substantial improvements over the baseline and stood the fourth runner up in terms of F1 score, and first runner up in terms of Recall with just 2.21 F1 score behind the best one.

## introduction a large amount of data is generated every year in the medical field. one of the most important generated data is the documentation of protocols. it provides individual sets of instructions that allow scientists to recreate experiments in their own laboratory. most of them are written in natural language which reduces its machine readability. the protocol gives a concise overview of the project which reduces its pre-processing needs but also make it less informative syntactically that eventually results in less accuracy. recent progress in named entity recognition was made possible by the advancements of deep learning techniques used in natural language processing (nlp). for instance, long short-term memory (lstm) @xcite and conditional random field (crf) 1 https://github.com/tejasvaidhyadev/ ner_lab_protocols @xcite have greatly improved performance in biomedical named entity recognition (ner) over the last few years. bio-bert @xcite outperform all the other previous approaches with the help of bert @xcite architecture pre-trained on bio-medical texts @xcite @xcite . in this paper, we are introducing our system for the ner tagging on the wlp dataset @xcite . we use a variant of bio-bert @xcite . the primary motivation to use the model is it's medical vocabulary and features encoded in the pre-trained model. ## task description and data set formally, the wnut 2020 shared task-1 named entity recognition, organized within, the 6th workshop on noisy user-generated text (wnut), 2020 @xcite ) is a ner prediction task. it can be expressed as 'tokens-level' classification task mathematically as: let the sentence s be defined as: @xmath0, s 2 , ..., s n } n is the number words in the sentences can be classified into the following label set @xmath1, l 2 , l 3 , ..., l m } where m is labels. given named entity of type xxx. whenever two entities of type xxx are immediately next to each other, the first word of the second entity will be tagged b-xxx in order to show that it starts another entity and the entities inside b-xxx will be represented as i-xxx. for example, the sentence {ncov-2019, sequencing, protocol} have the following labels {b-reagent, b-method, i-method}. dataset: all of the protocols @xcite were collected from protocols.io using their public apis by organising team. for the shared task-1 w-nut 2020: named entity extraction, the annotation of 615 protocols are re-annotated using bart styled annotated protocols by 3 annotators with 0.75 inter-annotator agreement, measured by span-level cohen's kappa. the re-annotators incorporate missing entity-relations and also corrected the inconsistencies. the task aims to create the system for protocol-named entities recognition (ner). the main difference that makes it difficult for traditional ner taggers is the vast vocabulary in medical filed and use of limited syntactic information. for instances "qiaprep spin miniprep" is device used in medical industry, but not present in our regular vocabulary that also makes it difficult for traditional ner tagger to learn. figure 1 provides the visualization of annotated datasets provided by wnut 2020 shared task-1 . ## approach in the section 3.1 we are going to define our proposed architecture and in section 3.2 briefly review the bio-bert @xcite used for final submission and also different domain specific bert based model used for experiment as shown in the baseline the organiser provided a simple linear crf model foot_1 . it utilized simple gazetteers and handcrafted feature to predict the entities from the test data. we replaced it with our proposed bert based architecture as describe in the section below. ## comparison and discussion we experimented with different bert models as shown in we also assumed protocols are relatively less noisy compare to other crowd source data with compact sentences. ## conclusion and future work in this paper, we present our system for the named entity recognition for bio-medical protocols a for shared task at w-nut task-1 2020. we built upon the recent success of pre-trained language models and apply them for protocols. our system achieves close to state-of-art performance on this task. as future work, we will try to experiment with xlnet @xcite and different ensembling between the models and would like to extend the work of @xcite by performing layer by layer analysis of bert.