B i T e M at WNUT 2020 Shared Task-1: Named Entity Recognition over Wet Lab Protocols using an Ensemble of Contextual Language Models (2020)

Abstract: Recent improvements in machine-reading technologies attracted much attention to automation problems and their possibilities. In this context, WNUT 2020 introduces a Name Entity Recognition (NER) task based on wet laboratory procedures. In this paper, we present a 3-step method based on deep neural language models that reported the best overall exact match F1-score (77.99%) of the competition. By fine-tuning 10 times, 10 different pretrained language models, this work shows the advantage of having more models in an ensemble based on a majority of votes strategy. On top of that, having 100 different models allowed us to analyse the combinations of ensemble that demonstrated the impact of having multiple pretrained models versus fine-tuning a pretrained model multiple times.

## introduction the last decades have seen both the amount and the complexity of biological experiments grow. coupling this phenomenon with the improvement in machine-reading technologies seem to have led researchers to look for ways to automate wet laboratory procedures. such technologies should allow reproducibility while reducing human errors in the process. however, as current protocols are usually written in a natural language, a collection of wet laboratory protocols annotated with entities and relations would help assess current machine-reading performances in this specific setting @xcite . in this context, wnut (workshop on noisy user-generated text foot_0 ) 2020 @xcite proposes two tasks, a named entity recognition (ner) task and a relation extraction (re) task. in this paper, we present a 3-step method we used for the ner task. our approach is essentially based on a deep neural language models supported by transformer-like architectures @xcite . first, we fine-tuned 10 different pretrained language models on the downstream task. then, we generated 10 instances of those pretrained models, each time with a new random initialization of the last layer, namely the classifier. finally, we used an ensemble strategy based on a majority of votes. our approach achieves the exact-match f 1 -score of 77.99% that ranks first in the shared task. ## related work deep learning approaches trained on large unstructured data have shown considerable success in nlp problems, including ner @xcite @xcite @xcite . these models use the learned representations over the large data and reuse them in a supervised setting for a downstream task. for domain-specific tasks, the models that are trained on large general text can be further trained on domain specific large data and then adapted for a downstream task @xcite @xcite or the models can be trained only on domain-specific data and then adapted for a specific task @xcite . ## data the data provided for this task is a subset of kulkarni et al.'s corpus @xcite . the dataset consists of 615 unique protocols annotated with 17 types of entities and action (an example is shown in figure 1 ). the organizers provided a set of protocols for training, development, and test. they further released a final set of unlabelled protocols for test during the competition (called test 2020). in ## method our models essentially focused on transformerslike @xcite language models that we fine-tuned on the ner task by adding a fully connected layer on top of the token representations. the models include bert (cased) @xcite , biobert (bert trained on pubmed abstracts and pmc full-text articles) @xcite , bio+clinicalbert (biobert trained on notes in the mimic-iii v1.4 database) @xcite , pubmedbert @xcite , roberta @xcite , biomed roberta @xcite , and xlnet @xcite . our method has been driven in 3 steps. first, we chose 10 different pretrained models and fine-tuned them on the downstream task. then, using a voting strategy, we created ensemble models. finally, we fine-tuned 9 more times each model, each time with a new random initialization of the fully connected layer, to see if sampling ensemble models from this set of models would improve the results even more. ## results and discussions in however, when looking at speed, it seems that the transformers-based models we used are not able to do a better job than the baseline. a closer look at the errors should be done in order to see what caused such a difference with the baseline (see section 5.4). when comparing the micro to the macro f 1score standard deviations across all the models, we can see that the macro f 1 -score standard deviations are systematically higher. this is probably due to the fact that some entities, namely generic-measure, mention, seal, size and ph, which account for less than 1% of the test set each (see ## conclusion with almost no preprocessing, we have seen that current pretrained language models seem to be quite efficient in any ner task @xcite . by analysing our voting strategy, we have also demonstrated the strengths as well as the weaknesses of such ensemble models. for instance, it looks like the more models we use, the more the performances tend to be high and stable, however, it appears that new pretrained model brings more information than fine-tuning again a pretrained model with a new fully connected weights random initialization. with this voting strategy, our submission achieved the best exact match overall f 1 -score of the competition. this clearly shows the power of such models. with almost no knowledge on the topic of wet laboratory protocols required, we think that those models open opportunity to out-of-field researchers. in future work, it would be interesting to improve the number of pretrained models selection and explore bootstrapping instead of fine-tuning multiple times the same pretrained model. it would also be interesting to see if some preprocessing tweaks could help us to improve the detection performance of speed where our models were outperformed by the baseline.