A Comparison between Named Entity Recognition Models in the Biomedical Domain (2021)

Abstract: The domain-specialised application of Named Entity Recognition (NER) is known as Biomedical NER (BioNER), which aims to identify and classify biomedical concepts that are of interest to researchers, such as genes, proteins, chemical compounds, drugs, mutations, diseases, and so on. The BioNER task is very similar to general NER but recognising Biomedical Named Entities (BNEs) is more challenging than recognising proper names from newspapers due to the characteristics of biomedical nomenclature. In order to address the challenges posed by BioNER, seven machine learning models were implemented comparing a transfer learning approach based on fine-tuned BERT with Bi-LSTM based neural models and a CRF model used as baseline. Precision, Recall and F1-score were used as performance scores evaluating the models on two well-known biomedical corpora: JNLPBA and BIOCREATIVE IV (BC-IV). Strict and partial matching were considered as evaluation criteria. The reported results show that a transfer learning approach based on fine-tuned BERT outperforms all others methods achieving the highest scores for all metrics on both corpora.

## introduction named entity recognition (ner) is a task that aims to recognise and classify mentions of named entities in unstructured text into pre-defined semantic categories such as person, organisation, location, time expression, monetary value, and so on. in natural language processing (nlp), ner not only acts as a tool for information extraction (ie), but plays an essential role in a variety of downstream applications such as information retrieval @xcite , text summarisation @xcite , machine translation @xcite , question answering @xcite , and many other nlp tasks. the interest in ner is not a novelty, but it has been increasing in recent years due to the exponential growth of digital information that stimulated domainspecific applications of ner in order to extract entity mentions not only from general texts, such as newspaper articles, but also from specialised texts. in many applied research domains, ner is directly used to alleviate the problem of the search and discovery of information, becoming an invaluable tool particularly in those research areas where it is difficult for researchers to keep up with relevant publications @xcite . one specialised application of ner is known as biomedical named entity recognition (bioner), which is defined as the task of identifying and classifying biomedical named entities (bnes), technical terms referring to key concepts that are of interest to biomedical researchers, such as gene, protein, chemical compound, drug, mutation, disease, and so on. bioner has gained increasing attention from the research community. in fact, many works in medicine focus on the analysis of scientific articles to find out hidden relationships between bnes, such as gene and protein, in order to drive experimental research @xcite . although a large body of systems are dedicated to extract bnes in scientific literature, bioner tools can be applied to find all kinds of entities in any kind of health related text, including radiology reports and clinical notes @xcite . generally, bioner is considered a more challenging task compared to domainindependent ner due to the characteristics of biomedical nomenclature. the lack of standardised naming conventions, the frequent crossover in vocabulary, the excessive use of abbreviations, synonyms and variations, the morphological complexity due to the use of unusual characters such as greek letters, digits, punctuation -these are just some of the factors making the recognition of bnes particularly difficult for bioner systems. moreover, biomedical text often contains complex multi-word bnes and, especially in the area of gene and protein names, multi-word bnes are rather the rule than the exception. not only multi-word bnes are more difficult to identify, but in many cases there is also no agreement on the exact borders of such names, making the evaluation of bioner tools complex @xcite . for example, many bnes may contain verbs and adjectives that are embedded in names, making a legitimate gene or protein name hard to distinguish from the general language text surrounding it. lastly, the biomedical domain is an expanding field where new concepts emerge daily and new names are coined on a daily basis. in addition, new variants are always created for already existing concepts since biomedical concepts are studied in different branches of medicine which use different naming conventions. to address these challenges, seven machine learning (ml) models were implemented following a sequence tagging (st) approach 4 . a transfer learning approach based on fine-tuned bert is compared to bi-lstm-based neural models and a crf model used as baseline. the impact of pre-trained word embedding models on the performances of neural models is also investigated. the comparison between models is carried out by evaluating the performances on two well-known bioner corpora. the rest of the paper is structured as follows. section 2 presents the data used in this study. section 3 outlines the models employed in our experiments, which are described in section 4. section 5 discusses the results obtained and finally section 6 summarises the conclusions of this study. ## data bioner models were evaluated using two benchmark corpora released during well-known and popular shared competitions. the first one is the corpus of the jnlpba 2004 shared task, which is derived from the popular genia corpus. the second one is the biocreative (bc-iv) corpus used for the track 2 of biocreative iv shared task. both corpora were made publicly available in the iob2 annotation format. 5 according to this schema, tokens are labelled with a b-class tag at the beginning of every sequence that represents an entity, with an i-class tag if the tokens are inside a sequence and with an o tag if the tokens are outside of a sequence that represents an entity. ## models considering bioner as a sequence tagging (st) task, seven models were implemented in order to solve bioner and compare performances of a traditional ml algorithm used as a baseline with the latest advanced neural models. ## experiments the first experiment is aimed at training a crf model, a traditional ml method widely used for solving bioner that is easy to implement, provides reasonable results, and does not require much expertise and time to build. the crf performance is considered as a baseline for evaluating the performance of the other models. for the crf model, a specific set of features is used to allow the algorithm to recognise bnes. linguistic features are selected exclusively to exploit the characteristics of biomedical bnes such as the morphological complexity. to enable the model to capture contextual information, context features are also provided in a 5-word window. the first neural model implemented is a bi-lstm-based architecture that uses softmax layer as the decoding layer. the embedding layer is initialised with random weights and computes word vectors during the learning process. learned representations of data are fed into a bi-lstm layer which extracts contextual information. the output is then passed to another bi-lstm layer so that the model learns even deeper, more abstract representations from data. the decoding layer uses a softmax function to transform scores into a probability distribution over classes. labels for each word are independently predicted without taking into account dependencies between labels. in a second model, softmax was replaced with crf to make the model capable of capturing relationships between entity labels. it has been shown indeed that for st tasks it is more beneficial to jointly decode label sequences using crf than decoding each label independently @xcite . in the next models, the bi-lstm+crf architecture was enhanced replacing randomly initialised word vectors with different pre-trained distributed representation models. first, bi-lstm+crf was combined with pre-trained vectors from fasttext. 6 next, these vectors were replaced with a concatenation of word-level and character-level representations using pre-trained word embedding from glove 7 and character embedding learned using an lstm model. character representations are able to capture sub-word level information such as prefix, suffix and orthographic characteristics enabling the model to handle the out-of-vocabulary (oov) problem, which causes glove to return many zero values. the last bi-lstm based neural model uses contextual embedding incorporating into the embedding layer a pre-trained elmo @xcite model. this model does not consider an additional character-level embedding, unlike the model with glove, since elmo already provides context-dependent character-level representations. finally, a fine-tuned bert-large model is employed. in the pre-processing step, a wordpiece tokenizer is used in order to allow the model to process words that it has never seen before by decomposing them into known sub-words. for restoring the original tokenisation, a post-processing step is needed in order to compare bert outputs with those of the other models. the hyper-parameter settings and all the fine-tuning procedure rely on the indication provided on the original paper by devlin et al. @xcite . due to the high number of parameters, the model is trained on an nvidia tesla k80 16gb gpu. ## results the results obtained on the test set for both corpora are shown in tables 1 and 2 . precision, recall and f1-score are reported according to the strict matching criterion, and the overall scores for jnlpba are computed using micro-average. the f1-score computed according to the partial matching criterion is also reported. for what concerns the neural models, each model is run five times and the final reported result is the average among the runs. ## conclusion seven machine learning (ml) models were implemented following a sequence tagging (st) approach for solving bioner on two well-known corpora. a transfer learning approach based on fine-tuned bert was compared with bi-lstmbased neural models and a crf model used as baseline. the fine-tuned bert model achieved the highest scores for all metrics on both corpora. thus, according to what emerged from these experiments, the use of pre-trained vectors has a significant impact on the performance of the bi-lstm models, leading to an oov error reduction and an increase of the recall. in addition, the inclusion of sub-word level information into the models proved to be particularly beneficial for solving bioner on both corpora. based on these results, the use of pre-trained transformer-based neural models such as bert for solving bioner looks promising. specifically, the advantage of using bert for bioner lies in the fact that it can be employed as a ready-to-use model that can be easily finetuned for solving the task, requiring significantly less time to train and achieving superior performance scores compared to other approaches.