B io BERT pt - A P ortuguese Neural Language Model for Clinical Named Entity Recognition (2020)

Abstract: With the growing number of electronic health record data, clinical NLP tasks have become increasingly relevant to unlock valuable information from unstructured clinical text. Although the performance of downstream NLP tasks, such as named-entity recognition (NER), in English corpus has recently improved by contextualised language models, less research is available for clinical texts in low resource languages. Our goal is to assess a deep contextual embedding model for Portuguese, so called BioBERTpt, to support clinical and biomedical NER. We transfer learned information encoded in a multilingual-BERT model to a corpora of clinical narratives and biomedical-scientific papers in Brazilian Portuguese. To evaluate the performance of BioBERTpt, we ran NER experiments on two annotated corpora containing clinical narratives and compared the results with existing BERT models. Our in-domain model outperformed the baseline model in F1-score by 2.72%, achieving higher performance in 11 out of 13 assessed entities. We demonstrate that enriching contextual embedding models with domain literature can play an important role in improving performance for specific NLP tasks. The transfer learning process enhanced the Portuguese biomedical NER model by reducing the necessity of labeled data and the demand for retraining a whole new model.

## introduction despite recent increases in the availability of machine learning methods, extracting structured information from large amounts of unstructured and noisy clinical documents, as available in electronic health record (ehr) systems, is still a challenging task. patient's ehr are filled with clinical concepts, often misspelled, abbreviated and represented by a variety of synonyms. nevertheless, they contain valuable and detailed patient information @xcite . natural language processing (nlp) tasks, such as named entity recognition (ner), are used for acquiring knowledge from unstructured texts, by recognizing meaningful entities in text passages. in the clinical domain, ner can be used to identify clinical concepts, such as diseases, signs, procedures and drugs, supporting other data analysis as prediction of future clinical events, summarization, and relation extraction between entities (e.g., drug-to-drug interaction). rule-based ner approaches, supported by dictionary resources, perform well in simple contexts @xcite . however, they are limited to work with the complexity of clinical texts. for complex corpora, machine learning approaches, such as conditional random fields (crf) @xcite and, lately, a combination with bidirectional long short-term memory (bil-stm) models, have been proposed @xcite . these supervised approaches have a considerable performance gain when trained on huge amounts of labeled data. neural network language models introduced the idea of deep learning into language modeling by learning a distributed representation of words. these distributed word representations, trained on massive amounts of unannotated textual data, have been proved to provide good lower dimension feature representations in a wide range of nlp tasks @xcite . the continuous bag-of-words and skip-gram models proposed to reduce the computational complexity were considered as a milestone in the development of the so-called word embeddings @xcite , followed by the global vector (glove) @xcite and the fasttext @xcite models. while these approaches work with a single global representation for each word, several context-dependent representations models have been recently proposed, such as embeddings from language models (elmo) @xcite , flair embeddings @xcite , the universal language model fine-tuning (ulmfit) @xcite and bidirectional encoder representations from transformers (bert) @xcite . contextual embedding models pretrained on large-scale unlabelled corpora, particularly those supported by the transformer architecture @xcite , reached the state-ofthe-art performance on many nlp tasks @xcite . nevertheless, when applying the general word representation models in healthcare text mining, the characteristics of clinical texts are not considered, known to be noisy, with a different vocabulary, expressions, and word distribution @xcite . therefore, contextual word embedding models, like bert, can be fine-tuned, i.e., have their last layers updated to adapt to a specific domain, like clinical and biomedical, using domain-specific training data. these transfer learning process allows the training of a general domain model with medical domain corpus, proving to be a viable technique to medical nlp tasks @xcite . despite the low availability of clinical narratives, given the sensitive nature of health data and privacy concerns @xcite , several models were trained on clinical and biomedical corpora. in 2013, the word2vec model was trained on biomedical corpora @xcite , creating a language model with high-quality vector space representations. biobert @xcite is a bert model trained from scratch using pubmed and pubmed central (pmc) scientific texts, reaching the state-of-the-art results on some biomedical nlp tasks. clinical bert @xcite demonstrated that the pre-trained model with clinical data improved performance in three common clinical nlp tasks. @xcite reached stateof-the-art for biomedical and clinical entity normalization with a model trained using ehr data. despite the essential contributions of contextual word embeddings on clinical ner, all these studies used english corpora. indeed, there are few studies in lower resources languages for the clinical domain. in portuguese, @xcite proposed a fasttext model trained with clinical texts, which achieved higher results when compared to out-ofdomain embeddings. in a recent work, @xcite explored the crf algorithm for the ner task on semclinbr @xcite , the same annotated corpus we used in this work. they classified three clinical entities (disorders, procedures and chemicals and drugs) and some medical text abbreviations, achieving promising results. a portuguese clinical word embedding model were trained using skip-gram with negative sampling and evaluated on a downstream biomedical nlp task for urinary tract infection disease identification @xcite . their results showed that larger, coarse-grained models achieve a slightly better outcome when compared with small, finegrained models in the proposed task. although these previous works achieved relevant results, we have not found studies for clinical portuguese using attention-based architectures, such as bert, which have been achieving the state-ofthe-art for most of english nlp tasks. even with the existence of multilingual models, like bertmultilingual, it is important to investigate what can be the contribution in creating a domain finetuned model for a lower-resource language. as demonstrated in the work of @xcite , pre-trained bert models with biomedical and clinical data achieves better results in the blue benchmark for english. this leads us to believe that the same is valid for portuguese. thus, the objective of this work is to assess the performance of a domain specific attention-based model, biobertpt, to support ner tasks in portuguese clinical narratives. we intend to investigate how an in-domain model can influence the performance of bert-based models for ner in clinical data. also, as knowledge encoded in transformer-based language models can be leveraged to several downstream nlp tasks, we release publicly the first bert-based model trained on clinical data for portuguese foot_0 . ## methods in this section, we first describe how biobertpt was developed using clinical notes and scientific abstracts. next, we introduce the corpora used for the ner tasks and the evaluation metrics used in our experiments. ## results in the semclinbr corpus, biobertpt(bio) improved 0.1 in precision, biobertpt(all), 2.0 in recall and 1.6 in f1-score, over the out-of-domain model with better performance. full f1-score values for each entity are provided on our repository. analyzing the performance by entity, the in-domain models in general were better at recall and f1-score. our models obtained better results in precision for 4 entities, recall for 8 and f1-score for 11. the out-of-domain models obtained better results for 9 entities in precision, 5 in recall and 2 in f1-score. the results of the friedman test evidenced that there is a difference between some models. the post-test wilcoxon signed-ranks pair-wise showed the statistical relevance between models over all entities, as shown in figure 2 . the results on the clinpt corpus, also presented in ## conclusions and future work we proposed a new publicly available portuguese bert-based model to support clinical and biomedical nlp tasks. our ner experiments showed that, compared to out-of-domain contextual word embeddings, biobertpt reaches the state-of-the-art on the clinpt corpus. additionally, it has better performance for most entities analyzed on the sem-clinbr corpus. our preliminary results are aligned with previous results in other languages, evidencing that domain transfer learning can benefit clinical tasks, in a statistically significant way. in the future, we would like to explore larger transformersbased models in the clinical portuguese domain and evaluate our model in different clinical nlp tasks, such as negation detection, summarization and de-identification.