Contextualized F rench Language Models for Biomedical Named Entity Recognition (2020)

Abstract: Named entity recognition (NER) is key for biomedical applications as it allows knowledge discovery in free text data. As entities are semantic phrases, their meaning is conditioned to the context to avoid ambiguity. In this work, we explore contextualized language models for NER in French biomedical text as part of the Défi Fouille de Textes challenge. Our best approach achieved an F1 -measure of 66% for symptoms and signs, and pathology categories, being top 1 for subtask 1. For anatomy, dose, exam, mode, moment, substance, treatment, and value categories, it achieved an F1 -measure of 75% (subtask 2). If considered all categories, our model achieved the best result in the challenge, with an F1 -measure of 72%. The use of an ensemble of neural language models proved to be very effective, improving a CRF baseline by up to 28% and a single specialised language model by 4%.

## introduction the large amount of raw text data available in the biomedical domain enables to leverage the wealth of the content. combined with manually curated data, it allows the development of automatic techniques to unlock the value of the raw resources for supporting healthcare and advance science. in particular, information extraction methods enable the extraction of specific data types from text data (e.g., entities). information extraction fosters several applications from tracking of technologies @xcite in patents to clinical decision support @xcite , biocuration assistance @xcite , and healthcare-associated infections detection @xcite in the biomedical domain. it has also important challenges associated with the application domain and the language in which the text is available. indeed, information extraction systems for recognizing entities are mostly focused on english. however, it is widely recognised that it is crucial that research expands to other languages in the same scale @xcite . named entity recognition (ner) is a key part in information extraction systems. named entities are phrases that contain names of persons, organizations, locations @xcite to name but a few examples. there are many studies for ner in french language, for instance in i) journalistic data @xcite with a set of entities like person, organization, company, location, point of interest, fiction character and product ; and ii) recognizing entities in tweets @xcite , including person, music artist, organisation, product and media, among others. in the biomedical domain, recognizing entities is mainly focused on semantic groups and concepts from unified medical language system (umls) on the quaero french medical corpus @xcite . the quaero corpus contains annotated french medline (titles and abstracts) and european medicines agency (emea) documents (drug labels). community challenges, such as clef ehealth, have been evaluating specific information extraction tasks for the clinical domain @xcite . erasmus mc, one of the clef ehealth top scorers, is a dictionary-based ner for french umls and translations for non-french terms, achieving 74.9% of f 1 -measure for emea and 69.8% of f 1 -measure for medline corpus in semantic groups annotation @xcite . in erasmus mc, semantics could be missed by the presence of compounded semantic groups or umls concepts. similarly, sifr annotator is a semantic annotator for french clinical narratives @xcite . it relies on mgrep, a concept recognizer based on label matching and heuristics, and achieves 62.6% of f 1 -measure in emea and 58.9% of f 1 -measure in medline for semantic groups. the tool is limited due to the lack of word disambiguator and scarce french ontologies concerning english ontologies. deep neural language models have been recently leveraged to improve ner methods @xcite . deep neural language models are self-supervised models that take advantage of free text to learn word representations using their context @xcite . with the advent of lowdimensional representation models supported by deep neural networks, such as word2vec @xcite , the importance of word representations has become more evident. further research has been taken to find out more accurate representations of words, such as in global vectors (glove) @xcite , and to the more recent contextualized representations, like elmo @xcite , @xcite , and bert @xcite . in particular, bert is based on the transformers architecture, which uses an attention mechanism, via bidirectional pre-training from unlabeled text, conditioned in left and right contexts in all layers @xcite . biobert @xcite , a bert-based model trained on large-scale biomedical corpora has shown significant improvements in downstream tasks in the biomedical domain, including ner. similarly, camembert 1 is a contextualized language model trained and optimized specifically for french language @xcite based on roberta model @xcite . in this paper, we investigate contextualized language models for french ner in clinical texts in the context of the information extraction task of the défi fouille de textes (deft) challenge @xcite . this task is divided in two subtasks, which aim to identify anatomie (anatomy), dose, examen, mode, moment, pathologie (pathology), sosy (symptoms and signs), substance, traitement and valeur (value) entities in clinical narratives. inasmuch as each language has its own peculiarities, our hypothesis is that it is worth designing a specific language model for french clinical corpora. thus, we explore a camembert-based model pre-trained on a biomedical corpus and fine-tuned on the deft information extraction task data. we compare its performance with multilingual bert, camembert and an ensemble of language models. in the following sections, we describe the design and results of the experiments. ## methods in this work, we explore two perspectives for ner : as information extraction and as word representation. for the first, named entities are considered as a sequence classification problem, for which we propose a baseline method using the conditional random fields (crf) framework. for the latter, our methodology is based on different deep neural language models derived from the bert architecture. these methods were used to extract named entities in subtask 1 -symptoms and signs, and pathology -and subtask 2 -anatomy, dose, exam, mode, moment, substance, treatment, and value of the deft information extraction task. ## results and discussion in this section, we present the results of our models for the training and test phases for both subtasks. during the training phase, we used only the training collection provided in the challenge in order to develop and tune our models. during the test phase, we evaluated over the test collection with the parameters identified in the training phase. ## conclusion among the experiments we have done, we can conclude that recognizing entities in biomedical domain is not a straightforward task and adding the complexity of language makes this task more difficult. for deft task 3 challenge, we proposed mainly two families of learning methods: a baseline based on crf and a set of transformers language models. we focused on exploring the performance of our ner models with the contextualized language models enriched with local text. our best results were given by the ensemble method based on a voting strategy between the bert based models, including camembert (trained specifically for french) and camembert-bio (trained on french biomedical texts), achieving 66% f 1 -measure in subtask 1 and 75% f 1 -measure in subtask 2. the ensemble of deep neural language models proved to be the most effective method for biomedical information extraction in french texts. as next steps, we will investigate whether fine-tuning the same model a number of times would improve performance. we are also interested to investigate nested entities approaches over deft task 3 data.