Clinical Flair: A Pre-Trained Language Model for S panish Clinical Natural Language Processing (2022)

Abstract: Word embeddings have been widely used in Natural Language Processing (NLP) tasks. Although these representations can capture the semantic information of words, they cannot learn the sequence-level semantics. This problem can be handled using contextual word embeddings derived from pre-trained language models, which have contributed to significant improvements in several NLP tasks. Further improvements are achieved when pre-training these models on domain-specific corpora. In this paper, we introduce Clinical Flair, a domain-specific language model trained on Spanish clinical narratives. To validate the quality of the contextual representations retrieved from our model, we tested them on four named entity recognition datasets belonging to the clinical and biomedical domains. Our experiments confirm that incorporating domain-specific embeddings into classical sequence labeling architectures improves model performance dramatically compared to general-domain embeddings, demonstrating the importance of having these resources available.

## introduction word embeddings are dense, semantically meaningful vector representations of a word. this method has proven to be a fundamental building block when constructing neural network-based architectures. however, the main drawback of using these embeddings is that they provide only a single representation of a given word across many documents. this is not optimal in practice, as the representation depends on the sentence in which the word appears. contextual word embeddings address this problem by capturing syntactic and semantic information at the sentence level to represent words according to their context. contextualized embeddings are commonly retrieved from language models trained on giant text corpora. these models are usually composed of sequential or attention neural networks, which allows obtaining sentence-level semantics. this method has contributed to major advances in several nlp tasks such as named entity recognition, text classification, and relation extraction. classic examples of contextual representation models are flair @xcite , elmo @xcite , and bert @xcite . regarding specific domains such as clinical and biomedical, there are widely used models for the english language, such as biobert @xcite , bioelmo @xcite , and the pubmed version of flair. these studies have shown that incorporating domain-specific contextual word embeddings contributes to a significant improvement in the performance of the models. however, although unstructured clinical texts are abundant in spanish, there is still a significant lack of language models. most of the domain-specific contextual representation models available for spanish focus on data obtained from scientific articles and not from texts written in a more realistic context. to fill this gap, we trained and publicly released clinical flair foot_0 , a character-level language model trained on a corpus with real diagnoses in spanish. to measure the potential impact of using these representations, we provide an empirical study of the effects of using language models trained on domain-specific against general-domain corpora. we evaluated the effectiveness of the proposed embeddings on four named entity recognition datasets belonging to the clinical and biomedical domain in spanish. the results suggest that the embeddings obtained from our model contribute to achieving a better model performance compared to the generaldomain contextualized embeddings by a wide margin. ## related work language models allow us to generate high-quality representations of words based on their surrounding context, better known as contextual word embeddings. these models are usually trained with large corpora, either general-domain or domainspecific. most of the available models have been trained with english resources, where the most popular ones are bert @xcite , elmo @xcite , gpt-2 @xcite , and flair @xcite . as pointed out in @xcite , building domain-specific language models allows to improve models performance compared to generaldomain language models. in relation to biomedical information retrieval (ir) tasks in english, the most well-known architectures are biobert @xcite , clinical bert @xcite , scibert @xcite , pubmed bert @xcite , bioelmo @xcite and pubmed flair. regarding the clinical domain in spanish, we found the models biomedical roberta @xcite and scielo flair @xcite . in the first case, the main difference with our model is that biomedical roberta was trained on a corpus formed by several biomedical and clinical corpora, while we only used clinical narratives. in the case of scielo flair, a point of differentiation is that they used data obtained from medical publications, whereas our data comes from primary care diagnoses. moreover, they only tested their model on the pharmaconer corpus, created from the same data source they trained scielo flair. in contrast, we tested the effectiveness of our model using four clinical and biomedical datasets. ## methods this section describes the clinical dataset used to train our language model, the details of the training process, and, finally, the task and datasets used in our experiments. ## results on the other hand, although we did not train our model on biomedical corpora, we observe that it is also beneficial for solving ner on those datasets. although we did not outperform the scielo flair model in pharmaconer, we obtained competitive results. however, as mentioned in their paper, they selected a subset of scielo texts to train the language model in line with the pharmaconer corpus. therefore, we expected that their results would be superior. compared with spanish flair, the major difference occurs in cantemist, reaching an average 6 https://github.com/plncmm/ clinical-flair difference of +0.028, while the slightest difference is observed in nubes with +0.007 according to the f1 measure. one possible reason for the similar performance between our model and spanish flair in nubes is that, although the dataset belongs to the biomedical domain, the task aims to identify entities associated with negations and uncertainties; therefore, the target labels are general-domain and distant from the original corpus on which we trained our model. finally, and as expected, in both corpora belonging to the clinical domain cantemist and clinical trials, our model outperforms both spanish flair and scielo flair. in the case of clinical trials, we reached an average difference of +0.023 and +0.012 compared to both models, respectively, while in the case of cantemist, we obtained improvements of +0.028 and +0.005 according to the f1 measure. ## conclusions and future work despite the growing interest of the nlp research community in contextualized embeddings, there is still a lack of language models for the spanish language, a gap that increases even more concerning domain-specific texts. to address this issue, this paper introduced clinical flair, a characterlevel language model for clinical nlp in spanish. specifically, we used a general-domain language model as a starting point and then fine-tuned it on chilean clinical narratives. our experimental results on four clinical and biomedical ner datasets show that incorporating our domain-specific embeddings outperforms by a wide margin the results obtained with general-domain embeddings, demonstrating the importance of having these resources available for languages not as widely explored. future work includes extending our study to other nlp tasks and using different combinations of embeddings, such as concatenating word2vec or character-level embeddings. in addition, to provide a variety of contextual representation models for clinical texts, we are training a clinical version of bert in spanish. although preliminary results have been inferior to those obtained with our clinical flair model, we expect to collect a larger clinical corpus to improve performance.