Development of pre-trained language models for clinical NLP in S panish (2023)

Abstract: Clinical natural language processing aims to tackle language and prediction tasks using text from medical practice, such as clinical notes, prescriptions, and discharge summaries. Several approaches have been tried to deal with these tasks. Since 2017, pre-trained language models (PLMs) have achieved state-of-the-art performance in many tasks. However, most works have been developed in English. This PhD research proposal addresses the development of PLMs for clinical NLP in Spanish. To carry out this study, we will build a clinical corpus big enough to implement a functional PLM. We will test several PLM architectures and evaluate them with language and prediction tasks. The novelty of this work lies in the use of only clinical text, while previous clinical PLMs have used a mix of general, biomedical, and clinical text.

## introduction clinical text is one of the richest forms of information in electronic health records. therefore, developing tools to extract useful information from clinical text has become relevant in clinical natural language processing (nlp). however, processing unstructured text is challenging due to the complexity of human languages. moreover, the clinical text has its own complexities, including non-standard abbreviations, misspellings, specific vocabulary, and jargon @xcite . clinical nlp aims to address several tasks in this complex scenario. these tasks can range from language tasks such as extracting entities, text classification, and relation extraction, among others, to prediction tasks such as predicting patient mortality, length of hospital stay, unplanned readmissions, etc. several works have been carried out to tackle these tasks generating specific models. however, since 2017, the nlp field has worked towards the creation of pre-trained language models (plms) that can be fine-tuned for any specific downstream task. these language models are built for a much simpler task, such as next-word or masked-word prediction in a huge amount of text. this process, known as pre-training, allows the language model to acquire language understanding that can be used for any text-related task @xcite . as soon as the nlp field started to work in plms, clinical nlp introduced this type of model into its set of techniques to improve performance in its own tasks. some examples of clinical plms are two different versions of clinicalbert @xcite . these models show a significant improvement in language tasks and a moderate improvement in prediction tasks. most of the research in clinical nlp has been done for text written in english, but not so much for other languages @xcite . in spanish, some publicly available plms relevant to clinical nlp are bsc-bio-ehr-es @xcite and spanish clinical flair @xcite . these plms were pre-trained heavily in general and biomedical text with some additions of clinical text. despite this drawback, they outperform general and biomedical plms in language tasks. there are two approaches to evaluate plms, intrinsic and extrinsic. an intrinsic approach measures the model quality independently of an application in any specific task. examples of intrinsic metrics are perplexity and word similarity. meanwhile, an extrinsic approach evaluates the model performance in downstream tasks, such as text classification or named entity recognition (ner). extrinsic evaluations are more expensive and timeconsuming than intrinsic evaluations @xcite . in biomedical nlp, some relevant benchmarks have been developed for extrinsic evaluations, such as blue @xcite and blurb @xcite . given their close relation to the clinical context, they are also used to evaluate clinical plms. however, these benchmarks are built with tasks in english. there are no such benchmarks in spanish, thus the research community evaluates their models in downstream tasks for biomedical and clinical corpora. most of these downstream tasks are language tasks, and few works focus on prediction tasks. in this research proposal, we aim to develop clinical plms in spanish using mostly clinical text and evaluate them with intrinsic and extrinsic approaches. the expected results are as follows: this research proposal shows a literature review of plms architectures for general, biomedical, and clinical domains. then, it states the research questions and their implications. later, it describes the methodology to carry out this research and details the expected results. ## literature review since the creation of transformers @xcite and ulmfit @xcite , we have witnessed the development of several plms that have become state-of-the-art on different tasks. starting with gpt @xcite and bert @xcite , continuing with roberta @xcite , and deberta @xcite , among others, the research community has focused on finding new architectures that can beat current benchmarks and apply them in many areas as possible. one of the areas where it is possible to find several plms is health. just a few months after the publication of bert, plms using clinical text were released @xcite . moreover, even though plms are built mostly for english text, it is still possible to find clinical and biomedical plms for spanish @xcite . this section introduces plms in the clinical and biomedical fields and describes significant clinical and biomedical plms in english and spanish. also, it shows some evaluation benchmarks and tasks for biomedical and clinical nlp in english and spanish. ## research questions from the literature review, it is possible to identify some research opportunities. these opportunities are mostly related to which data we can use to pre-train plms and which architecture best fits a specific task. also, the evaluation process should be clear and easy for comparison, which in the case of clinical nlp in spanish, there are still missing parts. additionally, there is an opportunity to advance clinical nlp in spanish by adopting strategies used in english and joining current efforts in the field. from the identified opportunities, we can state the research questions as follows: to answer question one, we have to build a clinical corpus in spanish big enough to implement a functional plm. however, building a clinical corpus is a difficult task since there are privacy and confidentiality issues that may arise. the plm trained over this corpus will be compared to biomedical and clinical plms for spanish described in the literature review. to answer question two, we have to test several plms architectures over the generated corpus. the literature review shows some interesting architecture to try in this research. however, as plm architectures are being developed periodically, there is always the risk of becoming obsolete. for both questions, we have to create an evaluation process that can include clinical intrinsic and extrinsic downstream tasks, as well as, language and prediction tasks. this evaluation process will allow us to clarify the research questions, but probably the answers will depend on the several tasks that we will include. hopefully, answering these research questions will contribute to the current efforts made by other laboratories working in clinical nlp in spanish and languages other than english. ## methods the methodology for this research proposal consists of three parts: build a clinical corpus for pretraining plms in spanish, test different plm architectures, and evaluate the implemented plms. this section will explain in detail every part. ## expected results the expected results for this proposal are as follows: ## conclusions this work presents a phd research proposal for developing clinical plms in spanish that can outperform general-domain, biomedical, and current clinical plms. as a result, it is expected to answer the research questions regarding whether using only clinical text to create a plm can outperform current approaches and if a particular plm architecture is better for clinical purposes. we expect this work can help to understand how plms work in the clinical domain and in languages other than english.