Pre-trained language models in S panish for health insurance coverage (2023)

Abstract: The field of clinical natural language processing (NLP) can extract useful information from clinical text. Since 2017, the NLP field has shifted towards using pre-trained language models (PLMs), improving performance in several tasks. Most of the research in this field has focused on English text, but there are some available PLMs in Spanish. In this work, we use clinical PLMs to analyze text from admission and medical reports in Spanish for an insurance and health provider to give a probability of no coverage in a labor insurance process. Our results show that fine-tuning a PLM pre-trained with the provider’s data leads to better results, but this process is time-consuming and computationally expensive. At least for this task, fine-tuning publicly available clinical PLM leads to comparable results to a custom PLM, but in less time and with fewer resources. Analyzing large volumes of insurance requests is burdensome for employers, and models can ease this task by pre-classifying reports that are likely not to have coverage. Our approach of entirely using clinical-related text improves the current models while reinforcing the idea of clinical support systems that simplify human labor but do not replace it. To our knowledge, the clinical corpus collected for this study is the largest one reported for the Spanish language.

## introduction clinical text is one of the most comprehensive data types in electronic health records. therefore, clinical natural language processing (nlp) has become relevant to extracting helpful information from clinical writing and supporting decision-making. the complexity of human languages makes it difficult to analyze unstructured text. additionally, the clinical text is complicated because of the heavy use of jargon, unusual spellings, and abbreviations @xcite . in this complex scenario, there are various tasks that clinical nlp aims to handle. these tasks might be anything from language-related ones like text categorization, relation extraction, and entity extraction to prediction-related ones like estimating patient mortality, length of hospital stay, unplanned readmissions, etc. several publications have addressed these tasks that have produced specialized models @xcite . however, since 2017, the nlp field has worked towards creating pre-trained language models (plms) that can be fine-tuned for any specific downstream task. these language models are built for a much simpler task, such as next-word or masked-word prediction in a massive amount of text. this process, known as pre-training, allows the language model to acquire language understanding that can be used for any text-related task @xcite . as soon as the nlp field started to work in plms, clinical nlp introduced this type of model into its set of techniques to improve performance. some examples of clinical plms are two different versions of clinicalbert @xcite . these models show a significant improvement in language tasks and a moderate improvement in prediction tasks. most of the research in clinical nlp has been done for text written in english, but not so much for other languages @xcite . in spanish, some publicly available plms relevant to clinical nlp are bsc-bio-ehr-es @xcite and spanish clinical flair @xcite . these plms were pre-trained heavily in general and biomedical text with minor additions of clinical text. despite this drawback, they outperform general and biomedical plms in language tasks. in this context, an insurance and health provider aims to analyze their clinical text to apply in a labor insurance coverage process. this provider receives patients who have suffered from a labor-related accident. when a patient is admitted to one of their clinics, admitting staff writes a report detailing the accident. this report includes rich contextual information about what happened in the accident. after admission, a physician checks the patient and writes medical information. every clinic's medical and administrative board decides a final coverage rating for each accident the next business day, considering both reports. currently, the provider has a model that gives a probability of not covering a patient given a specific diagnosis. this model serves as a ranking tool for the medical board to review cases with a high likelihood of no coverage. however, not all diagnoses are included in this model (considered as a categorical variable), and admission and medical reports are not considered to calculate the probability. additionally, the model can calculate the probability of no coverage just after a physician diagnoses a patient. this work aims to analyze clinical text from admission and medical reports to give a probability of no coverage. we try three approaches that use clinical plms in spanish to carry out this goal. first, we use a clinical plm. second, we do continual pre-training of the previous plm with text data from admission and medical reports. finally, we pre-trained a lm from scratch using admission and medical reports. all outcomes will be compared to the current model performance. ## problem statement in chile, employers must hire an insurance and health provider specialized in labor accidents. these providers should cover all labor accidents. to decide if the insurance will cover a worker, the providers have clinics where they admit and check the workers to make a decision. in this work, we use data from one of these providers. as this provider is specialized in labor accidents, the data that collects has some features. first, it has a high level of detail because that admission and medical reports are used to jus-tify insurance coverage decisions. second, many physicians can treat the same patient, requiring information in clinical records to be as complete as possible so that any medical staff can give better continuity to patient care over time. these features make the data cleaner compared to general health provider records. the asociación chilena de seguridad (achs), chilean safety association in english, is a preeminent non-profit insurance and health provider. its principal objective is to conceptualize and administer risk prevention programs alongside providing comprehensive coverage for occupational accidents. as evidence of its influence, achs accommodates more than 2.6 million affiliated workers and over 73,000 affiliated employing entities nationwide. moreover, with a record of the lowest average accident rate, achs unequivocally operates as the largest mutual association in chile. the stringent regulations under law no. 16,744 mandate that all chilean employing entities, regardless of their operational scale, must be affiliated with a social security administration agency. this agency is responsible for safeguarding against the risks of occupational accidents and diseases. as one of three private administrative bodies, achs is tasked with formulating risk prevention programs. it also offers health coverage and compensation for occupational accidents, transport mishaps, and professional illnesses. the type of labor accidents can be of two types, work-related and commuting accidents. workrelated accidents happen at the workplace or as a result of work. commuting accidents happen on the way to or from work with no stops in between (direct trips). the staff writes an admission report when the patient is admitted in both cases. later, when a physician receives the patient, a medical report is written. the medical report is based on three sources. the first includes the patient's anamnesis. the sec-ond information is from the physical examination performed on the patient. the third is the medical indication for treatment. each time a new or old patient passes through this healthcare provider and needs to be seen by a doctor, a new medical report entry is generated. the admitting staff and the physician give a label (covered or uncovered) classification on the reports. the final classification is made the next business day after the patient is seen by the medic. a board of physicians and administrative heads from each clinic determines a final coverage rating for each case. this final rating takes into account the medical and admitting staff reports. most admitting staff's labels will state that patients will be covered, and medics, after clinical examination, have a more robust filter to say whether a patient will be covered or not. the committee of physicians and administrative heads has a reviewer role, and some decisions are finally changed. figure 1 shows the flow diagram of the described process. the current model employed by the healthcare provider only makes predictions in 72.1% of cases, basing its predictions on structured diagnoses alone. unfortunately, this approach results in a lack of predictions for less frequently observed diagnoses. however, the majority of cases come with either admission records or medical reports, making it possible to improve coverage by utilizing these additional resources. we expect that the use of admission and medical reports can help to take better coverage classification compared to the current model that only uses diagnosis as a categorical variable with the most common ones. moreover, the classification prediction with the admission report can help the physician consider more information that may have been overlooked. ## datasets for this study, three different types of datasets were built, for fine-tuning, continual pre-training, and pre-training from scratch. here we list the details of these datasets: ## methods this section described the processes of pre-training and fine-tuning using the datasets described in the previous section. ## results as the metrics of all admission+medical models are close, we could select the least expensive and time-consuming when implementing it. in the case of this task, this process is the fine-tuning of the publicly available plm, bsc-bio-ehr-es. however, this evidence should not be generalized for other types of tasks like named entity recognition or question answering, which are more complex and may benefit from lexical specificity. in those tasks, a plm pre-trained with more clinical-related text could be better than a plm trained with a mix of biomedical and clinical text. interestingly, admission models perform 1 to 2% worse than medical models. therefore, there is an opportunity to make a coverage prediction before physicians check patients, helping the physicians review more medical details when their coverage decision does not match the predictions. moreover, the admitting staff can have a stronger opinion on a coverage decision. another benefit of providing a coverage prediction prior to the medical checkup is the possibility that the patient can manage his or her case more effectively. depending on the likelihood of coverage the model provides, the patient may seek resources to help better justify his or her accident. finally, implementing a pre-trained language model could help the healthcare provider increase model auc coverage current model 95.8 72.7 custom plm: admission+medical 97.7 96.1 ## related work nlp has been used in applications of the insurance industry in recent years. nlp techniques can be used to analyze vast amounts of unstructured data, such as customer interactions and policy documents, to gain insights and make informed decisions. in several areas within the insurance industry, nlp is being used, including customer service, claims processing, and fraud detection @xcite . one of the most significant uses of nlp in the insurance sector is customer service @xcite . to ascertain a client's wants and preferences, nlp techniques can be utilized to evaluate customer interactions such as phone calls and chat chats. customers may receive more individualized help and recommendations thanks to the utilization of this data. additionally, regular customer support operations like responding to frequently requested queries, have been automated using nlp algorithms. another area where nlp is employed in the insurance sector is claims processing @xcite . by automating the analysis and classification of claims, nlp approaches can cut down on the time and resources needed to process claims. to make more educated judgments about claims, nlp algorithms have been employed, for instance, to extract information from claim documents, such as the type of injury and the reason for the accident. fraud detection is another area where nlp is being used in the insurance industry @xcite . huge amounts of unstructured data, including policy documents and customer interactions, can be analyzed using nlp approaches to spot probable fraud cases. ## conclusion this work studied the performance of clinical plms in a coverage prediction task. three approaches were implemented, and the best model was compared to the current model used by the health provider. a plm from scratch was the bestperforming model but the most expensive and timeconsuming. clinical natural language processing has great potential to impact the insurance industry, not only because of the great predictive power they offer but also because it is unnecessary to implement expensive training in the models. as there are no significant differences in performance between the pre-trained model and the fine-tuning with the admission+medical data, by just fine-tuning a plm we can obtain good results at a lower cost for this downstream task. however, the situation might differ in other nlp tasks that benefit from lexical specificity.