Les grands modèles de langue biomédicaux préentraînés sur des données hors EHR sont moins performants en contexte multilingue réel (2025)

Abstract: Des travaux récents ont démontré que les grands modèles de langue (LLMs) sont capables de traiter des données biomédicales. Cependant, leur déploiement en zéro-shot dans les hôpitaux présente de nombreux défis. Les modèles sont souvent trop coûteux pour une inférence et un ajustement local ; leur capacité multilingue est inférieure par rapport à leur performance en anglais ; les ensembles de données de préentraînement, souvent issus de publications biomédicales, sont trop génériques pour une performance optimale, compte tenu de la complexité des scénarios cliniques présents dans les données de santé. Nous abordons ces défis et d’autres encore dans un cas d’usage multilingue réel à travers le développement d’un pipeline de normalisation de concepts de bout en bout. Son objectif principal est de convertir l’information issue de dossiers de santé non structurés (multilingues) en ontologies codifiées, permettant ainsi la détection de concepts au sein de l’historique médical d’un patient. Dans cet article, nous démontrons quantitativement l’importance de données réelles et spécifiques au domaine pour des applications cliniques à grande échelle.

## methodology in recent years, we observe an increase in the performance of medical llms along with an increase in the amount of data used for model pretraining and the size of medical language models, from the encoder-only biobert @xcite at 175m parameters to the autoregressive biogpt (luo et al., 2022) at 1.5b parameters, and 540b parameters of the medpalm model @xcite . oftentimes, state-of-the-art models are not publicly available, and hospitals in europe are reluctant to partner and share data with external providers, partly due to ethical and regulatory @xcite , as well as financial considerations (dubas-jakóbczyk et al., 2024). our goal was therefore to create a small, scalable pipeline that does not require high-end gpus for on-prem deployment. while we cannot reveal all the details involving the training of the proprietary pipeline, we can share the steps undertaken for the creation of each component. ## results the performance of the proprietary pipeline is compared against two generic, open-source end-to-end systems. as our baseline, we use quickumls @xcite that operates by the principle of approximate string matching of the concepts from the umls ontology onto the input data. due to the aforementioned challenges in the ehr data, we hypothesize that this baseline will result in lower scores on clinical narratives. the second pipeline consists of two open-source models for the (d)ner and el tasks. out of the available multilingual llms capable of processing medical data in a single-gpu setup, we selected the 4-bit quantized version of the mistral 7b instruct model @xcite for the (d)ner task. we found that even though the model was pretrained solely on english data, it could extract the necessary information from text in other languages if the prompt itself was composed in english. our choice was based on an empirical evaluation of the model's biomedical counterpart biomistral-7b @xcite , as we found the latter to be more prone to hallucinating in experiments, despite it being pretrained in multiple languages, among which dutch and french. ## discussion we observe that while all pipelines are capable of processing the data in three languages to various degrees, our custom pipeline outperforms the rest on clinical narratives, namely, the ehr and e3c datasets. this is a particularly interesting observation, since the e3c test subset is in english, while our model was fine-tuned only on dutch data, which highlights the importance of (sub)domain-tailored solutions. the performance of the pipelines that combine generic and custom llms equally demonstrate the importance of each component in the pipeline. for instance, the highest-scoring pipeline on the mantra dataset combines a custom dner model with the sapbert model, outperforming the mistral + sapbert pipeline's f1 by nearly 8 percentage points. we observe a similar performance on the e3c test set. with regard to the non-ehr test sets, we equally observe the strong performance of custom components. however, we emphasize that the annotations of the quaero and mantra datasets were tailored to the default ranking of the umls thesaurus @xcite . therefore, this explains the strong performance of the quickumls pipeline, particularly on the quaero dataset, as opposed to the custom entity linker that was fine-tuned according to the guidelines that specified the preferred umls terms among the many candidate concepts. the difference in custom annotation and ranking standards is equally reflected in the ehr dataset. for instance, we observe that mistral + sapbert outperform the custom pipeline by a small margin for the precision metric. we attribute this to mistral's tendency to generate simple, flat concepts (predominantly unigrams) that are further easily mapped to reference codes, while our custom dner model produces a hierarchical tree of nested concepts. this observation is supported by the precision scores for the quaero test set (74% of which consists of unigrams), as the dner + sapbert pipeline scores lower than mistral + sapbert. additionally, we must note that these scores are calculated only for a subset of entities that are currently listed in medical ontologies. other non-codified entities (e.g., those in languages other than english, full medication prescriptions and (discontinuous) measurements) are absent from the experiments, as there are currently no open benchmarks. finally, we underline that these results are often solely an indication of performance, rather than a representative evaluation thereof. clinical model (pre-)training can essentially never be complete due to the shifts in the data, such as new treatment trends. but, most importantly, general medical annotations might not reflect the desired output of normalization systems, as expected in applied subdomains. oftentimes, the final evaluation scores of clinical concept normalization pipelines change as per medical experts' validation, as they filter or adjust hierarchically the reference knowledge bases (ontologies), and adjust the ranking of preferred terms. we identify additional challenges related to the clinical normalization task in the following section. for instance, not all negated concepts can be standardized, unlike "no infection" and "no itching" in ## conclusion in this paper, we showed that bespoke in-domain solutions are superior to generic (biomedical) llms in their application to real-world data, and specified a number of challenges that hospitals can struggle with when crafting (multilingual) pipelines specifically applied to the concept normalization task. we underline the importance of smaller, scalable models, often tailored to certain subdomains (e.g., oncology, cardiology, psychiatry, etc.) and hope that the effort on multilingual alignment of biomedical datasets will continue to democratize research in healthcare worldwide.