An evaluation of current benchmarking strategies for F rench biomedical language models (2024)

Abstract: We describe the current state of benchmarking for French language biomedical natural language processing (NLP). We note two important criteria in biomedical benchmarking: first, that a biomedical benchmark clearly simulate a specific use cases, in order to offer a useful evaluation of a biomedical model’s real life applicability. Second: that a biomedical benchmark be created in collaboration with biomedical professionals. We note that many biomedical benchmarks, particularly in French, do not adhere to these criteria; however, we highlight other biomedical benchmarks which adhere better to those criteria. Furthermore, we evaluate some of the most common French biomedical benchmarks on an array of models and empirically support the necessity of domain-specific and language-specific pre-training for natural language understanding (NLU) tasks. We show that some popular French biomedical language models perform poorly and/or inconsistently on important biomedical tasks. Finally, we advocate for an increase in publicly available, clinically targeted French biomedical NLU benchmarks.

## introduction since the advent of the transformer architecture and the subsequent rise of deep language models (lms), the power of natural language processing (nlp) models has significantly improved @xcite . this improvement has led to the application of lms in various domains, such as grading at universities @xcite , policing the internet for hate speech (plaza-del @xcite , or helping doctors treat their patients @xcite . however, with great power comes great responsibility; as these models become increasingly ubiquitous and their decisions increasingly relied upon, potential deployers must have a nuanced understanding of their abilities. one must know as precisely as possible how well an lm will perform on its assigned task in order to gauge the expected error in its calculations, and thus afford it adequate human supervision. a model ought not be deployed until it has been properly and thoroughly evaluated. to perform this evaluation, the scientific community relies on benchmarks, which are series of tests designed to simulate real-life scenarios which an lm might encounter. in order for a new model to gain traction in the scientific community, it must perform well on certain benchmarks. for well established domains, these benchmarks have been studied for decades and have undergone multitudinous permutations and updates. at any given moment there are certain benchmarks that are understood by the community to be essential; a model not evaluated on these will not be taken seriously by the community, or reviewers at conferences or journals @xcite foot_0 . as the state-of-the-art (sota) improves, these benchmarks are continually updated or retired due to "degeneration", where human parity is reached @xcite . however, domains in which there are not yet well-established benchmarks, such as french biomedical nlp, lack such self-regulation @xcite . therefore, the publishers of models in cutting edge domains must choose, without relying on significant precedent, on which benchmarks to evaluate their models. this freedom of choice can lead authors to primarily include benchmarks on which their models perform well compared with their competitors, a process referred to by @xcite as "rigging the lottery". this is counterproductive for a nascent domain, as it can motivate the reverse-engineering of evaluation systems to promote individual models, rather than the engineering of better models to solve known tasks. in this paper, we show that french biomedical nlp benchmarking exhibits weaknesses consistent with an early stage domain as taxonimized by @xcite . we consider challenges inherent in biomedical benchmark creation, and discuss ways in which benchmarks can be created more effectively. we then perform a review of benchmarks used in french biomedical nlp, and perform an independent evaluation of them using sota models. we show that more work towards benchmarking is necessary in order to better prepare french biomedical lms for deployment. ## experimental setup the goal of this empirical section is two-fold: we train each model on each of the six benchmarks. we cross-validate the learning rate for each model and dataset using a random 80/10/10 train/valid/test split, each for up to 2000 steps, stopping early given validation set convergence. we repeat this training for frozen and unfrozen model weights. we replicate each experiment twenty times to gauge each model's consistency.