SweClinEval : A Benchmark for Swedish Clinical Natural Language Processing (2025)

Abstract: The lack of benchmarks in certain domains and for certain languages makes it difficult to track progress regarding the state-of-the-art of NLP in those areas, potentially impeding progress in important, specialized domains. Here, we introduce the first Swedish benchmark for clinical NLP: _SweClinEval_. The first iteration of the benchmark consists of six clinical NLP tasks, encompassing both document-level classification and named entity recognition tasks, with real clinical data. We evaluate nine different encoder models, both Swedish and multilingual. The results show that domain-adapted models outperform generic models on sequence-level classification tasks, while certain larger generic models outperform the clinical models on named entity recognition tasks. We describe how the benchmark can be managed despite limited possibilities to share sensitive clinical data, and discuss plans for extending the benchmark in future iterations.

## introduction the field of natural language processing (nlp) has seen several important breakthroughs in the past decade. currently, the field is dominated by pre-trained transformers models @xcite that can be used to solve a wide and -ideally -diverse set of tasks. the capabilities of these models have to a large degree been tracked through the use of benchmarks, significantly helping to drive progress in the area. these evaluation suites test how the models perform on different pre-defined tasks and allow for comparisons between models and approaches. while there are many benchmarks available, there are also many potential uses for nlp that they do not cover. frequently, evaluations rely on english data @xcite . however, a model performing well on an english benchmark in no way guarantees similar performance if the language changes. additionally, benchmarks such as glue @xcite tend to focus on tasks formulated for generaldomain data. with increasing calls for nlp to be applied to specific domains, such as the clinical domain, there is a pressing need for benchmarks that address these areas. the clinical domain, in particular, suffers from a lack of datasets for evaluating nlp systems. one critical reason for this is the inherently sensitive nature of clinical data. there are multiple studies @xcite demonstrating the potential risks of using sensitive data for machine learning -let alone sharing data in their raw form. that said, there are some widely used resources for clinical nlp. prominent examples include the various versions of mimic @xcite and the i2b2 datasets @xcite . crucially, these datasets predominantly evaluate nlp systems on data in english or other higher-resourced languages. in this paper, we introduce the first swedish benchmark based on real clinical nlp data: sweclineval. this benchmark consists of datasets built from electronic health records from the health bank @xcite and includes a wide range of clinical tasks. these tasks include three different document-level sequence classification tasks and three token-level named entity recognition (ner) tasks. this introduction of sweclineval includes nine different models, and future additions will be added to the benchmarks online leaderboard foot_0 . the evaluations presented in this paper show that many models targeting swedish data per-form strongly on our benchmark. however, the performances vary, and several interesting trends emerge from our results. these results highlight the importance of continuing to focus on domainspecific evaluations for languages other than english. our results demonstrate the current state of swedish clinical nlp, and the benchmark serves as an important tool for monitoring progress in this important nlp domain. ## related research the nlp community has seen impressive advances in the past few years with the advent of llms. several new model architectures have been proposed since @xcite described the transformer, and new models are released at a rapid pace. these llms aim to be generalpurpose models, with task-specific applications requiring only smaller adjustments in the form of fine-tuning or prompt engineering. in response to this new paradigm, there has been an increasing focus on creating benchmarks that capture the nuanced difference in performance in the growing plethora of models. ## methods and materials creating this first rendition of sweclineval involved collecting resources for evaluation and deciding how to conduct the evaluations. this section describes the datasets used for the benchmark and the models that were tested, and how they were chosen. the design of the evaluations and the metrics used for comparing models are also described. ## results nine models were evaluated using 10-fold crossvalidation for six different datasets, resulting in 540 evaluations. the average f 1 scores and their deviations are listed in for the sequence-level classification tasks, the highest average f 1 scores are consistently obtained using the domain-adapted models. the same is not true for the token-level ner tasks. for these tasks, the highest f 1 scores were obtained by the general-domain roberta large model from ai sweden. however, the domain-adapted swedeclin-bert model has the second-highest average f 1 scores for the factuality ner and clinical entity ner tasks. the different average f 1 scores vary substantially between the best-and worst-performing models. nevertheless, the standard deviations are large. this means that many of the averages are within a standard deviation of a competing model. this necessarily limits the analysis into which models are best, since randomness has a strong influence on the variability in the f 1 scores. in addition to the predictive performance, ## discussion a few trends emerge from the results in the previous section. there are also some limitations and pointers to future work that are important to discuss. however, we begin by discussing the findings from our results. as previously mentioned, the highest average f 1 scores in the sequence classification tasks are perhaps somewhat surprisingly, parameter count itself does not seem to be a determining factor in what models are the strongest. this is not only the case when comparing domain-adapted and general-domain models. for example, kb-bert base and kb-bert large were both trained by the same organization, and are from the same model family. the main difference between the while the large standard deviations call for cautious interpretations of the results, it is at least clear the larger model is not outperforming its smaller competitor. on the other hand, parameter count clearly influences the inference speed of the models, as indicated in ## conclusions in this paper, we present sweclineval -the first swedish benchmark for clinical nlp. we evaluate a wide range of encoder-style llms for six different swedish clinical nlp tasks. this effort represents the first such evaluation to be conducted, and forms a basis for future monitoring of the advances in swedish clinical nlp. the results of this first evaluation indicate several interesting trends. the benchmark results suggest that domain adaptation is an effective strategy for improving the performance of llms in the clinical domain, at least for small llms. future research should examine whether this also holds for larger models. furthermore, the evaluations also show that parameter count alone is not enough to perform strongly in the tasks included in our benchmark. the aim of this paper is to enable monitoring of the progress within swedish clinical nlp. due to privacy constraints, the data cannot be shared. we strongly encourage others interested in swedish clinical nlp to contact us for inclusion in the benchmark. this pragmatic approach to benchmarking enables us to monitor the progress that is being made, which sweclineval makes possible.