P olish- E nglish medical knowledge transfer: A new benchmark and results (2025)

Abstract: Large Language Models (LLMs) have demonstrated significant potential in specialized tasks, including medical problem-solving. However, most studies predominantly focus on English-language contexts. This study introduces a novel benchmark dataset based on Polish medical licensing and specialization exams (LEK, LDEK, PES). The dataset, sourced from publicly available materials provided by the Medical Examination Center and the Chief Medical Chamber, includes Polish medical exam questions, along with a subset of parallel Polish-English corpora professionally translated for foreign candidates. By structuring a benchmark from these exam questions, we evaluate state-of-the-art LLMs, spanning general-purpose, domain-specific, and Polish-specific models, and compare their performance with that of human medical students and doctors. Our analysis shows that while models like GPT-4o achieve near-human performance, challenges persist in cross-lingual translation and domain-specific understanding. These findings highlight disparities in model performance across languages and medical specialties, emphasizing the limitations and ethical considerations of deploying LLMs in clinical practice.

## introduction the potential of artificial intelligence, especially large language models (llms), is vast, but they come with considerable risks, particularly the issue of "hallucinations", where llms produce incorrect or misleading responses. this is especially concerning in fields like medicine, where errors can have serious consequences. therefore, rigorous evaluation of llm performance is essential before their clinical integration @xcite . llm performance varies significantly due to differences in training methods, datasets, and objectives, which affect their ability to perform specific tasks. the quality and diversity of training datasets are particularly important for specialized domains like medicine @xcite . while models trained on comprehensive, domain-specific datasets are expected to outperform those trained on general-purpose data, this assumption has been challenged @xcite . language also significantly impacts llm performance. most widely studied models are trained on multilingual datasets, predominantly in english, leading to better performance with englishlanguage inputs and challenges with non-english content @xcite . additionally, llms trained exclusively on non-english texts may lack important knowledge available only in english. modern medicine is evidence-based, and one might assume that the correct management of medical issues should be nearly universal. however, in practice, clinical practices are shaped by various factors, leading to significant variations in medical guidelines across countries. for instance, @xcite analyzed 22 clinical practice guidelines from 15 countries, highlighting notable differences in recommendations for managing lower back pain. llms trained primarily on english-language data are likely to align with disease prevalence and clinical guidelines typical of english-speaking countries. consequently, their diagnostic and therapeutic recommendations may be biased towards practices common in these regions. when presented with the same clinical scenario in different languages, an llm may produce varying responses, reflecting the diversity of healthcare practices across countries represented in the training data. such discrepancies could be revealed by evaluating llms on non-english medical tests, like those conducted in poland, where disease prevalence and medical guidelines may differ from those in english-speaking countries. to primarily assess the performance of llm models in medical question-answering tasks, we introduce a new benchmark based on publicly avail-able exam questions from medical and dental licensing exams, as well as specialist-level exams conducted in poland foot_0 . this dataset includes over 22,000 questions, primarily in polish, with a subset of licensing exam questions also available in english, enabling comparative analysis. we propose a benchmark that enables the study of llm behavior by addressing the following research questions: llms have the potential to revolutionize medicine by assisting medical professionals in key areas such as medical education, literature summarization, data extraction, manuscript drafting, and patientclinical trial matching @xcite . they streamline communication by converting unstructured data to structured formats and simplifying documentation, such as summarizing patient records and generating medical reports @xcite . this reduces administrative burdens on clinicians, allowing more focus on patient care @xcite . llms also enhance personalized, patient-centered care, improve clinicianpatient interactions, and may aid in diagnostics and management planning by analyzing medical data and monitoring patient parameters @xcite . integrating llms into healthcare requires thorough evaluation to ensure reliability, safety, and equity, while identifying weaknesses and addressing biases to improve clinical care and support healthcare professionals @xcite . this evaluation should go beyond traditional performance metrics to include factors such as accuracy, reasoning, and factual reliability, using benchmarks like medical licensing exams, as well as assessing real-world utility, including clinical impact and workflow integration @xcite . small, fine-tuned bert-style models continue to outperform llms in certain nlp tasks, such as text classification @xcite . however, the emergence of llms, such as gpt-3.5 and med-palm 2, has led to significant advancements in medical question-answering benchmarks, including medqa @xcite , medmcqa @xcite , and pubmedqa @xcite . for this specific nlp task, general-purpose llms enhanced with specialized prompting strategies @xcite or fine-tuned domain-specific models surpass small encoder-only models in performance @xcite . most of the current datasets focus on english, which reflects both the dominance of english in medical research and the initial english-centric development of llms. however, there is growing recognition of the need for multilingual and non-english datasets to ensure the broader applicability of medical llms. medqa is notable for its multilingual approach, incorporating questions from medical board exams in english, simplified chinese, and traditional chinese @xcite . additionally, there are datasets built around medical examinations in specific languages, including swedish medqa-swe @xcite , chinese cmexam @xcite , japanese igaku qa @xcite , and polish. for polish, lekarski egzamin końcowy (lek, eng. medical final examination) is used as a benchmark @xcite @xcite . lek is available in both polish and english, allowing researchers to evaluate the influence of language on llm performance. to date, analyses have primarily focused on gpt models, though several other llms, including llama and med42, have also been evaluated @xcite . regarding the państwowy egzamin specjalizacyjny (pes, eng. polish board certification examination), a few studies have assessed gpt's performance in specialized field exams @xcite @xcite . @xcite provided a comprehensive evaluation of gpt-3.5 and gpt-4 on the pes, utilizing 297 exams across 57 specialties in polish. we extend this research by incorporating additional pes specialties and introducing new exam types-the lek and ldek-in both polish and english. our study also includes cross-lingual evaluation of llms, comparisons with human performance, and assessments of publicly available llms, which were not done by @xcite . @xcite proposed a benchmark for the cross-lingual evaluation of llms. however, the questions they used were translated by a machine translation system, while the questions in our benchmark are translated by human medical experts from the examination center. furthermore, we evaluated new models that demonstrate much better performance @xcite . ## polish medical exams dataset overview the lek (lekarski egzamin końcowy, medical final examination) is a standardized exam for medical graduates and final-year students in poland. passing this exam, along with completing a postgraduate internship, is mandatory to obtain a medical license. starting from 2022, 70% of the questions come from a publicly available database, which includes 2,870 questions for lek. the exam is conducted twice a year and lasts four hours, consisting of 200 multiple-choice questions. candidates are allowed to retake the exam multiple times, even after passing, to improve their scores. the ldek (lekarsko-dentystyczny egzamin końcowy, dental final examination) is the equivalent exam for dentistry graduates and final-year students, following the same format and requirements as the lek. the pes (państwowy egzamin specjalizacyjny, national specialization examination) is a mandatory exam for physicians and dentists who have completed specialization training, including required internships and courses. it consists of a written test and an oral examination. the written test, held twice a year for each specialty, typically includes 120 multiple-choice questions, with one correct answer per question, and a passing score of 60%. candidates achieving at least 70% on the written part are exempt from the oral examination, a rule introduced in late 2022. pes is considered the most challenging exam in the professional career of a medical doctor in poland, and unlike lek and ldek, its questions are not made public before the exam. in poland, five types of exams for physicians and dentists are conducted: lek (lekarski egzamin końcowy, eng. medical final examination), ldek (lekarsko-dentystyczny egzamin końcowy, eng. medical-dental final examination), lew (lekarski egzamin weryfikacyjny, eng. medical verification examination), ldew (lekarsko-dentystyczny egzamin weryfikacyjny, eng. medical-dental verification examination), and pes (państwowy egzamin specjalizacyjny, eng. national specialization examination, board certification exam). lew and ldew are for graduates of medical or dental studies carried outside of the european union. passing these exams is necessary for them to legally practice in poland. 2 however, these lew and ldew are taken by a relatively small number of candidates, and access to previous exam questions is limited. therefore, they are not included in our work. the extensive descriptions of medical exams are included in appendix a. the dataset comprises medical exams from the medical examination center (centrum egzaminów medycznych -cem) and the supreme medical chamber (naczelna izba lekarska -nil), covering lek, ldek, and pes exams from 2008-2024. it sources the exams as html quizzes and pdf files, with missing data from 2016-2020 (lek/ldek) and 2018-2022 (pes) partially filled using archives published on the nil website. the exams are categorized by specialization, with questions and answers stored separately. automated tools scrape and process the data, balancing parallelization with server constraints. preprocessing ensures the dataset's suitability for text-only ai benchmarks by removing irrelevant files, questions containing images, and content misaligned with current medical knowledge. we refer to these as "invalidated questions" throughout the text. detailed descriptions of data sources, acquisition methods, and quality considerations appear in appendix e. finally, we create five sub-datasets: lek, ldek, pes, lek en (lek translated into english), and ldek en (ldek translated into english). not all of them are released in the same edition, particularly the polish and english counterparts. therefore, the results presented in section 4 should not be used to directly compare llm performance on polish exams with their english translations. to address this, we focus on the overlapping years and report these results in section 5. for the pes dataset, we collected a total of 180,712 questions. for the analysis in sections 4, 5, and 6, we select only the most recent exam from each specialty and base our analysis on these exams. detailed dataset statistics are provided in ## performance of llms on exams we categorize the models under study into the following groups: medical llms (models fine-tuned on english medical data), general-purpose multilingual llms, polish-specific models, and models with restricted apis. ## cross-lingual knowledge transfer to compare the performance of various llms on polish and english versions of the same datasets, in contrast, with meta-llama-meta-llama-3.1-70b-instruct, the difference is only 1.66 percentage points (80.94% for polish lek vs. 82.60% for english lek, or a 2% relative change). for gpt-4o-mini, which generally performs well, the results in english are only slightly better than in polish. interestingly, for gpt-4o, performance is actually higher on the polish version. the only polish llm, bielik, performs better on polish lek and slightly better on polish ldek, likely due to its fine-tuning from the multilingual model mistral-7b-v0.2 specifically for polish. this fine-tuning enables it to better capture the nuances of polish text to other models with a similar number of parameters. however, the tested bielik-v2.2-instruct, with only 11b parameters, is outperformed by models with double or even larger parameter counts on polish versions of the lek and ldek exams. the only exceptions to this trend are mistral-small-instruct-2409 and llama3-openbiollm-70b. overall, our observations suggest that language transfer is more effective as the model's general performance improves. refer to appendix f for detailed question-level analysis. ## comparison against human results meditron3-70b, meta-llama-3.1-70b-instruct, bielik-11b-v2.2-instruct, and gpt-4o-2024-08-06 are selected as the topperforming models for the groups mentioned in section 4, and compared against anonymized human results published on the cem webpage from the last four lek and ldek sessions (spring 2024, autumn 2023, spring 2023, autumn 2022), covering 977 lek and 984 ldek questions. the exams were taken by 33,929 participants for lek and 4,366 for ldek, totaling 38,295 results from medical graduates and final-year students in poland. while all selected models pass the chosen lek exams, only meta-llama-3.1-70b-instruct and gpt-4o-2024-08-06 score within the range defined by an average number of @xmath0. assuming a normal distribution of exam results, it could be concluded that these models perform as a typical medical student. notably, for the spring 2024 lek exam, meditron3-70b also achieves an average-level result, while gpt-4o-2024-08-06 exceeds the average student score. for the ldek exams, all models perform noticeably worse. assuming a normal distribution of exam results, only gpt-4o-2024-08-06 maintains a performance level comparable to that of an average medical student, consistent with its lek exam results. in contrast, meditron3-70b and bielik-11b-v2.2-instruct perform poorly, failing all exams, while meta-llama-3.1-70b-instruct score below the average but manage to pass each exam. these outcomes are summarized in the same models are used to compare their performance with humans on the pes exams. more details about joining the pes medical questions and human results are provided in appendix g. the best-performing model is gpt-4o-2024-08-06, which achieves results in above 60% of cases better than half of the test takers population and above 30% of cases is placed in the top 25% of scores. notably, this model outperforms all examinees in a thoracic surgery exam. however, it is important to note that the examinee population for this particular exam is relatively small, consisting of only six participants. however, it is worth noting that even the best model achieves results worse than half of the test takers population in over 30% of specializations. for the audiology & phoniatrics specialization, the model underperformes compared to all examinees. however, the test takers population for that particular case was relatively small, consisting of only nine participants. the secondbest model, meta-llama-3.1-70b-instruct, delivers significantly worse performance compared to the best model. only slightly above 11% of its results across specializations are above the population median, while in over 30% of medical specializations, its performance is above the 25th percentile and below the 50th percentile. the remaining models, meditron3-70b and bielik-11b-v2.2-instruct, perform extremely poorly, with most of their results falling below the 25th percentile or even below the lowest scores of the entire test takers population. the human results and additional explanations for ## conclusion in this paper, we propose a new benchmark for analyzing the performance of large language models in answering questions pertaining to the domain of medical knowledge. in contrast to the majority of previous medical datasets that collect the parallel sub-corpus composed of examination questions in polish aligned with their english counterparts is a distinguished feature of the presented benchmark which allows us to investigate the cross-lingual transfer of medical knowledge in llms. our findings show that models perform significantly better on english questions and that as the size of the model increases, performance improves, and the gap between languages narrows, an expected but difficult-to-measure result without an appropriate benchmark.