R o B iology D ata C hoice QA : A R omanian Dataset for improving Biology understanding of Large Language Models (2025)

Abstract: In recent years, large language models (LLMs) have demonstrated significant potential across various natural language processing (NLP) tasks. However, their performance in domainspecific applications and non-English languages remains less explored. This study introduces a novel Romanian-language dataset 1 for multiple-choice biology questions, carefully curated to assess LLM comprehension and reasoning capabilities in scientific contexts. Containing approximately 14,000 questions, the dataset provides a comprehensive resource for evaluating and improving LLM performance in biology. We benchmark several popular LLMs, analyzing their accuracy, reasoning patterns, and ability to understand domain-specific terminology and linguistic nuances. Additionally, we perform comprehensive experiments to evaluate the impact of prompt engineering, fine-tuning, and other optimization techniques on model performance. Our findings highlight both the strengths and limitations of current LLMs in handling specialized knowledge tasks in lowresource languages, offering valuable insights for future research and development.

## introduction large language models (llms) have achieved impressive results across a wide range of natural language processing (nlp) tasks. however, their performance often degrades in specialized domains and non-english languages, making romania's rich tradition in biology an ideal context for evaluating llms' scientific reasoning in a relatively lowresource setting. to rigorously examine and ultimately improve llm competence on such domain-specific tasks, we created a romanian-language dataset of multiple-choice biology questions. the dataset was developed to assess and enhance llm performance on authentic romanian biology tests. it enables the evaluation of model accuracy in a realistic multiplechoice setting and can also be used to fine-tune llms on domain-specific romanian biology terminology. our dataset comprises questions from two prestigious national sources: the romanian biology olympiad and medical school admission examinations. the olympiad is the country's largest biology competition, targeting middle-and high-school students, while medical entrance exams rigorously assess pre-university candidates on foundational biological knowledge. together, these sources provide a comprehensive and challenging collection of questions, covering a broad range of biological topics, difficulty levels, and linguistic complexity. this study goes beyond simple benchmarking. we conduct extensive experiments to explore how various factors such as prompt engineering strategies, model origin, and domain-specific fine-tuning influence model performance. our statistical analyses provide insights into how well llms grasp romanian biological concepts, reveal common failure patterns, and highlight differences across model types. our contributions are threefold: (1) we introduce a carefully curated romanian-language biology dataset suitable for benchmarking and domain adaptation; (2) we assess the capabilities of leading llms in scientific reasoning within a low-resource language setting, building on previous work that shows persistent challenges in this area @xcite ; and (3) we present an in-depth analysis of performance variation across experimental conditions, offering insights that can inform future model development and deployment in specialized domains. we aim to encourage research on llms for non-english and domain-specific tasks, advancing nlp for educational and scientific contexts. ## related work biomedical question-answering (qa) datasets have played a crucial role in advancing domain-specific language models. pubmedqa @xcite introduced a large-scale english-language biomedical qa dataset with 1,000 expert-annotated, 61,200 unlabeled, and 211,300 artificially generated yes/no/maybe questions. while valuable for scientific text comprehension, it does not include multiple-choice questions, which require more complex reasoning over structured information. a more relevant effort is medqa @xcite , an open-domain multiple-choice qa dataset collected from professional medical board exams. medqa covers three languages -english (12,723 questions), simplified chinese (34,251 questions), and traditional chinese (14,123 questions) -and requires models to select the correct answer from multiple options rather than extracting answers directly from text. similarly, medmcqa @xcite is an english-language multiple-choice qa dataset designed for medical entrance exams, containing over 194,000 questions. unlike medqa, which focuses on board exam questions, medm-cqa emphasizes a wide range of medical knowledge, testing over ten different reasoning abilities. efforts to develop language models specialized for romanian biology are quite limited. one notable contribution is roqllama, a lightweight romanian-adapted language model designed to enhance nlp performance in romanian-language applications @xcite . roqllama was evaluated using the romedqa dataset @xcite , a specialized collection of romanian medical school examination questions. our work surpasses this effort by introducing a carefully curated and extended romanian-language biology dataset extracted from multiple sources, going beyond single-choice questions. we also fine-tune promising models and perform multiple benchmarks. fine-tuning on our dataset significantly improves llm performance, making it a valuable resource for enhancing language models in biology. by focusing on this domain, our dataset diversifies the range of available domain-specific resources for romanian, complementing previous contributions in the medical field and aiming for deeper reasoning. guidance on creating and documenting highquality nlp datasets is essential for ensuring the utility of research outcomes. the dataset documentation framework proposed by @xcite provided foundational insights for structuring the description and documentation of our dataset. the use of llms in biology has shown significant potential for transforming research in the life sciences. @xcite explored the evolution of llms from textual comprehension tools to multimodal systems capable of analyzing complex biological data and contributing to advances in molecular biology and medicine. their findings highlight the importance of llms in handling scientific reasoning and specialized terminology, which is central to our work. 3 dataset composition (14,109) single choice (6021) multiple choice (4170) group choice (3918) college (1918) ix (1412) vii (812) x (802) xii (539) xi (538) college (4170) ix (1290) vii (801) x (761) ## experiments we conducted comparisons and benchmarks across several dimensions, including zero-shot vs. fewshot settings, group-choice heuristics, and combined vs. individual predictions. to ensure reproducibility, all experiments were run with temperature set to zero. these experiments were carried out using local hardware, a google colab pro subscription, and various api/runtime services, with a total cost of $48.73. although we do not have an exact runtime estimate, the work was completed over 2-3 months of intermittent activity. ## conclusion this study introduced robiologydatachoiceqa, a novel romanian-language dataset designed to evaluate biology comprehension in large language models (llms). sourced from the romanian biology olympiad and medical school entrance exams, it provides a diverse and challenging benchmark for assessing domain-specific reasoning in a lowresource language. our benchmarking experiments revealed significant variations in model performance, highlighting both strengths and limitations of llms in specialized tasks. while some models performed well on structured, single-answer university questions, their ability to handle grouped-choice and reasoning tasks remained inconsistent. fine-tuning gemini 1.5 flash and gemma 2 9b instruct improved accuracy in certain cases, demonstrating that targeted adaptation can be effective. beyond model evaluation, our study offers insights into the impact of prompt engineering, finetuning strategies, and dataset characteristics on llm performance. these findings contribute to the broader effort of advancing nlp applications in non-english languages and scientific domains. future research should focus on expanding the dataset with fine-grained subdomain annotations, improving ocr processing, experimenting with other fine-tuning strategies and model architectures, and addressing dataset biases by comparing performance across question sources. enhancing answer verification through expert validation will also be essential for benchmark reliability.