C asi M edicos-Arg: A Medical Question Answering Dataset Annotated with Explanatory Argumentative Structures (2024)

Abstract: Explaining Artificial Intelligence (AI) decisions is a major challenge nowadays in AI, in particular when applied to sensitive scenarios like medicine and law. However, the need to explain the rationale behind decisions is a main issues also for human-based deliberation as it is important to justify why a certain decision has been taken. Resident medical doctors for instance are required not only to provide a (possibly correct) diagnosis, but also to explain how they reached a certain conclusion. Developing new tools to aid residents to train their explanation skills is therefore a central objective of AI in education. In this paper, we follow this direction, and we present, to the best of our knowledge, the first multilingual dataset for Medical Question Answering where correct and incorrect diagnoses for a clinical case are enriched with a natural language explanation written by doctors. These explanations have been manually annotated with argument components (i.e., premise, claim) and argument relations (i.e., attack, support). The Multilingual CasiMedicos-arg dataset consists of 558 clinical cases (English, Spanish, French, Italian) with explanations, where we annotated 5021 claims, 2313 premises, 2431 support relations, and 1106 attack relations. We conclude by showing how competitive baselines perform over this challenging dataset for the argument mining task.

## introduction there is an increasingly large body of research on ai applied to the medical domain with the objective of developing technology to assist and support medical doctors in explaining their decisions or how they have reached a certain conclusion. for example, resident medical doctors preparing for licensing exams may get ai support to explain what and why is the treatment or diagnosis cor-* equal contribution rect given some background information @xcite . a prominent example of this is the recent proliferation of medical question answering (qa) datasets and benchmarks, in which the task often involves processing and acquiring relevant specialized medical knowledge to be able to answer a medical question based on the context provided by a clinical case @xcite @xcite . the development of large language models (llms), both general purpose and specialized in the medical domain, has enabled rapid progress in medical qa tasks which has led in turn to claims about llms being able to pass official medical exams such as the united states medical licensing examination (usmle) @xcite . thus, publicly available llms such as llama @xcite or mistral @xcite and their respective medicalspecific versions pmc-llama @xcite and biomistral @xcite , or proprietary models such as medpalm @xcite and @xcite , to name but a few, have been reporting high-accuracy scores in a variety of medical qa benchmarks foot_0 @xcite . while these results constitute impressive progress, currently the medical qa research field still presents a number of shortcomings. first, experimentation has been mostly focused on providing the correct answer in medical exams, usually in a multiple-choice setting. however, as doctors are also required to explain and argue about their predictions, research on medical qa should also address the generation of argumentative explanations. unfortunately, and to the best of our knowledge, no medical qa dataset, that currently exists, includes correct and incorrect diagnoses enriched with natural language explanations written by medical doctors. second, the large majority of medical qa benchmarks are available only in english @xcite , which makes it impossible to know the ability of current llms for medical qa in other languages. in this paper, we address these issues by presenting casimedicos-arg, the first multilingual (english, french, italian, spanish) dataset for medical qa with manually annotated gold explanatory argumentation about incorrect and correct predictions written by medical doctors. more specifically, the corpus consists of 558 documents with reference gold doctors' explanations which are enriched with manual annotations for argument components (5021 claims and 2313 premises) and relations (2431 support and 1106 attack). this new resource will make it possible, for the first time, to research not only on argument mining but also on generative techniques to argue about and explain predictions in medical qa settings. finally, strong baselines on argument component detection, a challenging sequence labelling task, using encoder @xcite , encoder-decoder @xcite and decoder-only llms @xcite demonstrate the validity of our annotated resource. data, code and fine-tuned models are publicly available ( https://github.com/ ixa-ehu/antidote-casimedicos ). ## related work in this section, we will focus on reviewing datasets for medical qa and on explanatory argumentation, the two main features of our main contribution, casimedicos-arg. ## casimedicos-arg annotation the spanish ministry of health yearly publishes the resident medical or médico interno residente (mir) licensing exams including the correct answer. every year the casimedicos mir project 2.0 foot_1 takes the published exams by the ministry and provide gold explanatory arguments written by volunteer spanish medical doctors to reason about the correct and incorrect options in the exam. the antidote casimedicos corpus consists of the original spanish commented exams by the casimedicos doctors which were cleaned, structured and freely released for research purposes @xcite . the original spanish data was automatically translated and manually revised into english, french, and italian. the corpus includes 622 documents each with a short clinical case, the multiple-choice questions and the explanations written by medical doctors foot_2 . in the rest of this section we describe the process of manually annotating argumentative structures in the raw antidote casimedicos dataset. ## experimental setup we first describe the process of projecting the manually annotated argumentation labels from the source english data to the other three target languages, namely, french, italian and spanish. since the annotators of the argument components were english speakers, we treated it as the source when projecting labels to the target languages. this process will result in the multilingual casimedicos-arg which will then be leveraged to produce strong baselines on argument component detection using a variety of lms, including encoders @xcite @xcite and decoder-only llms @xcite . ## empirical results in this section, we report the results obtained after performing the steps described in section 4. all the results and standard deviations reported in this section are obtained by averaging three randomly initialized runs. we evaluate using sequence level f1-macro score, a common metric for argument component detection. we first show the results on monolingual (using the manually annotated english data) and multilingual (fine-tuning on all four languages and evaluating in english) in the results for spanish, french and italian are displayed in for cross-lingual model transfer the best results are obtained by the medical mt5 models, which may be due to this model being trained on multilingual medical data @xcite . summarizing, in this section we present competitive baselines for argument component detection on casimedicos-arg, validating both the manual annotations and the strategy of projecting english labels to other languages to facilitate the application of cross-lingual and multilingual techniques. ## conclusion in this paper, we present casimedicos-arg, a multilingual (french, english, italian and spanish) medical qa dataset including gold reference explanations written by medical doctors which has been annotated with argumentative structures. this dataset aims to bridge a glaring gap in the medical qa ecosystem by facilitating the evaluation of explanations generated to argue or justify a given prediction. the final dataset includes 558 documents (parallel in four languages) with reference gold doctors' explanations which are enriched with manual annotations for argument components (5021 claims and 2313 premises) and relations (2431 support and 1106 attack). both inter-annotator agreement results and the baselines provided for argument component detection demonstrate the validity of our annotations. furthermore, experiments show the advantage of performing argument component detection from a multilingual data-transfer perspective.