F rench M ed MCQA : A F rench Multiple-Choice Question Answering Dataset for Medical domain (2022)

Abstract: This paper introduces FrenchMedMCQA, the first publicly available Multiple-Choice Question Answering (MCQA) dataset in French for medical domain. It is composed of 3,105 questions taken from real exams of the French medical specialization diploma in pharmacy, mixing single and multiple answers. Each instance of the dataset contains an identifier, a question, five possible answers and their manual correction(s). We also propose first baseline models to automatically process this MCQA task in order to report on the current performances and to highlight the difficulty of the task. A detailed analysis of the results showed that it is necessary to have representations adapted to the medical domain or to the MCQA task: in our case, English specialized models yielded better results than generic French ones, even though FrenchMedMCQA is in French. Corpus, models and tools are available online.

## introduction multiple-choice question answering (mcqa) is a natural language processing (nlp) task that consists in correctly answering a set of questions by selecting one (or more) of the given n candidates answers (also called options) while minimizing the number of errors. mcqa is one of the most difficult nlp tasks because it requires more advanced reading comprehension skills and external sources of knowledge to reach decent performance. in mcqa, we can distinguish two types of answers: (1) single and (2) multiple ones. most datasets focus on single answer questions, such as mctest @xcite , arcchallenge @xcite , openbookqa @xcite , qasc @xcite , social-iqa @xcite , or race @xcite . to our knowledge, few studies have been done to construct medical mcqa dataset. we can cite the medmcqa @xcite and head-qa @xcite corpora which contain single answer questions in spanish and english respectively. for the multiple answer questions, mlec-qa @xcite provides 136k questions in chinese covering various biomedical sub-fields, such as clinic, public health and traditional chinese medicine. the french community has recently greatly increased its efforts to collect and distribute medical corpora. even if no open language model is currently available, we can cite the named entity recognition @xcite and information extraction @xcite tasks. however, they remain relatively classic, current approaches already reaching a high level of performance. in this article, we introduce frenchmedmcqa, the first publicly available mcqa corpus in french related to the medical field, and more particularly in the pharmacological domain. this dataset contains questions taken from real exams of the french diploma in pharmacy. among the difficulties related to the task, the questions asked may require a single answer for some and multiple ones for others. we also propose to evaluate state-of-the-art mcqa approaches, including an original evaluation of several word representations across languages. main contributions of the paper concern (1) the distribution of an original mcqa dataset in french related to the medical field, (2) a state-of-the-art approach on this task and a first analysis of the results, and (3) an open corpus, including tools and models, all available online on demand. ## the frenchmedmcqa dataset in this section, we detail the frenchmedmcqa dataset and discuss data collection and distribution. ## methods the use alone of the question to automatically find the right answer(s) is not sufficient in the context of a mcqa task. state-of-the-art approaches then require external knowledge to improve system performances @xcite . in our case, we decide to build a two-step retriever-reader architecture comparable to unifiedqa @xcite , where the retriever job is to extract knowledge from an external corpus and using it by the reader to predict the correct answers for each question. figure 1 ## experimental protocol each studied discriminative and generative model is fine-tuned on the mcqa task with frenchmedm-cqa training data using an input sequence composed of a question, its associated options (i.e. possible answers) and its additional context, all separated with a "[sep]" token, e.g. for each question, the context is the text passage with highest confidence rate and can either be obtained using the bm25 algorithm or semantic search as described in section 3.1. concerning the outputs of the systems, we have for the bart generative model a plain text containing the letter of the answers from a to e separated with plus signs in case of the questions with multiple answers, e.g. a + d + e. for the other architectures (i.e. discriminative approaches), we simplify the multi-label problem into a multi-class one by classifying the inputs into one of the 31 existing combinations in the corpus. here, a class may be a combination of multiple labels, e.g. if the correct answers are the a and b ones, then we consider the correct class being ab, which explains the number of 31 classes. ## results as we can see, the best performing model is different according to the used metric. biobert v1.1 reaches best performance using the hamming score and bart-base in the case of the emr. these first observations are quite surprising since both models are trained on english data. while we could expect higher performance with french models (camem-bert for example), the fact that these models are trained on specialized data for one (biobert) and on a model designed for the targeted task (sota on question-answering for bart) finally shows that language models trained on generic data are inefficient for the mcqa task on medical domain. in all considered architectures, context seems to have a small impact on systems performance, with a limited increase or drop depending on the configurations. clearly, the roberta performance is much higher without context (i.e. without the use of the retriever part), while models based on bert generally (8 times on 12) outperform their own baseline performances with external context. the fact that we consider the 1-best passage only may explain this impact. concerning xlm-roberta-base (cross lingual representation), we obtain in the case of the context extracted using bm25 from wikipedia, the worst hamming score and emr out of all the discriminative approaches. this confirms our first observation that a non-specialized model does not allow to achieve the best performance on this task. using bm25 promotes better context than semantic search using minilmv2 on both wikipedia and hal for most of the runs. finally, the source depends of the retriever and model used. a majority of the experiments demonstrate that hal outperforms wikipedia on bm25 despite the fact that the best model was obtained using wikipedia. the scripts to replicate the experiments foot_1 as well as the pre-trained models foot_2 are available online. ## conclusion we proposed in this paper frenchmedmcqa, an original, open and publicly available multiple-choice question answering (mcqa) dataset in the medical field. this is the first french corpus in this domain, including single and multiple answers to questions. several state-the art systems have been evaluated to show current performance on the dataset. the analysis of these first results notably highlighted the fact that language models specialized to the medical domain allow us to reach better performance than generic models, even if these have been trained in a different language (here, english biomedical models applied to french). in future works, we will focus on improving the existing methods for the task of mcqa, considering other strategies for the retriever module (multiple passages, combining contexts...). likewise, we will also consider the construction of data representation models for french specialized for medical domain.