MLEC-QA : A C hinese M ulti- C hoice B iomedical Q uestion A nswering D ataset (2021)

Abstract: Question Answering (QA) has been successfully applied in scenarios of human-computer interaction such as chatbots and search engines. However, for the specific biomedical domain, QA systems are still immature due to expert-annotated datasets being limited by category and scale. In this paper, we present MLEC-QA, the largest-scale Chinese multi-choice biomedical QA dataset, collected from the National Medical Licensing Examination in China. The dataset is composed of five subsets with 136,236 biomedical multi-choice questions with extra materials (images or tables) annotated by human experts, and first covers the following biomedical sub-fields: Clinic, Stomatology, Public Health, Traditional Chinese Medicine, and Traditional Chinese Medicine Combined with Western Medicine. We implement eight representative control methods and open-domain QA methods as baselines. Experimental results demonstrate that even the current best model can only achieve accuracies between 40% to 55% on five subsets, especially performing poorly on questions that require sophisticated reasoning ability. We hope the release of the MLEC-QA dataset can serve as a valuable resource for research and evaluation in open-domain QA, and also make advances for biomedical QA systems.

## introduction as a branch of the qa task, biomedical question answering (bqa) enables effectively perceiving, accessing, and understanding complex biomedical knowledge by innovative applications, which makes bqa an important qa application in the biomedical domain @xcite . such a task has recently attracted considerable attention from the nlp community @xcite @xcite , but is still confronted with the following three key challenges: to push forward the variety of bqa datasets, we present mlec-qa, the largest-scale chinese multi-choice bqa dataset. questions in mlec-qa are collected from the national medical licensing examination in china (nmlec) 2 , which are carefully designed by human experts to evaluate professional knowledge and skills for those who want to be medical practitioners in china. the nmlec has a total number of 24 categories of exams, but only five of them have the written exams in chinese. every year, only around 18-22% of applicants can pass one of these exams, showing the complexity and difficulty of passing them even for skilled humans. there are three main properties of mlec-qa: (1) mlec-qa is the largest-scale chinese multichoice bqa dataset, containing 136,236 questions with extra materials (images or tables), question type example ( * represents the correct answer) a1 (public health) what should be used to compare the results of two samples? a. t test b. x 2 test c. µ test d. f test * e. rank sum test b1 (chinese western medicine) a. heart and liver b. spleen and lungs c. liver and kidneys d. heart and kidneys e. spleen and kidneys 1. what is the viscera that "yikui" in yikui homologous refers to? (e) 2. what is the viscera that "water and fire" in harmonization of water and fire refers to? (d) a2 (clinic) primipara, 29 years old, at 37 weeks of gestation. had jet vomiting once this morning, suddenly convulsed an hour ago and then went to hospital in a coma. physical examination: bp 180/120mmhg, urine protein (+++). the most likely diagnosis of this patient is: * a. eclampsia b. hematencephalon c. hysteria d. epilepsy e. cerebral thrombosis a3 (traditional chinese medicine) female, 28 years old. in the recent month, has oral ulcer attacks repeatedly, upset, difficulties to sleep at night, dry stool, defecates every 1-2 days, has dry mouth, does not like drinking water, and has yellow urine, red tongue, greasy fur, and rapid pulse. 1. the drug of choice for the treatment of the pattern is: a. mirabilite b. arctium lappa * c. bamboo leaf d. baikal skullcap e. gypsum 2. the appropriate compatible drug for the treatment of the disease is: a. angelica dahurica b. cassia twig c. rhizoma zingiberis * d. rheum officinale e. ash bark 3. which of the following drugs should be used with caution during menstruation? a. semen sojae praeparatum * b. rheum officinale c. coptis chinensis d. lophatherum gracile e. rhizoma phragmitis a4 (stomatology) a 50-year-old patient comes to the outpatient clinic 2 years after the end of radiotherapy for nasopharyngeal carcinoma outside hospital. examination: full mouth multiple teeth dental surfaces with different degrees of caries, some of the affected teeth have become residual crowns, residual roots, less intraoral saliva, more soft scaling of the dental surfaces and sulci. 1. the diagnosis of this patient is: a. acute caries * b. rampant caries c. chronic caries d. secondary caries e. smooth surface caries 2. there are several treatment designs as follows, except: a. design treatment of full mouth caries b. endodontic treatment of teeth with hypodontia * c. filling metal material d. remineralization adjunctive therapy e. regular review in summary, the major contributions of this paper are threefold: open-domain bqa the text retrieval conference (trec) @xcite has triggered the open-domain bqa research. at the time, most traditional bqa systems were employing complex pipelines with question processing, document/passage retrieval, and answer processing modules. examples of such systems include epocare @xcite , medqa @xcite @xcite and askhermes @xcite . with the introduction of various bqa datasets that are focused on specific biomedical topics, such as bioasq @xcite , emrqa @xcite and pubmedqa @xcite , pioneered by @xcite , the modern open-domain bqa systems largely simplified the traditional bqa pipeline to a two-stage retriever-reader framework by combining information retrieval and machine reading comprehension models @xcite . moreover, the extensive use of medical images (e.g., ct) and tables (e.g., laboratory examination) has improved results in real-world clinical scenarios, making the bqa a task lying at the intersection of computer vision (cv) and nlp. however, most bqa models focus on either texts or images @xcite @xcite ; as a result, bqa datasets that are composed by fusing different biomedical resources are relatively limited.