EXAMS : A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering (2020)

Abstract: We propose EXAMS – a new benchmark dataset for cross-lingual and multilingual question answering for high school examinations. We collected more than 24,000 high-quality high school exam questions in 16 languages, covering 8 language families and 24 school subjects from Natural Sciences and Social Sciences, among others.EXAMS offers unique fine-grained evaluation framework across multiple languages and subjects, which allows precise analysis and comparison of the proposed models. We perform various experiments with existing top-performing multilingual pre-trained models and show that EXAMS offers multiple challenges that require multilingual knowledge and reasoning in multiple domains. We hope that EXAMS will enable researchers to explore challenging reasoning and knowledge transfer methods and pre-trained models for school question answering in various languages which was not possible by now. The data, code, pre-trained models, and evaluation are available at http://github.com/mhardalov/exams-qa .

## introduction research on science question answering has attracted a lot of attention in recent years @xcite @xcite . such questions are challenging as they require domain and common sense knowledge @xcite , as well as complex reasoning and different forms of inference over a variety of knowledge sources @xcite . indeed, a combination of these was required to achieve noticeable performance gains @xcite . this inevitably made research in schoollevel science question answering (qa) hard for languages other than english due to the scarceness of resources @xcite @xcite . ## eχαµs dataset we introduce eχαµs, a new benchmark dataset for multilingual and cross-lingual question answering from high school examinations. in this section, we present the properties of the dataset, and we give details about the process of data collection, preparation and normalization, as well as information about the data splits, and the parallel questions. ## baseline models we divide our baselines into the following two categories: (i) models without additional training, and (ii) fine-tuned models. the first group contains common baselines, i.e., random guessing and information retrieval solver @xcite . in addition, we evaluate the knowledge contained in the pre-trained language model, i.e., mbert @xcite and xlm-r @xcite , and we use it as an answering mechanism. the second group of baselines compare the learning ability of state-of-the-art multilingual models on the task of multiple-choice question answering. since we have multi-choice questions, we adopt accuracy as an evaluation measure, as this is standard for this setup. ## experiments and results in this section, we evaluate the performance of the baseline models described in section 4 on the eχαµs dataset. in ## discussion our results show that initial fine-tuning on a large monolingual out-of-domain multi-choice machine reading comprehension dataset (race @xcite ) performs much better than no training baselines for answering multilingual eχαµs questions. moreover, additional training on english science qa in lower school levels has no significant effect on the overall accuracy. these results suggest that further investigation of finetuning with other multilingual datasets @xcite @xcite @xcite @xcite is needed in order to understand the domain transfer benefits to science qa in eχαµs, even if they are not in a multi-choice setting @xcite . using domain-adaptive and task-adaptive pre-training @xcite to the multilingual science qa might offer further potential benefits. moreover, we need a better knowledge context for a given question-choice pair (the last row in ## conclusion and future work we presented eχαµs, a new challenging crosslingual and multilingual benchmark for science qa in 16 languages and 24 subjects from high school examinations. we further proposed new fine-grained evaluation that allows precise comparison across different languages and school subjects. we performed various experiments and analysis with pre-trained multilingual models (xlm-r, mbert), and we demonstrated that there is a need for better reasoning and knowledge transfer in order to solve some of the questions from eχαµs. we hope that our publicly available data and code will enable work on multilingual models that can reason about question answering in the challenging science domain. in future work, we plan to extend the dataset with more questions, more subjects, and more languages. we further plan to develop new models to address the specific challenges we identified. with these parameters alone, the models did not perform well, and thus we added a warmup of 0.1 and a weight decay of 0.06, which stabilized the training. in all experiments, we used the adam optimizer with β 1 =0.9, β 2 =0.999, @xmath0. we further performed manual tuning of the hyper-parameters: we experimented with variations thereof, depending on the performance on the corresponding development sets, and we ended up with the values in fine-tuning xlm-r (550m parameters) on nvidia quadro rtx 6000 (24gb) with the given hyper-parameters took around three hours per epoch when fine-tuned on race (∼85k examples), 30 minutes per epoch when finetuned on sciens (∼9k examples), and 30 minutes on eχαµs on train m ul (∼8k examples). fine-tuning xlm-r base (270m parameters) and mbert (172m parameters) on nvidia gtx 1080 ti (pascal, 11gb memory) with the given hyperparameters took roughly 2 to 2.5 hours per epoch when fine-tuned on race (∼85k examples), 30 to 35 minutes per epoch when fine-tuned on sciens (∼9k examples), and additional 30 minutes on the eχαµs train m ul (∼8k examples).