Unsupervised multiple-choice question generation for out-of-domain Q & A fine-tuning (2022)

Abstract: Pre-trained models have shown very good performances on a number of question answering benchmarks especially when fine-tuned on multiple question answering datasets at once. In this work, we propose an approach for generating a fine-tuning dataset thanks to a rule-based algorithm that generates questions and answers from unannotated sentences. We show that the state-of-the-art model UnifiedQA can greatly benefit from such a system on a multiple-choice benchmark about physics, biology and chemistry it has never been trained on. We further show that improved performances may be obtained by selecting the most challenging distractors (wrong answers), with a dedicated ranker based on a pretrained RoBERTa model.

## introduction in the past years, deep learning models have greatly improved their performances on a large range of question answering tasks, especially using pretrained models such as bert @xcite , roberta @xcite and t5 @xcite . more recently, these models have shown even better performances when fine-tuned on multiple question answering datasets at once. such a model is unifiedqa @xcite , which, starting from a t5 model, is trained on a large number of question answering datasets including multiple choices, yes/no, extractive and abstractive question answering. unifiedqa is, at the time of writing, state-of-the-art on a large number of question answering datasets including multiple-choice datasets like openbookqa @xcite or arc @xcite . however, even if uni-fiedqa achieves good results on previously unseen datasets, it often fails to achieve optimal performances on these datasets until it is further finetuned on dedicated human annotated data. this tendency is increased when the target dataset deals with questions about a very specific domain. one solution to this problem would be to finetune or retrain these models with additionnal human annotated data. however, this is expensive both in time and resources. instead, a lot of work has been done lately on automatically generating training data for fine-tuning or even training completely unsupervised models for question answering. one commonly used dataset for unsupervised question answering is the extractive dataset squad @xcite . @xcite proposed a question generation method for squad using an unsupervised neural based translation method. @xcite and @xcite further gave improved unsupervised performances on squad and showed that simple rulebased question generation could be as effective as the previously mentioned neural method. these approches are rarely applied to multiple-choice questions answering in part due to the difficulty of selecting distractors. a few research papers however proposed distractor selection methods for multiple-choice questions using either supervised approaches @xcite or general purpose knowledge bases @xcite . in this paper, we propose an unsupervised process to generate questions, answers and associated distractors in order to fine-tune and improve the performance of the state-of-the-art model unifiedqa on unseen domains. this method, being unsupervised, needs no additional annotated domain specific data requiring only a set of unannotated sentences of the domain of interest from which the questions are created. contrarily to most of the aforementioned works, our aim is not to train a new completely unsupervised model but rather to incorporate new information into an existing stateof-the-art model and thus to take advantage of the question-answering knowledge already learned. we conduct our experiments on the sciq dataset @xcite . sciq contains multiple- question: what type of organism is commonly used in preparation of foods such as cheese and yogurt? (a) mesophilic organisms (b) protozoa (c) gymnosperms (d) viruses support text: mesophiles grow best in moderate temperature, typically between 25 • c and 40 • c (77 • f and 104 • f). mesophiles are often found living in or on the bodies of humans or other animals. the optimal growth temperature of many pathogenic mesophiles is 37 • c (98 • f), the normal human body temperature. mesophilic organisms have important uses in food preparation, including cheese, yogurt, beer and wine. figure 1: example of a question in sciq. the answer in bold is the correct one. ## question generation method we propose a method for generating multiplechoice questions in order to fine-tune and improve unifiedqa. this process is based on 3 steps. first, a set of sentences is being selected (section 2.1) from which a generic question generation system is applied (section 2.2). then a number of distractors are added to each question (section 2.3). ## training and implementation details to refine distractors, we use the "large" version of roberta and all models are trained for 4 epochs and a learning rate of 1 × 10 -5 . these hyperparameters are chosen based on previous experiments with roberta on other multiple-choice datasets. the final unifiedqa fine-tuning is done using the same multiple choices question answering setup as the one used in the original unifiedqa paper @xcite . we use the "large" version of unifiedqa and all the models are trained for 4 epochs using adafactor and a learning rate of 1 × 10 -5 . the learning rate is loosely tuned to get the best performance on the validation set during the supervised training of unifiedqa. we use the hugging face pytorch-transformers @xcite library for model implementation. experiments presented in this paper were carried out using the grid'5000 testbed @xcite , supported by a scientific interest group hosted by inria and including cnrs, renater and several universities as well as other organizations (see https://www.grid5000.fr ). fine-tuning unifiedqa on synthetic questions with random distractors improves the results as compared to the baseline and, as expected, the closer the unlabeled sentences are to the topics of the questions, the better is the accuracy. hence, generating questions from only the train set of sciq gives performances that are comparable but slightly lower to the ones obtained from the combined train, dev and test set of sciq. finally, questions selected from wikipedia also improve the results, despite being loosely related to the target test corpus. our distractor selection method further boosts the accuracy results in all setups. this suggests that a careful selection of distractors is important, and that the hard selection criterion used here seems adequate in our context. ## conclusion in this work, we proposed a multiple-choice question generation method that can be used to fine-tune the state-of-the-art unifiedqa model and improve its performance on an unseen and out of domain dataset. our contributions are: future work includes comparing our method to other question generation methods (including supervised methods: liu et al. (2020), puri et al. (2020) ) in order to assess the effect of both the generation method and the questions quality on the final performances of our models. also, we will further compare different variations of our question generation and distractor refining methods in order to more thoroughly understand the effect of hyper-parameters such as the number of candidate distractors.