Regularization of Distinct Strategies for Unsupervised Question Generation (2020)

Abstract: Unsupervised question answering (UQA) has been proposed to avoid the high cost of creating high-quality datasets for QA. One approach to UQA is to train a QA model with questions generated automatically. However, the generated questions are either too similar to a word sequence in the context or too drifted from the semantics of the context, thereby making it difficult to train a robust QA model. We propose a novel regularization method based on teacher-student architecture to avoid bias toward a particular question generation strategy and modulate the process of generating individual words when a question is generated. Our experiments demonstrate that we have achieved the goal of generating higher-quality questions for UQA across diverse QA datasets and tasks. We also show that this method can be useful for creating a QA model with few-shot learning.

## introduction machine reading for question answering (mrqa) is the task of answering questions from a context that contains the answer. this field has seen remarkable progress in recent years, with qa models outperforming humans on a question answering (qa) benchmark like squad @xcite . training a qa model requires a large amount of data, and constructing such a dataset is usually laborious or sometimes even impossible for some domains and languages. because of this, @xcite explored unsupervised question answering (uqa), a setting where manually constructed triples, (context, question, answer), are not available for training. they approached the problem with unsupervised question generation * equal contribution. ## preliminary study in order to ensure that using the two somewhat conflicting types of questions can result in a qa improvement, we set out to run a preliminary experiment. we simply combine the two datasets, copy-type and lm-type (question creation details are provided in section 4.1) to train a qg model to see if the regularization idea would be beneficial at all. we posit that a positive result would not only justify a more sophisticated regularization process but also serve as a baseline for the experiments. since the dataset used for training the qg model contains two types of questions, we use the following batch loss: ## proposed approach in an unsupervised setting, it is difficult to solve the problems related to the lm-type and copy-type qg models, primarily because the gold standard questions are not provided. however, the preliminary study (section 2) shows there is evidence that using two qg models, lm-type and copy-type, in an interleaving way has the potential to solve the problems of each model. instead of simply merging two datasets (as in the preliminary study), we propose a finer-grained approach of using token-level regularization rather than instance-level that selects full questions. figure 2 illustrates the difference. we posit that generated questions should not be easily classified into either lm-type or copy-type if the two styles were to be inter-mixed in a balanced way. in other words, the style of the questions should not be highly biased toward either side. in short, our unsupervised qg problem is reduced to devise a token-level regularization method that generates questions indistinguishable between the two question types and eventually mitigates their drawbacks. ## experiments the primary goal of our experiments is to demonstrate that the method of regularizing copy-type and lm-type questions yields questions that overcome the drawbacks of the two types. by measuring the qa performance on several qa datasets, we establish the generalizability of the proposed approach. dataset creation all our models use contexts from the dataset of @xcite foot_2 . in the case of the copy-type dataset, it also uses the questions and answers from the original dataset to train a qg model to create the copy-type questions. on the other hand, the qg-type dataset uses stanza @xcite to extract named-entities to obtain answers and gpt-2 to create the questions. figure 4 illustrates the process of lm-type question generation. since gpt-2 is not optimized to generate questions @xcite , we add a question filter to eliminate potentially unanswerable questions as a way to maximize its qa performance. the filter accepts a (c, a, q) iff the question generated by gpt-2 is answerable by itself. a further explanation is in appendix a.1. all the generated datasets, i.e. intermediate datasets to reproduce lm-type, copy-type, teacher and student qgs, and the dataset to train the final qa model, are publicly available in our github repository. ## conclusion and future work we have proposed a novel method for unsupervised question generation (qg), where the questions are generated with a teacher-student architecture. the teacher is composed of two distinctive qg models as assistants and a regularization module that attempts to stay unbiased between the two styles of qg. as the main thrust of this structure, the regularization scheme takes a novel approach of selecting the next tokens in a probabilistic way when a question is generated. this knowledge is implicitly transferred to the student module so that it can mitigate the drawbacks of the two qg models in the teacher module and generate higher quality questions. to encourage further development of unsupervised question answering, we release the qa dataset generated by our student model. with a series of experiments across the indomain mrqa shared tasks, we demonstrate the effectiveness of the proposed method as well as its generalizability. we also provide an insight as to how the proposed method can help progress toward zero-shot and few-shot learning. as reflected in the qualitative analysis, the current method still generates unreasonable questions that, for example, deviate too much from the context, end unnaturally, and fail to handle pronouns appropriately. to handle those cases, we need to look into what other types of qg can serve as new teachers and how the regularization needs to evolve. we leave it as future work in addition to the generalization of this approach to other generative models besides qg.