Generating Answer Candidates for Quizzes and Answer-Aware Question Generators (2021)

Abstract: In education, quiz questions have become an important tool for assessing the knowledge of students. Yet, manually preparing such questions is a tedious task, and thus automatic question generation has been proposed as a possible alternative. So far, the vast majority of research has focused on generating the question text, relying on question answering datasets with readily picked answers, and the problem of how to come up with answer candidates in the first place has been largely ignored. Here, we aim to bridge this gap. In particular, we propose a model that can generate a specified number of answer candidates for a given passage of text, which can then be used by instructors to write questions manually or can be passed as an input to automatic answer-aware question generators. Our experiments show that our proposed answer candidate generation model outperforms several baselines.

## introduction testing with quiz questions can help both learning and retention, e.g., it could be used for self-study or as a way to detect knowledge gaps in a classroom setting, thus allowing instructors to adapt their teaching @xcite . as creating such quiz questions is a tedious job, automatic methods have been proposed. the task is often formulated as an answer-aware question generation @xcite @xcite @xcite @xcite : given an input text and a target answer, generate a corresponding question. context: one of the most famous people born in warsaw was maria skłodowska-curie, who achieved international recognition for her research on radioactivity and was the first female recipient of the nobel prize. famous musicians include władysław szpilman and frédéric chopin. though chopin was born in the village of żelazowa wola, about 60 km (37 mi) from warsaw, he moved to the city with his family when he was seven months old. casimir pulaski, a polish general and hero of the american revolutionary war, was born here in 1745. q1: what was maria curie the first female recipient of? a1: nobel prize q2: what year was casimir pulaski born in warsaw? a2: 1745 q3: how old was chopin when he moved to warsaw with his family? a3: seven months old many researchers have used the stanford question answering dataset (squad1.1) dataset @xcite as a source of training and testing data for answer-aware question generation. it contains human-generated questions and answers about articles in wikipedia, as shown in figure 1 . however, this formulation requires that answers be picked beforehand, which may not be practical for real-world situations. here we aim to address this limitation by proposing a method for generating answers, which can in turn serve as an input to answer-aware question generation models. our model combines orthographic, lexical, syntactic, and semantic information, and shows promising results. it further allows the user to specify the number of answer to propose. our contributions can be summarized as follows: ## related work the success of large-scale pre-trained transformers such as bert @xcite , roberta @xcite , albert @xcite , and generative ones such as t5 @xcite or bart @xcite , has led to the rise in popularity of the question generation task. models such as bert @xcite , t5 @xcite and pegasus @xcite have been used to generate questions for the squad1.1 dataset and have been commonly evaluated @xcite using bleu @xcite , rouge @xcite , and meteor @xcite . strong models for this task include nqg++ @xcite , prophetnet @xcite , mpqg @xcite , unilm @xcite , unilmv2 @xcite , and ernie-gen @xcite . all these models were trained for answer-aware question generation, which takes the answer and the textual context as an input and outputs a question for that answer. in contrast, our task formulation takes a textual context as an input and generates possible answers; in turn, these answers can be used as an input to the above answer-aware question generation models. the quiz-style question generation for news stories task @xcite uses a formulation that asks to generate a single question as well as the corresponding answer, which is to be extracted from the given context. follow-up research has tried to avoid the limitation of generating a single question or a single question-answer pair by generating a question for each sentence in the input context or by using all named entities in the context as answer keys (montgomerie, 2020). finally, there has been a proliferation of educational datasets in recent years @xcite @xcite , which includes crowdsourcing @xcite , arc @xcite , openbookqa @xcite , multiple-choice exams in bulgarian @xcite , vietnamese @xcite , and exams, which covers 16 different languages @xcite ). yet, these datasets are not directly applicable for our task as their questions do not expect the answers to be exact matches from the textual context. while there are also span-based extraction datasets such as newsqa @xcite , searchqa @xcite and natural questions: a benchmark for question answering research @xcite they contains a mix of long and short spans rather than factoid answers. thus, we opted to use squad1.1 in our experiments, but focusing on generating answers rather than on questions. ## method given an input textual context, we first extract phrases from it, then we calculate a representation for each phrase, and finally, we predict which phrases are appropriate of being an answer to a quiz question based on these representations. ## experiments and evaluation we used our model to generate ten candidate answers per passage (taking the ones with the highest classifier confidence), and we compared the results to other commonly used methods. ## discussion figure 2 shows a passage from the development split of the squad1.1 dataset and the top-10 answers that our model proposed for it. we can see that these answers represent a diverse set, including named entities, noun chunks, and individual words. indeed, this is a typical example, as our analysis across the entire development dataset shows that on average, among the top-10 candidates, our model proposes 4.82 named entities and 6.40 noun chunks. note also that our evaluation setup could be unfair to the model in some cases, e.g., if the model proposes a good candidate answer but one that was not chosen by the human annotators, it would receive no credit for it. finally, note that our model can produce top-n results for user-defined values of n , which is an advantage over simple baselines based on entities or chunks, as well as over our setup for t5. ## conclusion and future work we proposed a new task: generate answer candidates that can serve as an input to answer-aware question generation models. we further created a dataset for this new task. moreover, we proposed a suitable model for the task, which combines orthographic, lexical, syntactic, and semantic information, and can generate a pre-specified number of answers. finally, we demonstrated improvements over simple approaches based on named entities, and competitiveness over complex, computationally expensive neural network models such as t5.