A Feasibility Study of Answer-Agnostic Question Generation for Education (2022)

Abstract: We conduct a feasibility study into the applicability of answer-agnostic question generation models to textbook passages. We show that a significant portion of errors in such systems arise from asking irrelevant or un-interpretable questions and that such errors can be ameliorated by providing summarized input. We find that giving these models human-written summaries instead of the original text results in a significant increase in acceptability of generated questions (33% \rightarrow 83%) as determined by expert annotators. We also find that, in the absence of human-written summaries, automatic summarization can serve as a good middle ground.

## introduction writing good questions that target salient concepts is difficult and time consuming. automatic question generation (qg) is a powerful tool that could be used to significantly lessen the amount of time it takes to write such questions. a qg system that automatically generates relevant questions from textbooks would help professors write quizzes faster and help students stay engaged when reviewing course material. previous work on qg has focused primarily on answer-aware qg models. these models require the explicit selection of an answer span in the input context, typically through the usage of highlight tokens. this adds significant overhead to the question generation process and is undesirable in cases where clear lists of salient key terms are unavailable. we conduct a feasibility study 1 on the application of answer-agnostic question generation models (ones which do not require manual selection of answer spans) to an educational context. our contributions are as follows: ## related work & background early attempts to use qg for educational applications involved generating gap-fill or "cloze" questions foot_0 @xcite from textbooks @xcite . this procedure has been shown to be effective in classroom settings @xcite ) and students' scores on this style of generated question correlate positively with their scores on human-written questions @xcite . however, there are many situations where gap-fill questions are not effective, as they are only able to ask about specific unambiguous key terms. in recent years, with the advent of large crowdsourced datasets for extractive question answering (qa) such as squad @xcite , neural models have become the primary methods of choice for generating traditional interrogative style questions @xcite . a common task formulation for neural qg is to phrase the task as answer-aware, that is, given a context passage @xmath0, ..., c n } and an answer span within this context @xmath1, ..., c k+l }, train a model to maximize p (q|a, c) where @xmath2, ..., q m } are the tokens in the question. these models are typically evaluated using n-gram overlap metrics such as bleu/rouge/meteor @xcite @xcite with the reference being the original human-authored question as provided by the extractive qa dataset. the feasibility of using answer-aware neural qg in an educational setting was investigated by @xcite , who used a bilstm encoder @xcite to encode c and a and a unidirectional lstm decoder to generate q. they trained on the squad dataset @xcite and evaluated on textbooks from various domains (history, biology, etc.). they showed that generated questions were largely grammatical, relevant, and had high n-gram overlap with human-authored questions. however, given that we may not always have a list of key terms to use as answer spans for an input passage, there is a desire to move past answeraware qg models and evaluate the feasibility of answer-agnostic models for use in education. shifting to answer-agnostic models creates new challenges. as @xcite claims, the task of deciding what is and is not important is, itself, an important task. without manually selected answer spans to guide it, an answer-agnostic model must itself decide what is and is not important enough to ask a question about. this is typically done by separately modeling p (a|c), i.e., which spans in the input context are most likely to be used as answer targets for questions. the extracted answer spans are then given to an answer-aware qg model p (q|a, c). this modeling choice allows for more controllable qg and more direct modeling of term salience. ## methodology to perform answer-agnostic qg, we follow work done by @xcite and @xcite who show that language models, when fine-tuned for both qa and qg, perform better than models tuned for only one of those tasks. we assume that answer extraction will aid both qa and qg and thus use a model that was fine-tuned on all three. we considered using unilm @xcite or prophetnet @xcite but ultimately chose a t5 language model @xcite on squad due to the clean separation between tasks afforded by t5's task-specific prefixes such as "generate question:" and "extract answer:". 3 the three fine-tuning tasks that were used to train the model we used are illustrated in figure 2 . for question generation, the model is trained to perform answer-aware question generation by modeling p (q|a, c). for question answering, the model is trained to perform extractive qa by modeling p (a|c, q). finally, for answer extraction, instead of modeling p (a|c), the model is trained to model p (a|c ) with @xmath3, ..., c s , ..., c e , ..., c n+2 } where c s and c e are highlight tokens that denote the start and end of the sentence within which we want to extract an answer span. to generate questions, we iteratively highlight the start and end of each sentence in a given passage and extract at most one answer span per sentence. 4 we then generate one question per extracted answer span using the same model in an answer-aware fashion. passages longer than 512 tokens are split such that no sentences are divided between subpassages and all sub-passages have a roughly equal number of sentences. ## experiments our first experiment evaluates the performance of the model on the original text extracted from jurafsky and martin (2020)'s textbook "speech and language processing 3rd edition." 5 to ensure proper comparison, we manually extracted the text from our three chapters of interest (chapters 2, 3, and 4). when extracting text, all figures, tables, and equations were omitted and all references to them were either replaced with appropriate parentheti-cal citations or removed when possible. in total, we generated 1208 question-answer pairs from the original text. our second experiment evaluates the performance of the model on human-written summaries. we recruited three research assistants (ras) as part of an undergraduate research experience to write abstractive summaries for each subsection of the same three chapters of the textbook. 6 they were encouraged to make their summaries easily readable by humans rather than to be easily understandable by machines but otherwise no specific guidelines were given. we report some statistics about these summaries in our final experiment evaluates the performance of the model on automatically generated summaries. to perform this automatic summarization we used a bart @xcite ) language model which was fine-tuned for summarization on the cnn/dailymail dataset @xcite . 7 the same chunking procedure as described in section 3 was performed on input passages that were larger than 512 tokens. the summarized output sub-passages were then concatenated together before running question generation. in total, we generated 318 question-answer pairs from our automatic summaries. ## evaluation for evaluation, we randomly sampled 100 questionanswer pairs from each of the three experiments to construct our evaluation set of 300 questions. we tasked the same set of ras to evaluate the quality of the question-answer pairs. all 300 pairs were given to all three annotators. we asked the following yes/no questions: @xcite . ( correct ) is the answer correct? we provided many example annotations to our annotators and wrote clear guidelines about each cat- in figure 3 we report the results of our evaluation across the three sources. we note that a majority of observed errors in the original text questions stem from them being either irrelevant or uninterpretable out of context. we also see that generating questions directly from human-written summaries significantly improves relevance and interpretability, resulting in over 80% being labeled as acceptable by annotators. finally, in the case of automatic summaries, we see that relevance and interpretability are improved as compared to the original text questions while grammaticality suffers. in ## conclusion and future work in this work we show that answer-agnostic qg models have difficulty both choosing relevant topics to ask about and generating questions that are interpretable out of context. we show that asking questions on summarized text ameliorates this in large part and that these gains can be approximated by the use of automatic summarization. future work should seek to further explore the relationship between summarization and qg. work done concurrently to ours by @xcite already has promising results in this direction, showing that training a qg model on synthetic data from summarized text improves performance on downstream qa. additionally, future work should focus on further refining and standardizing the metrics used for both automatic and human evaluation of qg. as noted by @xcite n-gram overlap metrics correlate poorly with in-context interpretability and evaluation on downstream qa fails to address the relevance of generated questions. would you directly use this question as a flashcard? (yes / no): a yes answer to this question means that the generated question is salient, grammatically correct, non-awkwardly phrased and has one correct answer. if you answer yes to this question you may skip the rest of the annotation for the given example -the answers for all other questions are assumed to be yes. if you answer no, then please continue on to the rest of the questions. importantly, if you *did* answer yes to all of the other questions, do not feel pressured to answer yes to this question. there are many reasons why you might not want to directly use a question as a flashcard (too easy, too general, etc.) that are not enumerated here. is this question grammatically correct? (yes / no): a yes answer to this question implies that a question has no grammatical errors. awkwardly worded questions that are grammatical should be annotated as such (answer yes for these questions). does this question make sense out of context? (yes / no): this question asks if there are any references made by the question to other items that have been "previously discussed". for our use case, questions should never refer to other specific items in the text from which they were drawn. a yes answer to this implies that the question is interpretable when taken on its own and is a question that someone would ask if there was no pre-existing context. is this question relevant? (yes / no): a yes answer to this question implies that the question being asked is important for understanding the main points that the chapter (and by extension the book) is attempting to teach. questions that are relevant should be ones that would plausibly be asked on a quiz or a test from a fairly thorough course on computational linguistics. questions that are about insignificant details or questions that are about specific illustrated examples that are not useful for understanding the main points of the chapter should be given a no. anything that is relevant (or tangentially relevant) to computational linguistics should be given a yes. is the answer to the question correct? (yes / no): a yes answer to this question implies that the answer given is one of a multitude of plausible correct answers to the question. if the question has multiple correct answers and the given answer is one of them, it should be annotated as a yes. if the question is bad/ungrammatical or underspecified to such an extent that you cannot judge the answer properly, you should annotate yes. however, irrelevant questions that are grammatical and reasonably interpretable should be annotated properly.