Enhancing Human Summaries for Question-Answer Generation in Education (2023)

Abstract: We address the problem of generating high-quality question-answer pairs for educational materials. Previous work on this problem showed that using summaries as input improves the quality of question generation (QG) over original textbook text and that human-written summaries result in higher quality QG than automatic summaries. In this paper, a) we show that advances in Large Language Models (LLMs) are not yet sufficient to generate quality summaries for QG and b) we introduce a new methodology for enhancing bullet point student notes into fully fledged summaries and find that our methodology yields higher quality QG. We conducted a large-scale human annotation study of generated question-answer pairs for the evaluation of our methodology. In order to aid in future research, we release a new dataset of 9.2K human annotations of generated questions.

## introduction automated generation of question-answer pairs for education can be used to assist students with selfguided reviews of educational materials or to support instructors with the creation of assessment materials. a key challenge for these question generation (qg) models is to ensure the relevancy of generated questions. most human evaluation of qg models often emphasizes the grammaticality and fluency of the generated questions, rather than their relevance @xcite . for educational applications, this shortcoming is critical. a recent study by @xcite showed that providing qg models with human-written summaries as input, instead of original textbook text, increases question relevance, acceptability, and interpretability. the study also demonstrated that using automatically generated summaries as input improved qg quality over original textbook input, but not as much as human-written summaries. we investigate whether advances to large language models (llms) like gpt-3 have closed this gap and introduce a novel methodology for generating summaries using student notes in the form of bullet points as input. the main contributions of our research are: ## related work prior work in question generation has focused primarily on using sequence-to-sequence models to generate questions from a given context passage. these methods can either be answer-aware (i.e., an answer span is given to the model, along with the passage) or answer-agnostic (i.e., just the context passage is given). our work focuses on the latter case, in which the model has the much more challenging task of generating the answer as well as the question. @xcite accomplished this by decomposing the generation process into two stages: answer-phrase extraction and answer-aware qg. follow-up work from sun et al. (2018) introduced a position-aware component to localize answers in the input context. work by wang et al. (2020) added joint training between the two stages of the pipeline. other work has found that transforming the input context passage can aid in answer-phrase extraction. lewis et al. (2021) filtered out passages that are unlikely to contain answers to human-written questions. qu et al. (2021) generated coarse keyphrases from input passages to help guide the answer extraction model. zhao et al. (2022) used an "event-centric" summarizer to generate a sequence of events, allowing them to ask better questions. more recently, @xcite showed that providing answer extraction models with humanwritten or lm-generated summaries significantly improved the relevance and interpretability of generated questions. we build on this insight and further investigate the gap in question quality between human-written summaries and lm-generated summaries. dugan et al. used a bart model @xcite for automatic summarization. however, recent work suggests that summaries generated by large language models such as gpt-3 are overwhelmingly preferred by human annotators @xcite . in what follows, we report the results of experiments that we conducted to evaluate whether large language models can, indeed, generate quality summaries for the task of generating question-answer pairs for educational materials. step -description 1. zero-shot -we generated new summaries from the human-written bullet style summaries with gpt-3 using the following prompt: "here's an outline, please expand it into full sentences and paragraphs: {human-written summary in bullet style with incomplete sentences}" 2. few-shot -we reviewed 10 examples by fact-checking and removing repeated phrases. we added these examples to the prompt and then generated 100 more summaries out of the hand-written summaries in bullet style. ## methodology as mentioned earlier, the central goal of this study is to address the problem of providing quality summaries of educational materials to qa models in order to generate important and relevant qa pairs. to investigate this problem, we ran two groups of experiments. first, we evaluated if gpt-3 generates better qa pairs than t5 which was used for the same task in prior work. in the second group of experiments, we investigated the impact of different types of input on the quality of the generation of qa pairs in addition to different ways of obtaining summaries. to this end, we collected summaries written by college students on course textbooks and classified them into two major categories: fully-fledged summaries and bullet-point summaries. fully-fledged summaries consisted of complete grammatical sentences that formed a coherent paragraph. bullet-point summaries consisted of bullet points or other fragments taken as short notes. in addition to these two types of input generated by humans, we introduced and compared a new method of generating summaries from bullet-point notes, which we call bull2sum (from bullets to summaries). bull2sum takes as input bullet-point summaries and rewrites them into fullyfledged summaries. ## experiments we compare the performance of two llms trained to do qg in 5 text input conditions. so, we ran a total of 10 experiments. each condition is a different type of input to the model, including a condition with summaries generated by a new model that we fine-tuned, bull2sum, which rewrites bullet points or short notes into fully-fledged sentences. we describe this model in section 3.3. ## evaluation we performed a human evaluation study to measure the qg performance of the models gpt-3 and t5 under our 5 different input conditions, as described in section 4. we had a total of 66 annotators, all university students enrolled in an advanced computer science course titled artificial intelligence. prior to the annotations, students signed a consent form to participate in the experiment and were rewarded with extra credit for their participation. moreover, we had a training session with the students to review the guidelines and demo the annotation tool. we employed the evaluation guidelines defined in @xcite . for each generated qa pair, the annotators evaluated the following criteria: our team created a web-based tool (as illustrated in appendix figure 2 ) in order to increase the scalability and ease of annotations. we randomly selected 10 qa pairs generated from each of our 5 input conditions by both the t5 and gpt-3 models. we divided our 66 annotators into groups of 3, for a total of 22 groups. each group would annotate the same group of questions generated by the different models for the same data. given that we had 22 groups of annotators, we collected 3,080 question-answer (qa) pairs annotated, i.e., 220 qa pairs annotated per input condition. we computed pairwise inter-annotator agreement (iaa) analysis using fleiss's multi-π method @xcite for finding the agreement for more than two coders and found iaa rates between 0.39-0.44 for our 5 evaluation criteria. we report the results in ## results and discussion our experiments show that although gpt-3 performs better than t5 in qg, it is not sufficient to improve a) the quality of qa pairs and b) the quality of the automated summaries as input. carefully written human summaries are still better than automatic summaries generated by gpt-3. however, our novel method of rewriting short bullet-point notes into summaries can be effectively used to generate quality qa pairs. ## limitations in this work, we explored question generation for computer science textbooks. we have not yet explored a broader range of course subjects, and it may be that the prevalence of computer science knowledge on the internet, including through forums like stack exchange, makes qg easier for this discipline than for others. furthermore, we examine a relatively narrow range of question types. other questions -like multiple choice questions, or compare and contrast questions-will require deeper exploration and substantial adaptation of the methodology that we proposed. ## ethics statement potential risks : as with all large language models, the models used in our research have the potential to generate factually incorrect information. this is a potential risk given that our intended application is for education. as reported in our paper, our best-performing models produce acceptable quality flashcard questions only 70% of the time. the remaining 30% is significant enough that manual review by course is necessary before questions are deployed to students. intended use : our models and methods shown here are for research purposes only. they should not be deployed in the real world as solutions without further evaluation. potential applications : bull2sum could be utilized in the field of education to convert course slides into summaries, which can then be used to generate pertinent and significant questions for the course. this application could enhance and facilitate students' exam preparation. type of input to qg model gpt-3 qa t5 qa original text from textbook q: are programs currently capable of doing all of these things? a: programs are currently capable of doing some of these things, but not all of them. q: what is a century-old technology that can make someone fall in love with it? a: teddy bear gpt-3 generated summary from text q: what is the argument from disability? a: the argument from disability claims that machines can never do x, where x is a characteristic of humans. q: what does the "argument from disability" claim machines can never do? a: x higher quality humanwritten summary q: how does a perceptron compute the and logical operation? a: a perceptron computes the and logical operation by adding the activation values of the multiple layers of units in the perceptron. q: perceptron can easily compute what? a: and and or poorer quality humanwritten summary q: what does a perceptron not have? a: a perceptron does not have a non-linear activation function. q: what is a simple neural unithas binary output -does not have a non-linear activation function? a: perceptron summary expanded into full sentences and paragraphs by zero shot q: what is the xor problem? a: the xor problem is a problem in which two inputs cannot be separated by a linear function. q: what problem is not linearly separable? a: xor few-shot generated summary q: what is the xor problem not linearly separable? a: the xor problem is not linearly separable. q: a network formed by many layers of what can always be reduced to a single layer of linear units with appropriate weights? a: purely linear units summary generated with our fine-tuned model q: what is a perceptron? a: a perceptron is a simple neural unit that has binary output and does not have a non-linear activation function. q: a perceptron that can solve the xor problem is said to be what? a: bi-dimensional