Comparing Neural Question Generation Architectures for Reading Comprehension (2023)

Abstract: In recent decades, there has been a significant push to leverage technology to aid both teachers and students in the classroom. Language processing advancements have been harnessed to provide better tutoring services, automated feedback to teachers, improved peer-to-peer feedback mechanisms, and measures of student comprehension for reading. Automated question generation systems have the potential to significantly reduce teachers’ workload in the latter. In this paper, we compare three differ- ent neural architectures for question generation across two types of reading material: narratives and textbooks. For each architecture, we explore the benefits of including question attributes in the input representation. Our models show that a T5 architecture has the best overall performance, with a RougeL score of 0.536 on a narrative corpus and 0.316 on a textbook corpus. We break down the results by attribute and discover that the attribute can improve the quality of some types of generated questions, including Action and Character, but this is not true for all models.

## introduction the task of automated question generation (aqg) has been proven to have significant potential for reducing teacher workload while effectively assessing reading comprehension for students @xcite . reading comprehension is indicative of a student's understanding of a subject, making it a critical metric for ensuring their future academic success. originally, advancements in question generation were isolated to broad question answering datasets including squad @xcite and narrativeqa @xcite . in recent years, aqg models pre-trained on these datasets have been applied to education-specific corpora to help generate questions that are more useful in the classroom setting. the domain shift from generic corpora to education-specific datasets is critical to model the unique characteristics of classroom discourse, but there is still much room for improvement. classroom texts vary greatly in terms of the age of the students, the subject material, and their discourse structure. prior work in question generation for education has focused on how language models perform on a single corpus with a single subject @xcite , but not on how these models perform across different subjects. this research also considers how different discourse representations for a particular neural architecture can improve the quality of generated questions as opposed to evaluating multiple systems. here, we analyze how different neural models perform on two corpora: the fairytaleqa corpus @xcite and the textbook question answering @xcite dataset. the fairytaleqa corpus is representative of narrative comprehension, whereas the textbook question answering dataset focuses on scientific topics including physical, earth, and life sciences. in addition to covering different subjects these datasets are quite different in terms of the passage structure. earlier research on aqg has also considered how different forms of discourse representation, such as question type and event summarization, can improve the quality of questions generated @xcite . the fairytaleqa corpus distinguishes questions by seven attribute types. these attributes indicate the semantic nature of the question as well as the type of information that the reader is searching for either implicitly or explicitly from the text. we incorporate the question attribute into each of our model architectures to see whether the attribute has more significant impact when combined with a particular neural structure. in this paper, we compare the performance of three different neural aqg architectures across two different datasets. we train baseline models for aqg on the fairytaleqa corpus, @xcite a narrative dataset for k-12 reading com-prehension, and the textbook question answering @xcite dataset focused on middle school science. these models include a t5 @xcite , bart @xcite , and gpt-2 @xcite . we also investigate the impact of incorporating question attributes into these different model types for the fairytaleqa corpus. the t5 models achieve the highest metric rankings across both datasets, with bart outperforming gpt-2 on both as well. including the question attribute as part of the input for training and inference improves the overall results for all model types, but leads to greater performance improvements for the gpt-2 and t5 models than for the bart model. additionally, our by attribute breakdown found that including question attribute does not increase rouge scores for setting attribute questions. to our knowledge, this is the first comparison of a broader set of neural architectures for aqg in the education domain. these baselines are intended to inform future work on aqg in the classroom while taking into account the nuances of different subjects. this prior work in the education space has focused experiments largely on a single model architecture -bart, but has not considered more recent improvements in neural generative architectures. @xcite explored the use of a pre-trained t5 transformer model for the task of question generation without answer supervision. their model was designed to take a passage as input and output multiple question-answer pairs related to the passage. it was trained and evaluated on the squad dataset @xcite for general question answering, but was not applied to education specific datasets. based on their results, we evaluate the effectiveness of t5 in the education domain in our experiments. @xcite looked beyond just generating quiz questions and conducted an experiment to evaluate generation errors. the result questions are categorized define a hierarchy of errors with three top-level justifications: disfluent, off target, and wrong context. included among their models are three gpt-2 based models as well as two bart models -all of which are fine-tuned on the squad dataset. the bart-large model has the second lowest rate of errors under their system with the gpt-2 based models all performing at the lower end of the range. their experiment setup for both bart and gpt-2 does not fine-tune on pedagogical texts, so we will be able to explore if this boosts performance in our experiments. ## results the result bleu and rougel scores across both datasets can be seen in ## limitations and future work future work on automated question generation for learning contexts could benefit from a number of potential research paths. in this paper, we tested three different architectures -but there are many more to be considered including those that incorporate knowledge graphs which have been shown to improve the richness and semantic correctness of generated questions @xcite . there is also room to explore different prompt strategies including a fill-in-the-blank approach which may be more appropriate fo the tqa data. for the attribute models, we used the single task objective of question generation, but it would be worthwhile to explore jointly generating the question attribute and the question itself. additionally, document level abstract meaning representations with resolved coreferences has been shown to improve the quality of knowledge based question generation @xcite . we also recognize that we focused on different context for the input, but not on the wide variety of generation strategies available for this task. on top of the variety of model architectures, we would like to evaluate a greater set of corpora that include additional topics such as history and economics. reading comprehension is critical to these fields as well and there is limited, if any, research on question generation for these topics. additionally, in future work we will conduct evaluation with expert annotators to incorporate into more complex models. ideally, we will have educators and students assess the output of our models for factual correctness, relevance, and fluency of the questions generated. this output can then be used to train an instruction fine-tuned model. in order to make a solution that is viable for the classroom, it is critical to think beyond the automated metrics and get real teacher feedback. this preliminary research demonstrates the potential for expanding automated question generation to multiple classroom subjects and the value of incorporating discourse information into different model architectures to produce high quality questions. ## conclusion in this paper, we conduct an initial comparison of automated question generation architectures for narrative stories (fairytales) and science textbooks. for each corpus, we trained bart, gpt-2, and t5 models to see which would perform best in which context. our results indicate that the t5 models achieve the highest scores in terms of automated metrics for both domains. the highest performing t5 model also outperforms the bart baseline for question generation on the fairytaleqa dataset put forth in @xcite . we also evaluated the effectiveness of encoding question attribute information in different model architectures. we saw improvements in performance for both character and outcome resolution questions when the attribute was included for multiple architectures suggesting that this information is beneficial for generating certain types of questions, but not all. additionally, the inclusion of attribute information led to a more significant improvement across question types for the gpt architecture.