Diverse Content Selection for Educational Question Generation (2023)

Abstract: Question Generation (QG) systems have shown promising results in reducing the time and effort required to create questions for students. Typically, a first step in QG is to select the content to design a question for. In an educational setting, it is crucial that the resulting questions cover the most relevant/important pieces of knowledge the student should have acquired. Yet, current QG systems either consider just a single sentence or paragraph (thus do not include a selection step), or do not consider this educational viewpoint of content selection. Aiming to fill this research gap with a solution for educational document level QG, we thus propose to select contents for QG based on relevance and topic diversity. We demonstrate the effectiveness of our proposed content selection strategy for QG on 2 educational datasets. In our performance assessment, we also highlight limitations of existing QG evaluation metrics in light of the content selection problem.

## introduction over the past few years, educational institutes have increasingly embraced digital tools and solutions to extend the purely classroom-based setting and enable wider access to high-quality education. an essential component of digital educational tools is the ability to test the learners' progress in acquiring the knowledge offered in a course. indeed, subjecting learners to such tests triggers reflection on and consolidation of the information they have consumed @xcite . yet, creating high-quality questions that comprehensively and accurately evaluate a learner's knowledge and/or skills is quite challenging due to the extensive human domain knowledge required. current automatic question generation (qg) solutions have already shown significant progress in reducing the time and effort to phrase suitable ques-tions. however, a common weakness is that they do not employ an education-oriented approach when processing full documents or book chapters @xcite . this implies human intervention is required to either select the input or filter suitable output questions afterwards. note that the latter likely implies a significant computational overhead associated with the generation of questions for each and every possible paragraph/sentence. in this research, we take one step back and focus on one of the earliest stages of developing a test called content identification or content selection, the process of reducing the amount of text in a chapter or document to its most meaningful subparts suitable for constructing questions @xcite . content selection is a crucial and challenging step in any assessment system. it is crucial because decisions regarding which content to include or exclude can significantly influence the inferences teachers make about their students' understanding of key concepts in the considered course material. more importantly, in some settings such as self-assessment and self-learning environments, leaving the content selection to users is not feasible @xcite . it is challenging because numerous trade-offs have to be considered, such as the type of exam (e.g., low stakes vs. high stakes), the subject (e.g., mathematics vs. history), and instructor preferences, among others @xcite . although natural language processing has been extensively employed in educational environments @xcite , only a few researchers have investigated content selection for generating educational questions. some studies @xcite used summarization techniques to identify important contents. however, because these methods aim to select sentences that maximize content coverage, they may not be suitable for generating questions in the context of education, as such sentences can be incoherent and complex @xcite . @xcite utilized a binary classifier trained on definitions from scientific textbooks to prioritize worthy over non-worthy content. this study, however, was limited to the definition of named entities or concepts rather than general pedagogical contents. related to our method is @xcite , that ranked sentences based on topic distributions obtained from a topic model for fill-the-gaps (cloze) questions. however, unlike our proposed method, the notion of relevance of contents to teachers is ignored. other methods @xcite @xcite jointly optimized content selection and qg in an end-to-end fashion. most of the previous studies validated their methodologies by evaluating qg performance using n-gram metrics (e.g., @xcite , @xcite ) instead of directly evaluating the content selection method. we will show ( §4) how these metrics are inadequate to evaluate this task. we frame content selection as a ranking problem that maximizes both relevance and topic diversity. the topic diversity is motivated by test development studies @xcite @xcite , that suggest reliable inference regarding students' understanding of contents is tied to the number of questions that cover main topics. this hypothesis is implicitly held by teachers during question construction (fig. 1 ). to this end, we propose a ranking model that assigns a score to each content (i.e., all paragraphs or sentences in a textbook), allowing us to prioritize relevant candidates. furthermore, we introduce a re-ranker that encourages topic diversity. our empirical results ( §4) not only show that our model leads to an improved content ranking compared to existing methods on two recently released educational datasets but also reveal difficulties in measuring ranking quality through evaluation of the question generation end task. ## methodology this section describes our strategy for obtaining a suitable ranking of a document's sentences or paragraphs. since optimizing relevance and diversity at the same time is an np-hard problem @xcite , heuristic approaches are typically used @xcite selected unselected figure 1: 2d visualization of the pooled hidden-state representations of the roberta @xcite on all paragraphs for four randomly sampled chapters of eduqg @xcite using t-sne (van der @xcite . each color stands for a different topic assigned by the gaussian clustering model, while the marker types represent the selected vs. unselected paragraphs by teachers. relevance-based ranking: we assume that the considered educational content involves textual documents d (e.g., course book chapters), each represented as a set of n content elements @xmath0, s 1 , s 2 , ..., s n -1 }. these elements s i can be, for example, the document's sentences or paragraphs (and will be referred to as the sentences). in our datasets, an associated question is available for a selection of the sentences, created by a teacher. although different teachers would likely not agree on which sentences are relevant (i.e., to create questions from) given the same document, we consider the sentences associated with the available questions as a good proxy. as such, we train a classifier to predict for each sentence whether it is relevant or not. in our experiments, this is achieved by training a logistic regression classifier on top of a pre-trained language model's representation @xcite , similar to the approach in @xcite . at inference, after ranking all sentences according to decreasing relevance scores as predicted by that model, each sentence s i is rep-resented by the relevance score r @xmath1, with ρ i its rank (i.e., the highest ranked sentence is scored 1, and the lowest one 1/n ). diversity-aware reranking: next, all sentences s i are iteratively reranked by combining the relevance-based score r i with a score that promotes topic-related diversity, in line with educational insights @xcite . the following paragraphs outline this procedure. first, topics are identified through a gaussian clustering model, fit to all (language model-based) sentence representations x @xmath2. the likelihood p(s i ) of a sentence s i over all topics is written as p(s i ) = z p(z) p(s i |z), with the topic probability p(z) and the gaussian mixture component p(s i |z) = n (x i ; µ z , σ z ) (with mean vector µ z and covariance matrix σ z ).after obtaining a fit for the topic probabilities and gaussian component parameters, the topic distribution for each sentence can be readily obtained as p(z|s i ) = p(s i , z)/ z ′ p(s i , z ′ ). next, we initialize the final sentence ranking s with the sentence that received the highest relevance-oriented rank r i . we then iteratively add sentence by sentence, by combining their relevance score and a diversity score that measures topic diversity with respect to the sentences already present in s. during every considered iteration, the diversity scores d(s i |s) are calculated as follows, for any sentence s i ̸ ∈ s: the method described above is inspired by a well-known technique for query reformulation for web search results diversification @xcite . it would likely work with alternative topic models as well. note that in our experiments, we do not predefine the number of topics, which is estimated through the bayesian information criterion @xcite . however, a teacher could alter the number of topics manually, depending on the desired level of granularity in the topics. ## experimental setup datasets. most existing qg datasets are neither educational (e.g., squad @xcite ) or do not provide an explicit link between questions and course content (e.g., learningq @xcite ) making it impossible to evaluate content selection methods directly. to the best of our knowledge, the only educational datasets that allow for such evaluation are eduqg @xcite and tqa-a @xcite . eduqg contains questions (i.e., phrased in cloze or closeended form) and correct answers that are sentencelevel aligned to source documents. tqa-a contains question-answer pairs where answers are annotated at the span level. we evaluate our content selection methods at the paragraph and sentence levels, respectively, for eduqg and tqa-a (an example entry of each dataset is presented in appendix §a). baselines. we compare different content selection baselines including: oracle (a perfect retrieval system), lexrank @xcite , svm @xcite , overgeneration and rank (ovr; @xcite . the baselines are compared against our methodology ( §2) for some selected λ values. the t5 model @xcite has been fine-tuned to function as the qg model for the baselines. we devised a fixed qg model for all content selection strategies in each dataset, in order to obtain a fair evaluation regarding diversity and generation quality (see appendix §b for further details). evaluation. to measure the content ranking performance, we report recall (r) and mean average precision (map) for the top 10 candidates. diversity, for the selected sentences/paragraphs as well as among the generated questions, is measured by average cosine distance between candidates (adc; @xcite , self-bleu (sbleu; @xcite , and distinct-unigram (dist1; @xcite . we also report bleu and meteor to measure the quality of the generated questions (more details in appendix §c). ## results and discussion we can see a clear correlation between contentdiversity and question-diversity columns. for instance, our (λ = 0.0) selection leads to the highest content diversity and, consequently, the highest question diversity. a higher degree of diversity can be obtained by decreasing λ, at the expense of retrieval effectiveness. the correlation between content diversity and question diversity further supports our suggestion to split up the evaluation of content selection and question generation. the ovr strategy gets better diversity scores on tqa-a. we observed the ranker prioritize cloze questions over "wh" questions. our hypothesis is that the lower word overlap on "wh" words in the highest ranked questions leads to the observed higher diversity. in eduqg, we do not observe this behavior, since all questions have the same format. in fact, we believe that selecting content first and then generating questions is a better strategy compared to ovr. this aligns more closely with how teachers typically create questions and is also more computationally efficient. foot_1 , and in terms of quality of the resulting question set (cf. higher generation scores for our models than ovr). summarized, our experiments lead to the following insights: (i) our proposed model is able to outperform all baselines in terms of retrieval metrics. (ii) it allows control over the trade-off between retrieval quality and diversity (through the parameter λ). (iii) content-diversity and question-diversity do correlate (which is less than surprising), but neither retrieval nor diversity seems to correlate well with established metrics to evaluate generated questions. ## conclusion this paper describes an educational-oriented strategy for content selection from educational documents to support question generation, using a ranker and a topic-wise diversifier. our empirical evaluations of two educational qg datasets demonstrate the effectiveness of the proposed model. however, we found that current ngram-based evaluation metrics of the generated questions, given the current level of generation quality, do not carry sufficient signal to evaluate the content selection problem.