C on Q uer: A Framework for Concept-Based Quiz Generation (2025)

Abstract: Quizzes play a crucial role in education by reinforcing students’ understanding of key concepts and encouraging self-directed exploration. However, compiling high-quality quizzes can be challenging and require deep expertise and insight into specific subject matter. Although LLMs have greatly enhanced the efficiency of quiz generation, concerns remain regarding the quality of these AI-generated quizzes and their educational impact on students. To address these issues, we introduce ConQuer, a concept-based quiz generation framework that leverages external knowledge sources. We employ comprehensive evaluation dimensions to assess the quality of the generated quizzes, using LLMs as judges. Our experiment results demonstrate a 4.8% improvement in evaluation scores and a 77.52% win rate in pairwise comparisons against baseline quiz sets. Ablation studies further underscore the effectiveness of each component in our framework.

## introduction quizzes are a widely used tool in modern education, serving as a means to test students' understanding of material and providing opportunities for reflection @xcite . well-designed quizzes can enhance active learning, provide valuable feedback, and stimulate curiosity @xcite . however, the process of creating quizzes is often labor intensive, requiring subject matter expertise, careful consideration of key concepts, and understanding of students' knowledge levels @xcite . this challenge becomes even more pronounced in fields where content is updated frequently or where educators need to generate quizzes on a scale. in recent years, the emergence of large language models has provided a promising solution to these challenges. llms can quickly generate quizzes that cover a wide range of topics. @xcite demonstrated that the llm-generated quizzes are promising for widespread use in the classroom. although this approach offers significant efficiency gains, it also raises concerns about the quality and relevance of the generated quizzes @xcite . specifically, there are questions about whether the quizzes accurately reflect key concepts in a given domain and whether they are grounded in reliable sources of knowledge @xcite . to address these concerns, we propose a conceptbased quiz generation method grounded in external knowledge corpora, such as wikipedia and conceptnet @xcite . using concepts instead of keywords to search for relevant information enables the capture of knowledge points that may not be explicitly mentioned in students' questions. by anchoring quiz generation in wellestablished knowledge bases, our approach ensures that quizzes are not only relevant but also comprehensive, covering critical concepts that learners must grasp. we employ comprehensive evaluation dimensions to assess various aspects of quiz quality. our concept-based approach achieves a 4.8% improvement in evaluation scores compared to traditional llm-generated quizzes. in pairwise evaluations, our method consistently outperforms other alternatives,with 77.52% of evaluations favoring our method over llm-generated quizzes. additionally, ablation studies reveal the critical contributions of the concept extraction module, knowledge source, and summary module in enhancing the overall effectiveness of our framework. in summary, our key contributions are as follows: retrieval-augmented generation while llms have demonstrated strong performance in various understanding and reasoning tasks, their ability to generate reliable and factually accurate text remains a challenge, particularly in knowledgeintensive tasks @xcite . this often leads to hallucinations, where models produce incorrect or fabricated information @xcite . retrieval-augmented generation (rag) @xcite has been proposed to address this issue by integrating llms with retrieval mechanisms, allowing models to refer to external databases and improve the factuality and credibility of their outputs. rag has shown promising results in qa tasks such as squad @xcite and hotpotqa @xcite , as well as in personal planning applications @xcite , where external knowledge is essential for generating accurate and contextually relevant responses. ## task previous studies have explored quiz generation based on predefined topics @xcite . however, such topic-centered approaches often fails to capture the complexities of real-world educational settings. in practice, students frequently ask vague or incomplete question, sometimes without fully grasping the underlying concepts they are struggling with @xcite . research in education has shown that students' questions can reflect their thought processes and serve as a valuable resource to enhance learning @xcite . inspired by this, our approach shifts from relying on predefined topics for llmbased quiz generation. instead, we focus on generating questions that mirror the types of inquiries students might pose to instructors, capturing their authentic learning challenges. the task then becomes generating quizzes that effectively support students with limited information about their current knowledge level. to enhance the diversity of student questions and broaden the framework's applicability across a wide audience, we selected 30 subject areas from the mmlu dataset @xcite and considered three educational levels: primary school, high school, and phd. for each subject and educational level, we tasked gpt-4o @xcite with generating five representative questions that students would typically ask. this approach yields a dataset of 450 questions, which we compiled into a comprehensive question set for experiment. the quiz generation task involves generating three quizzes for each student question, where each quiz consists of one question, one correct answer, and three incorrect options, with the correct answer always positioned as option a. we believe that a single quiz may only provide a limited perspective on the topic, and a set of quizzes offers a more comprehensive approach, thereby enhancing students' overall understanding of the subject matter. to verify that the difficulty of the student questions varies appropriately across different education levels, we tasked the llm with assessing the reasoning difficulty and knowledge depth required to answer each student question, assigning a score on a scale of 1 to 5. the results are presented in figures 2 and 3. as anticipated, the difficulty remains consistent within subject areas but increases progressively with the educational level, aligning with our goal for the dataset. example student questions can be found in the appendix a. ## framework the proposed framework conquer operates as follows. the system receives three inputs: the student's question, their educational level, and the subject area. we first use an llm to extract key concepts from the question. for example, given the question, "what happens to a plant when it doesn't get enough sunlight or water?", we identify several potentially relevant key concepts such as "plant", "sunlight", "water", "photosynthesis", "growth", "stress", "environment". after extracting the relevant concepts, we retrieve relevant information from a knowledge source based on these concepts. in this work, we primarily use wikipedia, which provides a wealth of information on a wide range of topics. to locate the most relevant content, we utilize sentence-bert @xcite to compute cosine similarity scores. this allows the system to pinpoint the most contextually appropriate sections of text. subsequently, an llm-based summarization module condenses the retrieved information into its key points. these summarized details are then passed to the quiz generator, which creates tailored quizzes based on the content. ## results we compare the performance of our conquer framework against a baseline, where the quiz is generated directly from the student's question without utilizing any external materials or concepts. the evaluation score for quizzes generated by gpt-4o-mini is shown in figure 4 , and the win rate of conquer in pairwise comparison is presented in figure 5 . for clarity, the evaluation score has been scaled to 100, and the win rate is expressed as a percentage. additional results for gemini-2.0-flash can be found in the appendix c. our results, based on both evaluation score and pairwise comparison win rates, demonstrate that conquer consistently outperforms the baseline across all five evaluation dimensions. although the average score improvement across these five dimensions is only 4.8%, conquer achieves a significant win rate of 77.52% in the pairwise comparison. we hypothesize that both conquer and the baseline produce quizzes that appear well-constructed when considered in isolation, leading to high evaluation scores for both. however, when evaluated together, conquer significantly outperforms the baseline because its quizzes are grounded in highquality knowledge sources that are closely aligned with key concepts. we have observed that the llm judge tends to prefer the second candidate, except in the diversity dimension. this preference may be attributed to the fact that the second candidate is closer to answer tokens, prompting the model to allocate more attention weights to it. in the diversity dimension, however, the model identifies more repeated content in the second candidate, as it has already seen all the quizzes from the first candidate. correlation analysis of the five evaluation dimensions can be found in appendix e. ## conclusion we introduced conquer, a concept-based framework for generating conceptually grounded and educationally effective quizzes. by prioritizing key concepts over surface-level keywords, conquer ensures alignment with essential learning objectives. our evaluations show a 4.8% improvement in quiz quality and a 77.52% win rate in pairwise comparisons, highlighting the superiority of our approach. ablation studies emphasize the importance of each component in driving these improvements. conquer offers a scalable, accurate, and pedagogically valuable tool for quiz generation across diverse educational contexts. future work could extend knowledge sources, refine quiz generation for adaptive difficulty, and personalize learning path. conquer represents a step forward in automating quiz creation while ensuring the accuracy and relevance that are critical to effective learning.