E d T ec- I tem G en: Enhancing Retrieval-Augmented Item Generation Through Key Point Extraction (2025)

Abstract: A major bottleneck in exam construction involves designing test items (i.e., questions) that accurately reflect key content from domain-aligned curricular materials. For instance, during formative assessments in vocational education and training (VET), exam designers must generate updated test items that assess student learning progress while covering the full breadth of topics in the curriculum. Large language models (LLMs) can partially support this process, but effective use requires careful prompting and task-specific understanding. We propose a new key point extraction method for retrieval-augmented item generation that enhances the process of generating test items with LLMs. We exhaustively evaluated our method using a TREC-RAG approach, finding that prompting LLMs with key content rather than directly using full curricular text passages significantly improves item quality regarding key information coverage by 8%. To demonstrate these findings, we release EdTec-ItemGen, a retrieval-augmented item generation demo tool to support item generation in education.

## introduction a key challenge in educational measurement is to construct high-quality exam questions or "test items" that effectively differentiate varying levels of student competency. assessment organizations rely on subject matter experts to extract essential content from domain-specific curriculum materials for item construction @xcite . thus, generative natural language processing-based techniques for automated item generation (aig) have gained interest in educational measurement to reduce the high costs and labor of manual test item creation @xcite . the widespread adoption of large language models (llms) has significantly encouraged employing generative nlp for aig (laverghetta jr. and licato, 2023; @xcite . despite their impressive performance across various subtasks, llms often struggle with hallucinations, bias, and limited domain-specific knowledge, diminishing their effectiveness in specialized tasks @xcite @xcite . retrieval augmented generation (rag) @xcite offers a solution by integrating domain-relevant knowledge, enhancing customization, reducing hallucinations, and improving access to authoritative and up-to-date information. to support bfz foot_0 , one of the largest german vocational education and training (vet) providers in manual item generation tasks, we deployed edtec-itemgen foot_1 , a rag platform for aig, that leverages a new mixed-integer linear programming driven method @xcite for key point extraction to assist educators in designing items for formative assessments. figure 1 summarizes the operational flow of edtec-itemgen for supporting item generation tasks. the process begins with users retrieving relevant vet passages, pre-extracted from educational materials, using semantic search. an extractive summarization step then identifies the most salient factoids or "key points" through the proposed milp-driven kpe approach and similarity scoring (see section 3). these key points are subsequently used to instruct an llm to generate new test items. finally, test item designers evaluate and filter high-quality items via edtec-itemgen's user interface, which are later added to an internal item base for exam assembly and construction. our work contributes as follows: section 3 presents our milp-based key point extraction; section 4 presents its evaluation under the trec-rag framework. section 5 provides an application and demo system overview, and section 6 concludes with future directions. ## related work to mitigate the complexity of manual test item construction, educators adopted automated approaches to simplify its development @xcite @xcite . prior research in automated item generation (aig) transitioned from classic nlp methods, including shallow parsing, term and topic extraction, and the use of semantic resources like wordnet @xcite @xcite @xcite , to neural architectures including graph-neural networks and transformer-based models for aig @xcite @xcite @xcite . the rise of llms has led educators to use promptbased generative nlp for aig @xcite ; for example, wu et al. (2024) propose a two-step multimodal framework that merges llm-generated sub-questions about related entities into coherent items. lin et al. (2024) proposed tase-cot, a few-shot method for type-aware semantic extraction of relevant item types and phrases to aid llms in generating refined items and answers requiring reasoning across multiple documents. guo et al. (2024) generate knowledge-base questions by extracting a skeleton of interrogatives and auxiliaries from graph triples to steer gpt-3.5. ashok kumar and lan (2024) fine-tune llama-2 with negative socratic example augmentation and direct preference optimization to boost programming-item validity. prior work includes pochiraju et al. (2023), which maps sentences via conceptnet/wordnet rules, and guinet et al. (2024), which fine-tunes llms for exam generation and filters items by syntax, incorrectness, self-containment, and embedding similarity. poon et al. (2024) show that few-shot llm prompts yield more higher-order chinese reading items than traditional methods, while mucciaccia et al. (2025) combine role-based prompts, glossaries, one-shot examples, and chain-of-thought reasoning to generate and evaluate university-level items. although llm prompting is widespread in aig, hallucinations, knowledge-reliability issues, and opaque reasoning persist without careful prompt design @xcite . to address these limitations, retrieval-augmented generation (rag) emerged as a popular strategy to enhance generative nlp performance @xcite . this work introduces a novel key point extraction method to enhance the retrieval-augmented item generation process in german vocational education and training (vet). we released a public fork of our industry partner's internal tool and apis to demonstrate this. the closest studies are pochiraju et al. (2023), which maps sentences via concept-net/wordnet rules, and guinet et al. (2024), which fine-tunes llms for exam generation and filters items by length, incorrectness, self-containment, and embedding similarity. however, our simpler approach applies extractive summarization to vet passages, isolating key nuggets and preserving only essential content. ultimately, we evaluated our ap-proach with pradeep et al. (2024a,b) framework. ## enhancing augmented item generation via key point extraction key point extraction (kpe) is an extractive summarization approach that selects high-level factoid statements capturing the main aspects of a passage @xcite . because item designers already distill such facts from domainspecific vet contents, we add kpe to our itemgeneration rag pipeline to replicate this manual process. milp-driven kpe after splitting the input passages into sentences, we employed jina-colbert 3 @xcite , an efficient multivector neural re-ranker model, to compute sentence embeddings and derive a similarity score for each pair of candidate sentences, ultimately producing a similarity matrix. then, we apply a mixed-integer linear programming driven formulation (milpdriven kpe) that aims to balance maximizing the relevance, while minimizing redundancy of a set of candidate sentences. essentially, our goal is to subselect k candidate sentences with maximum relevance to the original input passage, such that their pairwise similarity is minimized. assume we have m candidate sentences, each with a relevance score r 1 , . . . , r m relative to the original input passage and a pairwise similarity matrix @xmath0. then, we aim to solve: the objective in eq. ( 1 ) formulates the task as a quadratic knapsack problem @xcite . it selects exactly k sentences, rewarding their individual relevance while penalizing pairwise similarity (redundancy), with λ controlling the relevanceredundancy trade-off. we linearize it as a milp problem with @xcite method: ## experiments and results although llms have been shown to be more effective as weak labelers when combined with custom models like distilbart rather than for direct extractive summarization @xcite , training these models requires high-quality ground truth data, careful prompt design, and challenges for optimizing task-specific objectives. furthermore, llm's reliance on superficial features like sentence position rather than distinguishing content importance may make llms encounter challenges in extractive summarization tasks @xcite . therefore, we hypothesize that applying milpdriven kpe to instruct an llm in item generation can significantly improve both information coverage and the quality of test items, while reducing the reliance on resource-intensive training iterations. ## system overview edtec-itemgen aims to support educators in test item generation tasks at bfz, germany's largest provider of vocational education and training (vet) services. designing formative assessments is time-intensive, particularly when test items must cover specific curricular content. this process is slow and prone to errors. edtec-itemgen replaces the drafting process with retrieval-augmented item generation. a dense retriever searches from internal educational materials, while milp-driven kpe extracts essential concepts to instruct an llm in writing a candidate item. educators then review each suggested item, rating clarity and difficulty while discarding spurious items. edtec-itemgen also serves as a crowdsourcing infrastructure, collecting user interactions to drive future research on generative retrieval models for question banks (see appendix a2 for industry application details). frontend the platform interface, illustrated in figure 3 , implements the complete item-generation workflow in html and javascript. user interactions from the interface are logged in the backend using sqlite and made accessible through an internal endpoint for later analysis and review. to extend applicability to other languages, we expose language-agnostic apis that support the generation of multilingual items from user-custom data (see appendix a3 for more details). ## conclusions collaborating with bfz, one of germany's largest vet services providers, we explored employing retrieval-augmented generation to assist educators with manual item construction tasks 18.75% to 25%. in terms of item quality, kpe significantly improves grammar and readability by 2.70% and 2.73%, respectively, while increasing item length by 13.62%. overall, our approach enhances the coverage of essential curricular content and improves the clarity of language in test items. future work will analyze session logs to research how users generate and evaluate items using our method. we plan to further investigate how to employ and integrate linear constraint-based testassembly models (linden et al., 2005) with llms to control and generate different test item types (e.g., matching, cloze) to expand on item's psychometric coverage beyond recall.