E d T ec- QB uilder: A Semantic Retrieval Tool for Assembling Vocational Training Exams in G erman Language (2024)

Abstract: Selecting and assembling test items from a validated item database into comprehensive exam forms is an under-researched but significant challenge in education. Search and retrieval methods provide a robust framework to assist educators when filtering and assembling relevant test items. In this work, we present EdTec-QBuilder, a semantic search tool developed to assist vocational educators in assembling exam forms. To implement EdTec-QBuilder’s core search functionality, we evaluated eight retrieval strategies and twenty-five popular pre-trained sentence similarity models. Our evaluation revealed that employing cross-encoders to re-rank an initial list of relevant items is best for assisting vocational trainers in assembling examination forms. Beyond topic-based exam assembly, EdTec-QBuilder aims to provide a crowdsourcing infrastructure enabling manual exam assembly data collection, which is critical for future research and development in assisted and automatic exam assembly models.

## introduction examination forms consisting of validated, highquality test items are a crucial tool for estimating the current competence of students in education. while much research has covered the task of generating such items, relatively little research has focused on assembling new exams from a database of existing, validated test items @xcite . exam assembly is a challenging task on its own as exams need to cover all skills in a given topic comprehensively, at multiple levels of difficulty, and such that the resulting exam is psychometrically valid @xcite @xcite . to support educational experts in formative exam assembly, we present edtec-qbuilder 1 , a service 1 the edtec-qbuilder system demo version is publicly accessible at: https://www.dfki.de/kiperweb/about.html and tool to search a database of (validated and highquality) test items and assemble them for an exam. beyond its practical utility for exam assembly, our tool is also intended to be a crowdsourcing platform to collect data on exam assembly processes and thus provide data for future research on assisted and automated exam assembly. figure 1 summarizes the operational flow of our tool. after logging in, users query, filter, and annotate relevant items from a validated item database. while selecting, confirming, and submitting exam forms, users assess the quality of their search experience. finally, search sessions are stored for analysis and exam assembly model development. to implement edtec-qbuilder's semantic search functionality, we evaluated the performance of eight search and retrieval strategies in combination with the top twenty-five trending pre-trained sentence similarity models (representing over 176 different combinations, overall; refer to section 4). we selected the best-performing strategy and pretrained model from our evaluation to include it as a core search and retrieval module. our work makes the following key contributions: the evaluation and selection of the best-performing core semantic search functionality can be found in section 4. the implementation and features of edtec-qbuilder are described in section 5. finally, we discuss future directions and opportunities for improvement in section 6. ## related work research on assisted and automated exam assembly can be traced back to the evaling system (fairon, 1999), a platform that employed rule-based and finite state transducers to compile and update language proficiency exams from a question bank. @xcite presented a multidimensional adaptive test architecture and system that selects test items based on a student's profile and previous performance. @xcite utilized named entity recognition to build a knowledge graph of skills, which a recommendation method harnessed to suggest personalized interview questions. sangodiah et al. (2016) used text classification methods to estimate exam difficulty from question bank's items via bloom's taxonomy categories. han (2018) introduced selection methods for complying with test item criteria for automated item selection, fostering adaptive learning and individualized learning. ruan et al. (2019) developed a dialog agent to teach factual knowledge of science and safety. when contrasting system usage with a flashcard system, students increased their response accuracy significantly when using the agent. cole et al. (2020) leveraged classic natural language processing methods with a cluster-based test item generation approach aligned to a dashboard that enables test designers to select generated questions from the available clusters. datta et al. (2021) tackled the task of automatically generating an interview question plan from the applicant's resume, harnessing knowledge graphs and integer programming methods. upad-hyay et al. ( 2023 ) employed large language models to develop a suite for automated item generation and exam assembly. however, the suite delegates the assembly process to human experts to ensure the test items' quality. most of the prior research focuses exclusively on test item generation, utilizing either classical nlp methods such as text classification and topic modeling @xcite @xcite @xcite , or deep neural networks and transformer-based architectures @xcite @xcite @xcite . however, in real scenarios, where contextual knowledge, quality control, and topic alignment with curricular standards are necessary for effective student skill development, it is not sufficient to generate items; we also need to assemble them to exams that are psychometrically suited to test the competencies they ought to assess. therefore, in contrast to prior work, our contribution focuses on assisted exam assembly rather than item generation. further, we conceptualize exam assembly as an information retrieval problem. ## methods edtec-qbuilder aims to assist in tackling the topicbased exam assembly task. we formalize the task as an information retrieval problem. given a query @xmath0.g. a text describing the topic of a exam -and a database of possible items @xmath1, . . . , x n }, we wish to compute similarity scores s 1 = s(q, x 1 ), . . . , s @xmath2, x n ) for some similarity function s : q×@xmath3. based on these scores, we rank all possible items from most similar to least similar and let an educational expert assemble an exam from the ranked list. in other words, we support educational experts in exam assembly by assisting them in searching a large item database according to their query (refer to figure 1 ). therefore, the similarity function s needs to correspond to educational experts' notion of relevance. in other words, s must assign higher similarity values to the items experts want to include in their exam. this is precisely the criterion we evaluate in section 4. ## experiments and analysis our experiments aim to evaluate the capability of various (semantic) search strategies to retrieve matching test items for typical queries in our item database foot_2 . due to the lack of historical search data and insufficient human annotation capacity that would allow us to obtain ground-truth data, we opted to use an automated relevance labeling approach. using the following approach, we built a new synthetic trec-style @xcite @xcite data set of queries and corresponding relevancy items. testbed we randomly generated synthetic queries by selecting 15 syntactically different synonyms from the 34 available topic dimensions of the item database (refer to pairs with an average similarity score between 0.60 and 0.75 were labeled as moderately relevant; pairs above 0.75 were labeled as highly relevant. we acknowledge that this labeling strategy could introduce a bias as the assumed ground truth is influenced by the same language models later used as part of the search strategies. however, the labeling was obtained by averaging the similarity scores of all language models, thus reducing the bias toward any single model and enhancing the robustness of the obtained labeling. evaluation we employed the ranx library (bassani, 2022) to calculate standard information retrieval metrics as an evaluation method to identify and select the best-performing strategy to include as a search module in our tool. in addition to precision, recall, f1, and mean average precision (map), we used normalized discounted cumulative gain (ndcg) and mean reciprocal rank (mrr). the former expresses the relevance of documents ranked at the top, whereas the latter expresses relevant documents' (reciprocal) rank. both metrics are better if higher. ## tool overview this section presents edtec-qbuilder, a semantic search service and web tool to support exam assembly. figure 2 provides an overview of the tool. while the frontend is a website implemented in html and javascript, the backend is implemented in python and flask. below, we summarize each of the main components. user workflow figure 2a illustrates edtec-qbuilder's user workflow. first, after notifying users that we will only collect click data from the user interface and annotations for future research and development, users land on the search result page (serp) interface after successfully authenticating themselves. second, once in the serp view, the web application provides a search bar where users can browse and search across the test items in the item database. third, after triggering a search, the tool returns the top 100 items relevant to their query, grouped on pages with ten results each. we employed a cross-encoder as a re-ranking strategy because it showed the best performance (see section 4). fourth, as illustrated in figure 2b , to assemble exams, users click on the test items they wish to include in their exam. when users click on an item in the ranked list, the item changes to green, indicating its inclusion in the exam. any selected item can be de-selected by clicking again. users can flag items if they consider them outdated or inconsistent by pressing a red flag button. finally, after annotating, users submit the selected test items to another endpoint, e.g., to publish the exam or deliver it to a computer-based testing environment (cbte). note that our interface also crowdsources ground-truth annotation data by internally logging and sending the selected items to the tool's backend (refer to appendix 4 for more details about the user interface). ## conclusions more than item generation is required to build high-quality exams in education; items must also be assembled into comprehensive examination forms @xcite . we framed assisted topic-based exam assembly as an information retrieval problem, evaluating 176 search and retrieval strategies and language model combinations on a 5,624-item database from the vocational education domain. the most promising strategy is to embed queries via language models, identify the closest 100 items regarding cosine similarity, and re-rank these 100 results via a crossencoder, yielding 0.516 ndcg and 0.713 mrr. in other words, about half of the top-ranked results were relevant, and the highest-ranked relevant result was, on average, on rank 1 or 2. using the cross-encoder re-ranking strategy, we developed edtec-qbuilder 1 , a semantic search service and tool to support exam assembly, which is currently in use at bfz, one of the leading vocational training providers in bavaria. as competencies are known to be heterogenous constructs, test developers are well advised to combine items on multiple topics and items of varying difficulty @xcite . future work should investigate leveraging crowdsourced search sessions and developing models for estimating test item difficulty without entirely depending on respondent data beforehand. lastly, exploring and incorporating large language models could significantly aid the assembly process.