Mitigating Bias in Item Retrieval for Enhancing Exam Assembly in Vocational Education Services (2025)

Abstract: In education, high-quality exams must cover broad specifications across diverse difficulty levels during the assembly and calibration of test items to effectively measure examinees’ competence. However, balancing the trade-off of selecting relevant test items while fulfilling exam specifications without bias is challenging, particularly when manual item selection and exam assembly rely on a pre-validated item base. To address this limitation, we propose a new mixed-integer programming re-ranking approach to improve relevance, while mitigating bias on an industry-grade exam assembly platform. We evaluate our approach by comparing it against nine bias mitigation re-ranking methods in 225 experiments on a real-world benchmark data set from vocational education services. Experimental results demonstrate a 17% relevance improvement with a 9% bias reduction when integrating sequential optimization techniques with improved contextual relevance augmentation and scoring using a large language model. Our approach bridges information retrieval and exam assembly, enhancing the human-in-the-loop exam assembly process while promoting unbiased exam design

## introduction retrieving and assembling test items into exams from a pre-validated item base that accurately and comprehensively estimates examinees' competence remains a significant challenge in education @xcite @xcite . despite the practical importance of exam assembly, few methods exist to support educators during manual item retrieval for exam assembly tasks @xcite . a key limitation in high-quality exam assembly, especially when relying on a pre-validated test item base, is attribute bias, which typically arises when the retrieved items' ranking order reflects imbalances in specific attributes, such as difficulty or items are color-coded by difficulty: green (easy), yellow (medium), and red (hard). experts populate a test item base, then retrieve and assemble items into formative exams. the initial search skews toward harder items, but our milp-driven bias mitigation re-ranks difficulty distribution for a balanced ranking. information retrieval (ir) research has extensively documented how information access systems may retrieve specific content while systematically and inadvertently omitting relevant but underrepresented content @xcite . this phenomenon, also known as an instance of algorithmic bias, typically leads to "skewed or unfair" system behavior, potentially compromising system accuracy and integrity while perpetuating and reinforcing incomplete or distorted results @xcite @xcite . while several bias mitigation methods exist in ir, and although linear optimization methods could be employed to assemble exams @xcite , their application in supporting unbiased item retrieval for manual exam assembly still needs to be explored. this paper bridges this gap by introducing a new ir bias mitigation technique based on relevance and bias metric-based balancing. we addressed difficulty and source bias in item retrieval to enhance the human-in-the-loop exam assembly process, a critical requirement for testing and educational organizations @xcite . specifically, we examined bfz's foot_0 internal item retrieval platform for exam assembly, edtec-qbuilder. as germany's largest vocational education and training (vet) provider, bfz employs this system for test item selection, which we evaluated using an industry-standard trec-style benchmark comprising 5,624 validated items @xcite . on this benchmark, we employed an ad-hoc retrieval methodology to evaluate nine debiasing techniques to mitigate item difficulty and source bias, conducting 225 experiments overall 2 . we propose a new bias mitigation method incorporating a novel mixed-integer linear programming (milp) approach with enhanced relevance generation via llm-based contextual augmentation, finding that our approach best optimizes the trade-off between bias mitigation and the relevance of the retrieved test items (see ## related work bias and fairness in information retrieval (ir) pertain to how systems rank objects, potentially favoring or disadvantaging specific groups or categories unintentionally. numerous approaches have emerged to measure and address bias and unfairness in ir. for instance, @xcite proposed a probabilistic weighted sampling and horvitz-thompson inference approach to measuring bias based on proportional item exposure. raj and ekstrand (2020, 2022) evaluated and compared existing bias and fairness metrics, finding conceptual similarities but differences in the effect of ranking attributes, such as group/category distribution. recently, @xcite and @xcite surveyed 75 and 100 papers on bias and fairness in ir, respectively, finding that current notions of bias in ir are complex and multidimensional; most current approaches to tackle bias intervene at the in-or post-processing level. regarding in-processing interventions to address bias in ir, @xcite introduced a theoretical framework based on bipartite matching constraints, packing integer programming, and greedy-based diversification methods to incorporate fairness constraints during ranking generation. @xcite developed an efficient sequential greedy brute-force ranker that combines greedy selection to produce fair rankings when target groups are unknown. @xcite proposed a dynamic learning-to-rank approach that mitigated exposure bias by amortizing group allocation fairness while estimating relevance scores. @xcite mitigated bias in neural retrieval systems with an in-batch balancing regularization method enforcing fairness constraints during neural retrieval model training. @xcite proposed a hyperbolic mitigation model for news recommendations, which employs a re-weighting aggregation module to reduce conformity bias while improving user intrinsic interests. @xcite proposed a regression expectation maximization model for learning-to-rank to address position bias with click data. as for post-processing interventions to mitigate bias in @xcite debiased a bayesian personalized ranking method with an adversarial learning model that enhances predicted preferences among groups while ensuring statistical parity. @xcite introduced a candidate ranking multi-model aggregation method to enhance the protected group representation, enforcing fairness over hiring decisions. @xcite introduced an ϵ-greedy post-reranking method to tackle gender bias by reducing imbalanced representations over gender groups while maintaining the original ranking's relevance. in contrast to this prior work, we consider a new mixed-integer linear programming re-ranking formulation, which maximizes the retained relevance while minimizing the difference between actual and desired group distribution. furthermore, we are the first to consider bias in the context of item retrieval for manual exam assembly tasks, a sought-after capability by educational and assessment organizations. additionally, we explore llm prompting and optimization strategies for contextual query generation and improved relevance generation @xcite to boost our method's performance. ## bias framework below, we outline the bias measurement framework, testbed, and analysis of ann+ce, the core search and retrieval method used by edtec-qbuilder for manual item retrieval and exam assembly. bias measuring bias and unfairness in ir can be modeled from the user's perspective. as users' visual attention distribution is higher for top-ranked items, bias and unfairness increase if higher-ranked items from specific classes are over-represented among the top-ranked entries. for our use case, we operationalized bias measurement and mitigation using the framework proposed by @xcite . we measured bias as how balanced an item's difficulty and source classes are represented across the top search results (i.e., difficulty and source bias). due to its stability and robustness, we employed attention-weighted rank fairness (awrf) @xcite @xcite as a metric to evaluate the difficulty and source bias. additionally, to measure the tradeoff between relevance and bias equally, we calculated the following joint metric (jm): testbed to measure and operationalize bias mitigation methods that improve our industry partner's item retrieval and assembly platform's performance, we employed our previous trec-style testbed for the manual item retrieval and exam as-sembly task @xcite . the testbed includes 25 different top-performing frozen search runs, each comprising top-100 rankings for 15 queries across 5,624 items focused on vet for the german job market. each test item is accompanied by its corresponding 3-level graded query relevance judgments, attribute labels for difficulty (e.g., easy, medium, or hard), and source (i.e., manually created by a vet expert or generated via chatgpt3.5). ## milp-driven bias mitigation our task is to mitigate the difficulty and source bias in edtec-qbuilder (bfz's item retrieval and exam assembly platform). given a pre-ranked list retrieved items for a given query, we wish to re- to simplify optimization, we do not target eq. ( 1 ) directly but a linear surrogate objective, namely a linear combination of the sum of relevance scores in the subset and the total variation distance between the target distribution p and the actual class distribution among the included items. as such, our fairness term can also be regarded as a measure of demographic parity in the top-m results. based on our linear surrogate objective, our fair re-ranking scheme can be formulated as a mixed-integer linear program (milp): where x i is 1 if and only if item i is selected for the top m search results, d k is a slack variable representing the total variation distance for class k, and λ controls the trade-off between relevance and bias. equation 2 ensures that the final ranking maintains a class distribution close to the target dis-tribution p, balancing difficulty levels and sources. the fairness constraint minimizes the total variation distance between the observed and desired distributions across all classes. ## experiments section 3 bias analysis shows that edtec-qbuilder ann+ce search model only partially addresses difficulty and source bias. we applied a re-ranking approach on our testbed to evaluate bias mitigation, optimizing the relevance/bias tradeoff within the top 50 ranked results. for a given query, our reranking framework ensures fair representation of all relevant difficulty levels and sources at the top of the ranking. we assessed the proposed methods using ir metrics, awrf via the ranx and fairranktune libraries @xcite , and the jm metric to evaluate the relevance/bias tradeoff. mitigation methods below, we summarize the nine re-ranking methods benchmarked to mitigate difficulty and source bias on edtec-qbuilder for our task. we leveraged our testbed to benchmark the above re-ranking methods for bias mitigation. this evaluation enabled us to effectively address biases present in the current ann+ce-based search and retrieval method of edtec-qbuilder (see ## industry application we collaborated with bfz, germany's largest vet provider, to enhance edtec-qbuilder 3 , their internal exam assembly platform. performance and bias auditing (see section 4) showed that while palomino et al. ( 2024 ) method effectively retrieved relevant test items for assembling exams, it showed attribute biases related to the difficulty and source of the items, resulting in imbalanced exams, potentially compromising exams' comprehensiveness during the manual assembly process. to address this issue, we intend to deploy the milp-bopt re-ranking method (#16), which achieved a 9% reduction in awrf and a 17% improvement in ndcg compared by solely relying on the previous approach (see beyond improving fairness in ranking, our approach holds practical significance for vet services, particularly in manual exam assembly and assessment settings where exam validity depends on diverse and unbiased test item selection. edtec-qbuilder's former item retrieval and exam assembly system failed to account for difficulty and source-based imbalances, leading to biased test compositions that affected learners' evaluation outcomes. by integrating milp-driven bias mitigation, our method ensures that exams are more representative, supporting psychometric integrity in vocational assessment. this advancement aligns with broader trends in fair information retrieval and algorithmic transparency, where unbiased ranking is increasingly valued in education, commercial search applications, hiring platforms, and recommendation systems. the demand for unbiased exam assembly methods is growing among educational and high-stakes assessment organizations @xcite @xcite . more broadly, ensuring fairness in information retrieval is essential not only in education but also in commercial domains where ranking biases impact access to opportunities and decision-making, such as e-commerce and hiring platforms @xcite @xcite . by enhancing the fairness of test item retrieval and assembly, our approach contributes to both assessment quality in vet services and broader advancements in unbiased ranking methodologies. ## conclusions palomino et al. (2024) as a baseline to evaluate nine distinct bias mitigation rerankers, each designed to address the difficulty and source bias in edtec-qbuilder, bfz's item retrieval and exam assembly platform. enhanced by advanced contextualization through refined query and relevance generation and optimized via bayesian hyperparameter tuning, our new milp-driven reranking method achieved a 17% increase in ndcg while reducing awrf by 9% compared to previous results. our approach outperformed popular bias mitigation re-ranking methods in our task, underscoring the suitability of mathematical optimization techniques for mitigating bias in commercial search systems. future work should explore leveraging alternative optimization paradigms, such as multi-objective and nonlinear programming, and in-training techniques, including bias-aware loss functions and regularization for bias mitigation in neural ranking models.