MCQG - SR efine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback (2025)

Abstract: Automatic question generation (QG) is essential for AI and NLP, particularly in intelligent tutoring, dialogue systems, and fact verification. Generating multiple-choice questions (MCQG) for professional exams, like the United States Medical Licensing Examination (USMLE), is particularly challenging, requiring domain expertise and complex multi-hop reasoning for high-quality questions. However, current large language models (LLMs) like GPT-4 struggle with professional MCQG due to outdated knowledge, hallucination issues, and prompt sensitivity, resulting in unsatisfactory quality and difficulty. To address these challenges, we propose MCQG-SRefine, an LLM self-refine-based (Critique and Correction) framework for converting medical cases into high-quality USMLE-style questions. By integrating expert-driven prompt engineering with iterative self-critique and self-correction feedback, MCQG-SRefine significantly enhances human expert satisfaction regarding both the quality and difficulty of the questions. Furthermore, we introduce an LLM-as-Judge-based automatic metric to replace the complex and costly expert evaluation process, ensuring reliable and expert-aligned assessments.

## introduction in artificial intelligence (ai) and natural language processing (nlp), automatic question generation (qg) from knowledge bases, texts, and images @xcite plays a crucial role in enhancing question-answering (qa) models @xcite , supporting intelligent tutoring systems @xcite , improving dialogue systems, and aiding fact verification @xcite . multiple-choice question generation (mcqg), a specialized type of qg, is extensively used in exams to assess students' knowledge efficiently @xcite . however, creating mcqs is labor-intensive, requiring the design of effective stems, prediction of common errors as distractors, and provision of corrective feedback @xcite . in professional fields, mcqs often require field experts because they need to reflect real-world scenarios and involve complex multi-hop reasoning. these are unique challenges not typically encountered in general qg tasks. the united states medical licensing examination (usmle) exemplifies the need for highquality mcqg @xcite . preparing for the usmle costs medical students over $5000 on average @xcite . for exam boards and instructors, creating mcqs is both time-consuming and expensive @xcite . any application that can automate this process is highly valuable to medical educators @xcite . due to the high difficulty with the need for domain knowledge and complex reasoning, usmle questions are becoming important large language models (llms) benchmarks @xcite . top llms like gpt-4 have shown over 90% accuracy on sample usmle questions @xcite . recent research explores leveraging gpt-4's potential in usmle-mcqg to improve question generation efficiency for educators and assist students in exam preparation @xcite @xcite . however, relying solely on llms like gpt-4 to generate usmle questions presents several challenges. firstly, their performance is constrained by their training data, leading to two major issues: outdated knowledge @xcite and hallucination @xcite . outdated knowledge means that llms can only repeat or integrate old usmle questions or medical documents in their training data during the generation. consequently, they struggle to create new ques-figure 1 : usmle mcq generated by gpt-4 and mcqg-srefine. the gpt-4 question contains several errors and inconsistencies, such as extraneous information, a distractor option format mismatch, mentioning symptoms instead of conditions, and a context that contains the answer. the mcqg-srefine addresses these issues, resulting in a higher quality question that aligns the context, question, and answer options more coherently. irrelevant details are removed, the question focuses on the key clinical condition of sepsis, distractor options are presented in a consistent format, and the context no longer gives away the answer. to address these challenges, we propose a new task: converting medical cases that appear in case reports or clinical notes into usmle-style mcqs. our approach involves several key design elements to alleviate the above limitations: for limit1, previous research has attempted to prompt llms to generate usmle-mcqs by following expert-crafted instructions in zero-shot setting or further adding existing questions as fewshot examples @xcite . to our knowledge, we are the first to qualitatively and quantitatively study how to convert medical cases into usmle-mcqs. these medical cases provide valuable real-world information on disease progression, accurate assessments, diagnoses, and potential treatment plans. using the latest medical cases as input, llms can generate up-to-date questions, thereby minimizing the limitations of outdated knowledge. additionally, grounding questions' key elements in the original medical cases can help llms reduce hallucinations, enhancing the reliability of the generated content. for limit2, we follow the national board of medical examiners (nbme) guidelines foot_0 to establish a checklist of 41 target topics covering all potential exam areas. we then deployed a colbert retriever @xcite using usmle content outline 3 as a collection for test points retrieval. each input medical case in our experiments is evaluated by experts with exam experience to determine if it contains sufficient information to generate questions related to the specified target topics and test points. we also compared the topics and test points identified by experts with those generated by llms, assessing their impact on the quality and difficulty of the resulting questions. for limit3, as illustrated in figure 1 , we used the triplets (medical case, topic, and test point) as our question generation pipeline input. we work with experts for prompt engineering based on usmle guidelines. we then created our mcqg-srefine (e.g., self-refine with iterative critique and correction feedback) following three steps. s1 -initial mcq generation: generate an initial usmle-mcq based on the triplets. s2 -critique feedback: prompt the llm itself to provide feedback on the s1 usmle-mcq. s3 -correction feedback: correct the usmle-mcq based on the s2-generated critique feedback. through iterative critique and correction, mcqg-srefine significantly enhances the quality and difficulty of generated usmle-style mcqs. human evaluations confirm its effectiveness, showing a strong preference for mcqg-srefine generated questions, with a preference ratio of 72.5% in win, 10% tie, and 17.5% loss when compared to gpt-4 generated questions. in terms of difficulty, mcqg-srefine generates more challenging questions. specifically, when provided with expert-identified topics and test points, there is an 80% reduction in easy questions, a 2.25-fold increase in medium questions, and a 4-fold increase in hard questions. finally, designing a reliable reference-free metric to automatically evaluate the quality of systemgenerated usmle-mcqs is challenging. recent research indicates that llm-as-judge correlates more closely with human evaluations than traditional metrics @xcite @xcite @xcite @xcite , though these methods remain underexplored in medical nlp tasks. our goal is to replace the costly expert evaluation process in usmle-mcqg with llm-as-judge. we used 30 criteria designed by experts in their human evaluation, covering different aspects of usmle questions, to guide llm-as-judge in providing rating or comparison feedback on different systems' questions. by further exploring the filtering methods for the 30 criteria, we finally screened out a combination of 10 key criteria. this improved the correlation between llm-as-judge and expert evaluations, as measured by cohen's kappa, from 0.226 (slight reliability) to 0.539 (moderate reliability). using the results from this automated evaluation system, it was shown that the preference rate for questions generated by mcqg-srefine over those generated by gpt-4 was 79.97% in favor and 20.03% against. moreover, mcqg-srefine demonstrated overall improvements across 10 criteria within the 5 components, not just in a specific area. ## experimental design and setup experimental setup for all our experiments with mcqg-srefine, we use the chat completions api from openai and the gpt-4-0125-preview, which has a context window of 8192 tokens, and the values of the hyperparameters temperature and top-p are set to 1. similarly, for all other models used for the llm-as-judge comparison feedback generation, we used their default hyperparameters. dataset for the medical cases utilized in the generation of usmle questions, we used unidentified patient summaries from the pmc-patients dataset @xcite . the average length of these patient summaries was ~419 words. the frequency of topics used is listed in appendix experimental design our experimental design is motivated to answer the following research questions: evaluate whether mcqg-srefine improves both the quality and difficulty of the generated questions. specifically, we employed the mcqg-srefine pipeline to generate usmle questions and compared them with baseline questions generated by gpt-4 under identical inputs and settings. our inputs consisted of medical cases n from the pmc-patients dataset, with topics t and test points k that were either human-annotated or generated by the llm. we generated 373 questions from the human-annotated ⟨n, t, k⟩ set and 385 questions using the llm-generated set. to assess the quality of the questions (rq1), we first engaged two medical experts 8 to express their preferences between the two sets (gpt-4 and gpt-4 + mcqg-srefine) of system-generated questions based on an annotation guideline (appendix for evaluating the difficulty level of the questions (rq2), the human evaluators were also asked to classify the difficulty of both questions into one of three categories: easy, medium, and difficult. specifically, we randomly selected 50 real-world usmle-style questions from the amboss 10 dataset (10 for each difficulty level) as examples for the experts to reference. amboss categorizes question difficulty from 1 to 5, where 1 is the easiest and 5 is the most difficult. we grouped levels 1 and 2 as easy examples, levels 3 and 4 as medium examples, and level 5 as hard examples. llm-as-judge for evaluation metrics in addition to human evaluation, recent work has demonstrated that llm-as-judge (particularly gpt-4based) has a high correlation with human assessment in reference-free settings @xcite @xcite @xcite @xcite . recent work in the medical domain has also shown the potential of llm-as-judge to replace traditional evaluation metrics @xcite @xcite . in this work, finding reliable automatic evaluation metrics in a referencefree setting is crucial, as it can reduce the burden of expert evaluation and help improve llms in future work (e.g., as a reward model). to achieve this, we explored two common llm-as-judge modes: rating and comparison. regarding the evaluation 8 two medical students with 2+ years hospital experience. 9 one licensed physician. 10 https://www.amboss.com/us criteria for each part of the mcq, we found that directly using the criteria from the critique section of mcqg-srefine did not correlate well with the human evaluation results of expert x. therefore, we conducted a detailed correlation analysis between each criterion's score and human evaluation, ranking them accordingly. based on this analysis, we applied different filtering methods to identify the most relevant combination of criteria foot_6 . finally, we selected the following ten aspects: context (concision, relevance, misdirection), question (concluding, clarity), correct answer (occurrence, depth of understanding), distractor (common mistakes), and reasoning (logical flow, evidence-based reasoning). specifically, gpt4-as-judge provides two evaluation indicators: 1. detailed ratings and reasons for the above five sections and ten aspects; 2. preference between mcqa-srefine and gpt-4 generated questions. ## results main results figure 3 demonstrates the overwhelming advantage of gpt-4 + mcqg-srefine over gpt-4 with a 70-80% win rate in human preference about question quality (rq1). in figure 4 , we also observe a decrease of about 80% in easy questions, a 2.25 times increase in medium questions, and a 4 times increase in hard questions with the input of these medical cases with expertprovided topics and key points. for machineprovided topics and key point cases, the proportion of easy questions decreased by 33.3%, a 2 times increase in medium questions, but there was no increase in the proportion of hard questions (rq2). this demonstrates the effectiveness of mcqg-srefine in increasing the difficulty of questions. this also indicates that the quality of topics and key points provided by experts is higher, so llm can generate more difficult questions by thinking more deeply during the critique and correction steps. further improving the quality of the machine-provided topics and key points can be a future direction for improvement. figure 5 and another finding is that the questions generated by gpt-4 + mcqg-srefine are more concise compared to those by gpt-4 (case study 1). experts pointed out that this conciseness makes them more similar to real usmle questions. our experiments show that gpt-4 adopted a very conservative strategy when generating context due to our emphasis on hallucination issues in the prompts. this strategy involves copying and pasting much information from raw medical case inputs to avoid generating potentially incorrect new facts. although this does reduce the occurrence of hallucinations -our human assessment shows that 7.5% of gpt-4 problems contain factual errors -it inevitably sacrifices the typical simplicity and highly refined 12 the aspect-level score can be found in appendix ## discussion round-wise analysis since mcqg-srefine operates as an iterative system, an interesting question is whether gpt-4 can consistently provide meaningful critiques and corrections for itself in such a specialized and complex setting. to explore this, we conducted round-wise analyses, with key findings presented in figures 6, while additional analyses are discussed in the appendix. figure 6 shows the best-scoring rounds for human-and machinegenerated topic + key points. for human-generated llm-as-judge* preference t k (h.) t k (m.) mcqg-srefine win 79.80% 80.10% gpt-4 win 20.20% 19.90% ## conclusion in conclusion, mcqg-srefine improves llm ability to generate high-quality usmle-style mcqs by integrating expert-driven prompts with iterative self-refinement. the framework significantly enhances question quality and difficulty while aligning closely with expert evaluations. additionally, our llm-as-judge metric offers a scalable alternative to costly expert assessments. 10736 our research presents promising advances in the generation of medical exam questions. however, several limitations and ethical considerations require further exploration to ensure the robustness, fairness, and social value of this work. adaptability to other domains and languages: our study focuses on medical education, specifically the usmle format. although the methods and frameworks show potential, their generalizability to other domains, languages, or question types remains unexplored. future research should validate the adaptability of these methods to broader educational contexts and linguistic variations. evaluation bias and reliability of llm-as-judge: the llm-as-judge approach, where gpt-4 is used for both generating and evaluating output, carries a significant risk of self-evaluation bias. first, the evaluation and generation abilities are not equivalent. according to higher-order thinking theories 17 in educational research, evaluation is a cognitively higher-dimensional task compared to task completion. recent studies @xcite also reveal that generative ai models can excel at creating content while lacking deep understanding, which limits their reliability as evaluators. second, while gpt-4 was selected due to its relatively smaller position bias compared to other models such as claude3-opus, sonnet, haiku, and gpt-3.5 (as shown in appendix