Training Medical QA Models Based on Mixed Rewards from Multiple-Choice and Open-Ended Questions (2025)

Abstract: Reinforcement learning (RL) for large language models (LLMs) typically requires clear reward signals, which are often unavailable for open-ended (OE) questions where answer evaluation is ambiguous without scalable expert labeling. We investigate whether LLMs benefit from training on mixed data with varying reward clarity. Our approach combines Multiple-choice questions (MCQs), which offer clear binary rewards, with OE questions, for which we use simpler, potentially noisy rewards such as Jaccard similarity or LLM-based evaluators. We hypothesize that MCQs can stabilize training when mixed with OE questions. Our experiments show this mixed-data approach consistently improves medical question-answering performance across model scales.

## introduction reinforcement learning (rl) has shown promise in enhancing the reasoning capabilities of large language models (llms) @xcite . rl thrives on clear and consistent reward signals that provide unambiguous feedback. multiple-choice questions (mcqs) exemplify this ideal scenario by offering binary rewards: answers are either correct or incorrect. this clarity provides a stable learning signal for the model. however, specialized domains like medicine frequently require open-ended (oe) questions where evaluating answers involves greater complexity. for these questions, defining clear reward signals is challenging. consider this medical scenario: "what is the first step in the management of a patient with congestive heart failure, type 2 diabetes, altered mental status, and a serum glucose level of 500 mg/dl?" if the ground truth is "iv ns," and a model answers, "start iv dextrose-containing fluids," is this answer entirely wrong, partially correct, or acceptable? while human expert labeling could * corresponding author provide accurate assessments, this approach is expensive and not scalable for on-policy rl, where rewards are dynamically needed during training. this limitation raises a crucial research question: can llms benefit from training on data with noisy reward signals, and if so, how can such data be effectively utilized? reward models (rms) @xcite trained to mimic human expert evaluations provide numerical scores (e.g., 0 to 1), but potentially introducing additional biases, such as length bias @xcite . alternatively, simpler metrics like jaccard similarity @xcite between model outputs and reference answers provide more direct, but potentially noisier, reward signals. in this work, we investigate a novel strategy that leverages both the stability of clear reward signals from mcqs and the broader coverage of oe questions despite their inherently noisier rewards. we propose mixing mcqs and oe questions within the same training batches. the hypothesis is that the unambiguous feedback from mcqs serves as an anchor, stabilizing the training process, while the model still learns from diverse oe data. we explore various reward mechanisms for oe questions, including jaccard similarity and llm-based reward models. however, due to the complexity of medical terms, such as abbreviations and spelling errors, jaccard can fail to capture semantic equivalence. for instance, "administer intravenous normal saline" and the ground truth "iv ns" are identical in meaning but receive a score of 0 due to token mismatch. notably, after training with our mixed-reward strategy, the model still produced the correct answer "administer intravenous normal saline" to this question, demonstrating robustness even under noisy reward signals. in summary, our main contribution is demonstrating that rl can benefit from an expanded dataset, even if it includes noisy rewards. we evaluate the performance of our method on several medical qa benchmarks, including medqa-usmle, mmlu-pro, and cmb-exam, to show consistent accuracy improvements. this mixed-data, mixedreward strategy aims to effectively balance reward signal quality with data diversity, leading to more robust medical qa models. ## methodology dyanmic sampling in dapo ensures that each training batch contains prompts that provide effective gradients (eliminating zero-gradient groups). this is particularly useful when training with a mix of mcqs (clear rewards) and oe questions (potentially noisy rewards, examples shown in appendix a.2), as dapo can dynamically curate batches of appropriate difficulty. this process, where the algorithm adaptively selects data for optimal learning, aligns with the philosophy of curriculum learning and proved effective in our experiments. our preliminary experiments revealed that datasets composed exclusively of open-ended questions needed more iterations to identify useful training data and form effective learning batches (see appendix a.3). ## conclusion in this paper, we demonstrate the effectiveness of a mixed training approach combining english-chinese multiple-choice questions and open-ended qa data, using rule-based rewards and jaccard similarity with dapo for reinforcement learning. this strategy consistently outperforms single-dataset approaches on most of the benchmarks across both 3b and 7b models. the performance of the newly released qwen3-4b model aligns with the above conclusion while surpassing the larger qwen2.5-7b, indicating that the new model is more efficient and powerful. our approach strikes an optimal balance by leveraging the training stability of mcqs while still exposing the model to the diverse reasoning patterns essential for open-ended medical questions. this suggests a promising direction for future development.