Fine-Tuning Encoder-Decoder Models with Contrastive Learning for In-Context Distractor Generation (2025)

Abstract: Distractor generation is the task of automatically generating plausible yet incorrect options (i.e., distractors) for fill-in-the-blank and multiple-choice questions. In assessment, distractors must be contextually relevant to the given question and answer. Even though recent research works focus on fine-tuning pre-trained encoder-decoder models with data augmentation techniques to generate distractors, these models often fail to capture the full semantic representation of a given question-answer and related distractors. The augmentation methods often rely on expanding the quantity of proposed candidates (i.e., questions or distractors), which can introduce noise into the models without necessarily enhancing their understanding of the deeper semantic relationships between question-answer and related distractors. This paper introduces a novel distractor generation model based on contrastive learning to train the model to recognize essential semantic features necessary to generate in-context distractors. The extensive experiments on two public datasets indicate that contrastive learning introduces a strong baseline model to the distractor generation task. It significantly outperforms recent models, increasing the NDCG@3 score from 24.68 to 32.33 on the MCQ dataset and from 26.66 to 36.68 on the SciQ dataset.

## introduction in assessments, objective questions @xcite , including multiple-choice and fill-in-theblank questions, are widely used in education for fair evaluation across various domains and subjects @xcite . these questions require an examinee to select one correct answer from a set of wrong options. the quality of these questions relies on the quality of selecting these wrong options, known as distractors. distractor generation (dg) @xcite refers to the automated process of generating plausible yet incorrect options in objective types of questions. for decades, research works have shown interest in dg using several approaches, ranging from feature-based learning @xcite to deep neural networks @xcite . then, pre-trained language models (plms), based on transformer architecture @xcite , have notably enhanced dg through fine-tuning @xcite and prompting @xcite paradigms. two primary methods have been proposed for dg based on plms, as illustrated in figure 1 . first, candidate generation and selection framework (csg-ds) @xcite uses finetuning or prompting methods to generate a candidate set of distractors, then selects the top distractors based on embedding models or feature-based rules @xcite . second, text2text architecture @xcite utilizes fine-tuning pre-trained encoder-decoder models @xcite to generate distractors as a sequence-to-sequence (seq2seq) task, where the distractors are generated directly from plms rather than selected from the generated candidates. aligning text2text models with the dg task is still challenging. as shown in figure 1 , they may generate distractors (air, gas, electricity) that are only plausible to the answer (water) rather than in-context with the question (the common term for the chemical formula h 2 o is ...etc). recent state-of-the-art (sota) text2text works incorporate augmentation techniques @xcite and adopt a retrieval augmented pre-training method @xcite to enhance the knowledge of the encoder-decoder models, but these models are designed to restore and denoise entire text sequences during pre-training, rather than capturing fine-grained semantic distinctions required for generating contextually relevant distractors. this limitation causes the models to generate distractors (air, gas, electricity) that simply complete the denoised sequence for the given question-answer, rather than in-context distractors (helium, nitrogen, carbon) that semantically align with the question-answer, as shown in figure 1 . thus, we propose to integrate a contrastive learning (cl) approach inspired by computer vision and text generation works @xcite @xcite @xcite to enhance the semantic learning in these models. while cl is applied to the dg in ranking-based models @xcite , visual question answering task @xcite , knowledge graph reasoning @xcite , and most recently still under-explored in reading comprehension @xcite , it is not yet explored, particularly for the dg task, within the text2text architecture. initially, the encoding of the input (i.e., question-answer) and the decoding of the output (i.e., distractors) can be regarded as two representational views with respect to the same semantics, forming a positive pair. they are contrasted with negative pairs (i.e., a given question-answer with unrelated distractors in the mini-batch). integrating a contrastive loss with the generation loss encourages the model to bring semantically similar pairs closer in the representation space while pushing dissimilar pairs. this joint learning enhances the model to capture semantic features and generate in-context distractors. we explore two contrastive objectives: @xcite with multiple negatives and triplet loss @xcite with a single negative. our automatic and human evaluation results on two public datasets indicate that this method aligns the dg task with encoder-decoder models better than augmentation techniques. the main contributions can be summarized as follows: (i) introducing a contrastive-based learning approach within text2text architecture for the dg task (ii) benchmarking our approach against sota models using both automatic and human evaluation metrics; (iii) achieving new sota results, increasing ndcg@3 score from 24.68 to 32.33 on mcq and from 26.66 to 36.68 on sciq. we organize this paper as follows. sec. 2 reviews the related works on dg and cl. sec. 3 presents the details of the proposed methodology. sec. 4 reports the experimental details along with performance analysis, and sec. 5 offers a conclusion. ## methodology this section outlines the details of our approach. sec. 3.1 defines dg task formulation. sec. 3.2 and sec. 3.3 detail the training of text2text and the implementation of cl in encoder-decoder models, respectively. sec. 3.4 presents a two-stage training to incorporate contrastive loss with generation task. ## conclusion we propose a distractor generation model that integrates contrastive learning in a text2text architecture to better train pre-trained encoder-decoder models for generating relevant distractors for multiple-choice and fill-in-the-blank questions. we explore infonce and triplet losses, as contrastive objectives, with the generation task to align semantically similar question-answer and distractor pairs closer in feature space while distancing negative pairs. this joint training improves the models to capture semantic features to generate in-context distractors. we validate our work through automatic and manual evaluations across two datasets and recent state-of-the-art approaches. the contrastivebased approach represents a strong baseline model and an insightful contribution to the dg field. it shows the success in improving the automatic results and the quality of generated distractors.