LLM Distillation for Efficient Few-Shot Multiple Choice Question Answering (2025)

Abstract: Encoder models offer efficiency for specific tasks, but their performance depend on data availability. While Large Language Models (LLMs) excel at few-shot learning, their direct application in real-world scenarios is often hindered by their high computational cost. To address this challenge, we propose a simple yet effective approach that uses LLMs for data generation and scoring to improve encoder only model performance. We evaluate this framework on few-shot Multiple Choice Question Answering (MCQA), an important task where acquiring labeled data is costly. Our approach utilizes LLMs to create MCQA questions and choices (exploring both direct JSON and decomposed generation methods) and assigns probability scores to these choices. This generated data and the LLM scores are then used to fine-tune smaller and more efficient DeBERTa-v3-base using distillation loss. Extensive experiments on the MMLU benchmark demonstrate that our method improves accuracy from 28.9% to 39.3%, representing a gain of over 10% compared to a baseline finetuned directly on 5-shot examples. This shows the effectiveness of LLM-driven data generation and knowledge distillation for few-shot MCQA.

## introduction large language models (llms) have significantly advanced natural language processing (nlp) (brown, 2020), demonstrating strong capabilities in few-shot and even zero-shot learning @xcite . however, their large size, including powerful open-source versions @xcite , makes direct deployment costly and impractical for many real-world applications. techniques such as pruning @xcite , quantization @xcite , and distillation @xcite aim to mitigate these issues. yet, these approaches typically produce smaller * equal contribution. encoder models have shown powerful capability in many nlp tasks, including classification, information retrieval, natural language inference (nli), and named entity recognition (ner) @xcite @xcite . compared to llms, encoder models offer efficiency advantages due to their significantly smaller size, requiring fewer resources for finetuning on specific tasks. however, these models often require a significant amount of labeled training data to perform well. obtaining high-quality training data for encoder models can be expensive, which limits their applicability when available data is limited. to address the challenge of limited data for encoder models, distilling knowledge from llms into these smaller models is a promising strategy. current approaches often rely on using the llm to generate synthetic data or augment existing datasets for training the smaller model @xcite . however, we posit that relying solely on data generation limits the depth of knowledge transferred and typically results in performance significantly lower than training on real data. to bridge this gap, we argue that combining llm-driven data generation with probability-based distillation is crucial. while methods for data generation and llm knowledge distillation have been studied independently @xcite , research investigating their combination remains limited. to explore this combined framework, we focus on the task of few-shot multiple choice question answering (mcqa) @xcite . mcqa is extensively used to evaluate model performance, and obtaining sufficient labeled data for it can be difficult and costly. furthermore, mcqa is a crucial task in natural language understanding with wide applications in domains such as medicine @xcite , law @xcite , and education @xcite . this work introduces a framework combining llm-driven data generation and probability distillation for few-shot mcqa tasks, focusing on straightforward techniques to establish the potential of this approach. we employ probability-based distillation, a fundamental technique. for data generation, we explore two methods: 1) generating structured json data, which can yield high quality but is prone to parsing errors, and 2) a decomposed approach generating questions, correct answers, and distractors sequentially, which avoids parsing issues potentially at the cost of data quality. comparing these methods allows us to analyze the effect of generated data quality on the student encoder model. we emphasize that this study focuses on these foundational techniques and exploring more complex generation or distillation methods, while active research areas, is outside the scope of this paper. extensive experiments on the massive multitask language understanding (mmlu) benchmark demonstrate the effectiveness of our proposed framework. our approach significantly boosts the performance of an encoder-only baseline model (deberta-base-v3 trained on only 5 examples), achieving a 10.4 percentage point absolute improvement in accuracy (from 28.9% to 39.3%). remarkably, llm distillation enables this relatively small deberta model to surpass the 5-shot mmlu performance of significantly larger models, including llama-7b (35.1%) and flan-t5-250m (35.9%)the latter having been extensively trained on a large multi-task instruction dataset. this highlights the potential of our method to achieve strong mcqa performance with smaller, more efficient models. furthermore, our distilled deberta model achieves results comparable to a 4-bit quantized llama-3.2-1b model, while maintaining the inherent efficiency advantages of the encoder architecture. we outline our contributions as follows: few-shot multiple choice question answering (mcqa). few-shot mcqa remains a challenging problem, as achieving strong performance often requires large, computationally expensive language models @xcite @xcite . while efficient encoder-only models have shown promise @xcite , they typically rely on extensive multi-task training with hundreds of datasets. however, acquiring largescale mcqa datasets can be costly and timeconsuming @xcite . in this work, we aim to enable effective few-shot mcqa with encoder-only models by leveraging llm-generated data and knowledge distillation, addressing the limitations of both data scarcity and computational cost. llm distillation. llm distillation aims to transfer knowledge from large language models into smaller, more efficient ones @xcite . a common approach involves generating training data with llms and then fine-tuning smaller models on this data. this approach has proven successful in various tasks like classification @xcite , instruction following @xcite , and more @xcite @xcite . however, most research focuses on distilling into smaller but similar language models @xcite , primarily by creating synthetic datasets @xcite . directly distilling llm representations is challenging @xcite , and distilling into different model architectures, such as encoder-only models, remains largely unexplored. this gap is particularly pronounced in the context of few-shot mcqa, where the potential of distilling llms into encoder-only models remains largely unexplored. while some work has investigated llm distillation for other tasks, such as semantic search @xcite , to our knowledge our work is the first to systematically explore a combined approach of data generation and probability score-based distillation for enhancing encoder-only models specifically for few-shot mcqa. ## method our method addresses few-shot mcqa by leveraging the power of llms to generate synthetic training data and then distilling their knowledge into a smaller, more efficient encoder-only model, such as deberta. an overview of our method can be seen in figure 1 . we first generate an mcqa dataset and obtain probability scores for each answer choice using the llm. these scores serve as soft targets to guide the training of the encoder model, which learns from both the generated data and the distilled llm knowledge through distillation losses. this approach enables us to enhance the performance of the encoder model in few-shot scenarios while reducing the computational cost associated with deploying compute-intensive llms. ## experiments we conduct experiments to evaluate the effectiveness of our proposed framework for few-shot mcqa. we use a dataset consisting of only 5 mcqa examples covering the same topic, employing llama-3.1-8b-instruct 1 as the llm for data generation and scoring, and deberta-base-v3 (184m parameters) 2 as the efficient encoderonly student model. we chose deberta-base-v3 due to its strong performance and relatively small size, making it suitable for resource-constrained scenarios. we train the deberta-base-v3 model for 500 iterations with a learning rate of 1e-5, using a batch size of 4 and gradient accumulation for 2 steps, which is equal to using a batch size of 8. for the decompose generation method we set the number of negative examples to be 5 for all experiments, except explicitly mentioned. we average the results across 5 different random seeds for all experiments. unless otherwise specified, we generate 1024 mcqa examples from the initial 5-shot dataset for training using the temperature of 2. we first evaluate our approach on the mmlu benchmark in section 4.1 and then conduct an ablation study on the arc datasets in section 4.2 to analyze the impact of different components of our method. ## conclusion this work demonstrates the effectiveness of leveraging llms for both data generation and probability-based distillation to enable strong few-shot mcqa performance in smaller, more efficient encoder-only models. our findings show this combination is crucial for improving performance over using only llm-generated data, significantly reducing the gap compared to training on real data. our approach achieves encouraging results on the mmlu benchmark (+10.4% absolute gain over a 5-shot baseline), even approaching the performance of significantly larger models like llama-7b and flan-t5-250m, despite their more extensive training. furthermore, our distilled de-berta model performs comparably to few-shot results from lightweight llms like a 4-bit quantized llama-1b model, while offering substantial memory efficiency advantages during inference. this highlights the potential of our method to achieve strong performance with more compact and computationally efficient models. this work also demonstrates that using distillation is crucial to close the gap between training on real and artificial data. however, a performance gap remains compared to models trained with large-scale multi-task data, suggesting opportunities for further improvement. future work will focus on bridging this gap by exploring more advanced data filtering techniques to enhance the quality of the generated data and investigating novel distillation strategies to maximize knowledge transfer from llms to smaller models. additionally, extending our approach to effectively handle longer-context mcqa tasks is a crucial direction for future research.