Achieving Stronger Generation via Simple Contrastive Tuning (2024)

Abstract: Instruction tuning is widely used to unlock the abilities of Large Language Models (LLMs) in following human instructions, resulting in substantial performance improvements across various downstream tasks.Furthermore, contrastive decoding methods are employed to enhance instruction-tuned models. To further explore the potential of contrastive decoding, we introduce the Contrastive Tuning and Decoding (CTD) framework, which enhances model performance without requiring additional data or significant computational resources.When performing Contrastive Tuning, we optimize a correction model by targeting discrepancies between the original outputs and labels. During Contrastive Decoding, the correction model adjusts the logits of the SFT model using the same input to ensure better adherence to instructions.With the lightweight CTD framework, we refine the behavior of instruction-tuned models, improving their performance on the challenging SUPNATINST dataset with unfamiliar data distributions across various models and prompt formats.

## introduction recent years @xcite have witnessed remarkable progress in large language models (llms). some llms, such as llama @xcite , gpt-3 @xcite , and mistral @xcite , have acquired general abilities for solving various tasks. instruction tuning is a crucial technique to enhance the capabilities and controllability of llms @xcite ) by further fine-tuning llms with instruction-formatted data, resulting in better performance on downstream tasks. however, there is a concern that instructiontuned models show significant improvements mainly on tasks related to the instruction datasets * corresponding author. and may not perform as well on others @xcite . this limitation indicates that instruction-tuned models struggle to effectively generalize to tasks with entirely different data distributions. additionally, in practical scenarios, instruction tuning often faces challenges with limited data and tends to overfit, leading to performance degradation after multiple training epochs @xcite . to further enhance the performance of llms on text generation tasks, some researchers have explored decoding methods in a contrastive manner @xcite @xcite . these methods focus on modifying the model's initial output distribution with a specific distracted distribution. although existing contrastive decoding methods do not require additional training, they lack robustness across various models and scenarios and are sensitive to specific hyperparameters @xcite . to address this issue and further explore the potential of instruction-tuned models, we propose a novel contrastive tuning and decoding (ctd) framework to achieve stronger generation without additional data or much computational resource, as shown in figure 1 . with a supervised fine-tuned (sft) model derived from a pre-trained model as the original model, we initialize the correction model using the sft model and fine-tune it with parameter-efficient methods, utilizing data sampled from the sft dataset to obtain additional parameters for correction. finally, we apply contrastive decoding, which leverages the differences between the original and correction model's outputs. this process results in significant performance improvements with minimal resource requirements, as we use prompt tuning for contrastive tuning, which is lightweight. we are the first to explore employing contrastive tuning before contrastive decoding, which optimizes the correction model from a unique perspective, resulting in more reliable improvements on challenge tasks with unfamiliar data distribution. in summary, our contributions are as follows: (1) we introduce contrastive tuning, which develops a specific correction model for the original sft model. ## related work instruction tuning instruction tuning is a method for fine-tuning pre-trained llms on a collection of formatted instances presented in natural language @xcite . this approach enables llms to follow human instructions @xcite and perform specific tasks without requiring demonstrations, even for unseen tasks @xcite . however, in a data-constrained scenario, many prior works on llm @xcite show that training an llm with multiple epochs of repeated data leads to overfitting. our work also employs multiple rounds of tuning with repeated data and achieves better performance. ## motivation we draw inspiration from instructive decoding, which suggests that distracted logits from the sft model itself can be used to refine its original output @xcite . however, this approach is based on intuition, and the improvement is not stable @xcite . it has been observed that instructive decoding has misaligned training and decoding objectives. therefore, we design a novel optimization objective to align the language modeling objectives between training and decoding, enhancing the model's ability to correct its own outputs more effectively. so contrastive tuning and decoding framework can correct high-confidence errors, which instructive decoding alone cannot achieve. by incorporating prompt tuning, we avoid storing two sets of model parameters, reducing memory requirements.