RIFF : Learning to Rephrase Inputs for Few-shot Fine-tuning of Language Models (2024)

Abstract: Pre-trained Language Models (PLMs) can be accurately fine-tuned for downstream text processing tasks. Recently, researchers have introduced several parameter-efficient fine-tuning methods that optimize input prompts or adjust a small number of model parameters (e.g LoRA). In this study, we explore the impact of altering the input text of the original task in conjunction with parameter-efficient fine-tuning methods. To most effectively rewrite the input text, we train a few-shot paraphrase model with a Maximum-Marginal Likelihood objective. Using six few-shot text classification datasets, we show that enriching data with paraphrases at train and test time enhances the performance beyond what can be achieved with parameter-efficient fine-tuning alone. The code used for our experiments can be found at https://github.com/SaeedNajafi/RIFF.

## introduction multiple pre-trained language models (plms), such as bert @xcite , roberta @xcite , t5 @xcite , and gpt2 @xcite , have demonstrated remarkable performance when fine-tuned for downstream text processing tasks. plm variants with less than 1 billion parameters are easier to train endto-end with commodity hardware. however, very recent plms have been trained with a few hundred billion parameters, including palm-2 (540b) @xcite , gpt3 (175b) @xcite , opt (175b) @xcite , or llama-2 (70b) @xcite . training all parameters of these models end-to-end is not straightforward unless done with a dedicated cluster with specialized hardware. in response, nlp research have developed effective techniques to control or alter the behavior of plms by updating the input context through prompt optimization @xcite or adapting a few additional parameters within the network itself @xcite . however, current plm control techniques have not considered altering the original input text to improve the performance of the model. here, we investigate this idea by training a secondary smaller plm to paraphrase the original input at train and test time, thus augmenting the existing data and improving model performance. our inspiration comes from interactions with young children. determining what a child knows is challenging because they can be sensitive to the wording of the question @xcite . adults are also influenced by different wordings of a question. for example, opinion polling has been found to be sensitive to the wording of questions @xcite . just like we rephrase questions for humans, we should consider rephrasing input text while querying a plm. for instance, while classifying the topic of a sentence, phrases related to time may be irrelevant and could be removed to simplify the modeling problem. slight changes to wording could result in the model producing a correct prediction. we explore the integration of paraphrased input texts during both the training and testing phases. at training time, augmenting data through paraphrase generation has been shown to enhance performance while updating all parameters of the model @xcite @xcite . we broaden the scope of previous investigations by using paraphrase augmentation in tandem with recent prompt optimization and efficient tuning methods. at test time, recent works have used ensemble predictions with various optimized prompts and tuned weights @xcite . we further contribute to this line of work by incorporating ensemble predictions based on input paraphrases, again in concert with prompt optimization and efficient tuning techniques. we start by pre-training a smaller language model on paraphrases generated by a large language model (i.e. chatgpt or gpt3.5-turbo). subsequently, we explore various training objectives for fine-tuning this paraphrase generator with feedback from the main task's language model. our analysis shows that our proposed objective reduces hallucination in the generated paraphrases. then, we experiment on six text classification datasets demonstrating that incorporating paraphrase augmentation during both training and testing phases enhances the performance of discrete/soft prompt optimization and efficient tuning techniques. in summary, our contributions are as follows: ## problem formulation we focus on classification problems in natural language understanding (nlu) tasks where we have access to a mini-batch of supervised training examples b @xmath0, y i )} n @xmath1. our goal is to update the parameter set θ lm for a language model by maximizing the probability of the class label y i given the input x i : p θ lm (y i |x i ). here, we augment b supp with semi-supervised examples. in particular, we generate m paraphrases for each x i using the paraphrase generator p θpar (z i,j |x i ), where z i,j represents the jth paraphrase for the input x i . in the optimal case, this paraphrase will preserve semantic meaning but vary syntactic/lexical form. we then incorporate the generated paraphrases to create a new minibatch of examples b s+@xmath2. using this augmented mini-batch, we optimize the following objective j θ lm : we assume that each input x or its paraphrase z is preceded by the task instruction p, which is often specified in previous works. the task instruction, which we represent using the symbol p to be consist with prompt optimization literature, serves as a parameter-free, gradient-free technique for enhancing the performance of the plm across various downstream tasks @xcite @xcite . when using only the task instructions, no parameters for the language model are updated (θ @xmath3, and zero-shot predictions are made solely on the evaluation data. we further investigate multiple language model tuning techniques while incorporating these task instructions into the input or its paraphrases. ## related works to improve prompt optimization and efficient tuning techniques for lms, we incorporate the generated paraphrases into the training mini-batches. paraphrase generation represents just one technique of data augmentation. for a comprehensive overview of diverse data augmentation techniques for nlp tasks, we direct interested readers to a recent survey by @xcite . a recent work for few-shot prompt-based learning helps contrastive training by paraphrasing the inputs @xcite . our work proposes an objective to further fine-tune the paraphrase generator distilled from an llm that reduces hallucination. despite the previous work, which only studies the alltune technique, we investigate the impact of paraphrases for various language model tuning techniques. without contrastive learning, we can show that we can improve lm's robustness by paraphrase generation during training. prompt optimization & efficient tuning: recent research proposes various techniques for prompt optimization and efficient tuning. we have used successful techniques from each of these areas. appendix e provides our detailed description of these recent techniques. all of the recent techniques for prompt optimization and efficient tuning use the original input context provided within the dataset. paraphrase generation: recent techniques encompass various approaches, including the use of copy mechanisms, variational autoencoders, generative adversarial networks, and reinforcement learning techniques to generate diverse paraphrases @xcite . while previous studies have applied rl techniques for paraphrase generation, we propose the use of mml gradients instead of policy gradients to fine-tune our paraphrase model by the reward of a secondary classifi-cation task (see appendix e for more discussion). ## conclusion we investigated the impact of incorporating input paraphrases while fine-tuning plms with recent efficient tuning techniques. our results indicate that specific techniques, such as continuous and discrete prompt optimization methods like autoprompt or soft-prompt tuning, benefit significantly from the inclusion of paraphrases. we also conducted extensive experiments to reduce noise in a distantly supervised paraphrase generator. our ablation studies on fine-tuning the paraphrase generator demonstrate that policy gradient objectives lack robustness during training, while maximum marginal likelihood training remains a robust technique.