Monotonic Paraphrasing Improves Generalization of Language Model Prompting (2024)

Abstract: Performance of large language models (LLMs) may vary with different prompts or instructions of even the same task. One commonly recognized factor for this phenomenon is the model’s familiarity with the given prompt or instruction, which is typically estimated by its perplexity. However, finding the prompt with the lowest perplexity is challenging, given the enormous space of possible prompting phrases. In this paper, we propose monotonic paraphrasing (MonoPara), an end-to-end decoding strategy that paraphrases given prompts or instructions into their lower perplexity counterparts based on an ensemble of a paraphrase LM for prompt (or instruction) rewriting, and a target LM (i.e. the prompt or instruction executor) that constrains the generation for lower perplexity. The ensemble decoding process can efficiently paraphrase the original prompt without altering its semantic meaning, while monotonically decrease the perplexity of each generation as calculated by the target LM. We explore in detail both greedy and search-based decoding as two alternative decoding schemes of MonoPara. Notably, MonoPara does not require any training and can monotonically lower the perplexity of the paraphrased prompt or instruction, leading to improved performance of zero-shot LM prompting as evaluated on a wide selection of tasks. In addition, MonoPara is also shown to effectively improve LMs’ generalization on perturbed and unseen task instructions.

## introduction large language models (llms) have demonstrated remarkable proficiency in zero-shot decision making @xcite @xcite and instruction following @xcite @xcite @xcite . however, there can be significant variance in the performance of seemingly similar prompts @xcite @xcite @xcite . despite efforts of studies on prompting lms @xcite @xcite @xcite , it is still challenging to develop high-quality prompts that can induce better performance for varying tasks on evolving models in an effort-saving manner. one consensus reached by recent studies is the inverse relationship between a prompt's perplexity and its task performance @xcite . this stems from the intuition that the frequency of a prompt (or an instruction) foot_1 appearing in the (pre-)training data positively influences the model's familiarity with it, thereby enhancing its capability to perform the described task @xcite @xcite . as a result, existing attempts use specific criteria, especially perplexity for selecting prompts from a collection of candidates. for example, @xcite builds a large prompt pool for each task and selects the one with the lowest perplexity. however, the performance of such rewrite-then-select methods @xcite @xcite is limited by the size, quality, and even availability of the candidate pool. searching the prompt with the lowest perplexity for a particular task remains challenging due to the vast expanse of potential prompts. to address this challenge, we propose a novel progressive end-to-end prompt refining approach, namely monotonic paraphrasing (monopara), that can proactively rewrite the given prompt into its low-perplexity counterpart without compromis-ing its expressiveness of the task. during prompt (or instruction) refinement, monopara conducts an ensemble decoding process of a paraphrasing model together with the target lm. the paraphrase model is instructed to rewrite the given prompt of a specific task, while the target model, which is later on used for task inference, provides a constraint that aims to lower the perplexity of the generation. monopara iteratively decodes tokens based on the ensemble of these two models. intuitively, the paraphrase model can genuinely paraphrase the original prompt without altering its semantic meaning and remain instructive for the given task, while the target model restricts the search space of paraphrased tokens to those with low perplexity, resulting in a task prompt that the target model is more familiar with. further, for the decoding process of monopara, we explore two decoding schemes. in addition to greedy decoding ( §2.3) that iteratively generates the next token based on the weighted combination of predicted probabilities from both models, we also explored search-based decoding ( §2.4). the search-based decoding scheme follows the look-ahead decoding paradigm @xcite , which keeps several sequence candidates and maintains the one with the lowest perplexity scored by the target model. compared to recent prompt refinement approaches @xcite @xcite , monopara needs zero training effort and far less computational cost, and do not require the pre-existence of multiple prompt or instruction candidates @xcite . we find that by an ensemble of two lms, we are able to consistently decode lower perplexity prompts, which would, in turn, bring about better generalization of lms on downstream tasks. our contributions are three-fold. first, we propose monopara, a prompt refinement method that paraphrases prompts to be more familiar with the target model for boosted generalization and task performance. second, we explore and compare two distinct decoding strategies that proactively refine prompts by monotonically lowering their perplexity and maintaining their expressiveness of the task. third, we conduct experimentation for both prompt refinement for regular lms and instruction refinement for instruction-tuned lms to illustrate that monotonic paraphrasing improves the generalization of lm prompting. ## monotonic paraphrasing we propose monopara as an ensemble-based decoding method that can refine a given prompt by paraphrasing it into a low-perplexity counterpart. we first provide the intuition of monopara in §2.1 and the basics of prompt paraphrasing in §2.2. then, we formally define two decoding schemes of monopara in §2.3 and §2.4. ## https://huggingface.co/docs/datasets/index evaluation metrics to inspect whether monotonic paraphrasing can refine a prompt into its low-perplexity counterpart and further enhance the generalization of llm prompting, we use ppl (perplexity), acc (accuracy), and bs (bertscore, zhang* et al., 2020) for evaluation. (1) perplexity. following gonen et al. (2023), we define the perplexity of the prompt as the perplexity of the full prompt sequence, including the input itself and the choices of labels. to avoid noise when computing perplexity, the prompts are instantiated with 1,000 random examples of the dataset (we keep the same selected examples for performance evaluation), and the perplexity is averaged across all instantiated prompts. the correlation between the perplexity of a standalone prompt and the average perplexity over instantiated ones is illustrated in appx. §e. (2) bertscore. as a paraphrasing scheme, monopara is also evaluated by its performance in semantic alignment with the source sentence based on pre-trained bert @xcite . (3) accuracy. to compute accuracy, for each test sample, we obtain the model's predicted probability of all classes and choose the highest ranking class as the prediction of the model. the accuracy is calculated over the 1,000 samples for each task. ## related work paraphrasing. paraphrase generation has been a longstanding task in nlp. since the era of deep learning, studies have adopted the universal paradigm of training sequence-to-sequence generation models based on abundant learning resources @xcite @xcite . while more recent works have leveraged more robust pre-trained language models that captured rich supervision signals from various sequence-to-sequence generation tasks @xcite and achieving even more robust performance of paraphrasing with relatively limited end-task supervision @xcite @xcite . in addition to these advancements, a series of works have added auxiliary supervision signals such as syntactical guidance, lexical regularization, and cross-lingual supervision, to further improve the performance and controllability of paraphrasing in aspects such as syntactic structures, stylistic specification, or textual simplicity @xcite @xcite @xcite @xcite . different from existing paraphrasing techniques, this work conducts constrained decoding methods that seek to monotonically decrease the perplexity in the process of paraphrasing. this innovative approach serves as a general approach to rewrite and refine prompts or task instructions towards their more familiar counterparts for lms, seeking to enhance the generalization of lm prompting without the need for any training effort. prompt refinement. prompt refinement has been actively explored in recent years with the aim of selecting, retrieving, or even generating prompts that lead to improved zero-shot performance of lm prompting. when an end-task objective is known, prior works on prompt refinement approaches often set the end-task performance as the objective, and leverage approaches such as gradient-based search, similar to continuous prompts but with projections onto a discrete vocabulary @xcite . other works have explored edit-based enumeration @xcite , reinforcement learning @xcite , and large language model continuation and filtering @xcite . as for general-purpose llms, a dedicated prompt refinement approach may not assume the pre-existence of any specific end tasks. hence, more recent works @xcite ; @xcite ; @xcite explored the heuristic criterion based on the familiarity of the language model with the language contained in the prompt, as measured by its perplexity. lower perplexity prompts are preferred as they are expected to perform better across a wide range of tasks. while the last line of research has relieved the generic principle for improving zero-shot llm prompting generalizability, these studies are largely limited to selecting among pre-existing prompts or multiple beam search samples. our proposed monotonic paraphrasing approach, however, can proactively rewrite a prompt or task instruction into counterparts that better satisfy the above heuristic criterion, without the need of any pre-existing candidate prompts or any training effort. ensemble and search-based decoding. the proposed two decoding schemes of monopara are connected to both ensemble decoding and searchbased decoding approaches in recent studies. ensemble decoding has been previously proposed for purposes including debiasing @xcite , controllable generation @xcite , and cross-modality data processing @xcite . search-based decoding, on the other hand, is proposed to efficiently extend the search space of generation samples for satisfying specific constraints or criteria @xcite @xcite . representative approaches include look-ahead @xcite and look-back @xcite ones that search within the beam space towards different directions. the explored decoding schemes in monopara are inspired by both lines of studies on decoding approaches. specifically, the first line is related to our design choice for composing the probabilities of both the paraphrasing and the target lms in greedy generation. and our search-based decoding scheme is specifically inspired by the look-ahead decoding strategy in the second line of studies. ## conclusion in this paper, we propose monotonic paraphrasing (monopara) for automatically and iteratively generating low-perplexity prompts for given llms, which can lead to better task performance. following the high-level idea of paraphrasing prompts with perplexity constraints from the target llm, we design ensemble-based and search-based decoding strategies for efficient prompt refinement. experiments on prompt variations of unseen tasks and instructions demonstrate the effectiveness of monopara in reducing prompt perplexity while enhancing task performance and model robustness. future work may apply monopara on other scenarios, such as searching stealthy prompts for redteaming. replacing perplexity with other criteria, such as complexity @xcite , for different purposes of constrained generation is also a meaningful research direction.