Learning to Paraphrase for Alignment with LLM Preference (2024)

Abstract: Large Language Models (LLMs) exhibit the issue of paraphrase divergence. This means that when a question is phrased in a slightly different but semantically similar way, LLM may output a wrong response despite being able to answer the original question correctly. Previous research has regarded this issue as a problem of the model’s robustness to question paraphrase and proposed a retraining method to address it. However, retraining faces challenges in meeting the computational costs and privacy security demands of LLMs. In this paper, we regard this issue as a problem of alignment with model preferences and propose PEARL (Preference-drivEn pAraphRase Learning). This is a black-box method that enhances model performance by paraphrasing questions in expressions preferred by the model. We validate PEARL across six datasets spanning three tasks: open-domain QA, commonsense reasoning, and math word problem. Extensive experiments demonstrated not only the outstanding performance but also the composability, transferability, and immense potential of PEARL, shedding new light on the black-box tuning of LLMs.

## introduction the era of large language models (llms) has arrived. with the continuous increase in the scale of parameters and the ongoing development of pretraining methods, llm's overall performance is becoming increasingly powerful @xcite @xcite . however, the rapid development of llm has also brought about some issues. firstly, the surge in parameter scale has led to a substantial increase in computational resources required for tuning them. this limitation restricts the participation of many organizations and individuals, which is not conducive to the healthy development of the community. although some research has proposed methods to optimize the performance of llms with lower computational requirements @xcite @xcite , they primarily are white-box tuning methods, which necessitate access to model parameters. with the continuous evolution of regulations such as gdpr, the security and privacy concerns surrounding llm are increasingly being emphasized @xcite . as a result, some designers of llms are opting to provide services to users through api rather than exposing the complete model. this trend has made it increasingly challenging to apply white-box tuning. therefore, black-box methods are gaining attention, such as retrieving relevant examples @xcite @xcite and suitable prompts @xcite to add to llm's input. a notable phenomenon is the issue of paraphrase divergence in language models. indeed, when a question is phrased in a slightly different but semantically similar way, the language model can output a wrong response despite being able to answer the original question correctly. figure 1 shows three such examples. this can be explained as the language model does not only learn the knowledge itself from the corpus during pre-training but also learns the expression pattern associated with specific knowledge @xcite . @xcite regard this issue as a problem of the model's robustness to question paraphrase and proposed a retraining method to enhance the robustness of the model. however, this method is white-box and requires tuning all parameters of the model, making it difficult to meet the computational costs and privacy security demands of llms. to address these, we regard the paraphrase divergence of llm as a problem of alignment with model preferences and propose pearl (preference-driven paraphrase learning). this is a black-box method that enhances model performance by paraphrasing questions in expressions preferred by the model. specifically, we train a pearl generator to learn the model's preferences for expressions. during inference, the pearl generator generates paraphrases in alignment with model preference and then feeds them into the model. in this paper, the training of the pearl generator uses a seq2seq approach. considering the aspects of the diversity of paraphrase types and the preservation of semantics, we propose a prompt-based method for the automatic construction of final training sets. it's worth noting that the parameter scale of the pearl generator we train is significantly smaller than that of our target llm, and it requires only a small amount of training data. fundamentally, pearl is a process of learning model preference for expressions and incorporates the preference into questions to guide the model in producing correct answers. this parallels prompt learning, where adjusting prompts caters to the expressions that model is accustomed to during pre-training @xcite . however, prompt learning involves incorporating the preference information into the prompts that are independent of the questions. it is important to note that prompts are typically task-specific or domainspecific, meaning that the preference information learned through prompt learning is preserved in a fixed format within the prompts for a particular task or domain. however, questions are not specific to any domain or task, they are constantly changing. this necessitates dynamically incorporates the model's preference information into the questions. the pearl generator can incorporate preference adaptively during paraphrase by learning model preferences for expressions under different semantics. as depicted in figure 1 , the adaptability of the pearl generator can be concretely manifested in its ability to apply different paraphrase types to different questions. the paraphrase type represents the lexical variables manipulated during paraphrase generation @xcite . to validate the effectiveness of our approach, we conducted extensive experiments on six datasets across three tasks: open-domain qa, commonsense reasoning, and math word problem. the experiments demonstrate that pearl significantly enhances model performance. the contributions of this paper are as follows: ## related work paraphrases are texts that convey identical meanings while using different words or structures @xcite , which is a reflection of the complexity and diversity of human language. given a sentence, paraphrase generation aims to craft its paraphrases that are different from the original sentence, while maintaining the original meaning @xcite . many downstream tasks in natural language processing leverage paraphrase generation, such as question answering @xcite , semantic parsing @xcite , dialogue systems @xcite and machine translation @xcite . in the era of llms, leveraging language models for end-to-end paraphrasing ensures both substantial accuracy and diversity while maintaining flexibility and ease of use @xcite . therefore, this approach has become a mainstream choice for paraphrase generation. in this paper, all paraphrases are generated through this end-to-end approach. currently, deploying large language models as services has become a common practice, such as gpt-4 @xcite , claude 3 @xcite . on one hand, fine-tuned models in specific domains may possess knowledge that implicates privacy concerns. on the other hand, we cannot ascertain whether the answers directly provided by the model adhere to legal regulations and ethical norms. sun et al. were the first to propose the concept of lmaas (language model as a service) @xcite , where llms are provided to users through interfaces to accomplish various downstream tasks. and they advocated for optimizing model performance through black-box methods. the pearl method we propose is indeed a black-box approach, enabling optimization of model performance while requiring only access to input-output pairs of the llms. ## training of pearl generator we choose to directly treat preference-driven paraphrase generation as a seq2seq task. therefore, the key to training the pearl generator lies in constructing effective training sets. given an initial training set @xmath0, q 2 , . . . , q n } comprising n questions, our goal is to construct a set of paraphrased sentence pairs @xmath1, p c 1 ) , p b 1 , p d 1 , . . . , (p x n , p y n ) , where each pair consists of two sentences that are synonymous and all paraphrases originate from questions in the initial training set q. the former p x n in the sentence pair is the input to the pearl generator, which represents the question that is not aligned with model preference. the latter p y n in the sentence pair is the target output of the pearl generator, representing the question aligned with model preference. the process of constructing paraphrased sentence pairs can be summarized as generating paraphrases for the questions in the initial training set according to certain rules, and then determining the preference for each question and their paraphrases based on whether the target llm can correctly answer them. many existing pre-trained or tuned language models can be used as paraphrase generators directly given relevant prompts. however, their generation tends to be limited to a few types of paraphrases. for the pearl generator, we aim to include a sufficient variety of paraphrase types in its training data to increase its opportunities to produce effective paraphrases. therefore, we choose to include the type instruction in the prompts of the paraphrase generator, as shown in figure 2 . the type classification we use is detailed in figure 3 . secondly, maintaining the semantic essence of the paraphrases is also crucial. if the paraphrases generated by the pearl generator alter the fundamental meaning, the target llm would not be able to provide correct answers. but this aspect cannot be solely ensured by the capabilities of the paraphrase generator itself, we require the assistance of machine evaluation metrics to help filter out incorrect paraphrases. in this paper, we choose bluert @xcite as the metric for filtering. when the score between the original question and its paraphrase falls below a predetermined threshold θ, we filter out the paraphrased question. finally, we divide synonymous paraphrases into two sets: the preferred paraphrase set p + n and the non-preferred paraphrase set p - n based on feedback from the target llm. then obtain the set of paraphrased sentence pairs @xmath2. moreover, pearl generator does not need to paraphrase all questions since some of them may already align with the target llm's preferences. therefore, we augment the training set with a certain amount of identical sentence pairs i, allowing the pearl generator to understand that it can skip paraphrasing questions that are already in alignment with the model's preferences. we control the ratio between the two types of sentence pairs using a parameter λ to obtain the final training set @xmath3. ## conclusion in this paper, we regard the paraphrase divergence of llm as a problem of alignment with model preference. we propose pearl, a black-box method that enhances model performance by paraphrasing questions in expressions preferred by the model. extensive experiments on six datasets across three tasks validate the effectiveness, composability, and transferability of pearl. furthermore, our analysis experiments concretize llm's preferences for expression by analyzing paraphrase types and uncovering significant untapped potential in pearl.