PIP : Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation (2023)

Abstract: Syntactically controlled paraphrase generation requires language models to generate paraphrases for sentences according to specific syntactic structures. Existing fine-tuning methods on this task is costly, as all parameters of the model need to be updated during the training process. Inspired by recent studies on parameter-efficient learning, we propose Parse-Instructed Prefix (PIP), a novel adaptation of prefix-tuning to tune large pre-trained language models on syntactically controlled paraphrase generation task in a low-data setting with significantly less training cost. We introduce two methods to instruct a model’s encoder prefix to capture syntax-related knowledge: direct initiation (PIP-Direct) and indirect optimization (PIP-Indirect). Comparing to traditional fine-tuning methods for this task, PIP is a compute-efficient alternative with 10 times less learnable parameters. Comparing to existing prefix-tuning methods, PIP excels at capturing syntax control information, achieving significantly higher performance at the same level of learnable parameter count.

## introduction syntactically controlled paraphrase generation (scpg) has attracted increasing attention as it can diversify the generated paraphrases @xcite @xcite . given an input sentence and a target syntax specification, an scpg model aims to generate paraphrases that satisfy the specific syntax requirement. such generation systems are promising in benefiting multiple application areas in natural language processing (nlp), such as text summarization @xcite , dialogue systems @xcite , diverse question generation @xcite , creative generation @xcite , and improving the robustness of models @xcite . however, prior studies on scpg mainly explore fine-tuning strategies, which require updating the parameters of the entire language model to adapt to the newly included syntax information. therefore, many previously proposed methods suffer from tremendous training cost @xcite @xcite . with the recent rise of larger pre-trained language models (plms), this problem has become even more imminent. nevertheless, a lightweight and more resource-efficient tuning method would allow easier application of large plms on the scpg task. resource-efficient training methods such as prompt-tuning and prefix-tuning (li and liang, 2021; @xcite have proven to be effective in tuning large plms on various nlp tasks, such as text classification @xcite , sequence labeling @xcite , and summarization @xcite . prefix-tuning freezes a plm's parameters and optimizes a small taskoriented continuous prefix that is prepended to the model's transformer layers. it is a promising alternative to fine-tuning in a low-data setting with significantly fewer learnable parameters. however, no previous literature has explored the potential of prefix-tuning on the scpg task. in light of the lack of previous studies, we are amongst the first to study the application of resource-efficient training methods on the scpg task. our work has two main contributions. to begin with, we are among the first to study prefix-tuning's application on the scpg task as a compute-efficient alternative for fine-tuning. secondly, we propose parse-instructed prefix (pip), a novel adaptation of prefix-tuning for enhanced syntax control in paraphrase generation. similar to prefix-tuning, pip freezes all parameters of a plm and only optimizes the prefix parameters, reducing the number of tune-able parameters to almost 10× less than that required for fine-tuning. prefix-tuning methods initialize the prefix as continuous and completely free parameters. for the scpg task, this means that the prefix would need to learn the syntax control from scratch, since the plms were not pre-trained on any syntax-related task. in contrast, pip provides syntax-related guidance to the prefix, allowing for better capturing of syntax knowledge. specifically, we introduce two methods to guide the process of syntax knowledge capturing: direct initiation and indirect optimization. we prove that prefix-tuning-based methods achieve promising performance in a low-data setting with significantly fewer learnable parameters. in addition, our proposed pip methods outperform prefix-tuning at the same level of training cost. foot_0 2 related work syntactically controlled paraphrase generation. for the scpg task, given a source sentence and a target syntax structure, a language model is trained to output a paraphrase sentence of the source sentence that (1) is semantically similar to the source sentence, and (2) conforms to the given target syntax structure, or the "syntax control". prior works mainly adopted encoder-decoder model structures and used sequence-to-sequence training for the scpg task @xcite @xcite , while exploring different means to include syntax control signal during training. the first type of approach encodes the source sentence and the target syntactic tree separately, then concatenates them at decoder input @xcite . the second type of approach concatenates linearized target constituency parse and source sentence at model input @xcite @xcite . however, the aforementioned methods require updating all model parameters during tuning at a high training cost. prompt-tuning and prefix-tuning. prompting @xcite provides plms with a discrete task-specific "prompt" to generate task-related outputs without task-specific fine-tuning. prompt-tuning-based methods @xcite @xcite @xcite , prefix-tuning (li and liang, 2021) and p-tuning v2 @xcite derived from prompting and propose to only optimize a small sequence of continuous vectors. however, since prefix-tuning learns a prefix that was initiated as a continuous vector with completely free parameters, the prefix would need to learn task information from scratch during training. in addition, the training process for prefixtuning does not allow for incorporation of any taskspecific guidance. in summary, existing prefixbased methods fail to consider both specific task instruction and model-learned knowledge. prefix-tuning. we investigate a resourceefficient method for training an scpg model based on the prefix-tuning method @xcite . li and liang (2021) freezes all pre-trained lm parameters and only optimizes a small sequence of continuous prefixes that are then prepended to keys and values of the attention module in the input layer of the model's encoder and decoder. @xcite further extends this approach and applies prefixes to every layer of the model encoder and decoder. we follow the previous approach (li and liang, 2021) and consider prepending additional prefix parameters to the key and value matrices of each transformer layer in the plm. specifically, we establish a prefix p with length |p| for a plm with l layers and hidden dimension dim(h) and produce a set of key prefixes k @xmath0, k 2 , ..., k l } and a set of value prefixes v @xmath1, v 2 , ..., v l }, where k i , v @xmath2, respectively. for an encoder-decoder plm, the key and value prefixes will then influence the model's encoding and decoding process through the attention mechanism, where the prefixes directly attend with the hidden states of the model. figure 1 visualizes structure of the prefix-tuning method. ## experiments we conducted experiments on our proposed pip-direct and pip-indirect methods, as well as two baseline methods for comparison. all four training methods are implemented on the bart-base model @xcite . for all models, we concatenate the source sentence and target parse as input, and train the models to output a paraphrase that follows the given target syntax. dataset. we use paranmt @xcite as the training and validation dataset for all models. specifically, we sample 30,000 and 6,400 data entries from paranmt as our training set and dev set, respectively. to test the models' abilities to generate paraphrases with syntax control in unseen domains, we follow previous work @xcite and apply the trained models on three mainstream test datasets: quora @xcite , mrpc @xcite , and pan @xcite . evaluation metrics. conforming to prior works @xcite , we evaluate generations of models on both alignment-based and syntactic conformation metrics. alignment-based metrics measure the similarity between target paraphrases and model-generated paraphrases. we consider 4 alignment-based evaluation metrics: bleu @xcite , rouge-1, rouge-2, and rouge-l @xcite . syntactic conformation metrics measure the quality of syntactic control in generated paraphrases. we consider 2 syntactic conformation evaluation metrics: template matching accuracy (tma) and tree-edit distances score @xcite at height three (ted-3). baselines. we establish 2 baseline training methods for our study. the first baseline is the vanilla fine-tuning method that updates all parameters in the plm, which we denote as seq2seq, in addition, we consider prefix-tuning, which freezes the plm and learns a small set of parameters as the prefix, as the second baseline. experiments on prefix-tuning in this study are based on our implementation of li and liang and liu et al.'s work. implementation details. we train all baselines and our proposed pip methods for 10 epochs with batch size 64. at decoding stage, we use beam search with beam size 4. we use the adamw optimizer @xcite and apply gradient clipping for training all models. for finetuning, we set learning rate to 10 -5 with a linear scheduler. for prefix-tuning and pip methods, we set learning rate to 3 × 10 -4 . ## conclusion this research is amongst the first to study resourceefficient training methods for syntactically controlled paraphrase generation tasks. in this work, we proposed parse-instructed prefix (pip), a compute-efficient method that only requires 10× less learnable parameters than traditional finetuning methods. we introduce direct and indirect pip methods to further improve prefix-tuning's performance by providing task-specific guidance and augmenting task-related knowledge at the finetuning stage. through extensive experiments, we find out that both pip-direct and pip-indirect outperform prefix-tuning in a low-data setting at the same level of parameter count, and are promising as a resource-efficient alternative to fine-tuning. with ablation studies, we further validate that the performance gain of the proposed pip methods is due to the effectiveness of the design.