Multi-task Learning for Paraphrase Generation With Keyword and Part-of-Speech Reconstruction (2022)

Abstract: Paraphrase generation using deep learning has been a research hotspot of natural language processing in the past few years. While previous studies tackle the problem from different aspects, the essence of paraphrase generation is to retain the key semantics of the source sentence and rewrite the rest of the content. Inspired by this observation, we propose a novel two-stage model, PGKPR, for paraphrase generation with keyword and part-of-speech reconstruction. The rationale is to capture simultaneously the possible keywords of a source sentence and the relations between them to facilitate the rewriting. In the first stage, we identify the possible keywords using a prediction attribution technique, where the words obtaining higher attribution scores are more likely to be the keywords. In the second stage, we train a transformer-based model via multi-task learning for paraphrase generation. The novel learning task is the reconstruction of the keywords and part-of-speech tags, respectively, from a perturbed sequence of the source sentence. The learned encodings are then decoded to generate the paraphrase. We conduct the experiments on two commonly-used datasets, and demonstrate the superior performance of PGKPR over comparative models on multiple evaluation metrics.

## introduction the task of paraphrase generation is to rephrase a given sentence by preserving its key semantics. while the problem was solved using rulebased approaches @xcite and traditional machine learning techniques @xcite , recent attentions have been shifted to devising effective deep neural networks @xcite @xcite , which generally adopt the encoder-decoder framework. more recently, controllable paraphrase generation has been extensively investigated and offers the mechanisms to guide the generation process by providing a reference such as a syntactic template @xcite @xcite , a sentential exemplar @xcite and so on. src what are good workouts to lose belly fat ? pos [wdt] [vbp] [to] [.] tgt what is the best way to lose belly fat ? pos [wp] [vbz] [dt] [to] [.] gnt what are some good exercises to get rid of belly fat ? pos [wdt] [vbp] [dt] [to] [.] although the problem has been studied from different aspects, the fundamental goal of paraphrase generation is to preserve the key semantics of a source sentence and rewrite the rest of the content. taking the paraphrase pair in specifically, we propose a novel two-stage model, pgkpr, for paraphrase generation with keyword and part-of-speech reconstruction. in the first stage (section 3), we fine-tune a bert model to identify the keywords in a source sentence. the identification is based on a prediction attribution technique @xcite ) that computes the gradient vector of each input word. we compute as the attribution score of each input word the l2norm of the corresponding gradient vector, where the words with higher scores are more likely to be the keywords. in the second stage(section 4), we adopt transformer @xcite and devise a multi-task learning model for paraphrase generation. given a pair of paraphrase sentences, the learning tasks include 1) reconstructing the keywords and the pos tags of all words in the source sentence, 2) distinguishing the latent features of the pair from the features of non-paraphrase pairs, and 3) generating the paraphrase sentence. finally, the objective function is the combination of the loss function in each learning task. in the experiments, we show that pgkpr outperforms the comparative models by a notable margin on both bleu and rouge scores. the ablation study shows the effectiveness of each learning task, and the case study and user study show that pgkpr could produce paraphrases with higher quality. a similar study was conducted by @xcite , where they proposed a novel identification algorithm, psi, to identify the primary and secondary content in a source sentence. our work differs from theirs at least on the following three aspects. first, our strategy for keyword identification is purely data-driven, whereas the psi algorithm uses a rule-based method and is sensitive to the similarity measurement used in the algorithm. second, the pgkpr model is trained with multiple learning tasks, whereas the ianet model proposed in @xcite only has the learning task of predicting the target sentence. third, pgkpr determines the keywords in a source sentence with the probability transformed from the attribution scores, which gives the model a more flexible way to separate the keywords and the other content, whereas the ianet model deterministically separates the primary and secondary content using a manually-tuned threshold based on the psi scores. ## stage one: keyword prediction in the first stage, we train a bert model to predict the keywords in a source sentence. the prediction is based on an attribution technique that computes the gradients of the input elements @xcite . in particular, given a binary classification model f and an input sequence @xmath0, x 2 , . . . , x l }, where l is the number of input elements (i.e., the sequence length), the gradient vector g i of x i (1 ≤ @xmath1, based on the technique, we devise the following training task for keyword prediction. denote by n the number of paraphrase pairs in the training set, and (s i , t i ) the source sentence and the target sentence of the i th pair, respectively, 1 ≤ @xmath2. we first construct n positive data points (i.e., each data point corresponds to a paraphrase pair) where the i th data point consists of s i , the special token after fine-tuning, given a new data point consisting of a source sentence and its paraphrase (the source sentence itself during inference), we first compute the output in the forward pass, and then compute the attribution scores of all the input words in the backward pass. since the attribution score reflect how much each word contributes to the paraphrase prediction, the words with higher scores are more likely to be the keywords that capture the common semantics of the two sentences. for keyword prediction, we just use the attribution scores of the words in the source sentence. figure 1 shows the inference process for predicting the keywords of the source sentence in the running example. we observe the five words "good", "workouts", "lose", "belly" and "fat" are more likely to be the keywords. 4 stage two: multi-task learning for paraphrase generation figure 2 shows the overview of the second stage. the model is trained simultaneously with three tasks: reconstruction of keywords and pos tags, contrastive learning for distinguishing paraphrase pairs from others, and paraphrase generation. ## performance evaluation we implement all the models using pytorch 1.4 and run all experiments on a centos machine installed with tesla v100. ## conclusion we propose a new model with multi-task learning for paraphrase generation. the motivation is to simultaneously capture the key semantics of a source sentence and the relations between the keywords. the proposed model, pgkpr, has two stages. in the first stage, pgkpr leverages a datadriven technique to identify the possible keywords in the source sentence. in the second stage, pgkpr adopts the transformer model and devises three learning tasks, including 1) reconstructing the keywords and the pos tags of all words in the source sentence, 2) contrastive learning for distinguishing the latent features of the paraphrase pair from others, and 3) generating the paraphrase sentence. we conduct extensive experiments to show the superior performance of pgkpr over comparative models, as well as the effect of the keyword identification strategy and the multiple learning tasks. our future research would focus on how to apply the model in the current study to controllable paraphrase generation and produce more diverse sentences.