Guiding Zero-Shot Paraphrase Generation with Fine-Grained Control Tokens (2023)

Abstract: Sequence-to-sequence paraphrase generation models often struggle with the generation of diverse paraphrases. This deficiency constrains the viability of leveraging paraphrase generation in different Natural Language Processing tasks. We propose a translation-based guided paraphrase generation model that learns useful features for promoting surface form variation in generated paraphrases from cross-lingual parallel data. Our proposed method leverages multilingual neural machine translation pretraining to learn zero-shot paraphrasing. Furthermore, we incorporate dedicated prefix tokens into the training of the machine translation models to promote variation. The prefix tokens are designed to affect various linguistic features related to surface form realizations, and can be applied during inference to guide the decoding process towards a desired solution. We assess the proposed guided model on paraphrase generation in three languages, English, Finnish, and Swedish, and provide analysis on the feasibility of the prefix tokens to guided paraphrasing. Our analysis suggests that the attributes represented by the prefix tokens are useful in promoting variation, by pushing the paraphrases generated by the guided model to diverge from the input sentence while preserving semantics conveyed by the sentence well.

## introduction paraphrasing is a way of conveying some given meaning using different wording. automatic paraphrase generation aims to produce sequences that carry similar semantics to some arbitrary input sentence but are realized in different surface forms. to increase variation, we propose the training of a guided multilingual neural machine translation (nmt) system that can be applied to diverse zero-shot paraphrase generation by leveraging dedicated prefix tokens designed to enhance variation. we train our multilingual translation system in english, finnish, and swedish, and apply it to guided zero-shot paraphrasing in the three languages. the model does not see parallel monolingual sentence pairs during training, but we guide it to produce monolingual paraphrases during inference. during training, our proposed model learns the semantics of a set of dedicated prefix tokens that are designed to capture certain attributes of language, and can be used for promoting diversity in generated text during inference. the attributes we consider are length, lexical variation, word order, and negation. when generating paraphrases, we can thus guide the model to produce sentences that vary in the given attributes by assigning corresponding values to the prefix tokens. apart from a original paraphrase they are excellent dancers. they dance extremely well. the dinner will be served in the dining area. the dining area is where the dinner will be served. he enjoys playing the guitar. playing the guitar brings him joy. by evaluating the applicability of multilingual nmt pretraining with prefix tokens to paraphrasing, we analyze whether the dedicated prefix tokens increase variation in sequence-to-sequence-based paraphrasing. we asses the generated sequences with respect to the references using bleu @xcite , and analyze the ranking of generated correct references using mean reciprocal rank. additionally, we analyze how faithful the model is to the given instructions during decoding by comparing the accuracy of the guided model outputs to the prefix tokens. we also apply the models to a novel test suite @xcite , designed for analyzing how language models represent negation. our analysis suggests that the paraphrases generated by the proposed model are more diverse compared to the baseline model, especially when selecting hypotheses from n-best lists with smaller n-sizes, and preserve semantics of the original sentence well. the main advantage of our approach is that we train our system on parallel cross-lingual translation pairs rather than monolingual paraphrase data. translation examples are available for a far larger number of languages and in larger quantities than monolingual paraphrases. as a result, our approach can be extended to a considerably larger number of languages than models that depend on existing paraphrase data. furthermore, our model is not tied to diversity in the monolingual paraphrase examples in obtaining variation in the generated sequences. as we use cross-lingual training examples, the model can learn characteristics that might not be prominent in the existing paraphrase data sets. for instance, large language models do not reliably represent negation @xcite , which can be a result of not having a sufficient number of such examples in the training data. we show that the proposed model can learn the semantics of a set of dedicated guiding tokens, for instance a token for negation from sentence pairs where an explicit negation occurs, and that these tokens can then be used to guide the decoder to produce sentences with desired characteristics. finally, we show that, especially when selecting hypotheses from smaller n-best lists, the guided paraphrase generation model goes beyond variation that can be achieved by filtering beam search @xcite , as the prefix tokens provide more control for variation. ## previous research previous research has studied paraphrase generation inspired by nmt systems. @xcite use a deep lstm network for paraphrase generation using monolingual parallel training data. @xcite train encoder-decoder-based paraphrase generation systems for six languages, likewise using paraphrastic sentence pairs. as an alternative to paraphrase data, crosslingual parallel data has been used for finding paraphrases @xcite @xcite ; inter alia). @xcite generate paraphrases via bilingual pivoting using a nmt system. similarly, models based on nmt have been used in generating synthetic paraphrase pairs for learning paraphrastic sentence embeddings @xcite . additionally, multilingual nmt systems have been applied for paraphrase generation leveraging both parallel and monolingual data @xcite , and assessing generalization to zero-shot paraphrasing while also promoting variation in the generated sequences by penalizing matching tokens in the source and output sentences @xcite . zero-shot paraphrasing using large multilingual language models has also been explored @xcite . exploiting various linguistic features to control the decoding process of sequence-to-sequence models has been studied in different nlp tasks and granularities. auxiliary control tokens have been used for controlling the language of the output in multilingual nmt @xcite . @xcite use various features for controlling output translations from a nmt system. in addition to prefix-based control tokens, they use vectorbased interventions that guide the decoding process to certain directions. the complexity of the generated translations have been controlled by utilizing reading level tags, and by partitioning data based on reading or grade levels @xcite . @xcite and @xcite control length of the translated sequences with control tokens. additionally, control tokens have been used with nmt systems for instance in domain adaptation @xcite , formality transfer @xcite , and voice control @xcite . outside of machine translation, control tokens have been used successfully for instance in sentence simplification @xcite . additionally, control tokens have been applied to sentences mined from the internet to obtain synthetic simplification data @xcite . in paraphrase generation, additional linguistic information obtained from the training data has been used for example in syntactic guiding @xcite @xcite . our approach to promoting variation is inspired by @xcite and @xcite . we leverage existing translation corpora to learn controlled zero-shot paraphrasing using dedicated prefix tokens whose semantics the model learns directly from the training data. our control tokens are designed to affect various properties of natural language. however, unlike @xcite , we do not assess our model on machine translation, but take one step further, and evaluate it in zero-shot paraphrasing. we do not only attempt at increasing variation in lexical choices or divergence in syntactic realizations, for instance, but aim to affect both concurrently. ## guiding attributes to guide the decoding process, we need a method for signaling which decisions the decoder should take. here, we use a prefix token-based approach, where we extract certain features from the sourcetarget pairs in the training data, and concatenate the extracted information to the source side in the form of prefix tokens. we let the model learn to represent the semantics of each prefix token from the information incorporated in the translation pairs. consequently, we can guide the decoding process of the proposed paraphrase model by applying these prefix tokens in monolingual transformation triggered by a target language token. to promote variation in the generated paraphrases, we use the following attributes to control for various properties of natural language: length, lexical variation, word order, and negation. ## experiments we train two multilingual nmt models for english, finnish and swedish from scratch: a baseline model without prefix tokens apart from the target language token, and our proposed model with prefix tokens. both models are based on the transformer architecture @xcite , and trained using opennmt @xcite with standard hyperparameters for training a transformer. we gather training data from opensubtitles @xcite using opus-tools @xcite , by filtering for one-toone aligned sentences with a time stamp overlap threshold of 0.85. 1 the obtained training set consists of approximately 17 million sentence pairs in three language pairs (en-fi, en-sv, fi-sv). we extract 10 000 sentence pairs from each direction to serve as validation data for tuning the translation models. we train the models for all cross-lingual directions on two gpus for one million steps or until early stopping criteria is met. we evaluate the models on true paraphrase pairs extracted from the opusparcus test sets @xcite . the sizes of the filtered english, finnish and swedish test sets are 723, 669, and 732 sentence pairs, respectively. opusparcus is a sentential paraphrase corpus that consists of paraphrastic bi-texts in six languages, english, finnish, and swedish included. the data is collected from the opensubtitles corpus, and therefore matches the domain of the training data. consequently, the translation models may have seen some of the sentences included in the test sets during training, either on the encoder or the decoder side, but not as parallel monolingual pairs. ## conclusions we propose a paraphrase generation model that is based on multilingual nmt, leveraging crosslingual parallel examples as diverse paraphrase data. we apply dedicated diversity-promoting prefix tokens to the training of the model in order to obtain a paraphrase model designed for guided zero-shot paraphrasing, and compare the model to a baseline paraphrase generation model based on multilingual nmt without prefix guiding. compared to the baseline model, the results suggest that the proposed guided paraphrase generation model benefits significantly from the guiding information, and produces paraphrases that deviate more from the original sentence but maintain the meaning of the original sentence well, especially with lower n-sizes of n-best decoding. the analysis also suggests that there is still room for improvement, and especially the prefix tokens promoting lexical and word order variation are not perfectly optimized. in future work, we would like to further improve the aforementioned prefix tokens by either optimizing the bucketing based on the observed values better, or by modeling the variation promoting attributes directly within a paraphrase generation model. we would also like to evaluate the applica-bility of dedicated guiding attributes with different data sets or transfer tasks, such as simplification. the method could also be expanded to a larger number of languages by fine-tuning existing multilingual nmt models for guided paraphrasing. finally, we plan to explore modular architectures for diverse paraphrasing.