Visual Information Guided Zero-Shot Paraphrase Generation (2022)

Abstract: Zero-shot paraphrase generation has drawn much attention as the large-scale high-quality paraphrase corpus is limited. Back-translation, also known as the pivot-based method, is typical to this end. Several works leverage different information as ”pivot” such as language, semantic representation and so on. In this paper, we explore using visual information such as image as the ”pivot” of back-translation. Different with the pipeline back-translation method, we propose visual information guided zero-shot paraphrase generation (ViPG) based only on paired image-caption data. It jointly trains an image captioning model and a paraphrasing model and leverage the image captioning model to guide the training of the paraphrasing model. Both automatic evaluation and human evaluation show our model can generate paraphrase with good relevancy, fluency and diversity, and image is a promising kind of pivot for zero-shot paraphrase generation.

## introduction paraphrase generation is a long-standing problem for natural language processing that aims to rewrite a text in other forms while preserving its original semantics. paraphrase generation has many applications in other down-stream tasks, such as machine translation @xcite , semantic parsing @xcite and so on. with the development of supervised seq2seq generation, most paraphrase systems depend on the large-scale aligned paraphrase corpora to train the seq2seq model. this leads to the fact that the quality of aligned corpora is extremely important for training a paraphrase system. however, high-quality paraphrase datasets are still lacking in many domains. to solve this problem, there are a few works focusing on zero-shot paraphrase generation such as back-translation. back-translation makes use of language as pivot and treats the backtranslated text as the paraphrase of the original text. for example, @xcite leveraged multilingual neural machine translation to generate paraphrase @xcite proposed to employ semantic representation as the "pivot language" of back-translation to generate paraphrase. all these works show that back-translation can generate high-quality paraphrase. inspired by back-translation based paraphrase generation, we explore to leverage visual informa-tion (i.e. image in this study) to guide the zero-shot paraphrase generation as similar images or similar partial images may have different captions or descriptions that can be treated as paraphrases. figure 1 shows an example. a naive method is that we can use image as the "pivot language" and generate paraphrase by back-translation with a text-to-image model and an image-to-text model. unfortunately, text-to-image generation is still a challenging task and it is hard to generate an image of sufficient quality from the text. the semantic loss in text-toimage generation is so huge that generating paraphrases using this method is barely possible. another method is that we can use an image captioning model to generate a caption from the image corresponding to the original text, and regard this caption as the paraphrase of the original text to train a supervised paraphrasing model. however, the generated caption may describe different elements in the image with the original text, which leads to huge semantic shift. in this study, we propose visual information guided zero-shot paraphrase generation (vipg), which leverages image information to guide the paraphrase generation based only on paired imagecaption data. we jointly train a specific image captioning model and a paraphrasing model, and leverage the output of the image captioning model to guide the training of the paraphrasing model. this can be regarded as distilling the knowledge of the image captioning model to the paraphrasing model at the word level. experiment results on two datasets show our model substantially outperforms the supervised paraphrasing model trained on paired captioncaption data and it can generate valid paraphrases with high diversity. we also compare our model with other zero-shot paraphrase generation methods such as autoencoder and back-translation, and analyze the strengths and weaknesses of these methods. in all, our contributions can be summarized as follows: there are several works leveraging image caption datasets like mscoco to train the paraphrasing model. prakash et al. (2016) proposed residual-lstm model to generate paraphrase. gupta et al. (2018) found deep generative model such as variational auto-encoder can achieve better performance in paraphrase generation. fu et al. (2019) regraded the bag of word as the latent variable of vae to control the semantic of paraphrase. chen et al. (2020) proposed a semantically consistent and syntactically variational encoder-decoder framework, which uses adversarial learning to ensure the syntactic latent variable be semantic-free. cao and wan (2020) leverage gan to generate multiple diverse paraphrases. lin and wan (2021) raised multi-round paraphrase generation to improve the diversity and leveraged back-translation to maintain the semantic. all these works regard different captions of the same image as paraphrase and leverage caption-caption pairs to train paraphrasing model. there are also some works focus on zero-shot paraphrase generation. mallinson et al. (2017) revisited back-translation paraphrase generation with neural machine translation. cai et al. (2021) leveraged amr as the new pivot of back translation. thompson and post (2020) proposed a novel decoding strategy to generate diverse paraphrase via multilingual translation. liu et al. (2020) leveraged simulated annealing to train unsupervised paraphrase generation model. ## methodology as mentioned earlier, the pipeline back-translation with the "image pivot" can not generate valid paraphrase as the performance of the text-to-image generation model is poor. different from the pipeline method, our proposed method jointly trains an image captioning model and a paraphrasing model, and leverages the output of the image captioning model to guide the training of the paraphrasing model. the rationale is that an image may correspond to different captions with same meaning 1 . the image captioning model with our specific design (i.e., with additional input of object representations) may generate a caption that is different from the original caption for an input image while keeping the same meaning, and this output caption can be treated as the paraphrase of the original caption and it can be used for training the paraphrasing model. our model relies only on image captioning dataset consisting of pairs of image and caption, and it does not need any text paraphrasing corpus and any data of caption pairs of same image. each pair in the training dataset includes an image i and a corresponding caption sentence @xmath0, • • • , w n }, where w i is the i-th word of the sentence and n is the sentence length. in this section, we begin by introducing the initial embeddings of the image and text, followed by describing our multi-modal joint encoder, which employs partial attention to encode the image and 1 note that the different captions provided by human judges for a same image in most existing datasets like mscoco are often semantically inconsistent, so we do not aim to make use of the caption pairs to train the paraphrase model in this study. the text together. then we introduce a decoder with masked object copy mechanism to guide text generation. finally, the objective functions will be detailed. the overall architecture of our model is shown in figure 2 . ## conclusion in this paper, we propose a visual information guided zero-shot paraphrase generation approach. we explore employing image as the "pivot" of the back-translation. instead of using a pipeline backtranslation, we jointly train an image captioning model and a paraphrasing model together. we leverage the image captioning model to guide the training of the paraphrasing model. both automatic evaluation and human evaluation show the competitive performance of our model. in the future, we will explore huge-scale image caption dataset to train our model and test the model's ability on other domains. moreover, leveraging video as pivot for paraphrase generation is also an interesting research direction.