Improving the Diversity of Unsupervised Paraphrasing with Embedding Outputs (2021)

Abstract: We present a novel technique for zero-shot paraphrase generation. The key contribution is an end-to-end multilingual paraphrasing model that is trained using translated parallel corpora to generate paraphrases into “meaning spaces” – replacing the final softmax layer with word embeddings. This architectural modification, plus a training procedure that incorporates an autoencoding objective, enables effective parameter sharing across languages for more fluent monolingual rewriting, and facilitates fluency and diversity in the generated outputs. Our continuous-output paraphrase generation models outperform zero-shot paraphrasing baselines when evaluated on two languages using a battery of computational metrics as well as in human assessment.

## introduction paraphrasing aims to rewrite text while preserving its meaning and achieving a different surface realization. it is an eminently practical task, useful in educational applications @xcite @xcite , information retrieval @xcite @xcite , in dialogue systems @xcite , as well as for data augmentation in a plethora of other tasks @xcite @xcite @xcite . generating diverse and coherent paraphrases is a difficult task. unlike in machine translation, where naturally occurring parallel data in the form of translated news, books and talks are available in abundance on the web, naturally occurring paraphrase corpora are scarce. most common approaches to paraphrasing are based on translation, in the form of bilingual pivoting @xcite or back-translation @xcite . this stems from the hypothesis that if two sentences in a language (e.g. english) have the same translation in another, (e.g. french) they must be paraphrases of each other. while these pipeline approaches bypass the problem of missing data, they propagate errors. further, all neural paraphrasing models (e.g., @xcite @xcite predict discrete tokens through a final softmax layer. we hypothesize that softmax-based architectures restrict the diversity of outputs, biasing the models to copy words and phrases from the input, which has an effect opposite to the intended one in paraphrasing. in this work, we introduce paravmf -a simple and effective method of training paraphrasing models by generating into embedding spaces ( §2). since parallel paraphrasing data is not available even in otherwise high-resource languages like french, we focus on an unsupervised approach. using bilingual parallel corpora, we adapt multilingual machine translation @xcite to monolingual translation. we propose to train this model with translation and autoencoding objectives. the latter helps simplify the training setup by using only one language pair, whereas prior work required multiple language pairs and more data to stabilize training @xcite @xcite . to encourage diversity, we propose to replace the final softmax layer in the decoder with a layer that learns to predict word vectors @xcite . we show that predicting into word meaning representations increases diversity in paraphrasing by generating semantically similar words and phrases which are often neighbors in the embedding space. we evaluate our proposed model on paraphras-ing english and french sentences ( §3). in several setups, standard automatic metrics and human judgment experiments show that our zero-shot paraphrasing model with embedding outputs generates more diverse and fluent paraphrases, compared to state-of-the-art methods ( §4). ## the paravmf model let the language to paraphrase in be l 1 . our goal is to learn a mapping f (x; θ) parameterized by θ. f takes a text @xmath0, x 2 , • • • , x m ) containing m words as input, which can be a sentence or a segment in l 1 . it then generates @xmath1, y 2 , . . . , y n ) of length n in the same language such that x and y are paraphrases. that is, y represents the same meaning as x using different phrasing. we assume that no direct supervision data is available, but there exists a bilingual parallel corpus between l 1 and another language l 2 . we are also given pre-trained embeddings @xcite for words in both l 1 and l 2 . the dimension of both the embedding spaces is d. we use a standard transformer-based encoderdecoder model @xcite as the underlying architecture for f . as visualized in the system diagram presented in the appendix, f is jointly trained to perform three tasks with a shared encoder and decoder: (1) translation from l 1 to l 2 , (2) translation from l 2 to l 1 and (3) reconstructing the input text in l 1 (autoencoding). 2 towards our primary goal of meaning preservation, the translation objectives help the encoder map the inputs in both the languages to a common semantic space, whereas the decoder learns to generate language-specific outputs. on the other hand, with the autoencoding objective, we expose the model to examples where the input and output are in the same language, biasing the model to adhere to the start token supplied to it and decode monolingually. using this training algorithm, we find in our experiments ( §4), that the resulting paraphrases albeit meaning-preserving still lack in diversity. we identify two reasons for this issue. first, the model overfits to the autoencoding objective and just learns to copy the input sentences. we address this issue by using only a small random sample of the total training sentences for training with this objective. 3 second, we find that cross-entropy loss used to train the model results in peaky distributions at each decoding step where the target words get most of the probability mass. this distribution being another signal of overfitting also reduces diversity @xcite . we find in our preliminary experiments, that prior work to address this issue by augmenting diversity inducing objectives to the training loss @xcite often comes at a cost of reducing meaning preservation. in this work, we propose using a different training loss which naturally promotes output diversity. we follow @xcite , and instead of treating each word w in the vocabulary as a discrete unit, we represent it using a unit-normalized pre-trained vector e learned using monolingual corpora @xcite . at each decoding step, instead of predicting a probability distribution over the vocabulary using a softmax layer, we predict a d-dimensional continuous-valued vector ê. we train our proposed model by minimizing von mises-fisher (vmf) loss-a probabilitistic variant of cosine distance-between the predicted vector and the pre-trained vector. at each step of decoding, the output word is generated by finding the closest neighbor (using cosine similarity) of the predicted output vector ê in the pre-trained embedding table. since this loss does not directly optimize for a specific token but for a vector subspace which contains many similar meaning words, we observe that it has a higher tendency to generate diverse outputs than softmax-based models, both at the lexical and syntactic level as we show in our experiments. overall, the contribution of this work is twofold: (1) a translation and autoencoding based training objective to enable paraphrasing while preserving meaning without any parallel paraphrasing data, and (2) optimizing for vector subspaces instead of token probabilities to induce diversity of outputs. implementation we modify the standard seq2seq transformer model in opennmt @xcite to generate word embeddings @xcite , and train it with the vmf loss with respect to target vectors. we initialize and fix the input embeddings of the encoder and decoder with off-the-shelf (sub-word based) fasttext embeddings @xcite for both en and fr and align the embeddings to encourage crosslingual sharing @xcite . with a vocabulary size of 50k for each language, the combined vocabulary size of the encoder and the decoder is 100k. both encoder and decoder consist of 6 layers with 4 attention heads. the model is optimized using adam (kingma and ba, 2015), with batch size 4k, and 0.3 dropout. the hidden dimension size is 1024, the dimension of the embedding layers is 512. we add a linear layer to transform 300-dimensional pre-trained embeddings to 512-dimensional input vectors to the model. after decoding, we postprocess the generated output to replace words from l 2 by a look-up in the dictionary induced from the aligned embedding spaces. baselines although unsupervised methods of paraphrasing with only monolingual data have been explored in recent works @xcite @xcite @xcite they have not been shown to outperform translation based baselines @xcite . hence we compare our proposed approach with translation-based baselines only. first, we compare with bilingual pivoting baselines @xcite which pipeline two separate translation models, l 1 → l 2 , and l 2 → l 1 . we use two bilingual pivoting baselines, one based on continuous-output model (bp-vmf; the output vectors of the first model are first converted to discrete tokens before being fed to the next) and another based on softmax-based model (bp-ce). to evaluate the impact of embedding outputs, we also compare our proposed model paravmf to softmax-based baseline parace, leaving other model components unchanged. parace is a modified bilingual version of the multilingual method proposed in @xcite , the current state-ofthe-art in zero-shot paraphrasing. evaluation setup there are many ways to paraphrase a sentence, but no manually crafted multireference paraphrase datasets exist, that could be used as test sets (and there are no datasets in languages other than english). we thus evaluate the generated paraphrases on semantic similarity and lexical diversity compared to the input text. following prior work, we use the n-gram based metric meteor @xcite . despite accounting for synonyms, it is not well-suited to evaluate paraphrases, since it typically assigns lower scores to novel phrasings, due to incomplete synonym dictionaries. we thus also include bertscore (zhang et al., 2020), computing cosine similarity between the contextual embeddings of two sentences. naturally, just copying the inputs can also lead to high scores in these metrics. to evaluate lexical diversity, we follow @xcite and include iou -intersection over union (also called jaccard index) and word error rate (wer). to measure structural diversity we use (constituency) parse tree edit distance (pted). 4 note that model outputs that do not preserve meaning in paraphrasing (and generate totally different sentences) will also obtain high diversity scores, but these are not indicative of quality paraphrasing but will falsely contribute to high diversity scores if averaged across the entire test set. we thus measure the diversity only on subsets of the test set for which the strongest baseline (parace) and our model generate meaning-preserving paraphrases measured using bertscore thresholds. we report the diversity scores for three such thresholds: 0.95, 0.9, 0.85, selected empirically such that the sample size is sufficiently large. finally, we also evaluate that our results hold on a larger dataset in different domain. we retrain paravmf and parace on 2m en-fr corpus described in §3. 7 the results of automatic evaluation are presented in the appendix. we conduct human evaluation on a sample of 200 sentences from this test set following the same a/b testing procedure as described above, with each sample rated by three annotators, resulting in a pairwise-average kappa agreement index of 0.21. foot_7 42.9% paravmf outputs were selected as better paraphrases, compared to 24.5% outputs from parace, supporting our main results on the iwslt dataset. ## related work bilingual pivoting is a common technique used with bilingual data @xcite @xcite . paranmt @xcite is a large psuedo-parallel paraphrase corpus constructed through back-translation @xcite . @xcite augment it with syntactic constraints for controlled paraphrasing; parabank @xcite improves upon paranmt via lexical constraining of decoding; and parabank 2 @xcite improves the diversity of paraphrases in parabank through a clustering-based approach. note that these works are focused on english. here, we propose a language-independent approach relying only on abundant bilingual data. our approach is most similar to @xcite who use bilingual and multilingual translation for zero-shot paraphrasing. they, however, observe that bilingual models are insufficient for paraphrasing and are often unable to produce the output in the correct language. we incorporate an autoencoding objective which simplifies and stabilizes training, and embedding-based outputs improving the diversity in paraphrasing. ## conclusion we present paravmf, an end-to-end model for generating paraphrases, trained solely with bilingual data, without any paraphrase supervision. we propose to generate paraphrases into meaning