Delexicalized Paraphrase Generation (2020)

Abstract: We present a neural model for paraphrasing and train it to generate delexicalized sentences. We achieve this by creating training data in which each input is paired with a number of reference paraphrases. These sets of reference paraphrases represent a weak type of semantic equivalence based on annotated slots and intents. To understand semantics from different types of slots, other than anonymizing slots, we apply convolutional neural networks (CNN) prior to pooling on slot values and use pointers to locate slots in the output. We show empirically that the generated paraphrases are of high quality, leading to an additional 1.29% exact match on live utterances. We also show that natural language understanding (NLU) tasks, such as intent classification and named entity recognition, can benefit from data augmentation using automatically generated paraphrases.

## introduction paraphrases provide additional ways in which the same semantic meaning can be communicated through text or voice. automatic paraphrase generation can benefit various applications, including question answering @xcite , summarization @xcite and machine translation @xcite . recently, neural paraphrasing methods have been proposed that utilize sequence-to-sequence models @xcite or generative models @xcite . similar to other work @xcite , we apply an encoder-decoder model for paraphrasing, inspired by neural machine translation (nmt). delexicalization unlike general paraphrases, which are typically reformulations of utterances, we paraphrase delexicalized sentences, in which named entities are replaced with generalized slot names. for example, "i want to listen to taylor swift 's shake it off " will be transformed into "i want to listen to {artist}'s {music}." as a result, it is expected that the paraphrasing model will learn more about syntactic variations rather than semantic similarities among words. an example application of our paraphrasing model is third-party skill systems in digital voice assistants such as amazon's alexa. users can extend alexa's capabilities by "skills." these skills are built by third-party developers, using the alexa skills kit (ask), and may cover any specific domain-starbucks orders, uber reservations, jeopardy quizzes, and so on. developers can build skills on the alexa developer console, and start by defining an interactive model including an intent schema, slot types, sample utterances, and an invocation phrase @xcite . the sample utterances can be delexicalized, and include general slots that can be filled by provided slot values. sample json for a developer-defined skill can be found below. our paraphrasing model generates delexicalized utterances that help developers create sample utterances for alexa skills, augmenting the training data of nlu (natural language understanding) models and improving the performance of such models. ## related work to the best of knowledge, our research is the first to generate delexicalized paraphrases by leveraging entity information directly within a neural network. @xcite introduce a similar notion of paraphrasing and apply variational autoencoders to control the quality of paraphrases. @xcite tackle a similar problem of paraphrasing utterances with entity types, but implement the slot copy mechanism via pre-processing and post-processing. in addition, @xcite apply paraphrases to improve natural understanding in an nlu system, both for augmenting rules and for enhancing features. ## model we use the encoder-decoder sequence-to-sequence model @xcite . the encoder embeds an input sentence via transformers @xcite . the decoder is also a transformer model that generates output one token at a time, using information from the encoder and the previous time steps. an attention mechanism @xcite helps the decoder to focus on appropriate regions of the input sentence while producing each output token. a good paraphrase should contain all the entity slots names from the source; some words remain the same in the paraphrase. to facilitate such copies, we use pointers that directly copy input tokens to the output @xcite . as a result, in cases where an input token does not exist in the vocabulary, the model will learn to make the copy based on its embedding and context. figure 1 depicts our proposed architecture. ## experimental setup in this section we introduce the training and evaluation datasets we used, and the deep sequence-tosequence model training environment. dataset we train paraphrase models using data from 58,000 skills, live non-skill utterances from broader domains, and the public dataset ppdb. we then apply these models to the 88 most popular skill in order to obtain paraphrases and calculate evaluation metrics. we generate 6.4 million paraphrase pairs from skills, which form the bulk of the training dataset. we also create another training set by appending an extra 500,000 non-skill paraphrase pairs, and two million pairs from ppdb. however, model performance here is not as remarkable. the public dataset ppdb we used only includes lexicalized paraphrase pairs, and those are generally sentences from the web and from various documents, which are a little different from our use case. in our task, including the public dataset does not seem to provide much extra gain. thus, the discussion below focuses the analysis on results from the skill-only dataset. utterances are delexicalized, and each signature (as defined in section 1) corresponds to a set of delexicalized utterances. we create two source-target pairs for each utterance, by randomly sampling its target from the same set. when training the model with pointers, slots in the target are replaced by respective pointers. we also cleaned up noisy data, so that utterances have reasonable length and contain enough contextual words around entities. as described in section 3.4, the training dataset is reformatted into four different types, with various paraphrasing models trained on each of them. training and inference we implemented the special input embedding layer and transformers with pointers in mxnet 1.5.0 (gluon api). all models are trained with the same hyperparameters for fair comparison. both the transformer encoder and the decoder include 8 heads, 6 layers, a hidden size of 512, and a 0.1 dropout @xcite ratio. the adam optimizer (kingma and ba, 2014) and noam learning rate scheduler are used, with an initial learning rate of 0.35 and 4000 warm-up steps. the model is trained for 40 epochs with a batch size of 1400 and with 8 batches per update. inference is performed with a beam-search decoder. the beam size is 5 and 3-best paraphrases are kept for each input. ## evaluation there are numerous evaluation metrics for sequence generation problems, such as bleu @xcite and rouge @xcite . however, in our case we do not have ground truth for paraphrases and thus it would be hard to directly apply these metrics. we now describe how to evaluate paraphrase generation for the use case of data augmentation, and propose several intrinsic metrics that emphasize different characteristics. we hypothesize that paraphrases which benefit downstream models should have the following properties: divesity, novelty, and good coverage of test data. we describe each of these in detail below. ## results the intrinsic and extrinsic results are presented in ## conclusion we trained and evaluated multiple types of models for paraphrasing delexicalized utterances, motivated to assist skill developers and ultimately to improve the user experience of virtual assistant customers. we experimented with anonymizing entity slots in utterances, applying cnns and pooling on slot entities, and using pointers to locate slots in the output. the generated paraphrases bring about 1,669 exact matches with human utterances in the best model, and also improve nlu tasks, especially for those skills with insufficient training samples. in addition, we showed the benefit of including slot value information in paraphrasing for some skills.