Sound Natural: Content Rephrasing in Dialog Systems (2020)

Abstract: We introduce a new task of rephrasing for a more natural virtual assistant. Currently, virtual assistants work in the paradigm of intent-slot tagging and the slot values are directly passed as-is to the execution engine. However, this setup fails in some scenarios such as messaging when the query given by the user needs to be changed before repeating it or sending it to another user. For example, for queries like ‘ask my wife if she can pick up the kids’ or ‘remind me to take my pills’, we need to rephrase the content to ‘can you pick up the kids’ and ‘take your pills’. In this paper, we study the problem of rephrasing with messaging as a use case and release a dataset of 3000 pairs of original query and rephrased query. We show that BART, a pre-trained transformers-based masked language model, is a strong baseline for the task, and show improvements by adding a copy-pointer and copy loss to it. We analyze different trade-offs of BART-based and LSTM-based seq2seq models, and propose a distilled LSTM-based seq2seq as the best practical model

## introduction virtual assistants have achieved very high accuracy in parsing queries for execution @xcite , such as reciting the weather or setting a reminder. however, in some scenarios, parsing alone is not enough to execute a request as expected. for example, when the user says "tell alice i'll meet her in 10 minutes", executing the parsed message would send the tagged content "i'll meet her in 10 minutes" instead of a more appropriate message such as "i'll meet you in 10 minutes". the other scenario where rephrasing is needed to better represent the user's request is when the user asks "remind me to brush my teeth tonight". a more natural response * equal contribution would be "ok, i'll remind you to brush your teeth tonight". to make a virtual assistant sound more natural, it needs to rephrase the user's query content before executing it. the task is different from paraphrasing, as we do not want to change the user's wording, i.e., the language formality or choice of words. instead, we need to make minimal syntactic changes to make the utterance sound natural. as a use case, we work on the messaging domain, where we focus on rephrasing a message that needs to be inferred from the user query. this domain is so named as it covers requests about sending and receiving text and instant messages. unlike the confirmation case, the message rephrasing is more complicated and can involve syntactic, pronoun or verb changes. note that our goal is not to paraphrase the user's message but to rephrase it minimally, making it sound more natural when being sent to another user. as such, we need to maintain the semantics and style of the original content. our contributions are as follows: (1) we introduce a new task and release a message content rephrasing (mcr) dataset for this task consisting of 3k queries with tagged content and possible rephrases, (2) we explore various modeling approaches to achieve high accuracy on the mcr and modify existing pre-trained models to accommodate for the nature of this task, and (3) we show that distilling the pre-trained models into simple models can significantly close the performance gap. ## data we first collected a task-oriented dataset of messaging utterances by asking our annotators to come up with natural scenarios in which a user wants to send a message to a second user. we observed that the collected queries contained two distinct types of messaging content: 1) where the content needs to be rephrased (rephrase) and 2) where the content should be used verbatim (exact). as such, the utterances were sent to another set of annotators to mark the message content in the utterances, disregard the ones that do not contain one, and mark whether the utterance belongs to the exact or rephrase class. two annotators needed to agree on the labelling for this task, with a possible third for disagreement resolution. in the event of no resolution after three annotators, the query was reviewed individually. examples of each class is shown in next, we sent the utterances belonging to the rephrase class to a different set of annotators and asked them to rephrase the message content in a way that it would be natural to send to the second user without any additional context. during annotation, our goal was to minimally rephrase a sentence, e.g., keeping the words and attributes (e.g., formality) of the original content as much as possible. in order to ensure high quality, we asked three annotators to independently rephrase the utterances. in around 30% of the cases, there was not a majority (i.e., two or more annotators agree) and we asked a fourth annotator to resolve. most of the disagreements were due to changing words that did not need to be changed for minimal rephrasing but there were cases where the minimal rephrase was not obvious. we will discuss this further when introducing our metrics. overall, we have around 3k examples (almost half for each class) which we split by 70/20/10 for train/test/validation, respectively. we can see from the training data that rephrasing mostly involves making a question and/or changing the subject pronoun. there are other linguistically complex scenarios such as deciding when to use politeness strategies (e.g. "could you pick up milk" as opposed to "can you pick up milk?") among these queries as well. we decided that these complex edge cases were best addressed in future work. as such, we cluster the rephrasing into three main categories. in ## evaluation our goal is to maximize the rephrasing accuracy while also maintaining a very high accuracy on the exact class. our first metric is the exact match (em) accuracy in which the predicted rephrase should be the same as the original content for the exact class and equal to the top rephrased candidate for the rephrase class. the downside of this metric is that for utterances such as 'ask her to pick up her phone', we would penalize rephrases such as 'can you pick up your phone' if the gold label was 'pick up your phone'. in order to smooth this metric, we also use em any in which the rephrased content is correct if it matches any of the provided annotations. since the required changes to rephrase the content are usually small, the bleu score may not be useful. on the other hand, not all the wrong rephrases are equal, e.g., when the model hallucinates. metrics such as bleu can penalize these phenomena more than the em metrics. we also use sari @xcite , which is commonly used for text-editing tasks. it measures the average f1 score of three editing actions for ngrams: keep, add, and delete. ## modeling approaches we assume that the gold tagging for the content inside the query is provided. our base model is an lstm seq2seq model with two-layers for both encoder and decoder using glove @xcite initialized word embeddings (20k vocab size) concatenated with elmo @xcite embeddings to represent the tokens. we also use the pointer-generator mechanism @xcite , which can choose between copying from the source or generating new tokens using a pointer-attention mechanism. as we can see in the copy pointer works as follows: we calculate two token output probabilities; one over the full vocab p t vocab using the standard softmax and another p t copy over the source tokens. to obtain p t copy we use a learned attention between the decoder to generate the output, we weigh between copying and generation using a parameter α mix which is also computed as a function of the hidden states, i.e., p t @xmath0. more precisely: we have shown the lstm results alongside ablation on the elmo and copying mechanism in we also experiment with using bart @xcite for this task. bart is a powerful pre-trained seq2seq model trained on a de-noising objective over massive amount of web data. the training details are listed in the appendix. during our initial experiments with bart, we realized it can replace proper nouns when rephrasing. even though bart is a de-noising autoencoder and it has a high proclivity to copy the source through its encoder-decoder attention heads, it is still done over the whole vocabulary space (50k bpe tokens) and not the dozen of source tokens. to address this pegasus @xcite is pre-trained by generating a selected masked sentence from the input, where some of the selected sentences are not masked. we instead opt to add an explicit copying to bart in the fine-tuning stage. since the pre-trained model has no explicit copy mechanism, adding it naively during the fine-tuning phase as above is not effective. in this case, the decoder prefers to use the well-trained generator instead of a randomly initialized attention head for copying. we use two strategies to mitigate this: (1) we initialize the copying attention head with the average of the last layer's pre-trained decoder attention head, and (2) we also add an explicit loss that forces the decoder to use the copying mechanism when it can. for all the target tokens that can be found in the source, we add a hinge loss: λmax(t -p, 0) to the cross-entropy loss which forces the copying probability p for those token to be above a threshold t . hyper-parameters λ and t are optimized over the validation set, 0.25 and 0.9, respectively. we show results using the bart large model in transferring the language model of massive pretrained models into smaller models has been of high interest recently @xcite @xcite . knowledge transfer to simple models has also been discussed in lesser extent @xcite . we use the sequence-level distillation introduced in @xcite and train the lstm model using the bart output. we found that fine-tuning on the gold labels after the kd step is also beneficial to the performance. ## conclusion in this paper, we introduce a new task of message rephrasing in task-oriented dialog. we release a dataset, mcr, for this task and propose a new model (bart with copy). we show that adding an explicit loss to a pre-trained generative model during fine-tuning can improve the copying performance without hurting its generation power. we also show that by distilling the pre-trained model into a much smaller lstm seq2seq model with copy pointer, we can significantly improve the lstm seq2seq model's language model capability while still keeping its accurate copying.