Don’t Change Me! User-Controllable Selective Paraphrase Generation (2021)

Abstract: In the paraphrase generation task, source sentences often contain phrases that should not be altered. Which phrases, however, can be context dependent and can vary by application. Our solution to this challenge is to provide the user with explicit tags that can be placed around any arbitrary segment of text to mean “don’t change me!” when generating a paraphrase; the model learns to explicitly copy these phrases to the output. The contribution of this work is a novel data generation technique using distant supervision that allows us to start with a pretrained sequence-to-sequence model and fine-tune a paraphrase generator that exhibits this behavior, allowing user-controllable paraphrase generation. Additionally, we modify the loss during fine-tuning to explicitly encourage diversity in model output. Our technique is language agnostic, and we report experiments in English and Chinese.

## introduction notions of semantic similarity and paraphrase are highly context dependent. consider "i'm looking for cheap hotels in new york." vs. "what are cheap lodging options in beijing?": from the perspective of intent classification, both express similar intents, but from the perspective of paraphrasing in a community qa application, a user looking for the answer to one question would not find the other response helpful. this is because the location "new york" anchors the information need, and any changes to it would be unacceptable to the user. it is not always the case that named entities are "immutable" in this respect: consider a user looking for vacation destinations in the south of france. from the perspective of an advertiser, there might be good reason to tempt the user with alternative locations such as the italian riviera; the user may even welcome these suggestions. also, it is not always the case that these immutable anchors are named entities: for example, some metrics assign high similarity to antonyms, and so "cheap hotels" and "expensive hotels" might be considered semantically close, but obviously not from the perspective of an end user looking for inexpensive lodging. although whether or not certain words can be changed without affecting the meaning of a sentence is highly dependent on context, the user of a paraphrase generation system usually would know. consider the application of paraphrase generation in a community qa application, where a developer wishes to automatically generate question variants to increase the chances of a semantic match: a naïve system will indeed generate "what are cheap lodging options in beijing?" as a paraphrase to "i'm looking for cheap hotels in new york." what if we provide the user with a way to explicitly tag portions of the input so that a paraphrase generator knows what parts of the input to repeat verbatim? for example, a simple annotation scheme like "what are cheap lodging options in tag beijing /tag ?", where words between tag and /tag should not be paraphrased. we present a paraphrase generator that implements such tags, allowing user-controllable paraphrase generation. on a standard sequence-to-sequence model (mbart), our contribution is a novel data generation technique via distant supervision to fine-tune a paraphrase generator that supports this tagging behavior. to our knowledge, we are the first to describe such a capability-this solves a practical problem that hinders deployment of models in real-world applications. foot_0 our technique is entirely language agnostic, and we report experiments in both english and chinese. additionally, we modify the loss during fine-tuning to explicitly encourage diversity in the paraphrase generation process. ## paraphrase generator we treat paraphrase generation as a standard supervised sequence-to-sequence task by fine-tuning mbart-large @xcite . although our technique can be applied to any sequence-to-sequence framework, we selected mbart because a multilingual model is widely available. following standard practice, custom language tags are used to denote the desired behavior of the model, but otherwise everything in our model is language agnostic. building on our running example, a training pair might be ("what are cheap lodging options in tag beijing /tag ?", "i'm looking for cheap hotels in tag beijing /tag ?"). multiple such corpora exists (without these tags), and this is a straightforward use of mbart, but the challenge is this: starting with an existing paraphrase dataset, how do we know where the tags should go? at inference time, the user supplies the tags, but during training, such knowledge is not available. obviously, we could go through and manually insert tags to existing paraphrase datasets, but such large annotation efforts are impractical. instead, we adopt a solution based on distant supervision by building three different taggers, discussed below. ## experimental setup we describe the data we used for experiments and the basic setup. at a high level, we have two types of datasets, with or without paraphrase clusters (sentences with the same meaning) to evaluate our tagging behavior and model output diver-sity improvement. three english paraphrase cluster datasets were utilized, but no chinese cluster datasets since we did not find such a dataset to be available. one english and one chinese paraphrase pair dataset were used. some of the datasets are widely used in paraphrase generation tasks, the others are adopted from other natural language tasks. below we briefly describe each dataset: ms coco captions 2017 (mscoco) @xcite is an english dataset designed for automatic image caption generation. five humanwritten caption descriptions were collected for each image. the total number of images is about 118k with 590k captions. parabank @xcite is an english sentence re-writing dataset. we took its eval dataset (parabankeval), which contains 400 semantic paraphrase with an average of 14 paraphrases in each cluster. comqa is an english dataset of real user questions from the wikianswers community qa platform. the questions are grouped into paraphrase clusters, of which 1,809 clusters have more than one question. quora question pairs (qqp) contain humanannotated duplicate english questions, with 50k training paraphrase pairs and 20k testing instances. atec is a chinese dataset that comes from ant technology exploration conference developer competition. it contains 14,946 financial question pairs that are semantically similar. for mscoco, parabankeval, and comqa, we randomly picked one sentence from each cluster as the paraphrase source sentence and the rest as the ground-truth references. for the other datasets, we pick one sentence as the source and the other one as the reference. we divided all datasets with 80% in the training set and 20% in the test set. ## results as a preface to our results, we emphasize that, to our knowledge, user-controllable paraphrasing in the manner that we have described is a novel capability. that is, no previous work has addressed this problem-and thus, points of comparison are limited. further note that the point of our technique is not to establish state-of-the-art performance on these various datasets, but rather to illustrate our tagging feature. nevertheless, it is worth noticing that our model outperforms the previous state-of-the-art paraphrase generation model (fu et al., 2019) on mscoco and qqp datasets in terms of rouge scores. we only present the results of rouge 2-gram scores for brevity, but the results of other rouge scores are consistent. we choose rouge @xcite instead of bleu @xcite as our evaluation metrics for both tagger performance and diversity evaluation because of the diversity loss that we introduced. in an ideal dataset, the references of a source sentence should be able to cover all possible paraphrases. in this case, precision-based metrics like bleu should be an even more suitable evaluation metric of generated paraphrases. however, the datasets we experimented on only contain a handful of references, far from covering all possibilities. the substituted words and diversified descriptions encouraged by our minimized mutual information will all be false negatives since they are not covered by the reference, thus leading to low precision. ## conclusions and future work this paper tackles a practical, real-world problem in paraphrase generation that to our knowledge has not been previously addressed: there are tokens that a user might wish to preserve verbatim for a variety of reasons. we further assume, that in many scenarios, the user knows exactly what those tokens are. this leads to our relatively straightforward solution-we provide the user with tags whose semantics are "don't change me". the contribution of our work is a languageagnostic implementation of this capability using a pretrained sequence-to-sequence model (mbart), coupled with an objective that encourages diversity in the parts of the input that are not surrounded by tags. evaluations on both english and chinese paraphrase datasets demonstrate the empirical success of our proposed model, and additional experiments reveal interesting observations about cross-lingual transfer effects, potentially paving the way for future studies.