Advancing Seq2seq with Joint Paraphrase Learning (2020)

Abstract: We address the problem of model generalization for sequence to sequence (seq2seq) architectures. We propose going beyond data augmentation via paraphrase-optimized multi-task learning and observe that it is useful in correctly handling unseen sentential paraphrases as inputs. Our models greatly outperform SOTA seq2seq models for semantic parsing on diverse domains (Overnight - up to 3.2% and emrQA - 7%) and Nematus, the winning solution for WMT 2017, for Czech to English translation (CzENG 1.6 - 1.5 BLEU).

## introduction natural language provides a vast number of alternative ways to state something or to ask a question @xcite . this poses a daunting challenge to natural language processing methods because there is no possible way to enumerate all these alternatives. as a result, many popular machine learning systems trained on benchmark datasets are surprisingly fragile to such previously unobserved variations of the training input at test time @xcite @xcite 1 . an attempt to ameliorate this is to augment the training data with paraphrases. regardless of the magnitude of data augmentation, unseen instances may still break the model; data augmentation alone is an insufficient remedy for model brittleness. we propose to go above and beyond data augmentation in handling model generalization for sequence-to-sequence (seq2seq) architectures and improve model generalization to test sets that entirely consist of unseen paraphrases of the training set. assuming that data augmentation already took place in the training set, either by annotation or off-the-shelf paraphrase generation, we propose new models that actively employ the properties of paraphrase-augmented data as part of the training objective. we examine sequence models for diverse tasks across diverse domains, consider state of the art models for each of those tasks and incorporate multi-task paraphrase detection and generation learning. we show that our models compare over and above other popular generalization schemes, such as feature-based or fine-tuned word embeddings @xcite or paraphrase-based methods such as paraphrase embeddings @xcite . the proposed models outperform state-of-the-art models @xcite @xcite when evaluated across a variety of settings on emrqa @xcite and overnight for semantic parsing and czeng 1.6 @xcite with @xcite for machine translation. moreover, even when paraphrase augmentation is not available, we demonstrate that the proposed multi-task models improve model generalization with only synthetic, noisy paraphrases from off-theshelf models. the main contributions of our work are as follows: (1) we propose novel multi-task learning seq2seq models that significantly improve model generalization to unseen paraphrases at test time, in diverse domains (clinical text, 7 domains of overnight, english subtitles). (2) proposed models bring additional major performance boost on top of paraphrase-augmentation, but also work when the training set does not come with paraphrases at all. (3) we introduce new methods of splitting data into train/ test sets that more realistically evaluates model generalization to paraphrases. (4) we present the first competitive baseline for semantic parsing on the emrqa dataset. ## related work dealing with unseen paraphrastic variants of the input has been a fundamental problem @xcite . recently, multiple works have shown that models easily "break" when evaluated on adversarial examples, which are noisy variants of the training inputs @xcite . however, there is relatively little work that go beyond augmentation and actively optimizes paraphrastic generalization along with learning the main nlp task at hand, in neural settings. in non-neural settings, the idea that leveraging paraphrases facilitates modeling sentential semantics has been repeatedly verified across various nlp tasks. in semantic parsing, @xcite deal with understanding the myriad paraphrastic variants in which knowledge base relations can be expressed in human language. they use a paraphrase of the original input utterance as an intermediary, which is used as an ancillary factor in ranking the likelihood of each candidate logical form. in machine translation, @xcite handles unseen source language phrases by substituting paraphrases of those phrases and then translating the paraphrases. in neural settings, the most widespread approach is to simply generate paraphrases for data augmen-tation, as used by @xcite in question answering and @xcite in semantic parsing. there are relatively few approaches that explicitly incorporate pairwise paraphrastic equivalence of inputs as part of the model. in semantic parsing, @xcite applies cnn to learn paraphrase detection in a multi-task manner; @xcite generate the simplest paraphrases for input utterances and use them as intermediaries for mapping input to ouput. in question answering, several multi-task learning works learn paraphrase detection along with the main task; @xcite optimizes a multitask objective (negative cosine similarity) that encourages embeddings of paraphrases to have small angular distance in every other iteration of training. additionally, @xcite uses an auxiliary multi-task learning objective for paraphrase detection in training multi-column convolutional neural networks for structured question answering. both of these works leverage the paraphrase clusters of the wikianswers @xcite dataset. however, @xcite found that their multitask learning method gives almost no advantage. moreover, both works did not analyze which domains or types of validation inputs benefited from paraphrase learning. most importantly, these works are fundamentally and methodologically different from ours, in that they leveraged the paraphrases from wikianswers not as inputs to the main model, but only for learning paraphrase detection. on the other hand, our work uses paraphrase instances for both multi-task paraphrase learning and the main task, which is the driving factor behind the significant performance boost by our models. unlike past methods applicable to single tasks, our work shows improvements across several different problems and domains. also, we introduce the first framework in a neural-sequence-to-sequence setting, unlike past works that apply cnn's or nonneural settings. furthermore, our approach can be generally applied to any state-of-the-art variants of seq2seq, such as nematus @xcite . ## paraphrases & datasets paraphrases are sentences or phrases that convey the same meaning using different wording @xcite . methods to construct paraphrases are largely divided into syntactic variation and substitution @xcite . "does the patient have a history of leukemia?" and "is para. types emrqa syntactic para's    what medication has the patient used for |problem| what medications have been previously used for the treatment of |problem| substitution para's is there any mention of |problem| in the patients record has been the patient ever been considered for |problem| para. types t overnight syntactic para's find an additional author to an efron article who is the other author for the article written by efron substitution para's article that at least two article cites articles cited by two or more articles para. types paraphrase augmented czeng 1.6 syntactic para's it was good in spite of the taste despite the flavor, it felt good substitution para's i took a stool sample from his heart. i took the stool sample after his lungs failed. ## problem statement our setup assumes (1) a paraphrase-augmented dataset, either by annotation or simple off-the-shelf paraphrase generation models, and (2) a baseline seq2seq model @xcite which maps an input sequence to an output sequence. we propose methods to endow additional improvement in model generalization given this setup. naive & strict splitting schemes not all paraphrases are created equal; some paraphrases are much less challenging than others in evaluating model performance. a common yet undesirable scenario in nlp datasets is that the form of input utterances can be repeated across training/ test splits. for example, in overnight's recipe domain, there are several questions in the form of "how many x are there" where x is some recipe-related entity, such as "recipes", "ingredients", "meals", etc. with such datasets, the test set often contains too many repeating forms of the training set. such a train/ test split is an unrealistic evaluation of model generalization. there are numerous ways a user can phrase one's information needs, and all possible forms cannot be seen at training even in the most well-augmented datasets. we propose a new, more realistic way to split paraphrase-augmented data, with the emrqa dataset as an example (figure 2 ). emrqa consists of paraphrase groups of inputs. within a single group, "templates" are filled with clinical entities to produce actual input instances (fig 2 purple box). 2(a) shows a naive splitting scheme where the input instances are split at random. on the other hand, 2(b) is a more realistic scenario where a form that was seen during training never appears at test time. for example, all instances of "has the pat. ever been exposed to |problem|?" belong to training and never when the model is evaluated. on the other hand, all instances of "does this patient have a history of |problem|?" never appear during training yet do so at test time; thus, the model is tested whether it can infer the meaning of this form only from its paraphrased forms seen during training (such as "has the pat. ever been exposed to |problem|?"). while this split is more challenging than the naive one, test instances are still semantically equivalent to some training instance, so the model is expected to catch this and generalize to unseen form. while we used a paraphrase-annotated dataset as an example, 2(b)'s splitting scheme works with nonannotated data, via automatic augmentation with off-the-shelf models. thus, given a seq2seq task where we have a training and test set of input-output pairs and several unseen observations in the test set that are paraphrases of the training observations, we want to learn a model that can generalize accurately to these unseen observations, and preferrably, even to unseen forms in the strict split of fig 2(b) . ## methods: seq2seq with joint paraphrase learning we incorporate auxiliary multi-task learning to the main seq2seq task -learning paraphrase generation (paragen), paraphrase detection (paradetect), and a combination of both tasks (paragen + paradetect). these methods work with any task whose inputs and outputs are sequences, on paraphraseaugmented data. the goal of our models is to inject paraphrastic inductive bias to the encoder hidden state -so that when it is passed to the decoder, paraphrase inputs have the same end results. we achieve this by actively employing natural properties that rise from data augmentation as part of the training objective. first, paragen, pa-radetect, paragen + paradetect sample a paraphrase of the input and leverage it to reduce intraclass variance of paraphrases in the representation space. the sampled paraphrase is a term inside each model's respective multi-task objective, which affects the encoded input embedding in directions that reward paraphrastic homogeneity when back-propagated. however, paraphrase sampling is only required during training; at test time, the multi-task portion of the model is discarded, and the input is passed through the seq2seq model only. this is a realistic test scenario that does not require paraphrase identification among test inputs; the expectation is that the multi-task training has optimized the backbone seq2seq model's parameters for generalization at test time. we introduce notations, with semantic parsing on emrqa as a running example (fig 1 ). x is an input utterance (e.g., "does the patient have a history of leukemia") and p is a paraphrase of it sampled from the training set (e.g., "is leukemia in his clinical history?"). y is the desired output sequence (e.g. "conditionevent ( leukemia ) or symptomevent ( leukemia)"); mapping from x to y is the main task, and l t is the neagtive-log-likelihood (nll) loss for this main task. ŷ, p are output sequence and paraphrase generated by the models. finally, we note that we regard an attention-based @xcite seq2seq with a bidirectional lstm encoder and a lstm decoder, with the dropout probability set to 0.1 @xcite , as the backbone baseline model (figure 3b ). ## experiments we evaluate the proposed models over state-of-theart methods and above existing methods of generalization, on two paraphrase-annotated datasets (emrqa, overnight) and one that is automatically augmented with noisy paraphrases (czeng 1.6). ## discussion: cosine distance analysis (emrqa) to understand the contribution of the proposed paraphrase models, we study the cosine similarity between embeddings of sentence pairs, a general metric in textual similarity and paraphrase detection @xcite in figure 3 . we do this by calculating the similarity between the last hidden state of the encoder for a pair of input utterances in the test set, for the four splits of emrqa's "realistic" splitting scheme. we calculate two metrics: (1) the average cosine similarity between pairs of paraphrase utterances (avg. para cos, blue bar in fig 4 ) and (2) the average difference between cosine similarity of paraphrase pairs and that of non-paraphrase pairs (avg. cos gap, orange bar in fig 4 ). they respectively quantify (1) how homogenously paraphrase utterances are embedded as vectors, and (2) how nonhomogeneously non-paraphrase utterances are embedded; high numbers in both quantities are ideal, if our models behave as intended in the methods section. we observed that paradetect achieves noticeably the highest avg. para cos, and paragen the highest avg. cos gap; paragen + paradetect shows something in between the two but closer to pa-radetect. these cosine statistics of embeddings seem to be indicative of model performance. para-gen + paradetect, which embeds both paraphrases homogenously and non-paraphrases nonhomogenously, performs the best in terms of exact match accuracy; the other two models also achieve higher performance than baseline, with much higher avg. para cos and avg. cos gap than baseline. ## conclusion we presented a new general seq2seq framework where the main task is trained together with a paraphrase-learning objective to enhance model generalization. we also introduced new splitting schemes that reflect realistic evaluation for practical use. our proposed approaches outperform the state-of-the-art across three datasets across diverse domains and tasks.