Iterative Paraphrastic Augmentation with Discriminative Span Alignment (2021)

Abstract: We introduce a novel paraphrastic augmentation strategy based on sentence-level lexically constrained paraphrasing and discriminative span alignment. Our approach allows for the large-scale expansion of existing datasets or the rapid creation of new datasets using a small, manually produced seed corpus. We demonstrate our approach with experiments on the Berkeley FrameNet Project, a large-scale language understanding effort spanning more than two decades of human labor. With four days of training data collection for a span alignment model and one day of parallel compute, we automatically generate and release to the community 495,300 unique (Frame,Trigger) pairs in diverse sentential contexts, a roughly 50-fold expansion atop FrameNet v1.7. The resulting dataset is intrinsically and extrinsically evaluated in detail, showing positive results on a downstream task.

## introduction data augmentation is the process of automatically increasing the size or diversity of a dataset with the goal of improving performance on a task of interest. it has been applied in many areas of machine learning including computer vision @xcite and speech recognition @xcite . with text-based datasets in particular, paraphrastic augmentation, a technique to automatically expand datasets in their overall size and lexico-syntactic diversity via the use of a paraphrase model, may be applied. in general, a paraphrase model outputs a sentence s ′ given an input sentence s such that meaning(s) ≈ meaning(s ′ ) and @xmath0. prior work has demonstrated that paraphrastically augmented datasets are beneficial when applied to a variety of sentence-level tasks including machine translation, natural language inference, and intent classification @xcite @xcite . often in paraphrastic augmentation an input sentence is rewritten one or more times, with the assumption the transformed sentence(s) preserve the original label. for example, in sentiment analysis, data consists of (sentence i , label i ) pairs, where each label i is in {0, 1}, indicating negative or positive sentiment. to augment this kind of dataset, we can paraphrase each sentence i with a model f and thereby produce an additional (f (sentence i ), label i ) pair, doubling the size of the dataset. in many natural language understanding tasks, however, data contains span labels of the form: (sentence i , {(start i,1 , end i,1 , type i,1 ), ...}), where the latter element is a set of tuples indicating each label's location (as a contiguous subsequence of the input tokens) and type. in this paper, we develop a data augmentation strategy for span labeling problems where we are concerned with balancing the joint objectives of finding different ways to express meaning at the level of a word or phrase while ensuring the paraphrase is sensitive to the context of the surrounding sentence. although a paraphrase is expected to have the same meaning as the sentence from which it was generated, words and phrases are usually added, removed, or reordered. thus for a given sentence annotated with span labels, while we expect the same label types to still apply to a paraphrase, the locations (start and end) are expected to shift. to address this issue, we introduce a new model for span-based discriminative alignment. given an input sentence s, a paraphrase f (s), and a span of tokens in s representing a label location, the alignment model finds a semantically equivalent span in f (s). we present the architectural details of this model, a dataset for span alignment, and corresponding results in §4. a second problem is that most paraphrase models offer no control over specific words or phrases figure 1 : framework for iterative paraphrastic augmentation illustrated on an actual system output. the original, manually annotated sentence contains a tag over the word ''corroborate''. in iteration 1, the sentence is paraphrased using a lexically constrained decoder with negative constraints on ''corroborate'' and all associated inflectional forms, guaranteeing that it will not appear in the paraphrase. next, a span alignment model is used to obtain a link between ''corroborate'' in the original sentence and ''confirm'' in the paraphrase. all inflectional forms of ''confirm'' are then unioned with the prior set of negative constraints and the process repeats for a predetermined number of iterations. to be included in or excluded from the final output. as text-based data augmentation typically aims to increase lexical diversity, it is useful to force span label text to be rewritten in the paraphrase as a synonymous or semantically similar phrase via lexically constrained decoding ( §3). in §5 we describe an augmentation framework that utilizes lexically constrained paraphrasing and alignment together, iteratively, to expand datasets for span labeling problems. an illustrative diagram is given in figure 1 . finally, we demonstrate the application of this framework to framenet in §6, resulting in a new dataset with 495,300 unique (frame, trigger) pairs in diverse sentential contexts. the intrinsic quality of the dataset is evaluated manually and its utility on external tasks is demonstrated with positive results on the task of frame id. ## lexically constrained paraphrasing sentential paraphrasing is a sequence generation problem where the goal is to find an output sequence conveying similar semantics to the input sequence while also ensuring that the two sequences are lexically or syntactically distinct. recent prior work has approached this problem with sequence-to-sequence neural networks @xcite , where an encoder embeds the input sequence into a fixed-dimensional space and a decoder produces a sequence auto-regressively. often, the decoder uses beam search to explore the output space more efficiently. lexically constrained decoding allows one to dynamically include or exclude token sequences from the output via user-supplied positive or negative constraints. when combined with paraphrasing, it can boost external nlp task performance via data augmentation @xcite . our work uses negative constraints, which exclude certain token sequences from the output by setting the likelihood of the last token in the negative constraint phrase to zero when all preceding tokens in the phrase have been generated @xcite . in our experiments, we follow the model architecture foot_2 described by @xcite with minor changes: 1) we use sentencepiece @xcite unigrams instead of tokenization, following @xcite ; 2) we do not use source factors, as sentencepiece unigrams are case-sensitive. these changes allow us to rewrite raw text without tokenization. the model is trained to convergence on a corpus @xcite with rich lexical and syntactic diversity, as measured by human judgment and parse-tree edit-distance, respectively. 4 alignment models ## iterative augmentation procedure our alignment model ( §4) is paired with a lexically constrained paraphrase model ( §3) to form an iterative procedure for augmenting data of the form: (sentence i , {(start i,1 , end i,1 , type i,1 ), ...}). the process consists of three steps: constraint expansion, paraphrasing, and alignment. in constraint expansion, we negatively constrain on a text span of interest, including its upper/lowercase counterparts and morphological variants using the pattern software package @xcite . by applying negative constraints, the paraphrase model is forced to generate a semantically equivalent sentence with a different surface form of the labeled text, creating a target for the alignment model. in the alignment stage, we score the original text span's representation together with each candidate span in the paraphrase and choose the one with the highest score under the model. using the newly obtained aligned phrase as input to constraint expansion, we repeat the process for a predetermined number of iterations. although we apply the iterative augmentation procedure to english language text, the method could be applied to other languages as long as a dataset exists on which to train a monolingual paraphrase model, which could then be used to generate data to be manually annotated for the span alignment training set. our paraphrase model is trained on data that ultimately needs backtranslation, which requires a set of aligned bilingual sentence pairs, though there are other types of paraphrase models that use only monolingual data @xcite . software such as pattern that allow the procedure to negatively constrain on morphological inflections of a given word would speed up the rate at which new lemmas are generated; however, even without such software, the paraphrase model would eventually discover inflections independently and negatively constrain on them. languages with richer morphological structure would benefit more from this kind of software as the paraphrase model might otherwise waste computational resources generating sentences with many inflections of the same word. ## experiments our approach lends itself to two applications: in §6.1 we are concerned with building a semantic resource from scratch, whereas in §6.2 we are concerned with expanding a pre-existing resource. we demonstrate the usefulness of our approach on downstream tasks in §6.3, where we apply our generated paraphrastic dataset to the task of frame id. following @xcite , we consider framenet as an illustrative resource motivating augmentation. in all experiments we treat each system output (paraphrase and alignment) as evoking the same frame as the original framenet input sentence. ## conclusion we introduced a novel approach for iterative construction of semantic resources via automatic paraphrasing. to demonstrate two possible uses of our framework, we simulated the rapid creation of a new semantic resource from a small seed corpus and generated a large-scale expansion of an existing resource. the latter experiment, run on framenet data, generated a lexically diverse dataset with 495,300 unique (frame, trigger) pairs in diverse sentential contexts, 50x the number of such pairs originally in framenet, which we release to the community alongside our 36,417-instance span alignment dataset.