Coordination Generation via Synchronized Text-Infilling (2022)

Abstract: Generating synthetic data for supervised learning from large-scale pre-trained language models has enhanced performances across several NLP tasks, especially in low-resource scenarios. In particular, many studies of data augmentation employ masked language models to replace words with other words in a sentence. However, most of them are evaluated on sentence classification tasks and cannot immediately be applied to tasks related to the sentence structure. In this paper, we propose a simple yet effective approach to generating sentences with a coordinate structure in which the boundaries of its conjuncts are explicitly specified. For a given span in a sentence, our method embeds a mask with a coordinating conjunction in two ways (”X and [mask]”, ”[mask] and X”) and forces masked language models to fill the two blanks with an identical text. To achieve this, we introduce decoding methods for BERT and T5 models with the constraint that predictions for different masks are synchronized. Furthermore, we develop a training framework that effectively selects synthetic examples for the supervised coordination disambiguation task. We demonstrate that our method produces promising coordination instances that provide gains for the task in low-resource settings.

## introduction pre-trained language models (lms) have brought a large shift from preparing a substantial number of training examples and learning task-specific representations to collecting tons of unlabeled text and learning task-agnostic representations. for instance, fine-tuning lms, such as bert @xcite and bart @xcite , has improved performances of many natural language processing (nlp) tasks, even when using a small amount of training data. this paradigm shift has led to many challenges to exploit knowledge of lms rather than manually annotated data, including zero-or few-shot learning @xcite @xcite and data augmentation @xcite @xcite . however, many of these studies focus on sentence classification tasks, such as question answering and sentiment analysis, and thus are not applicable to structured prediction tasks, such as syntactic parsing. although some studies propose data augmentation methods that manipulate the structure of a sentence @xcite , producing syntactic structures using recent pre-trained lms has not been explored. as an attempt to produce syntactic structures using lms, we explore approaches to generating coordinate structures, in which two or more elements, known as conjuncts, are linked by a coordinating conjunction (coordinator), and yielding the corresponding annotation that explicitly marks which elements in the sentence are coordinated. coordination frequently appears in natural language sentences and causes ambiguities, leading to errors in many nlp tasks @xcite @xcite . we expect that performances of supervised methods for coordination disambiguation (e.g., @xcite @xcite would be improved if we can generate good-quality coordinate structures with annotation. in this paper, we present a method that exploits large-scale pre-trained lms to inject a coordinate structure into a given sentence with the corresponding annotation of coordination boundaries. specifically, we use masked language models (mlms) to generate text that seems to be in parallel with a given reference span in a sentence. to enhance mlms to make use of the similarity and replaceability properties of conjuncts described by @xcite and @xcite , our method utilizes a pair of sentences, each of which embeds a mask with a conjunction differ-figure 1 : overview of our approach. an input sentence with a reference span ("retain its gains") is masked in two ways. then, an mlm fills the masks with an identical text ("rise further") according to the constraint that predictions for different masks are synchronized. ently, and forces the models to fill the masks with an identical text (figure 1 ). for bert and t5 @xcite as instantiations of mlms, we introduce decoding techniques that follow the constraint that predictions for different masks are made to be synchronized, which can be potentially applied to other generation tasks, such as paraphrasing and summarization. we also describe a training framework to effectively select synthetic examples for supervised models, where a classifier is employed to filter out erroneous samples as similarly used in self-training. experimental results of the coordination disambiguation task in low-resource settings show that synthetic coordinate structures with annotation provide gains and indicate that our method yields promising instances of coordination. the contributions of this work are summarized as follows: as synonym-based augmentation can be applied to only a small percentage of the vocabulary covered by a hand-crafted ontology, kobayashi (2018) instead exploited a recurrent neural network-based lm to produce replacement words that fit in the context. this lm-based method was later extended by @xcite . they instead used bert @xcite for replacing words, which is a transformer-based lm trained on an objective where the model has to predict a token for a masked token in a sentence. anaby-tavor et al. (2020) proposed a method to generate a sentence for a category by providing the class label to gpt-2 @xcite . @xcite further generalized data augmentation methods that exploit pretrained lms. our work is similar to these studies in that we employ a pre-trained lm to generate a new sentence by filling masks with words, but differs in that it does not require a human-annotated label for an input sentence. self-training. self-training is a semi-supervised approach to effectively utilize unlabeled data. selftraining employs a model (teacher) trained on a limited amount of labeled data to assign synthetic labels to unlabeled data, which are later used to train another model (student). self-training has been successfully applied to a wide variety of nlp tasks, such as word sense disambiguation @xcite , syntactic parsing @xcite , and machine translation @xcite . the training framework used in our method is inspired by self-training, but unlike general settings of self-training, it employs a trained model only for inspecting synthetic examples that are generated by another model. coordination disambiguation. the task of coordination disambiguation is to identify the conjuncts for a given coordinator. prior to the recent advancement of neural networks, many efforts have been made to explore a range of approaches to this task: rule-based methods @xcite , statistical methods @xcite , and machine learning-based methods using handcrafted rules @xcite . recently, ficler and goldberg (2016b) proposed a neural network-based method that considers the similarity and replaceability between conjuncts. as opposed to the pipeline approach of ficler and goldberg (2016b), @xcite developed an end-to-end approach using neural networks, which was further extended by @xcite . ## method this section describes our approach to coordination generation exploiting lms pre-trained with the objectives for text-infilling. figure 1 shows the overview of our approach. formally, for a given sentence s 1:@xmath0, . . . , w n , the goal of our method is to embed a coordinate structure with annotation indicating a set of conjunct spans {(i (1) , j (1) ), . . .} together with a coordinator w k . we currently restrict each generated coordination instance to have two conjuncts linked by the conjunctive coordinator "and". to this end, our method first selects a reference span (i, j) in s and generates text homologous to the span using an lm. the produced text is then inserted with a coordinating conjunction after the reference span, which results in a coordinate structure {(i, j), (i ′ , j ′ )} accompanied by the coordinator. we explain the details of the method in the following sections. ## training framework for coordination disambiguation a synthetic example generated by the method described in section 3 can be immediately seen as an annotated sentence, but the resulting sentence can be inconsistent with its coordination boundary. for example, we expect a sentential conjunct for the reference "he said mary had dinner with john.", but an mlm can produce "bill" rather than a sentence because both "john and bill" as objects in s ′(1) and "bill and mary" as subjects in s ′( 2 ) are algorithm 1 training with generation and filtering input: labeled sentences l; unlabeled sentences u ; predictor m ; generator g constants: batch size n batch ; number of generated examples in a batch k (≤ n batch ); maximum number of generation trials k max ; threshold for filtering δ output: trained model m 1: train m on l 2: repeat 3: b ← {} 4: k ← 0 5: for the purpose of improving supervised methods for the coordination disambiguation task, we instead propose a training framework inspired by self-training. the algorithm of our training method is described in algorithm 1. at the first step of the procedure, a supervised model m for the task is trained on a limited amount of labeled examples l. in the following steps, the model is further trained on labeled examples l and synthetic examples generated from a large number of unlabeled sentences u . concretely, in each training iteration, we pick one unlabeled sentence x that contains no coordination from u and feed it to an lm g (that encapsulates a span selection mechanism) to obtain a sentence x ′ that has a coordinate structure with the corresponding annotation y ′ using the method explained in section 3. after generation, we employ the model m to assign a score to the ex- ## experiments we evaluate our method using generated examples to train supervised models for the coordination disambiguation task. ## limitations the main weakness of the proposed approach is its dependence on a pre-trained large lm, which may not be available in an extremely low-resource setting in terms of the amount of unlabeled text or computational resources. furthermore, the effectiveness of our method has been examined for gen-erating a limited form of coordination in english (i.e., two conjuncts joined by "and"). thus, additional treatments for other forms of coordination in english or for coordination in other languages using multilingual lms would need to be developed and tested with more thorough qualitative analyses through human evaluation. ## conclusion we present a method that exploits pre-trained lms for generating coordination with boundary annotation. synchronized decoding by bert and t5 for two masked sentences with conjunctions leads to coordinate structures that are consistent with synthetic annotation. the experimental results for coordination disambiguation show that examples generated by our method provide gains to supervised methods. the extensive study also supports the effectiveness of the proposed training framework for removing erroneous examples. as future work, we intend to develop a fine-tuning method on lms specialized for coordination generation and apply synchronized decoding to other lms and to other generation tasks.