AMR-DA : D ata Augmentation by A bstract M eaning R epresentation (2022)

Abstract: Abstract Meaning Representation (AMR) is a semantic representation for NLP/NLU. In this paper, we propose to use it for data augmentation in NLP. Our proposed data augmentation technique, called AMR-DA, converts a sample sentence to an AMR graph, modifies the graph according to various data augmentation policies, and then generates augmentations from graphs. Our method combines both sentence-level techniques like back translation and token-level techniques like EDA (Easy Data Augmentation). To evaluate the effectiveness of our method, we apply it to the tasks of semantic textual similarity (STS) and text classification. For STS, our experiments show that AMR-DA boosts the performance of the state-of-the-art models on several STS benchmarks. For text classification, AMR-DA outperforms EDA and AEDA and leads to more robust improvements.

## introduction data augmentation (da) techniques automatically generate additional data from existing data set for training machine learning models. they are widely used in computer vision (see, e.g. @xcite and can boost the performance of the trained models. in nlp, da methods can be roughly classified into token-level ones and sentence-level ones @xcite . token-level da methods generate new sample sentences from the original ones by changing some of their tokens (words). they include the method in @xcite that replaces some random tokens by their synonyms using a thesaurus, the now widely used easy data augmentation (eda) methods in @xcite that allow some random token insertion, deletion and swaps, and the more recent one in @xcite that performs token replacement using their embeddings. one advantage of these token-level da methods is that they are easy to implement. however, they can sometimes generate ill-formed or incoherent sentences as they do not take the sentence structures into account. in contrast, sentence-level methods generate new sample sentences by modifying the whole original sentences. they typically work by having an encoder that converts the input sentence to an intermediate representation and a decoder that generates new sentences from the intermediate representations. for example, in back translation @xcite , the intermediate representation is a sentence in another natural language. in generation methods @xcite , the intermediate representation is a hidden state. one advantage of sentence-level da methods is that they can preserve the semantics of the sentences. a major limitation of current sentence-level da methods is that there is not much variation in the generated sentences as the intermediate representations used are not easily controllable @xcite . for example, modifying the sentences in back translation requires knowledge of other languages, and minor changes of hidden states severely increase training difficulty. in this paper, we propose a new da method called amr-da that uses the abstract meaning representation @xcite as the intermediate language. amr is a well-known semantic meaning representation. it aims to remove syntactic idiosyncrasies and to represent the semantic structure of a sentence as a rooted, directed graph. it works well as an intermediate language for data augmentation as it allows us to combine the token-level and sentence-level methods in a single framework. like the sentence-level method, our method encodes the entire sentence as an amr graph. like the token-level methods, our method manipulates amr graphs at the node i' m a slightly skeptical because it' s so cheap . i' m a little skeptical because it' s so cheap in the hospital. i' m so skeptical because it' s a little cheap . i' m a little skeptical because it' s so cheap . rs rd ri sr no modification (token) level. thus our method can augment the original sample sentence in various ways without the need to retrain the decoder. this overcomes a key weakness of the current sentence-level methods. figure 1 shows an overview of our amr-da: amr parser first transduces the sentence into an amr graph, followed by an amr graph extender to diversify graphs with different augmentation strategies; finally, the amr generator synthesizes augmentations from amr graphs. to demonstrate the effectiveness of our method, we evaluated amr-da on two downstream tasks, semantic textual similarity (sts) and text classification tasks. experimental results show that our methods boosted unsupervised contrastive learning models to achieve new state-of-the-art results on several benchmarks in sts tasks and outperformed eda and aeda in text classification tasks. ## experiments we conduct experiments on two nlp tasks, semantic textual similarity tasks and text classification tasks, to evaluate our augmentation method. ## analysis effect of amr generators from the introduction in section 2.1, paraphrased sentences correspond to the identical amr graph. in other words, amr graph to sentences is a one-to-many relationship. since there is no uniform evaluation of amr generators, it is necessary to study the impact of amr generators on the performance of amr-da. we compared amr-ori with various generators based on bart base , t5 small and t5 base . ## related work our proposed data augmentation method is based on manipulating amr graphs. similar tree-edit techniques on syntax trees have been found to be useful in paraphrases generation @xcite . other applications of amr have also been based on graph manipulation. for example, @xcite used amr in summarization task by first parsing the source text to a set of graphs, transforming it to a summary graph, and then generating a summary using the summary graph. @xcite represented text and questions as amr graphs and reduced the machine comprehension problem to a graph containment problem. we have seen a growing body of work that makes use of amr in other applications such as dialogue modeling, information extraction and commonsense reasoning @xcite @xcite . based on the influence scope of augmentation, related data augmentation methods can be roughly classified into token-level and sentence-level methods @xcite . in token-level, synonyms replacement, random swap, random insertion, random deletion @xcite have been proven to improve the performance in classification tasks. in sts task, plenty of data augmentation techniques have been utilized such as shuffling, cutoff @xcite , synonyms replace @xcite , word repetition @xcite , etc. however, these methods all risk impairing structure information, resulting in incoherent augmentations. in contrast, sentence-level take the whole sentence into consideration. widely used back translation @xcite @xcite translates sentences into intermediate languages and then translates back. some studies attempt to incorporate syntactic information @xcite or latent variables @xcite to guide generators synthesize various augmentations. but these methods significantly increase the training difficulty. amr-da uses amr as an intermediate language, which can modify graphs as easily as in token-level methods, and synthesizes high-quality and diversified augmentations without grinding in training. ## conclusion and future work we propose a novel data augmentation method called amr-da. amr-da transduces sentences to amr graphs, applies multiple strategies to modify graphs, and then generates diversified augmentations. to the best of our knowledge, this paper is the first work that utilizes amr for data augmentation. amr-da overcomes the deficiency of previous sentence-level generation methods and diversifies augmentations without retraining decoders. our experiments show that amr-da boosts the performance of models to achieve state-of-the-art results in several sts benchmarks and outperforms eda and aeda in text classification tasks. in this paper, we mainly use amr-da to generate positive augmentations. further research could use amr-da to carefully construct adversarial samples for specific tasks and. downey. 2020. generative data augmentation for commonsense reasoning. in findings of the association for computational linguistics: emnlp 2020, pages 1008-1025, online. association for computational linguistics. xiang zhang, junbo zhao, and yann lecun. 2015. character-level convolutional networks for text classification. advances in neural information processing systems, 28:649-657. zixuan zhang, nikolaus parulian, heng ji, ahmed elsayed, skatje myers, and martha palmer. 2021. fine-grained information extraction from biomedical literature based on knowledge-enriched abstract meaning representation. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 6261-6270.