Avoiding Overlap in Data Augmentation for AMR -to-Text Generation (2021)

Abstract: Leveraging additional unlabeled data to boost model performance is common practice in machine learning and natural language processing. For generation tasks, if there is overlap between the additional data and the target text evaluation data, then training on the additional data is training on answers of the test set. This leads to overly-inflated scores with the additional data compared to real-world testing scenarios and problems when comparing models. We study the AMR dataset and Gigaword, which is popularly used for improving AMR-to-text generators, and find significant overlap between Gigaword and a subset of the AMR dataset. We propose methods for excluding parts of Gigaword to remove this overlap, and show that our approach leads to a more realistic evaluation of the task of AMR-to-text generation. Going forward, we give simple best-practice recommendations for leveraging additional data in AMR-to-text generation.

## introduction deep learning has made remarkable progress in many areas of natural language processing, including language generation @xcite and semantic parsing @xcite . nevertheless, neural models are usually data-hungry, and sophisticated use of data augmentation can often go a long way @xcite @xcite . one common method of data augmentation is to leverage large amounts of out-of-domain data for semi-supervised learning. however, without proper examination of the data being used, the external data may contain significant overlap with the test set, leading to unfair gains as a result. this issue is a unique problem for natural language generation (nlg) tasks with data augmentation, because training with data that overlaps with the test set is akin to training on the answers. in this work, we study the task of amrto-text generation and scrutinize the datasets used for training and evaluation. our contributions are two-fold: (1) we develop an examination procedure to confirm that there are serious overlaps between one of the amr datasets and gigaword @xcite , and conduct experiments showing that some of the performance gains are indeed "unfair"; (2) we propose several strategies to apply when collecting external data for training, and empirically show that these strategies can mitigate the aforementioned unfair gains. for best practice, we suggest future work on amr-to-text generation exclude gigaword articles that are written in the nearby months of those covering proxy to be on the safer side (strategy no-3months described in section 5). ## related work abstract meaning representation (amr) @xcite has gained growing interest as a semantic formalism. the first amr-to-text generator was developed using tree transducers @xcite . more recent work heavily adopted neural models, explored different architectures, and commonly employed gigaword data to boost results @xcite @xcite . the most common approach is to use jamr @xcite to bootstrap labels for the additional data and then add them to the training data. prior work on amr generation has used automatic metrics such as bleu @xcite and human evaluations @xcite . currently, there is increased research on evaluation metrics for nlg @xcite 2019; dataset # sentences domain bolt 133 web consensus 100 news dfa 229 web proxy 823 news xinhua 86 news ## origin of amr and gigaword overlap in this section, we describe the reason for the overlap between the amr dataset and gigaword. in standard ldc releases of amr, for example ldc2015e86 and ldc2017t10, the dev and test set consist of 5 datasets from different sources. information about these datasets are listed in ## measuring overlap we use the following procedure to quantitatively examine the overlap between proxy and gigaword dataset. for each proxy sentence in the validation and test split, we find the gigaword article whose id is associated with the proxy sentence id. then we tokenize and split the article into sentences. we measure the overlap between the proxy sentence and each of the gigaword sentences with 3 different metrics: (1) absolute count of common words, which is the number of distinct words that appear in both sentences, (2) bleu score, and (3) rouge-l score. ## exclusion strategies we propose and investigate 3 sampling strategies for constructing semi-supervised training datasets from gigaword, and these strategies differ by how to exclude certain gigaword articles: no-id excludes articles whose id appeared in the proxy dataset; no-month excludes articles that are written in the same month as those excluded by no-id ; no-3months excludes articles that are written in the same month or neighboring months from those excluded by no-id . we use reservoir sampling @xcite to sample sentences from gigaword. we first collect a set with 200k sentences without any exclusion as a baseline. we then filter out sentences that are from articles excluded by no-id , and sample same number of sentences as those being filtered from articles that are included by no-id . this yields a set of 200k sentences representing no-id . we collect the sample sets for no-month and no-3months based on the baseline set in a similar fashion. 2 we use the ggnn-dual-encoder model by @xcite as our model to study the effects of different exclusion strategies. for each exclusion strategy, we obtain 3 different samples using different random seeds and repeat experiments. we keep most of the hyperparameters from the original paper. we adjusted the learning rate schedule to accommodate larger sets of training sentence score ## conclusion and recommendation in this paper, we examined gigaword, the commonly used dataset for improving amr-to-text generation, and found sentences that almost duplicate the test set of proxy, one of the amr datasets. we developed a procedure that utilizes a word overlap measure to find overlapping sentences, and found several metrics that may be good at finding duplicating sentences. we proposed 3 different strategies for excluding overlapping data from gigaword, and validated the idea that without filtering certain articles, the evaluation results may be unfair. for best practice, we suggest future work on amrto-text generation exclude gigaword articles that are written in the nearby months of those covering proxy to be on the safer side (no-3months). additionally, we suggest future work report results on each amr dataset separately so that techniques favoring one dataset can be detected.