Investigating Paraphrase Generation as a Data Augmentation Strategy for Low-Resource AMR -to-Text Generation (2024)

Abstract: Abstract Meaning Representation (AMR) is a meaning representation (MR) designed to abstract away from syntax, allowing syntactically different sentences to share the same AMR graph. Unlike other MRs, existing AMR corpora typically link one AMR graph to a single reference. This paper investigates the value of paraphrase generation in low-resource AMR-to-Text generation by testing various paraphrase generation strategies and evaluating their impact. The findings show that paraphrase generation significantly outperforms the baseline and traditional data augmentation methods, even with fewer training instances. Human evaluations indicate that this strategy often produces syntactic-based paraphrases and can exceed the performance of previous approaches. Additionally, the paper releases a paraphrase-extended version of the AMR corpus.

## introduction abstract meaning representation (amr) is a widely popular semantic representation. it encodes the whole meaning of a sentence into a labelled directed and rooted graph, including information such as semantic roles, named entities, and coreferences, among others @xcite . moreover, it has been successfully used in diverse applications/tasks such as automatic summarization @xcite , and paraphrase detection @xcite . its popularity is partly attributed to its extensive use of mature linguistic resources, like propbank @xcite , and its effort to abstract from syntax. figure 1 and bateman, 1991) (sub-figure b) for the sentence "the boy must go" along with other alternative surface forms that, while syntactically and lexically different, convey the same meaning. interestingly, amr corpora, as far as we know, include only one reference per amr graph, not leveraging their syntax-independent nature. in contrast, other semantic representations, such as those in the webnlg challenge @xcite or the e2e dataset @xcite , typically provide multiple references for each representation. having multiple references is advantageous for developing natural language generation systems, as it helps them handle potential noise by increasing data diversity @xcite . on the other hand, manually creating additional references can be costly. specifically, the words used in surface forms are tightly connected to the concepts in an amr graph @xcite . thus, references generated for an amr graph should ideally include only its concepts in their canonical form or possible derivatives as much as possible. for instance, the concept "boy" in figure1 should not be replaced with "guy" in a surface form, even if both terms are interchangeable. an alternative to manual annotation is the automatic generation of new references using paraphrase generation models. however, we must still adhere to the aforementioned guideline. paraphrase generation has been valuable for data augmentation in various tasks such as natural language understanding @xcite , and taskoriented dialogue systems @xcite . however, to our knowledge, this technique has not yet been explored to enhance amr-to-text generation performance or to develop a more robust amr corpus (apart from the work of @xcite ). moreover, other methods in the literature that utilize amr parsers to generate new instances @xcite @xcite might outperform paraphrase generation. nevertheless, we focus on low-resource scenarios where amr parsing could negatively impact the amr-to-text generation task. this work seeks to assess the helpfulness of paraphrases in the context of low-resource amrto-text generation for brazilian portuguese (bp). more, specifically, we try to answer the question to what extent can paraphrase generation contribute to improvement of the amr-to-text generation in a low-resource scenario? to answer this question, we investigate two approaches for generating paraphrases. the first approach employs a portuguese paraphrasing model @xcite . the second approach uses english as pivot language and is divided into two sub-approaches: one relies solely on machine translation models, while the other also includes an english paraphrase generation model. in addition, we compare this strategy with other well-known data augmentation strategy based on automatic parsing. due to the possibility of adding unrelated paraphrases introducing noise into the models, we explore using three selection criteria. these criteria help select a specific number of high-quality paraphrases. finally, we examine if added paraphrases can benefit when included in the development set in a multi-reference training. in general, our main contributions are: ## paraphrase generation for producing multiple references to evaluate the helpfulness of paraphrasing for the low-resource amr-to-text generation task, we explore generating paraphrases for each reference in the amr corpus. in particular, we explore two approaches for performing it. the first one assumes the existence of paraphraser models for the target language (in our case, portuguese). the second one is a cross-lingual approach that tackles the problem under the assumption that there is no paraphraser model for the target language; however, there is a bilingual corpus or a translation model between the target language and another richer-resource language (e.g., english) and, possibly, a paraphrasing model in the richer-resource. this way, we can use this language as a pivot. figure 2 shows an example of both approaches. the sub-figure a corresponds to the first approach, whereas the other two (b and c) correspond to the cross-lingual approach. in b, we only use machine translation models, whereas in c, we also use a paraphrasing model for the pivot language. ## manual revision to gain insights into some results, we conduct a manual revision. we select 112 instances from the development set to identify the primary mistakes and phenomena generated by the models. we define two categories in the evaluation: valid and invalid outputs. valid outputs are further divided into three sub-categories: "equivalent", where the system output and the reference are the "same" (with minor modifications such as the use of determiners); "semantic", where the system output is equivalent to the reference but uses different words or non-syntax paraphrases; and "syntactic", where the output is equivalent to the reference but exhibits some syntax differences (e.g., changing from active to passive voice). invalid outputs include 3 sub-categories: "missing", when the system output is similar to the reference, but omitted a few words; "partial hallucination", when the output contains part of the reference and part of extra information not related to the input/reference; and "total hallucination", when the output is totally unrelated to the reference. the analyzed approaches include the baseline, the data augmentation by parsing approach, the portuguese paraphrasing approach (under the setting t and b), and the two english-pivot sub-approaches under the setting t. more details about the selected models are described in a.3. on the other hand, english-pivot approaches are also promising to generate syntactic-based paraphrases; however, they are unsuitable for generating equivalent outputs, being overcome by the bosque-augmented approach almost twice (7.14%). in addition, we note that the overall percentage of valid outputs is lower than the obtained by the baseline and the bosque-augmented approach (19.64% and 18.76% vs 22.32% and 20.54%), showing that automatic metrics can hide some undesirable behaviour as english-pivot approaches gets better results in automatic evaluation. it could be explained by the fact that generating more diverse (and less related) paraphrases during training can add noise, thus being prone to generate more hallucinations. analyzing the invalid outputs, we see that paraphrase approaches tend to omit some words in the outputs, particularly portuguese ones. this way, some models generate "ele ficou só" ("he was alone.") instead of the reference "ele ficou literalmente só" ("he was literally alone."), omitting the word "literalmente" ("literally"). concerning the hallucinations, it is worth noting that all approaches produce a high number of hallucinations (47.32%-64.29%). this can be produced by the limited size of the original dataset and the high relation/node sparsity, however, more research should be done to confirm this hypothesis. about the approaches, paraphrase approaches are less prone to generate total hallucinations, being the best portuguese approach and the worst englishpivot approach that applies back-translation and paraphrase generation. we can see an example in figure 3 . as we can see in figure 3 , paraphrase approaches produce outputs more related to the reference, demonstrating the effectiveness of the approach. another interesting finding we found is that the major gain of this approach raises in the ability to produce the tokens included in the amr representation, i.e., paraphrase approach helps to better identifying concepts but not relations between them. we analyze this by using a sample that comprises only totally hallucinated outputs in the baseline model and verifying to what class (valid/invalid) they belong after applying the paraphrase approach. the results show that 13.23% of the outputs are fixed in the paraphrase approach, but 17.65% and 17.65% are classified as missing and partial hallucination classes, respectively. finally, we find the occurrence of partial hallucinations in the outputs produced by the paraphrase approach. even though models can be better than the baseline, they are more prone to generate additional expressions to the original one. for instance, the model generates "outro problema político tem um fundo político." ("another political problem has a political background.") when the reference no total , 20000 serão gastos em 2017 . ## related work paraphrase generation has been widely studied in natural language understanding tasks such as dialogue systems @xcite , intent classification @xcite and slot filling @xcite . for natural language generation (nlg), we have found that using multiple references leads to a more robust evaluation @xcite . besides, it has been successful in neural translation tasks @xcite . in the case of low-resource nlg, as far as we know, there are few works. @xcite proposes a paraphrase-augmented response generation framework that jointly trains paraphrasing and response generation models to improve dialogue generation. besides, the authors describe a strategy to generate paraphrase training sets. on the other hand, @xcite proposes a target-side paraphrase-based data augmentation method for low-resource language speech translation. ## conclusion and further work this study investigates the effectiveness of paraphrases for the amr-to-text generation task in brazilian portuguese. two paraphrase generation strategies were explored: one using a model trained on brazilian portuguese and the other using english as a pivot. the quality of generated paraphrases was evaluated using three automatic criteria, and the impact of the number of paraphrases on model performance was examined. experiments were conducted in two settings: adding paraphrases only to the training set and adding them to both the training and development sets. key findings include that paraphrase generation is a powerful data augmentation strategy, outperforming the baseline and traditional data augmentation in low-resource settings. however, not all metrics respond equally, and careful selection of paraphrases is crucial. the paraphrase-extended amr corpus showed slight improvement, with better performance seen when more paraphrases per instance were added. regarding human evaluation, portuguese-based models generated more valid outputs but also omitted words, while english-pivot models had lower performance and were more prone to hallucinations. as future work, we plan to curate the amr corpus with paraphrases and to explore new methods for generating syntax-focused paraphrases. this study acknowledges that its approach can only add a limited number of paraphrases and suggests combining it with classical data augmentation methods to expand the amr corpus. finally, the amr corpus for brazilian portuguese and the associated code will be made publicly available at https: //github.com/msobrevillac/amr-paragen .