GPT 4 AMR : Does LLM -based Paraphrasing Improve AMR -to-text Generation Fluency? (2025)

Abstract: Abstract Meaning Representation (AMR) is a graph-based semantic representation that has been incorporated into numerous downstream tasks, in particular due to substantial efforts developing text-to-AMR parsing and AMR-to-text generation models. However, there still exists a large gap between fluent, natural sentences and texts generated from AMR-to-text generation models. Prompt-based Large Language Models (LLMs), on the other hand, have demonstrated an outstanding ability to produce fluent text in a variety of languages and domains. In this paper, we investigate the extent to which LLMs can improve the AMR-to-text generated output fluency post-hoc via prompt engineering. We conduct automatic and human evaluations of the results, and ultimately have mixed findings: LLM-generated paraphrases generally do not exhibit improvement in automatic evaluation, but outperform baseline texts according to our human evaluation. Thus, we provide a detailed error analysis of our results to investigate the complex nature of generating highly fluent text from semantic representations.

## introduction abstract meaning representation (amr; @xcite is a graph-based semantic representation which captures the meaning of a phrase or sentence, with particular emphasis on semantic roles such as "who does what to whom." the substantial efforts towards amr-to-text generation (producing text from an amr graph, see an example amr graph and generated sentence in figure 2 ) and text-to-amr parsing (producing the graphs from the text) have enabled the amr schema to be incorporated into a range of downstream tasks @xcite . figure 2 : example text generated by amrbart from an amr graph in amr2.0 dataset. the reference text's amr graph is in 'penman' notation @xcite . currently, amr-to-text generation models can produce fairly fluent and adequate sentences that reflect the meaning of the graph. still, the quality of the generated text from amr-to-text generation models can be improved, both according to automatic metrics and human evaluation: state-of-theart amr-to-text generation models achieve approximately 50 bleu points @xcite out of 100, and @xcite find that amr-to-text generated output occasionally suffers from repetition of words or anonymization of low-frequency tokens. in recent years, large language models (llms) show the incredible ability to generate highly fluent text for a range of natural language processing tasks, such as machine translation and summarization. therefore, in this work, we examine the ability of several prominent llms, including a reasoning model, to improve the fluency of amr-totext generation output. specifically, we investigate whether passing the output of an amr-to-text generation model through a prompt-based llm tasked with paraphrasing the text output can enable heightened fluency (see figure 1 ). paraphrases generally refer to varied expressions that convey the same meaning @xcite . here, we aim to preserve semantic meaning while improving fluency. we first generate texts from four state-of-the-art amr-to-text generation models to serve as baselines. then, we prompt the llms to output paraphrases for these texts through multiple prompting protocols. finally, we compare the baseline texts and the llm-generated paraphrases via four automatic metrics and a survey of human judgments. our contributions include: ## approach in our experiments, we first pass amr2.0 and amr3.0 data into amr-to-text generation models to generate baseline texts ( §2.1). then, we prompt llms ( §2.2) to produce more fluent paraphrases of these texts through several prompting protocols ( §2.3). finally, we compare the results via automatic metrics and human evaluation ( §2.4). ## related work the rise of llms and the subsequent development of prompt engineering @xcite have led to recent work prompting llms to generate text in a variety of domains, such as paraphrasing math problem to improve solve rates @xcite and to produce specific types of paraphrases following linguistic instructions @xcite . however, it has been noted that llms tend to provide overly complicated lexical expressions @xcite and struggle to understand sentence structure @xcite when paraphrasing, which presents a challenge for our approach. although in-context learning (icl) prompting is common, work integrating amr graphs has been sparse. one such study @xcite discovers that amr-augmented prompting may improve llms' performance in tasks involving long context, such as summarization, which suggests that amrs may help with certain text generation tasks. in regard to amr-to-text generation, the output is mostly evaluated with automatic metrics, such as bleu @xcite , that compare the generated text with the human-annotated reference. however, it is unclear whether these metrics are suitable for assessing paraphrases, as they punish results with less n-gram overlap despite successful semantic preservation @xcite . bertscore @xcite , on the other hand, relies on comparing contextual embeddings to more accurately reflect semantic similarity. in addition to automatic metrics, using human eval-uation has been emphasized for a fuller analysis of amr-to-text output @xcite . ## conclusion & future work in this work, we explore the extent to which prompting llms to paraphrase can improve amr-totext generated output fluency, experimenting with variations of prompts such as constrained rewording, role-playing, and amr augmentation. our findings are mixed. through automatic evaluation, we find that none of the prompts lead to better llm-generated paraphrases compared to the baseline. specifically, we reveal llms' tendency to relate paraphrasing to synonym substitution, which may result in meaning drift. we discover llms' sensitivity to prompt wording, especially when given rewording constraints. few-shot and amr-augmented prompting improve llms' performance in most cases, but this may have arisen from llms extracting the surface form instead of truly utilizing the semantic content of the amr graphs. human evaluation, on the other hand, shows that the best zero-shot prompt leads to a statistically significant increase in fluency. the higher ratings may be due to the fact that the zero-shot prompting has not been exposed to the rigid amrgenerated outputs and still has sufficient freedom to use more natural phrases and grammar. additionally, applying role-play exhibits potential in aiding output fluency, given llms' massive training and thus the need to specify a trigger of specific domain knowledge. our study highlights the complex nature of generating fluent text from a semantic representation that abstracts away from the surface form, as we find that leveraging a wide range of llm prompts post-hoc to paraphrase the amr-totext generation system output generally does not improve performance.