PAM : Paraphrase AMR -Centric Evaluation Metric (2025)

Abstract: Paraphrasing is rooted in semantics, which makes evaluating paraphrase generation systems hard. Current paraphrase generators are typically evaluated using borrowed metrics from adjacent text-to-text tasks, like machine translation or text summarization. These metrics tend to have ties to the surface form of the reference text. This is not ideal for paraphrases as we typically want variation in the lexicon while persisting semantics. To address this problem, and inspired by learned similarity evaluation on plain text, we propose PAM, a Paraphrase AMR-Centric Evaluation Metric. This metric uses AMR graphs extracted from the input text, which consist of semantic structures agnostic to the text surface form, making the resulting evaluation metric more robust to variations in syntax or lexicon. Additionally, we evaluated PAM on different semantic textual similarity datasets and found that it improves the correlations with human semantic scores when compared to other AMR-based metrics.

## introduction paraphrase generation is a relevant task in natural language processing (nlp), which has been widely applied in versatile tasks, such as question answering @xcite @xcite , machine translation @xcite @xcite , and semantic parsing @xcite @xcite . recent years have witnessed rapid development in paraphrase generation algorithms @xcite @xcite . however, little progress has been made in the automatic evaluation of this task. it is even unclear which metric is more reliable among many widely used metrics. most evaluation metrics used in previous paraphrase generation research were not designed specifically for the task but adopted from other tasks, such as machine translation (mt) @xcite and summarization @xcite . however, the evaluation of paraphrases is inherently different from the evaluation of most other text-generation tasks because a good paraphrase typically obeys two criteria @xcite @xcite : semantic similarity and lexical diversity. semantic similarity means that the paraphrase maintains similar semantics to the input text, whereas lexical diversity requires that the paraphrase possesses lexical or syntactic differences from the input. in contrast, a task like machine translation does not have such lexical diversity requirements. it is, therefore, uncertain whether the metrics borrowed from other tasks are sensible in paraphrase evaluation. recent semantic similarity metrics leverage pretrained language models to compute the graded similarities between two texts @xcite . however, because of the way these models work, there will always be some attachment between the output score and the texts' specific ordering, structure, and wording. in hopes of reducing this coupling between semantics and syntax when learning a semantic similarity evaluation metric, we propose to explore abstract meaning representation (amr). given that amr is a formalism designed to capture the semantic content of a sentence, abstracting away from the surface form and focusing on the underlying meaning, we hypothesize that embedding representations of these structures instead of the traditional plain texts will help the trained models to better capture the semantics with less emphasis in the surface structure. we propose to learn text similarity through a similar pipeline used in plain-text automatic similarity assessment methods that encode sentences into latent semantic representations to measure the similarity of the two representations @xcite ; however, this time we encode latent semantic representations from amr graphs extracted from the two input texts. our proposed metric pam, paraphrase amr-centric evaluation metric, builds on top of sentence-bert @xcite , adding extra layers to process amr graphs and fuse these newly computed graph representations into the text-based backbone. inspired by other similarity scoring works @xcite , we utilize selfsupervised learning methods to overcome the high cost of collecting training data. we experiment with pam on two popular semantic textual similarity (sts) datasets and demonstrate that pam achieves considerable improvements in correlation with human annotations. in further analysis, pam retains the highest performance under various challenges compared to previous metrics. the code to reproduce the experiments is available at https://github.com/afonso-sousa/pam . ## related work neural networks have seen significant progress in paraphrase generation, leading to systems capable of producing more natural and human-like outputs @xcite @xcite @xcite . alongside these developments, extensive research has focused on automatically extracting paraphrases to build benchmarks, supporting further advancements in the field @xcite @xcite @xcite . despite these advancements, relatively little attention has been given to evaluating the quality of generated paraphrases. earlier work @xcite @xcite typically relied on metrics that measure surface-level overlap between generated outputs and reference paraphrases, such as bleu @xcite , rouge @xcite , and meteor @xcite . however, these metrics are limited in their ability to capture semantic equivalence and lexical diversity -two key aspects of paraphrasing. as a result, recent research has increasingly shifted toward compound metrics that explicitly balance semantic fidelity with lexical variation @xcite @xcite . a prominent example is ibleu @xcite , which combines bleu and self-bleu @xcite to quantify both meaning preservation and novelty. in parallel, modern embedding-based approaches like sbert @xcite leverage sentence embeddings to measure semantic similarity, have also been employed for evaluating paraphrases as they offer robustness to lexical variations in paraphrases. alignscore @xcite further advances the field with a fact-aware alignment model that evaluates factual consistency between the source and paraphrase, addressing critical challenges in meaning preservation. to our knowledge, @xcite are the only researchers to systematically assess the adequacy of existing paraphrase evaluation metrics. they identified key shortcomings and introduced parascore, a composition of metrics combining semantic similarity and lexical divergence. we focus on improving the semantic similarity component of paraphrase evaluation, which can function independently or within a framework like parascore. ## pam we introduce pam, an evaluation metric that computes graded similarity scores between sentences using amr graph representations. for each sentence, an amr graph is extracted, linearized, and fed into a siamese network, which outputs a similarity score as the cosine similarity between the graph embeddings. ## training data to train pam, we first use data parsed by @xcite , which entails one million sentences (randomly sampled from english wikipedia, used in simcse @xcite ) parsed to amr graphs using spring @xcite . this data is used in the self-supervised training, where we set the positive ratio to be 4/16, following @xcite , in a batch of 16, there are 4 positive graph pairs and 12 negative pairs. that is, in a batch we sample 4 graphs and create one positive pair and three negative pairs for each graph. we further train pam in a supervised learning fashion using 101 762 entries from the quora question pairs (qqp) dataset foot_0 . these entries were compiled by @xcite as anchorpositive-negative triplets foot_1 . ## semantic textual similarity results following @xcite , we select, as baselines, well-known metrics often used to evaluate paraphrase generation. we categorize evaluation metrics into three main types based on their focus: lexical overlap: bleu @xcite (ngram precision with brevity penalty), rouge @xcite (e.g., rouge-l, measuring recall of overlapping n-grams), and meteor @xcite (precision, recall, and synonym matching). token-level semantic similarity: bertscore @xcite (contextual token similarity using bert) and bartscore @xcite ) (text quality scoring with bart). ## reframing robustness analysis a basic assumption of pam is that it is concerned with the entire amr graph and is unaffected by focus change -or reframing -of the amr graph, that is, restructuring the graph by having as root node a different concept. for this, we create synthetic versions of the stsb and sick datasets, where, for each entry pair, we compute all possible rearrangements of the amr graphs by changing its focus @xcite to other concepts (see reframing examples in figure 3 ). we randomly sample these reframed versions of the original amr and pair them randomly, creating up to three new entry pairs per original pair. since pam can also be seen as an amr similarity metric, we assess its performance against various metrics often used to evaluate amr similarity: smatch (cai and knight, 2013): this metric evaluates the structural overlap between amr graphs by treating each graph as a conjunction of triples. smatch identifies a one-to-one variable matching that maximizes the number of exact triple matches using a greedy hill-climbing algorithm. sembleu @xcite : this metric extends bleu @xcite to amr graph comparison. sembleu linearizes amr graphs through breadth-first traversal and assesses text quality by comparing n-grams. wwlk @xcite : weisfeiler-leman amr similarity metric view amr graphs as highdimensional objects. the method iteratively propagates node embeddings with contextualization and employs the wasserstein weisfeiler-leman kernel (wwlk) to compute the minimum cost of transforming one graph into another. to understand the contribution of each component of the approach, we conduct an ablation study whose results are shown in ## score distribution we compare the score distributions of pam and sbert. as observed in the histogram in figure 4 , pam shows a higher dispersion of scores when compared with sbert. pam's greater dispersion provides finer granularity for distinguishing paraphrase quality, enabling it to more easily capture subtle differences between near-paraphrases, partial paraphrases, and unrelated pairs. in contrast, sbert's concentrated distribution risks overstating similarity for non-paraphrastic pairs and may fail to penalize meaningful deviations. ## conclusion we have proposed a learning-based semantic similarity metric for paraphrase evaluation called pam. it leverages amr graphs that are extracted from input texts by a pretrained amr parser and processed by the model's underlying siamese network structure that incorporates gnns for explicit graph connectivity representation. pam is trained under self-supervised and supervised learning. our proposed similarity metric addresses the main problem with currently employed metrics for paraphrase generation evaluation: the assumption of a high coupling between semantic meaning and surface form. pam effectively encodes the meaning of the texts by leveraging amr graphs, attaining the highest correlations with human evaluations compared to other test metrics, and maintaining robustness against reframed amr variants. it also has a more discriminative distribution than sbert.