Measuring Fine-Grained Semantic Equivalence with A bstract M eaning R epresentation (2023)

Abstract: Identifying semantically equivalent sentences is important for many NLP tasks. Current approaches to semantic equivalence take a loose, sentence-level approach to “equivalence,” despite evidence that fine-grained differences and implicit content have an effect on human understanding and system performance. In this work, we introduce a novel, more sensitive method of characterizing cross-lingual semantic equivalence that leverages Abstract Meaning Representation graph structures. We find that parsing sentences into AMRs and comparing the AMR graphs enables finer-grained equivalence measurement than comparing the sentences themselves. We demonstrate that when using gold or even automatically parsed AMR annotations, our solution is finer-grained than existing corpus filtering methods and more accurate at predicting strictly equivalent sentences than existing semantic similarity metrics.

## introduction translation between two languages is not always completely meaning-preserving, and information can be captured by one sentence which is not captured by the other. semantic divergence (or conversely, semantic equivalence) detection aims to pick out parallel texts which have less than equivalent meaning. though semantic divergence across sentences in parallel corpora has been wellstudied, current detection methods fail to capture the full scope of semantic divergence. state-of-theart semantic divergence systems rely on perceived sentence-level divergences, which do not entirely encapsulate all semantic divergences. for example, consider the parallel french and english sentences from the refresd dataset @xcite shown in figure 1 . the french sentence says "tous les autres édifices" (all other buildings) while the english specifies "all all other religious buildings are mosques or koranic schools founded after the abandonment of old ksar in 1957. tous les autres édifices sont des mosquées ou des écoles coraniques fondées à l'époque postérieure à l'abondance du vieux ksar en 1957. figure 1 : two parallel sentences from the refresd dataset marked as having no meaning divergence, for which the amrs diverge. fine-grained semantic equivalence detection is not widely studied-in spite of evidence that: (1) implicit information can be critical to the understanding of the sentence @xcite , (2) fine-grained divergences in parallel training data have a negative effect on neural machine translation system performance @xcite , and finally, that (3) fine-grained semantic equivalence detection holds promise for a number of applications. most notably, translation studies, semantic analyses, and language learning contexts could all benefit from the distinction between semantically equivalent sentence pairs and sentence pairs which have subtle or implicit differences @xcite . a fine-grained divergence detection system would enable the probing of machine translation models for semantic equivalence @xcite and could point to areas where the source language itself affects semantics in parallel sentences @xcite . other potential uses include: reducing the workload of human translators in post-editing of machine translation output by filtering out exactly semantically equivalent sentence pairs @xcite and crosslingual text reuse detection (plagiarism detection) @xcite . given the wide-ranging motivation for the development of a fine-grained equivalence detection system, coupled with the notable gap in research on this task, we argue that a finer-grained measure of semantic equivalence is needed: a way to detect strictly semantically equivalent sentence pairs. we leverage explicit semantic information in the form of abstract meaning representation (amr; @xcite to fill this gap. in this work, we demonstrate that parsing sentences into amr graphs and comparing those graphs enables a finergrained semantic comparison than simply comparing the sentences. we suspect that amr may be useful in this case because it makes explicit every concept and relationship between those concepts present in the sentence, taxonomically categorizing each concept's role and argument. with analysis of data in two language pairs (english-french and english-spanish), we demonstrate that sentence-level divergence annotations can be coarse-grained, neglecting slight differences in meaning ( §3). we find that comparing two amr graphs is an effective way to characterize meaning in order to uncover finer-grained divergences ( §4), and this can be achieved even with automatic amr parsers ( §5). finally, in §6 we evaluate our amr-based metric on a cross-linguistic semantic textual similarity dataset, and show that for detecting semantic equivalence, it is more precise than a popular existing model, multilingual bertscore @xcite . our primary contributions include: ## background on semantic divergence semantic divergences can arise when translating from one language to another. these divergences can arise due to different language structure, syntactic differences in the language, or translation choices @xcite . additional divergences can be introduced when automatically extracting and aligning parallel resources @xcite @xcite . to address these divergences, a number of systems have been developed to automatically identify divergences in parallel texts @xcite @xcite @xcite . the approach taken by @xcite to detecting sentence-level semantic divergences involves training multilingual bert @xcite to rank sentences diverging to various degrees. they introduced a novel dataset called rational english-french semantic divergences (refresd). refresd is a subset of the french-english wikimatrix @xcite with crowdsourced annotations classifying the sentences as having no meaning divergence, some meaning divergence, or being unrelated. recent work has investigated the differences in cross-lingual (english-spanish) amr pairs within the framework of translation divergences @xcite . specifically, this work developed an annotation schema to classify the types and causes of differences between cross-lingual amr pairs. we use this dataset to test the performance of our system on english-spanish gold amr pairs. (for english-french, we produce our own gold judgments of amr divergence to test our algorithm.) additional prior work has explored the role of structural divergences in cross-lingual amr parsing @xcite . the relationship between abstract meaning representation metrics and measures of semantic similarity has been explored in @xcite . recent work has also integrated sentence-level embeddings and comparison of amr graphs @xcite @xcite . ## amr for identification of semantic equivalence semantic representations are designed to capture and formalize the meaning of a sentence. in partic- he later scouted in europe for the montreal canadiens. (s / scout-02 :arg0 (h / he) :arg1 (c / continent :wiki "europe" :name "europe") :arg2 (c2 / canadiens :mod "montreal") :time (a / after)) il a plus tard été dépisteur du canadiens de montréal en europe. (he later scouted for the montreal canadiens in europe.) (d / dépister-02 :arg0 (i / il) :arg1 (c / continent :wiki "europe" :name "europe") :arg2 (c2 / canadiens :mod "montreal") :time (p / plus-tard)) figure 2 : a pair of sentences and their human annotated amrs, for which the sentences receive a "no meaning divergence" judgment in the refresd dataset, and are also equivalent per amr divergence. ular, the abstract meaning representation (amr) framework aims to formalize sentence meaning as a graph in a way that is conducive to broad-coverage manual annotation @xcite . these semantic graphs are rooted and labeled, such that each node of the graph corresponds to a semantic unit. amr does not capture nominal or verbal morphology or many function words, abstracting away from the syntactic features of the sentence. we leverage the semantic information captured by amr to recognize semantic equivalence or divergence across parallel sentences. figure 2 , for example, illustrates a strictly meaning-equivalent sentence pair along with the amrs. though the sentences differ with respect to syntax and lexicalization, the amr graphs are structurally isomorphic. if the amr structures were to differ, that would signal a difference in meaning. two particularly beneficial features of the amr framework are the rooted structure of each graph, which elucidates the semantic focus of the sentence, as well as the concrete set of specific non-core roles, which are useful in classifying the specific relation between concepts/semantic units in the sentence. for example, in figure 3 , the emphasis on the english sentence is on possession-your planet-but the emphasis on the spanish sentence is on place of origin, asking, which planet are you from? this difference in meaning is reflected in the diverging roots of the amrs. which is your planet? (p / planet :poss (y / you) :domain (a / amr-unknown)) ¿ de qué planeta eres ? (which planet are you from?) (s / ser-de-91 :arg1 (t / tú) :arg2 (p / planeta :domain (a / amr-desconocido))) figure 3 : two parallel sentences and amrs from the migueles-abraira et al. english-spanish amr dataset, which diverge in meaning. the spanish role labels are translated into english here for ease of comparison. finally, we identify the fact that non-core roles (such as :manner, :degree, and :time) are particularly helpful in identifying parallelism or lack of parallelism between the sentences. this is because amr abstracts away from the syntax (so that word order and part of speech choices do not affect equivalence), but instead explicitly codes relationships between concepts via semantic roles. furthermore, amrs use special frames for certain relations, such as have-rel-role-91 and include-91, which can be useful in enforcing parallelism when the meaning is the same but the specific token is not the same. for example, if the english and french both have a concession which the english marks via "although" and the french marks with "mais" (but), the amr special frame role will still preserve parallelism by indicating them both as a concession. ## examining and automatically detecting differences in gold amrs in this section, we evaluate the ability of amr to expose fine-grained differences in parallel sentences and how to automatically detect those differences. in order to do so, we produce and examine english-french amr pairs, which is the first annotated dataset of french amrs; we also examine a number of english-spanish amr pairs. this is a relatively small dataset (100 english-french items and 50 english-spanish items) because it serves as a manually annotated precursor to validate our hypothesis, ahead of our extensive automatically-produced amr experimentation ( §5) which uses 1033 items. ## strictness results using automatic english-french amr parses in §4, we confirmed our hypothesis by demonstrating that we are able to use gold (human annotated) amrs to capture a finer-grained level of semantic divergence, quantifiable via smatch. we extend this further by determining whether fine-grained semantic divergences can be detected well even when using noisy automatically parsed amrs. to do so, we compare the smatch scores of automatically parsed amr pairs with the human judgments output on the corresponding sentences by @xcite . to take the expensive human annotation piece out of the process, we show that automatic amr parses can be used instead of gold annotations by establishing a threshold, instead of via binary classification. therefore, we use the f1 score output by our pipeline algorithm as a continuous score and establish thresholds (described later in this section) to divide the data between divergent and equivalent. we automatically parse cross-lingual amrs for the entirety of the english-french parallel re-fresd dataset (1033 pairs). the refresd dataset is parsed using the mbart-st version of sgl, a state-of-the-art multilingual amr parser @xcite . the (monolingual) smatch score for the sgl parser, comparing our gold amrs with the automatically parsed amrs, is 0.41 for the 100 french sentences using smatch (0.43 using our pipeline algorithm) foot_3 and 0.52 for the 100 parallel english sentences using smatch. in doing error analysis, we find that the data points which are classified as having no meaning divergence but have extremely low f1 scores are largely suffering from parser error. we do find that there are pairs classified in refresd as having no meaning divergence at the sentence-level that do correctly receive low f1 scores. for example, the sentence pair in figure 4 , which has a refresd annotation of sentence-level equivalence and a gold amr-level annotation of divergence, was assigned an f1 score of 0.3469. despite smatch scores of 0.5 between the gold and automatic parses, both are usable for the task of detecting finer-grained semantic equivalence. to demonstrate the usefulness of our continuous metric of semantic divergence using automatically parsed amr pairs, we develop potential thresholds at which you could separate data as being equivalent vs. divergent. because our metric is more sensitive, a practitioner could choose their own threshold by determining appropriate precision (how semantically equivalent they wanted a subset of filtered data to be) and recall (how much data they are willing to filter out) needs. this tradeoff is depicted in figure 5 . for example, if all pairs are marked as equivalent, precision would be approximately 40% on the refresd dataset if considering solely the "no meaning divergence" pairs equivalent. comparing with model probabilities. though it is reasonable to assume that if the gold amr annotations provide a distinctly finer-grained measure of divergence than sentence-level divergence then this would also be the case when using automatically parsed amrs, we want to ensure the continued strictness of our methodology. to do this, we compare the values of our continuous metric and the probabilities produced by the @xcite system. because the probabilities produced by the system described in @xcite are always very close to 1 (equivalent) or very close to 0 (divergent) and there are far more divergent instances than equivalent instances, median and mode serve as a more effective form of comparison than mean between our f1 score and their probability score. above the 0.7 threshold, the median f1 for our system is 0.7869 and mode is 0.8; the median probability for the @xcite system is 0.9990 and the mode is 1.0. for the 0.6 threshold, our median is 0.6667 and our mode is 0.6667; their median is 0.9871 and mode is 1.0. above the 0.5 threshold, our median is 0.5814 and our mode is 0.5; their median is 0.8907 and mode is 1.0. because these numbers are lower for our system than their system, we confirm that our measure is a stricter measure of equivalence even when using the automatically parsed amrs. if the goal is to prioritize items for a human to look at on a fixed budget, the absolute scores may matter less than rankings, though the rankings additionally differ drastically. of the top 50 sentences ranked by amr divergence (which range in amr similarity score from 0.96 to 0.67), only 19 of the 50 appear in the 166 sentences scored 1.0 by the briakou and carpuat (2020) system. ## sentence similarity evaluation with automatically parsed english-spanish amrs as we have shown in previous sections, our amr-focused approach in general is stricter than sentence-based measures of equivalence, in particular corpus filtering methods. because our system is a stricter measure of semantic equivalence, it may be the case that our system can more precisely identify the most similar sentences than existing measures of sentence similarity. in this final results section, we look at the most semantically equivalent sentences in the dataset (as judged by our approach and as judged by multilingual bertscore (mbertscore; @xcite ) in comparison to their human judgments of equivalence. specifically, we aim to investigate: (1) whether the average human similarity score for the most similar n sentences is higher when ranked by our amrbased metric versus when ranked by mbertscore, and (2) whether human judgments of sentence similarity for the most similar sentences are more correlated with our amr-based metric than with mbertscore (an embedding-based automatic evaluation metric of semantic textual similarity). we compare our amr-based metric to mbertscore because it has been shown to work well in crosslingual settings when comparing system output to a reference @xcite . semantic textual similarity considers the question of semantic equivalence slightly differently because it rewards semantic overlap as opposed to equivalence. ## conclusion in this work, we have proposed a stricter measure of semantic divergence than existing systems which rely on perceived differences at the sentence level. we have effectively demonstrated that parsing sentences into abstract meaning representations and comparing those graphs facilitates a more detailed semantic comparison, when using either gold or automatically parsed amr pairs. we are excited by the numerous possible applications of this finer-grained measure of meaning (mentioned in §1), both from an engineering standpoint and the potential it has in translation and language-learning environments to highlight specific differences in language pairs.