Semantic Similarity as a Window into Vector- and Graph-Based Metrics (2022)

Abstract: In this work, we use sentence similarity as a lens through which to investigate the representation of meaning in graphs vs. vectors. On semantic textual similarity data, we examine how similarity metrics based on vectors alone (SENTENCE-BERT and BERTSCORE) fare compared to metrics based on AMR graphs (SMATCH and S2MATCH). Quantitative and qualitative analyses show that the AMR-based metrics can better capture meanings dependent on sentence structures, but can also be distracted by structural differences—whereas the BERT-based metrics represent finer-grained meanings of individual words, but often fail to capture the ordering effect of words within sentences and suffer from interpretability problems. These findings contribute to our understanding of each approach to semantic representation and motivate distinct use cases for graph and vector-based representations.

## introduction deriving sentence-level semantics is a nontrivial task and is fundamental to natural language understanding. word embeddings (vectors) and graphbased formalisms are two kinds of sentence meaning representations that are widely used in nlp and nlg. one way to evaluate semantic representations is to compare their judgments on semantic similarity, often using human judgments as a reference, and there are automatic sentence similarity metrics that have been developed which make use of vector and graph-based models. vector-based models, such as sentence-bert @xcite and bert-score @xcite , rely on bert embeddings. though they are robust and highly efficient, they often suffer from interpretability issues and do not meet all of the expectations of a distributional semantics model @xcite . on the other hand, semantic formalisms such as abstract meaning representation (amr; ba-what is the difference between a stock and a bond? what is the difference between a mode and a scale? figure 1 : a sentence pair in the sts @xcite dataset which receives a human judgment of 0 (no similarity), an s 2 match similarity score of 0.75, a sentence-bert similarity score of 0.10, and a bertscore score of 0.94. all three automatic metrics range from 0 to 1, where 1 indicates that the sentences are identical. @xcite are more explicit, and can be compared via graph-based similarity measures @xcite . while the explainability of these metrics is high, some studies have shown that they do not correlate with cross-lingual humanlevel judgments of similarity as well as embeddingbased metrics @xcite . amr-based metrics reflect the semantics of a sentence while abstracting away from syntactic features, while word-embedding based-metrics compare the tokens with contextualized embeddings. to this date, there is not a single approach that fully captures sentence meaning in a transparent fashion, so we need to investigate the strengths and weaknesses of different approaches in order to develop a better representation. given the importance of reflecting semantic similarity in formalisms of meaning, we hypothesize that semantic similarity is an informative way to investigate these representations. in this work, we investigate these semantic models through the lens of semantic textual similarity to investigate the differences between the various representations. these graph-based and the bert-based approaches to automatically assessing semantic similarity have different strengths, but no work to date has thoroughly compared these metrics as a way to better understand their applicability and utility. we investigate how these metrics compare to human judgments of similarity on a semantic similarity task. specifically, we compare and analyze the scores of two amr-based metrics-smatch and s 2 match-as well as two bert-based metrics-bertscore and sentence-bert-in relation to each other and in relation to human similarity judgments. for example, figure 1 features a sentence pair with vastly different similarity judgments via human annotation and three of our automatic metrics. our primary contributions include: wein and schneider (2022) introduce three methods for comparing cross-lingual pairs of amrs and evaluate the amr-based metrics against human judgments of sentence similarity and bertscore. cross-lingually, @xcite found that bertscore was more correlated with human judgments than the amr-based metrics. semantic textual similarity. semantic textual similarity (sts) is the task of judging the semantic equivalence of two sentences @xcite . in the most recent semeval semantic textual similarity for monolingual data in 2016 @xcite , the top performing team at that time incorporates wordnet information into word embeddings in their model @xcite . @xcite to compute sentence similarity. wordnet and framenet focus on lexical information and relations between words or frames, which is distinctively different from amr which represents lexical concepts and their relations within the same sentence. the state-of-the-art systems most correlated with human judgments consistently make use of transformers, such as smart-roberta large @xcite , which achieves state-of-the-art 92.9 and 92.5 on spearman's and pearson's correlations. @xcite introduced a similarity metric, semantically structured sentence-bert (s 3 bert), which combines the explainability of amr metrics with the high performance of bert-based approaches. for the sts task, s 3 bert obtains a correlation score of 83.7 on spearman's rank correlation coefficient. s 3 bert separates sentence-bert embeddings into partitions and trains the sub-embeddings on individual aspects of amr metrics. @xcite developed s 3 bert with the motivation of uncovering the semantic features that contribute to similarity ratings, and in doing so, develops hypotheses and conjectures about the reasons to combine these two methods based on their potential strengths and weaknesses. for example, amr metrics are said to be able to capture specific aspects related to semantic similarity, such as semantic roles, but are less correlated with human judgments; the bert-based metrics are more similar to human judgments but might lack sensitivity to word order. relatedly, @xcite combine semantic roles labels and dependency grammar on top of the bert transformer model @xcite with the aim of computing semantic textual similarity. in order to provide a more fine-grained evaluation of existing amr parsers, @xcite compare their parser with jamr @xcite and camr @xcite on various sub-tasks, such as named entity identification and semantic role labeling, and conclude that there is not a single parser that outperforms others in all sub-tasks. while prior research efforts have focused on either combining explicit information from graphbased resources with vectors, to the best of our knowledge, there has not been a direct, fine-grained comparison between these two formalisms. in our work, we perform a comparative analysis testing the hypotheses proposed in @xcite and analyzing the distinct strengths of graph versus vector-based representations on the task of semantic similarity. ## sentence similarity metrics in this work, we investigate the performance of and differences between four metrics of sentencelevel semantics: smatch @xcite , s 2 match @xcite , sentence-bert @xcite , and bertscore @xcite . we select these four metrics, two amr-based and two bert-based, because they are popularly used to compute sentence similarity @xcite , and are often used for downstream nlp tasks which depend on sentence semantics, such as paraphrase detection @xcite and coreference predictions @xcite . moreover, we wish to contrast the weaknesses and strengths of graph and bert based metrics. smatch: smatch is a widely used metric that measures whole sentence semantic structure similarity by computing the overlaps of structures between sentences that are represented in amr graphs @xcite . since amr abstracts away from syntax, syntactic paraphrases are expected to be represented with the same graph. a smatch score of 1 indicates semantic equivalence between two sentences, and a smatch score of 0 indicates that two sentences are completely dissimilar. figure 2 shows the amr graph of two semantically identical sentences. in order to compute sentence similarity, sentences are first parsed into amr graphs. smatch then aligns the graphs by finding the maximum number of triple matches (there are two types of triple matches: <var, instance, @xmath0, relationship, @xmath1, which is achieved by using a hill-climbing method with a smart initialization and 4 random starts to increase the probability in finding the highest number of matches @xcite . figure 2 : the amr graph for two syntactic paraphrases: he gives the school money, and he gives money to the school s 2 match: this metric is an extension of smatch which incorporates word-embeddings to match synonyms or near-synonyms @xcite . during graph alignment, if the cosine similarity between the word-embeddings of two nodes meets a threshold, these two nodes, even if they have a different surface form, are considered a match. as a result, the s 2 match score will go up according to the cosine similarity score. this addresses the disadvantage of smatch @xcite ) that graded meanings are not measured. for example, <var, instance, @xmath2, instance, @xmath3, but not in smatch since "skinny" and "thin" are two different words. following @xcite , we set the s 2 match alignment threshold value to 0.5. therefore, we only consider the similarity between nodes if their cosine similarity reaches 0.5 or higher. in order to use smatch and s 2 match to compare sentence similarity, we first use an automated amr parser @xcite to convert sentence pairs into amr graphs. the parser is a bartbased model @xcite ) that is trained on a self-supervised graph-to-graph method. the accuracy of the parser is 83.6% on the amr2.0 (ldc2017t10) dataset. sentence-bert: sentence-bert @xcite is a bert-based model that is pre-trained on the snli @xcite and the multi-genre nli @xcite datasets. it generates sentence embeddings using the siamese bert model architecture @xcite . to measure sentence similarity using sentence-bert, we pass sentence pairs into this model to obtain sentence embeddings, and compute their cosine similarity. bertscore: this metric was designed with the intention to evaluate text generation quality @xcite . to obtain bertscores between reference and candidate sentences, this metric first matches the tokens in the sentences using a greedy method: each token in a sentence is matched to the most similar token in the other sentence. therefore, tokens are potentially matched more than once. after, the normalized pairwise cosine similarity between their word embeddings are computed with an optional idf-importance weighting which can put more weight on more indicative words of sentences during computation. ## data & evaluation protocol we use the test data from the semeval-2016 semantic textual similarity english subtask @xcite to evaluate the metrics on their degree of alignment with human judgments. the data contains 1,189 pairs of english snippets from five sources: newswire headlines, short-answer plagiarism, machine translation post-editing, q&a forum answers, and q&a forum questions. all the pairs are labeled on an ordinal scale from 0 to 5, with 0 indicating the texts are completely dissimilar, and 5 indicating they are semantically equivalent. for example, the sentences the bird is bathing in the sink and birdie is washing itself in the water basin are labeled as 5, while john went horse riding at dawn with a whole group of friends and sunrise at dawn is a magnificent view to take in if you wake up early enough for it are labeled as 0. to measure the correlation between each metric with human judgments, we use both the spearman's and pearson's rank correlation statistics. sentence-bert @xcite also use the same task to evaluate their model and report 74% spearman's rank correlation. since the amr parser @xcite used in this experiment may output multiple alternative amr graphs due to linguistic ambiguity, we only select sentences that are only parsed into a single graph to avoid impact caused by ambiguity on the correlation results. as a result, a total of 1138 sentence pairs are selected in our test set. this pre-processing step is very likely the reason why there is a discrepancy between our reported score as shown in ## results and analyses in this section, we present qualitative and quantitative analyses of the performance of each of the four metrics: smatch, s 2 match, sentence-bert, although sentence-bert has the highest similarity with human judgments, it suffers from low interpretability. in particular, we find it hard to account for seemingly inconsistent predictions. for example, the sentence pair what is the best way to store fresh berries? vs. what is the best way to cite an anonymous writer? receives a similarity score of 0.06 from sentence-bert, but this pair what is the best way to treat a feline ringworm? vs. what is the best way to clean a grater? receives a similarity of 0.4 from the same model. intuitively, the differences in these two sentence pairs are very similar, but they have very different similarity scores. based on the correlations with human judgments and the mean scores of the metrics, we conclude that sentence-bert's scores are the most indicative of semantic similarity between sentences. ## conclusion we compared four graph-and vector-based semantic metrics via a semantic similarity task. in the task, we used human judgments as a reference and explored various scenarios to investigate the strengths and weaknesses of these metrics, both qualitatively with examples and quantitatively via correlation with human judgments. we found that graph-based metrics are highly accurate in capturing meaning variations driven by change in sentence structures, whereas vector-based metrics allow more fine-grained meanings of individual words due to contextual embeddings. as we used automatic parsers in our experiments, the results were certainly affected by some amount of parser error. in future work, it would be interesting to see how gold amr graphs perform in the same experiment. we also hope that our analyses can motivate more robust research on utilizing the strengths of both vector-and graph-based meaning representations, allowing more effective semantic representation at both the word and sentence levels.