SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features (2022)

Abstract: Models based on large-pretrained language models, such as S(entence)BERT, provide effective and efficient sentence embeddings that show high correlation to human similarity ratings, but lack interpretability. On the other hand, graph metrics for graph-based meaning representations (e.g., Abstract Meaning Representation, AMR) can make explicit the semantic aspects in which two sentences are similar. However, such metrics tend to be slow, rely on parsers, and do not reach state-of-the-art performance when rating sentence similarity. In this work, we aim at the best of both worlds, by learning to induce Semantically Structured Sentence BERT embeddings (S ^3 BERT). Our S ^3 BERT embeddings are composed of explainable sub-embeddings that emphasize various sentence meaning features (e.g., semantic roles, negation, or quantification). We show how to i) learn a decomposition of the sentence embeddings into meaning features, through approximation of a suite of interpretable semantic AMR graph metrics, and how to ii) preserve the overall power of the neural embeddings by controlling the decomposition learning process with a second objective that enforces consistency with the similarity ratings of an SBERT teacher model. In our experimental studies, we show that our approach offers interpretability – while preserving the effectiveness and efficiency of the neural sentence embeddings.

## introduction abstract meaning representation (amr) represents the meaning of a sentence as a directed, rooted and acyclic graph @xcite . it shows events and entities referred to in a sentence, their semantic roles and key semantic relations such as cause, time, purpose, instrument, negation. the explicit representation of meaning in amr has motivated research into amr metrics that measure meaning similarity of the underlying sentences. e.g., amr metrics are used for semantics-focused nlg evaluation @xcite @xcite , a semantic search engine @xcite , comparison of cross-lingual amr @xcite , and argument similarity @xcite . moreover, fine-grained amr metrics can assess meaning similarity of semantic sub-aspects that amr explicitly captures, e.g., semantic roles or negation @xcite . however, when measuring similarity rating performance against human ratings in the typical zeroshot setting on tasks like sts @xcite or sick @xcite , the (untrained) amr metrics tend to lag behind large models such as sbert @xcite that computes sentence embeddings with a siamese bert transformer model @xcite . notably, sbert alleviates the need for end-toend similarity inference on each sentence pair. instead, it infers the embedding of each sentence individually, and calculates similarity with simple vector algebra, which greatly reduces clustering and search time. amr metrics, by contrast, tend to be slower, are often np-hard @xcite and rely on a parser. hence, we find complementarity in these two approaches of rating sentence similarity: amr metrics offer high explainability -but tend to be slow and need improvement to compete in benchmarking. by contrast, neural embeddings show strong empirical performance and efficiency -but lack explainability. aiming at the best of these worlds, we propose to leverage multi-aspect amr metrics as a means to teach a pre-trained sbert model on how to structure its sentence embedding space such that it explicitly captures specific abstract aspects of meaning similarity, in terms of semantic roles, negation, quantification, etc. this has to be undertaken with care, to prevent catastrophic forgetting @xcite , which could negatively impact sbert's empirical performance and the overall effectiveness of its embeddings. our contributions: amr metrics: the cost of interpretability amr graphs @xcite explicate aspects of meaning, such as entities, events, coreference, or negation. metrics defined over amrs therefore show specific aspects in which two sentences are similar or different, which makes them attractive for tasks going beyond parser evaluation, such as nlg evaluation @xcite , semantic search @xcite , explainable argument similarity rating @xcite , or investigation of cross-lingual divergences @xcite . while classical amr metrics assess semantic similarity structurally via binary matches of triples @xcite , recent metrics target larger contexts and graded similarity scoring @xcite , e.g., to match a subgraph cat :mod young against a node kitten. but this high degree of explainability comes at a price: amr metrics tend to be slow since they i) compute costly graph alignments @xcite and/or ii) require amr parsers @xcite that are typically slow due to auto-regressive inference of large lms @xcite . iii) they are untrained, and thus tend to lag behind sbert-based metrics in empirical settings @xcite . we aim to overcome these weaknesses by making sentence embeddings capable of expressing amr metrics while preserving the full power of neural sentence embeddings. sentence and argument similarity several works and resources aim to capture human sentence similarity ratings. e.g., sick @xcite rates semantic relatedness and sts @xcite ) semantic similarity, on 5-point likert scales. relatedness and similarity have been argued to be very similar notions, albeit not the exact same @xcite . 1 an emergent branch of sentence similarity is the similarity of natural language arguments @xcite @xcite , which finds broad application scenarios, e.g., in argument search engines @xcite @xcite @xcite . while much research has been devoted to improving the accuracy of similarity rating systems, little attention has been paid to uncovering the features that (in the eyes of a human) make two sentences similar or dissimilar @xcite . in our work, we propose a method that can potentially help uncover such features, while provably preserving strong rating accuracy. preliminary ii: amr and amr metrics an amr @xmath0. the amr graph makes key aspects of meaning explicit, e.g., semantic roles or negation. hence, given a pair of amr graphs ⟨a, a ′ ⟩ ∈ a × a, an amr metric can measure overall graph similarity, or similarity with respect to specific aspects. we denote such a metric as ## amr metrics and data construction in section 3, eq. 2, we formally described an amr metric. now we consider the concrete metric instances we will use for s 3 bert decomposition. we distinguish general metrics that assess global amr graph similarity, and aspectual metrics that aim at assessing amr similarity with respect to specific semantic categories, e.g., semantic roles. ## evaluation study our two objectives aim at creating s 3 bert embeddings by partitioning sbert's output space into features that capture different semantic amr aspects, while controlling the decomposition process such that we prevent any forgetting of knowledge and preserve the power of the neural embeddings. hence, two key questions need to be addressed: basic setup we use a standard sbert model foot_4 with 11 layers and allow tuning of the last two layers. the sentence embedding dimension is @xmath1, the sub-embedding dimension is set to @xmath2, which implies that the dimension of the residual is 384-(15×16) = 144. more details on the model architecture and the training hyper-parameters can be found in appendix a.1. in all result tables, † indicates statistically significant improvement over the runner-up (student t-test, @xmath3.05, five random runs) ## conclusion we propose a method for decomposing neural sentence embedding spaces into different sub-spaces, with the goal of obtaining sentence similarity ratings that are accurate, efficient and explainable. the sub-spaces express facets of meaning as captured by amr and amr metrics, such as negation or semantic roles. the decomposition objective partitions the semantic space via targeted synthesis of amr metrics. the effectiveness of neural sentence embeddings is preserved by a consistency objective that controls the decomposition process and routes global semantic information not expressed by amr into a residual embedding. the s 3 bert embeddings are more explainable and are on par, or even outperform, sbert's accuracy. our approach allows straightforward extension to customized metrics of meaning similarity.