Toward Interpretable Semantic Textual Similarity via Optimal Transport-based Contrastive Sentence Learning (2022)

Abstract: Recently, finetuning a pretrained language model to capture the similarity between sentence embeddings has shown the state-of-the-art performance on the semantic textual similarity (STS) task. However, the absence of an interpretation method for the sentence similarity makes it difficult to explain the model output. In this work, we explicitly describe the sentence distance as the weighted sum of contextualized token distances on the basis of a transportation problem, and then present the optimal transport-based distance measure, named RCMD; it identifies and leverages semantically-aligned token pairs. In the end, we propose CLRCMD, a contrastive learning framework that optimizes RCMD of sentence pairs, which enhances the quality of sentence similarity and their interpretation. Extensive experiments demonstrate that our learning framework outperforms other baselines on both STS and interpretable-STS benchmarks, indicating that it computes effective sentence similarity and also provides interpretation consistent with human judgement.

## introduction predicting the semantic similarity between two sentences has been extensively studied in the literature @xcite @xcite . several recent studies successfully utilized a pretrained language model such as bert @xcite by finetuning it to capture sentence similarity @xcite . to be specific, they define a similarity score between sentence embeddings, which are obtained by aggregating contextualized token embeddings (e.g., avg pooling) or using a special token (e.g., @xcite ), then optimize the score for natural language inference (nli) or semantic textual similarity (sts) tasks @xcite . along with the quality of sentence similarity, interpreting the predicted sentence similarity is also important for end-users to better understand the results @xcite @xcite . in general, finding out the cross-sentence alignment and the importance of each aligned part is useful for analyzing sentence similarity @xcite . for example, there were several attempts to use explicit features (e.g., tf-idf) for easily analyzing the interaction among the shared terms @xcite or to adopt sophisticated metrics (e.g., word mover's distance) for explicitly describing it by the importance and similarity of word pairs across two sentences @xcite . however, for recent approaches that leverage sentence embeddings from a pretrained model, it has not been studied how the cross-sentence interaction of each part contributes to the final sentence similarity. in this work, we propose an analytical method based on optimal transport to analyze existing ap-proaches that leverage a pretrained model. we consider a sentence similarity measure a solution to a transportation problem, which aims to transport a collection of contextualized tokens in a sentence to the ones in another sentence. as byproducts of the problem, we obtain a cost matrix and a transportation matrix, which encode the similarities of all token pairs across sentences and their contributions to the sentence similarity, respectively. using this analytical method, we point out that the existing approaches suffer from the rank-1 constraint in the transportation matrix; this eventually keeps the model from effectively capturing the similarities of semantically-aligned token pairs into sentence similarity. for example, considering transportation in a contextualized embedding space (figure 1 ), the distance between averaged token embeddings (orange arrows) cannot clearly represent the distance of semantically-aligned token pairs (blue arrows). to resolve the above limitation and enhance the interpretability of a model, we present a novel distance measure and a contrastive learning framework that optimizes the distance between sentences. first, we apply optimal transport in a contextualized embedding space and leverage the optimal solution for a relaxed transportation problem as our distance measure. this sentence distance is composed of the distances of semantically-aligned token pairs; this makes the result easily interpretable. furthermore, we present a contrastive learning framework that adopts the proposed distance to finetune the model with token-level supervision. it optimizes the model to learn the relevance of semantically-aligned token pairs from that of sentence pairs, which further enhances interpretability. we extensively evaluate our approach and validate the effectiveness of its sentence similarity and interpretation. the comparison on 7 sts benchmarks supports the superiority of sentence similarity predicted by the model trained by our framework. in particular, the evaluation on 2 interpretable-sts datasets demonstrates that the proposed distance measure finds out semantically relevant token pairs that are more consistent with human judgement compared to other baseline methods. our qualitative analysis shows that both the token alignment and their similarity scores from our model serve as useful resources for end-users to better understand the sentence similarity. ## method we first analyze the similarity measure used by existing models from the perspective of a transportation problem. considering the above analysis, we present a novel distance measure and a contrastive sentence learning framework to enhance the interpretability of a sentence similarity model. ## experiments to analyze our approach in various viewpoints, we design and conduct experiments that focus on the following three research questions: ## conclusion in this work, we present clrcmd, a learning framework for an interpretable sentence similarity model based on optimal transport. first, we view each sentence similarity measure as a transportation problem, pointing out the unexpressiveness of the existing pooling-based similarity. integrating the concept of optimal transport into a pretrained language model, clrcmd defines the distance measure by using the semantically-aligned token pairs between two sentences and furthermore, it finetunes a model with this distance based on contrastive learning for better interpretability. we empirically show that clrcmd accurately predicts sentence similarity while providing interpretable token pair contributions consistent with human judgements. with the belief that the ability to interpret model behavior is critical for future ai models, we focus on enhancing this virtue targeted on sts task throughout this research.