Explaining Text Similarity in Transformer Models (2024)

Abstract: As Transformers have become state-of-the-art models for natural language processing (NLP) tasks, the need to understand and explain their predictions is increasingly apparent. Especially in unsupervised applications, such as information retrieval tasks, similarity models built on top of foundation model representations have been widely applied. However, their inner prediction mechanisms have mostly remained opaque. Recent advances in explainable AI have made it possible to mitigate these limitations by leveraging improved explanations for Transformers through layer-wise relevance propagation (LRP). Using BiLRP, an extension developed for computing second-order explanations in bilinear similarity models, we investigate which feature interactions drive similarity in NLP models. We validate the resulting explanations and demonstrate their utility in three corpus-level use cases, analyzing grammatical interactions, multilingual semantics, and biomedical text retrieval. Our findings contribute to a deeper understanding of different semantic similarity tasks and models, highlighting how novel explainable AI methods enable in-depth analyses and corpus-level insights.

## introduction modern foundation models provide flexible text representations that enable the detection of semantic structure in vast amounts of unlabeled data. while traditionally supervised settings have taken a predominant role for nlp research, many widely used unsupervised tasks have received growing attention. the similarity structure of text embeddings herein provides a central starting point for many tasks, including information retrieval, semantic search, ranking and knowledge extraction @xcite @xcite , clustering @xcite , and visualization (van der @xcite @xcite . in the context of language generation, the retrieval-augmented generation (rag) @xcite approach computes semantic closeness to index relevant text data, resulting in numerous new information retrieval systems @xcite . furthermore, as the number of models and tasks increases, the need for quantitative evaluation of embeddings has become apparent, leading to the introduction of embedding benchmarks @xcite . complementary to the evaluation of nominal performance, the field of explainable ai aims to provide insights about the inner model mechanisms by highlighting relevant features for a specific prediction @xcite @xcite . explanations play a crucial role in verifying that predictions are grounded in task-relevant features, fostering trust and verifiability into complex machine learning models, and enabling the discovery of data patterns and novel insights. in this context, the prediction of supervised models can often be explained using heatmaps over input features. beyond heatmaps, specific explanation approaches have been proposed in the context of unsupervised models @xcite ) and higher-order explanations @xcite @xcite . in this paper, we focus on transformer-based similarity models that motivate the use of second-order attributions to highlight feature interactions. our main contributions are as follows: similarity models for nlp similarity models are designed to capture the meaning and context of the input texts and provide a quantitative measure of their semantic similarity or relatedness. commonly used approaches include end-to-endtrained siamese networks @xcite @xcite and universal encoder models with a pooling layer to extract text summary embeddings @xcite . combined with transformers, these approaches have become powerful frameworks to build similarity models. sentence transformers, specifically sentence-bert (sbert), @xcite have emerged as a flexible and widely-used method to compute compact text representations. similarity models typically process pairs of inputs to compute a similarity score, often based on dot products such as cosine similarity, while extensions to dataset-level dot products have enhanced semantic sentence matching @xcite and guided document retrieval in rag systems @xcite . recent efforts in evaluating similarity models and text representations include benchmarks on information retrieval @xcite and comparing performance across diverse embedding tasks @xcite . explaining transformers the limitations of raw attention scores as explanations have emphasized the need for improved explanation methods for transformers @xcite . notably, aggregating relevant information across attention heads has proven to be a promising direction @xcite , with further empirical evidence supporting the benefits of gradient information @xcite @xcite . the naive computation of explanatory gradients in transformers could be further improved by considering the conservation of relevance during backpropagation, which results in a modified layer-wise relevance propagation (lrp) scheme for transformers @xcite . interpretable feature interaction several works have focused on determining the effect of joint features in supervised classification scenarios. to identify such pair-wise attributions, multivariate statistics @xcite , hessian-based approaches @xcite , as well as methods inspired by co-operative game theory @xcite @xcite , have been proposed. beyond classification, the interaction between features provides an appropriate level of complexity to assess why a pair of inputs produces a high or low similarity score. to compute joint feature relevance, hessian × product and bilrp have been proposed @xcite . these methods can be derived directly from a deep taylor decomposition @xcite of similarity models and can be seen as extensions of the widely used gradient × input and lrp explanation methods to bilinear models. ## explainable ai for similarity models in the following section, we briefly outline how the specific structure of deep similarity models motivates the consideration of second-order terms, as well as the need for tailored propagation rules to compute robust and accurate explanations for transformers. ## experiments we now introduce the semantic similarity datasets and models used in this paper before proceeding with the evaluation of second-order explanations. ## corpus-level use cases to go beyond the mere evaluation of nominal correlation, which is at risk of obfuscating undesired model strategies, i.e. clever-hans-type and shortcut learning @xcite , we hereby explore how bilrp explanations can be used to uncover general model strategies in three distinct use cases. ## discussion and conclusion our evaluation and use cases have provided a framework to analyze textual similarity models using explainable ai. in the following, we contextualize our findings and discuss further implications.