An Attribution Method for S iamese Encoders (2023)

Abstract: Despite the success of Siamese encoder models such as sentence transformers (ST), little is known about the aspects of inputs they pay attention to. A barrier is that their predictions cannot be attributed to individual features, as they compare two inputs rather than processing a single one. This paper derives a local attribution method for Siamese encoders by generalizing the principle of integrated gradients to models with multiple inputs. The output takes the form of feature-pair attributions and in case of STs it can be reduced to a token–token matrix. Our method involves the introduction of integrated Jacobians and inherits the advantageous formal properties of integrated gradients: it accounts for the model’s full computation graph and is guaranteed to converge to the actual prediction. A pilot study shows that in case of STs few token pairs can dominate predictions and that STs preferentially focus on nouns and verbs. For accurate predictions, however, they need to attend to the majority of tokens and parts of speech.

## introduction siamese encoder models (se) process two inputs concurrently and map them onto a single scalar output. one realization are sentence transformers (st), which learn to predict a similarity judgment between two texts. they have lead to remarkable improvements in many areas including sentence classification and semantic similarity @xcite , information retrieval (ir) @xcite and automated grading @xcite . however, little is known about aspects of inputs that these models base their decisions on, which limits our understanding of their capabilities and limits. @xcite analyze sts with sentences of pre-defined lexical and syntactic structure and use regression analysis to determine the relative importance of different text properties. @xcite analyze ir models with samples consisting of queries and contrastive documents that differ in certain aspects. @xcite train an st to explicitly encode amr-based properties in its sub-embeddings. more is known about the behavior of standard transformer models; see @xcite for an overview. hidden representations have been probed for syntactic and semantic information @xcite @xcite . attention weights have been analyzed with regard to linguistic patterns they capture @xcite and have been linked to individual predictions @xcite . however, attention weights alone cannot serve as explanations for predictions @xcite . to obtain local explanations for individual predictions @xcite , @xcite suggest the use of feature attribution methods @xcite . among them, integrated gradients are arguably the best choice due to their strong theoretic foundation @xcite (see appendix a). however, such methods are not directly applicable to siamese models, which compare two inputs instead of processing a single one. in this work, we derive attributions for an se's predictions to its inputs. the result takes the form of pair-wise attributions to features from the two inputs. for the case of sts it can be reduced to a token-token matrix (fig. 1 ). our method takes into account the model's full computational graph and only requires it to be differentiable. the combined prediction of all attributions is theoretically guaranteed to converge against the actual prediction. to the best of our knowledge, we propose the first method that can accurately attribute predictions of siamese models to input features. our code is publicly available. additionally, let r be reference inputs that always result in a score of zero for any other input c: f (r, c) = 0. we extend the principle that sundararajan et al. ( 2017 ) introduced for single-input models (appendix a) to the following ansatz for two-input models, and reformulate it as an integral: this ansatz is entirely general to any model with two inputs. in the last line, we then make explicit use of the siamese architecture to derive the final attributions (details in appendix b). indices i and j are for dimensions of the two inputs a and b, respectively. individual summands on the righthand-side can be expressed in an attribution matrix, which we will refer to as a ij . by construction, all terms involving a reference input on the left-hand-side vanish, and the sum over this attribution matrix is exactly equal to the model prediction: in the above result, we define the matrices j as: the expression inside the integral, ∂e k /∂x i , is the jacobian of the encoder, i.e. the matrix of partial derivatives of all embedding components k w.r.t. all input components i. we therefore, call j an integrated jacobian. the integral proceeds along positions α on an integration path formed by the linear interpolation between the reference r a and input a: x(α) = r a +α(x-r a ). intuitively, eq. 4 embeds all inputs between r a and a along the path x(α) and computes their sensitivities w.r.t. input dimensions @xcite . it then collects all results on the path and combines them into the matrix j a ; analogously for j b . eq. 2 combines the sensitivities of both inputs and computes pairwise attributions between all feature combinations in a and b. in a transformer model, text representations are typically of shape s × d, where s is the sequence length and d is the embedding dimensionality. therefore, a quickly becomes intractably large. fortunately, the sum in eq. 2 allows us to combine individual attributions. summing over the embedding dimension d yields a matrix of shape s a ×s b , the lengths of the two input sequences. figure 1 shows an example. since eq. 3 is an equality, the attributions provided by a are provably correct and we can say that they faithfully explain which aspects of the inputs the model regards as important for a given prediction. for efficient numerical calculation, we approximate the integral by a sum of n steps corresponding to equally spaced points α n along the integration path (eq. 4). the resulting approximation error is guaranteed to converge to zero as the sum converges against the integral. it is further perfectly quantifiable by taking the difference between the left-and right-hand side in eq. 3 (cf. § 3.2). ## experiments and results in our experiments, we evaluate the predictive performance of different model configurations and then test their attribution accuracy. generally, the two are independent, so that a model with excellent attribution ability may not yield excellent predictions or vice versa. in the following, we analyze statistical characteristics of attributions. to demonstrate our method, we perform a pilot on which parts of speech (pos) models attend to. ## conclusion our method can provably and accurately attribute siamese model predictions to input and intermediate feature-pairs. while in sentence transformers output attributions are not very expressive and attributing to inputs can be computationally expensive, attributions to deeper intermediate representations are efficient to compute and provide rich insights. referring to the terminology introduced by @xcite our feature-pair attributions are single cognitive chunks that combine additively in the model prediction. importantly, they can explain which feature-pairs are relevant to individual predictions, but not why @xcite . improvements may be achieved by incorporating the discretization method of @xcite , and care must be applied regarding the possibility of adversarially misleading gradients @xcite . in the future, we believe our method can serve as a diagnostic tool to better analyze the predictions of siamese models.