Approximate Attributions for Off-the-Shelf S iamese Transformers (2024)

Abstract: Siamese encoders such as sentence transformers are among the least understood deep models.Established attribution methods cannot tackle this model class since it compares two inputs rather than processing a single one. To address this gap, we have recently proposed an attribution method specifically for Siamese encoders (Möller et al., 2023). However, it requires models to be adjusted and fine-tuned and therefore cannot be directly applied to off-the-shelf models. In this work, we reassess these restrictions and propose (i) a model with exact attribution ability that retains the original model’s predictive performance and (ii) a way to compute approximate attributions for off-the-shelf models.We extensively compare approximate and exact attributions and use them to analyze the models’ attendance to different linguistic aspects. We gain insights into which syntactic roles Siamese transformers attend to, confirm that they mostly ignore negation, explore how they judge semantically opposite adjectives, and find that they exhibit lexical bias.

## introduction siamese encoders (se) are a class of deep-learning architectures that are trained by comparing embeddings of two inputs produced by the same encoder. in nlp they are often realized in the form of sentence transformers or sts @xcite , which have been successfully applied to the prediction of semantic similarity @xcite , natural language inference @xcite , and in information retrieval @xcite . despite their wide use, our understanding of which aspects of inputs sts base their decisions on is still limited, partly due to the fact that established attribution methods like integrated gradients @xcite cannot be directly applied to ses as they compare two inputs rather than processing a single one. in a recent publication @xcite , we have derived an attribution method specifically targeted for ses by generalizing the concept of integrated gradients to models with two inputs and introduced integrated jacobians (ij). resulting attributions take the form of token-token matrices (cf. figure 1 ) and they inherit theoretical guarantees from integrated gradients. however, they require models to be adjusted in two ways: (1) embeddings need to be shifted by a reference input and (2) the usual cosine similarity is replaced by a dot product. this has a number of disadvantages: the (unnormalized) dot-product is not a sufficient similarity measure, the adjustments lead to a drop in predictive performance, and models need to be fine-tuned. in this work, we address these drawbacks. our main contributions are twofold: these updates to our original method close the performance gap between standard and interpretable sts. our additional evaluations provide important guidance for the use and the limitations of approximate attributions for off-the-shelf models. our code is available on github at https://github.com/lucasmllr/xsbert . ## related work model explainability. a large number of concepts and methods are associated with model explainability, and no unified definition exists @xcite . feature-attribution methods, showing which parts of an input the model consults for a given prediction, are a means of local explainability for individual predictions @xcite . they provide post-hoc explanations for models that are not inherently interpretable, because we cannot decompose their decision making process into intuitively understandable pieces at prediction time @xcite . the framework of integrated gradients (ig; @xcite provides a way to do this in a provably correct way and with measurable accuracy. in the terminology introduced by @xcite , such feature attributions are individual cognitive chunks that may be cumulated across input dimensions and add up to the total prediction. ## experiments 1: analysis of attributions our approximate attributions do not provide a theoretical guarantee to be correct. therefore, in this section, after evaluating predictive performances in different settings, we first quantify the influence of reference contributions to approximate attributions, and then evaluate how well exact and approximate attributions agree. we work with attributions to layer nine, because they are expressive, while still being accurate with reasonable computational cost @xcite . ## experiments 2: analysis of sentence transformers the attributions derived by our method let us directly analyze the decision making process inside sts for the first time. in this section, we extend the analysis to concrete levels of linguistic structures including syntactic functions, negation, adjectives, and general lexical effects. ## conclusion the updates to our original method proposed in this paper (i) result in a siamese transformer with exact attribution ability to retain the predictive per-2066 formance of the equivalent unmodified model, and (ii) enable a way to compute approximate attributions for siamese encoders which can be directly applied to off-the-shelf models without the need for fine-tuning. unlike their exact counterparts, these approximate attributions do not come with the theoretical guarantee to exactly reflect the model prediction. our evaluation, however, shows that for deep intermediate representations they are reliable to a certain extent and often agree with exact attributions. analyses carried out based on our attributions show that siamese transformers primarily match subjects, predicates and objects but also considering different syntactic relations. they mostly do not attend to negation and often assign small yet positive contributions to semantic opposites. on a lexical level, some words always obtain high attributions with small variance whenever they appear. on the methdological level, we suggest that due to the practicality of approximate attributions, they may be used to obtain a first round of insights into off-the-shelf models. whenever reliable attributions of predictions are required, however, an exact attribution model should be employed. therefore, an interesting future perspective will be to train large siamese models with exact attribution ability from scratch.