Representation biases in sentence transformers (2023)

Abstract: Variants of the BERT architecture specialised for producing full-sentence representations often achieve better performance on downstream tasks than sentence embeddings extracted from vanilla BERT. However, there is still little understanding of what properties of inputs determine the properties of such representations. In this study, we construct several sets of sentences with pre-defined lexical and syntactic structures and show that SOTA sentence transformers have a strong nominal-participant-set bias: cosine similarities between pairs of sentences are more strongly determined by the overlap in the set of their noun participants than by having the same predicates, lengthy nominal modifiers, or adjuncts. At the same time, the precise syntactic-thematic functions of the participants are largely irrelevant.

## introduction transformer-based encoder-only models derived from the bert architecture and pre-trained using similar objective and training regimens @xcite have become the standard tool for downstream tasks at the level of individual tokens and token sequences @xcite . whole-sentence representations can also be easily extracted from the outputs of these models by either using the embedding of the special @xcite token, in cases where the model was trained on the next-sentenceprediction task, or averaging or max-pooling the embeddings of all tokens produced by the model @xcite . while both approaches are widely used in practice, it has been argued that these representations are not well suited for sentence-level downstream tasks. several modifications to the architecture and training regime were proposed, which are known collectively as sentence transformers (sts; @xcite . sts have achieved state-of-the-art performance on downstream tasks such as semantic search and question answering @xcite . their analysis, however, has received considerably less attention than the analysis of the vanilla bert model and its variants @xcite . in fact, these models are often considered to be uninterpretable @xcite . a common feature of sts is that they are finetuned to produce similar vector-space representations for semantically similar sentences. this objective induces a complex loss landscape shaped by the available training data. the original sentence-bert model @xcite was trained on natural language inference data, and sentences were considered to be semantically similar if their nli label was that of entailment. sota models were trained on a much larger web-crawled corpus including more than 1 billion sentence pairs mined from sources such as reddit conversations, duplicate question pairs from wikianswers, etc. 1 the richness and variability of this dataset begs the question of what notion of semantic similarity is implicitly learned by the models trained on it. in this study, we begin addressing this question through analysis of natural-looking synthetic sentences with controlled syntactic and lexical content. we concentrate on three questions. first, we test if sts have part-of-speech biases. we show that, all other things being equal, information provided by nouns plays more important role than the information provided by verbs, both in simple sentences and in sentences with coordinated verbal phrases. second, we compare the relative importance of the overlap in the sets of participants in two sentences with that of how many participants have identical syntactic functions. we show that raw lexical overlap is relatively more important than having the same nouns in the same syntactic slots. third, we check how strongly sentence representations are affected by other sentential elements, such as adverbials and nominal modifiers of different types and lengths. we show that, unlike bert with token averaging, sts seem to largely disregard these components in favor of nominal participants. the paper is structured as follows: § 2 presents the methodology that we follow in our analyses and the models we employ; § 3 presents the case studies and their results; § 4 provides an overall discussion; § 5 surveys related work; § 6 concludes the paper. ## methods and experimental setup we experiment with representations produced by three models. two are sota sts: allmpnet-base-v2 (mpnet) is an instance of mpnet-base @xcite fine-tuned on the 1b sentence-pair corpus using the training architecture from @xcite ; all-distilroberta-v1 (distilroberta) is a distilled instance of roberta-base @xcite fine-tuned in the same way. the third model is the vanilla pre-trained bert-large-uncased (bert), as a point of comparison for the first two. all models were downloaded from huggingface. standard apis from the sentence transformers library foot_1 were used to compute embeddings using mpnet and distilroberta; for the vanilla bert model, we averaged the embeddings of all sentence tokens, including @xcite and @xcite . 3 we structure the presentation as a series of case studies. for each case study, we construct a set of sentences controlled for lexical content and syntactic structure. sentences are created in such a way as to be grammatically correct, look naturalistic, and as far as possible not bias the analysis. 4 they are arguably less complex and variable than examples sampled from real-word corpora; however, we believe that an analysis based on simple sentences is a reasonable first step towards a better understanding of model representations, as previous work has shown for sentiment analysis @xcite and syntactic analysis @xcite . for each case study, we compute embeddings for all sentences, together with cosine similarities between embeddings of sentence pairs. we analyze the similarities by means of regression modelling. more precisely, we regress cosine similarities, zscored to improve comparability between encoders, on the properties of sentence pairs, such as lexical overlap, presence of identical participants in identical syntactic positions, or pos tags of participants. we inspect the coefficients of the resulting regression fits to assess the relative importance of these properties. since (almost) all properties are coded as binary variables, their magnitudes are directly comparable in terms of importance. for terminological clarity, we will use the term models to refer to the regression models we use to analyse the impact of sentence properties on representational similarity. we call the transformers computing these embeddings encoders. where the features of sentence pairs can be straightforwardly related to simple properties of individual sentences (e.g., in case when we are testing if they have the same subject or direct object), we also project sentence embeddings on a 2-d surface using umap @xcite 5 and check if the spatial organisation of the points is in line with our observations. lexical choice a potential confound of our experimental setup is lexical choice, which is never completely neutral. for example, by taking a semantically close pair of verbs, we can considerably reduce the effect of predicate mismatch between two sentences. moreover, encoders can react idiosyncratically to particular words and word combinations. including all combinations of words and their positions in sentence pairs as predictor variables is not a solution, however, as it defeats the purpose of identifying structural patterns and, in the limit, amounts to replicating the encoders. we address this confound in three ways. first, we select nouns to be always at least as interchangeable as words of other parts of speech in terms of belonging to similar mid-to-high frequency bands and referring to conceptually simple, concrete objects. this follows from our working hypothesis that encoders give preferential treatment to nominal elements, whose (generally entity referring) semantics is arguably easier to capture than, for example, that of (generally event referring) verbs @xcite . second, we compare the analysis of the st encoders against the analysis of the vanilla bert encoder. as they are derived from averaging, vanilla bert embeddings treat all words equally, so if our sentences, e.g., undersell differences in adverbs because we chose two nearly synonymous ones, this should be visible in the small coefficient tracking the impact of adverbs in the regression model based on bert embeddings. as will be shown below, however, the hierarchy of coefficients for regression models of sts is very different from that for vanilla bert, which arguably indicates that the role of lexical effects is minor. third, we re-run all reported models on sentences of the same structure with different lexical content; see the appendix for details. we observe high stability of coefficients across replications, higher for sts than for vanilla bert. this further corroborates the validity of our generalisations. ## case studies this section presents a series of case studies testing the sensitivity of embeddings produced by sentence transformers and bert token averages to properties of input sentences. we start with analysing simple intransitive sentences ( § 3.1) and simple transitive sentences ( § 3.2). we then make specific aspects of the structure more complex, analysing the effect of lengthy nps ( § 3.3) and coordinated vps ( § 3.4). finally, we look more closely at the syntax-semantics interface by inverting the prototypical alignment of pos tags and syntactic functions (predicative nominals and gerund subjects, § 3.5) and by testing the degree to which encoders track particular syntactic functions of verb arguments ( § 3.6). ## discussion our analysis arguably goes some way towards explaining why sentence transformers beat vanilla bert-based models with token averaging on sentence-modelling tasks. token averaging makes it impossible to distinguish between semantically rich and impoverished sentence elements, nor between syntactically central vs. peripheral elements: punctuation signs and determiners contribute on the same level as the matrix-clause predicate and main participants, while lengthy modifiers, such as relative clauses, and multi-word copula forms dominate the representation. sentence transformers, on the other hand, learn to discount elements that only serve a grammatical function or present background information and focus instead on the semantic kernel of the sentence. the latter is in effect largely synonymous with the set of nominal elements in the main clause, first of all participants, but also predicative nominals. importantly, despite their evident syntactic-analytic capabilities (e.g., in our setting they can distinguish between participants of main and relatives clauses and between main and auxiliary verbs), sts seem to not pay much attention to the distinction between subjects and direct or indirect objects. instead they prioritise raw overlap in the set of nominal participants of the matrix clause. this can be seen, by slightly abusing terminology of theoretical linguistics, as a focus on the aboutness/topic of sentences, what things they describe, and not on their predication/comment, what they actually say about those things @xcite . we believe that this focus is not inherent to the architecture of sentence transformers but reflects the nature of the datasets used for fine-tuning sts. the size of these datasets makes it impossible to convincingly reason about their contents, but their genres (qa pairs, reddit threads, etc.) makes it plausible to expect a high degree of topic-based overlap: questions and conversations tend to revolve around entities (persons and things), with their actions and properties repeating less often. this naturally leads to a focus on nouns referring to prominent entities, which are known to appear preferentially as subjects or objects for reasons of coherence @xcite , arguably a good match to the patterns we observe. ## related work analysis of transformer-based models for sentencelevel tasks, such as nli, question answering, or text classification, has largely followed the same approaches as found in the general bertology @xcite : probing, analysis of the geometry of the embedding space, extraction of parts of input that are particularly important for model performance, and behavioural analysis. in this vein, @xcite and @xcite analyse the attention patterns powering the performance of transformer models on different types of sentence classification, and @xcite show that embeddings of sentences computed by bert-based models, including siamese-fine-tuned sentence transformers, are anisotropic and can be improved via normalisation. @xcite survey the existing methods for extracting rationales from input sentences in the context of text classification and propose an improved approach, while @xcite demonstrate that sentence embeddings derived by averaging bert token representations suffer from artefacts arising from positional embeddings. @xcite argue that averaging should be replaced with maxpooling. very similar to ours is the approach adopted by @xcite , who construct a series of probes to analyse the performance of several models on the task of information retrieval. while their methodology relies on high-level document statistics and wholistic document manipulation (word and sentence shuffling, token-frequency similarity between the document and the query, textual fluency, etc.), our study analyses the role of linguistically motivated structural factors and thus complements their findings. @xcite aim at directly decomposing the representations produced by sentence transformers into several parts capturing different properties of sentences reflected in amr annotations (presence of negation, concepts included in the sentence, etc.). while our study tries to ascertain what meaning components dominate the representations, opitz and frank assume that these components are known in advance and are equally important: sentence embeddings in their modified sbert model are split into 15 segments, each of which corresponds to one amr-based meaning component, plus a residual part to capture everything not covered by amr annotations. ## conclusion this paper aims at making a contribution towards a better understanding of sentence transformers, which are often seen as black boxes. we have demonstrated that we can make surprisingly precise inferences about sentence-pair similarities using simple linguistic features such as lexical overlap. the crucial difference between bag-of-words distributional models and current encoders is that sts have became quite adept at disregarding 'irrelevant' parts of the sentence and concentrating on its key elements. unlike vanilla bert sentence embeddings obtained by token averaging, sts yield more structured embeddings that focus on the matrix clause and are less tied to individual lexical items and strings of function words. this progress, however, comes with a particular type of bias: the structures that lead to high sentence similarity in sts, i.e. the overlap in nominal 'participant sets', seem to mirror the dominant type of paraphrases found in the data the sts were tuned on, and sts are not compelled to look at finer structures of input sentences. at least without further fine tuning, this would appear to make them unsuitable for downstream tasks that require knowledge about more fine-grained aspects of sentence structure, such as semantic roles @xcite , or extra-propositional aspects, such as monotonicity, negation, or modality @xcite . an interesting direction for future research would be to explore the ways of decomposing sentence representations into additive aspects such as participant structure, main predication, etc. the additional challenge here is that while theoretical semantics has a lot to say about aspects of sentence meaning @xcite , there remains a lack of analysis linking the notion of one-dimensional semantic similarity @xcite that underlies the optimisation of current sentence transformers with theoretically more substantial concepts.