Is anisotropy really the cause of BERT embeddings not being semantic? (2022)

Abstract: In this paper we conduct a set of experiments aimed to improve our understanding of the lack of semantic isometry in BERT, i.e. the lack of correspondence between the embedding and meaning spaces of its contextualized word representations. Our empirical results show that, contrary to popular belief, the anisotropy is not the root cause of the poor performance of these contextual models’ embeddings in semantic tasks. What does affect both the anisotropy and semantic isometry is a set of known biases: frequency, subword, punctuation, and case. For each one of them, we measure its magnitude and the effect of its removal, showing that these biases contribute but do not completely explain the phenomenon of anisotropy and lack of semantic isometry of these contextual language models.

## introduction the introduction of the transformer architecture @xcite has had an enormous impact over natural language processing (nlp) in the last years. these models, trained for language modelling via self-supervision over huge text collections, have become the state-of-the-art in many nlp tasks by applying a fine-tuning process with a small supervised dataset. semantic textual similarity (sts) is one of these tasks, where training a linear layer on top of bert @xcite to get the similarity score of two concatenated sequences has emerged as a new standard in this field. however, this approach, called cross-encoder, has its own problems. it takes two sentences as input, so it has to rerun the whole model for each pair of sentences. this makes infeasible the application of this approach to most tasks. in order to solve this problem of efficiency, a lightweight approach called bi-encoder, consisting on obtaining the distance of two sentences within the embedding space, has emerged. in this case, each sentence is passed separately to the transformer, which obtains an embedding for each of them. a similarity metric is then computed between the two embeddings. unfortunately, this approach does not perform well with vanilla pre-trained transformers. @xcite showed that representing sentences by average contextual bert embeddings perform worse than averaging static glove embeddings in semantic tasks @xcite , despite their lack of contextuality. on the other hand, studying this mismatch between the contextual word embedding and meaning spaces, @xcite diagnosed high anisotropy in transformer language models, named as the representation degeneration problem. this means that embeddings do not follow a uniform distribution with respect to direction, i.e. the embeddings concentrate in an hypercone instead of occupying the whole space. they stated that this degeneration of the representation space could be closely related with the lack of semantic isometry to the point that some authors have tried to correct it by applying isotropy correction techniques @xcite . on the other hand, approaches based on contrastive learning have recently achieved remarkable improvements in this respect @xcite @xcite . contrastive learning methods pull together semantically similar sentence vectors and push apart semantically dissimilar ones, partially correcting the semantic isometry of embedding spaces. despite their good results, these techniques also have their own problems. the main difference between these methods is how they perform the selection of the positive (similar) and negative (dissimilar) sentence pairs they need for training. for example, @xcite used back translation from english to german to obtain augmented views of a sentence, and @xcite used near text spans in a document as positive samples. finally, @xcite used dropout as a data augmentation technique for the unsupervised model, and the nli dataset @xcite annotations for the supervised model. despite their originality, all of these approaches have different weaknesses, and they are not easily improved, as authors just find better ways of creating positive and negative pairs, without really building on previous work. consequently, improvements in semantic isometry correctness obtained by one method do not serve as a starting point for the next method, and no progress is made in solving the overall problem. contrastive bi-encoder models achieve unprecedented results in semantic tasks, but the root cause of the anisotropy and lack of semantic isometry observed in pre-trained transformer language models is not fully understood. we think that any finding in this area can be very relevant and open new research lines that can lead to future improvements in bi-encoder methods. therefore, our main aim in this research is the improvement of our understanding of the bert embedding space, and the relationship between semantics and isotropy through empirical results. after our experimentation, we conclude that there is not enough evidence to say that anisotropy is the root cause of the lack of semantic isometry of bert embeddings, while some biases seem to affect both isotropy and semantic isometry. we name bias to any information from a sentence that is encoded in the embedding space and that is not relevant to its meaning. this paper is structured as follows. in section 2 we review the related work. next, in section 3 we describe the experiments carried out and present their corresponding analysis of results. in section 4 we draw our conclusions, derived from our empirical results, and leave some ideas for future work. finally, we state the limitations of this work. ## related work the anisotropy found by @xcite when training a model for natural language generation tasks through likelihood maximization is produced by the combination of the zipfian nature of natural language and the log-likelihood loss function. this representation degradation makes the most frequent tokens to concentrate in a hypercone in the embedding representation space, having a more sparse space for infrequent tokens. contrastive learning methods seem to correct anisotropy to some extent as well, in addition to semantic isometry, which could make us think that the anisotropy was, in fact, part of the problem. however, @xcite realized through a series of experiments that there exist certain biases in the bert model, and that high anisotropy is not always equivalent to poor semantic isometry. although the insights by @xcite are certainly interesting, they seem to contradict to some degree previous works like @xcite @xcite . on the other hand, @xcite and @xcite had found that a big portion of the anisotropy of bert comes from outlier dimensions, related with positional information. @xcite showed that, in spite of bert embeddings having global anisotropy, each cluster in the embedding space is isotropic, and that this local isotropy could be enough for transformer models to achieve their full representation power. this hypothesis is supported by recent empirical results from @xcite . if the anisotropy comes from the existence of different clusters, and these clusters encode non-semantic information like token frequency, this can be matched with the biases described by @xcite and the representation degeneration by @xcite . as it can be observed, there is no consensus in the literature regarding the cause of the poor performance of transformer embeddings in semantic tasks, and there is also no consensus about the reasons for the anisotropy observed in these embedding spaces. this disagreement is amplified by the fact that there is not a standard method for evaluating anisotropy; for example, @xcite evaluates it at the word level, while @xcite consider the sentence level, by averaging the word embeddings. we therefore find that there is room for research and reflection on these aspects. ## experimentation and results we initially carried out an exploratory analysis to better understand the magnitude of the different biases studied. next, we compared several transformer-based language model configurations, particularly: different pooling strategies and models, and a bias removal technique; all in terms of isotropy and semantic isometry. a variety of studies like @xcite @xcite have shown that different transformer-based language models have similar behaviours regarding high isotropy and poor semantic isometry, even when they differ in number of parameters, architecture or learning objective. we considered to evaluate the following models that share the same bert architecture: bertbase-uncased, bert-base-cased, unsupervised-simcse-base, and supervised-simcse-base. the conclusions we extract for bert can be extended to other models, as a variety of studies like @xcite @xcite have shown that different transformer-based language models have similar behaviours regarding high isotropy and poor semantic isometry, even when they differ in number of parameters, architecture or learning objective. the cased and uncased models were included in order to study the case bias reported by @xcite . we also include both supervised and unsupervised variants of simcse @xcite because it is a contrastive learning bi-encoder model that achieves state-of-the-art results in semantic tasks and it is interesting to see how much does a successful model actually increase isotropy to understand to what extent is the anisotropy related with the poor semantic performance of the non fine-tuned bi-encoders. ## conclusions in this paper we have carried out a set of experiments intended to confirm and measure known biases, and to understand their impact over anisotropy and semantic isometry in fine-tuned and non finetuned bert models. in our results we have not found a clear correlation between isotropy and semantic isometry. in fact, models or pooling methods with a higher anisotropy are sometimes more semantic than others that are more isotropic. however, there exists a correlation between the biases and the semantic isometry. these biases are present in the embedding space, encoding information that is not semantic, like the frequency of a token or its case. this non semantic information distorts the embedding representation space, which leads to poor performance on semantic tasks. due to this distortion, biases naturally contribute to anisotropy, so there is, in fact, sometimes a certain correlation between isotropy and semantic isometry. but this correlation is spurious and comes from both the anisotropy and poor semantic isometry being a consequence of a high bias. we don't think that there is a causality relation between isotropy and semantic isometry. this means that isotropy correction methods will not achieve substantial improvements over their base models, which has been recently proven by @xcite . therefore, it could be said that assuming that the lack of isotropy of the embedding spaces is the cause of the lack of semantics is a post-hoc fallacy. methods that correct the embedding space to a certain degree, like the ones based on contrastive learning, also decrease anisotropy as side effect of removing biases, but we can't expect it to work the other way around, which is, to remove biases by increasing isotropy, as we could just be opening the general cone, while keeping the same islands with the same biases inside. even after manually removing the known biases (frequency, subword, case, puntuation marks), we are still far away from state-of-the-art unsupervised contrastive learning models. we think this indicates that there are a series of biases that we still don't understand. one promising line of work, taking into account the results from @xcite and @xcite , could be to try to find a positional bias, and a way to correct it.