On the Dimensionality of Sentence Embeddings (2023)

Abstract: Learning sentence embeddings is a fundamental problem in natural language processing. While existing research primarily focuses on enhancing the quality of sentence embeddings, the exploration of sentence embedding dimensions is limited. Here we present a comprehensive and empirical analysis of the dimensionality of sentence embeddings. First, we demonstrate that the optimal dimension of sentence embeddings is usually smaller than the default value. Subsequently, to compress the dimension of sentence embeddings with minimum performance degradation, we identify two components contributing to the overall performance loss: the encoder’s performance loss and the pooler’s performance loss. Therefore, we propose a two-step training method for sentence representation learning models, wherein the encoder and the pooler are optimized separately to mitigate the overall performance loss in low-dimension scenarios. Experimental results on seven STS tasks and seven sentence classification tasks demonstrate that our method significantly improves the performance of low-dimensional sentence embeddings.

## introduction learning sentence representation is a fundamental problem in natural language processing. sentence embeddings represent sentences as fixed-length vectors, which can be used in various downstream tasks, such as semantic textual similarity (sts) @xcite @xcite , information retrieval @xcite @xcite , and sentiment analysis @xcite @xcite . existing work usually focuses on improving the quality of sentence embeddings by introducing novel model architectures or training strategies @xcite @xcite @xcite the old pooler pooler @xcite tok 1 tok 2 ... ## the impact of the dimension of sentence embeddings we conduct a study to examine the impact of the dimension of sentence embeddings on the performance of various downstream tasks. we select roberta base @xcite as the sentence representation learning model and made its output dimension configurable. we utilize the unsupervised simcse @xcite as the training method, which takes an input sentence and predicts itself in a contrastive objective with dropout used as noise. similar to simcse, we train the model on one million randomly sampled sentences from english wikipedia, then apply the model to the following downstream tasks: it can be concluded that the optimal 1 dimen1 although there is no strict definition for "optimal", it can 2048 1024 768 512 256 128 64 32 16 8 sion of sentence embeddings varies across different downstream tasks. specifically, the optimal dimensions for trec, sts-b, cr, and mrpc are 768, 256, 256, and 16, respectively. one possible explanation for this variation is that downstream tasks exhibit different levels of difficulty, requiring varying amounts of information to be stored in embeddings to achieve the best performance. this observation motivates further exploring the dimensionality of sentence embeddings, particularly to enhance model performance in low-dimension scenarios. ## conclusion this paper presents a comprehensive and empirical study on the dimensionality of sentence embeddings. first, we propose customizing the dimension of sentence embeddings by directly modifying the pooler's output dimension. subsequently, we demonstrate that the default dimension (768 or 1,024) of sentence embeddings commonly used in the literature are usually suboptimal. to enhance the performance of low-dimensional sentence embeddings, we decompose the performance loss into the encoder's loss and the pooler's loss. we then introduce a two-step training method that separately addresses the two parts of the performance loss. experimental results demonstrate that our proposed training method consistently enhances the performance of sentence embeddings with low dimensions across all tasks.