Whitening Not Recommended for Classification Tasks in LLM s (2024)

Abstract: Sentence embedding is a cornerstone in NLP. Whitening has been claimed to be an effective method to improve embeddings obtained from Large Language Models (LLMs) for sentence embedding. However, we find that the effectiveness of whitening is model-dependent and task-dependent. In particular, whitening degenerates embeddings for classification tasks. The conclusion is supported by extensive experiments. A by-product of our research is embedding evaluation platform for LLMs called SentEval+

## introduction sentence embedding plays a fundamental role in nlp @xcite . despite the widespread success of large language models (llms) in generative tasks, embeddings obtained from pre-trained models are not impressive @xcite . sometimes, they are not even competitive with traditional word2vec-based approaches on machine learning tasks such as classification and semantic text similarity (sts). consequently, there has been a flurry of research aimed at improving the quality of embeddings from pre-trained models @xcite @xcite . among this group of work, whitening has been shown to be an effective post-processing method for improving embeddings obtained from llms @xcite @xcite . we find that the efficacy of whitening is both model-dependent and task-dependent. although we reproduced the result that whitening does work for some models on sts tasks, it does not work for other models. more importantly, the effectiveness of the whitening operation is restricted to sts tasks. for classification tasks, whitening degrades embedding quality consistently and sometimes with a large margin. the result is 1 here is the link to the github for senteval+ supported consistently for all the evaluated models and all the datasets in senteval @xcite . to further consolidate the surprising results, we explored a variety of whitening operations, including principal component analysis (pca) @xcite , cholesky matrix decomposition @xcite , and zero-phase component analysis (zca) @xcite . although some variants of whitening induce different performances, the overall conclusion remains unchanged. a by-product of our research is an embedding evaluation platform for llms, which we call senteval + , to streamline the evaluation of embedding quality. llms are big and costly to run. senteval @xcite provides a platform for embedding evaluation on a variety of models, tasks, and datasets. it works well on smaller models such as bert. to facilitate the evaluation of llms on commodity machines, we provide the embeddings for all sentences in our evaluation datasets. there is not much detailed comparison of the performance of embeddings from openai, maybe partially due to the cost for api calls. we observe that embeddings from openai are on par with llama overall. another interesting observation is that llama and llama2 are very close in terms of embedding performance. our work is important for both practitioners and researchers in llms. for llm providers such as openai, various post-processing are commonly applied to the embeddings they serve. they may want to serve different types of embeddings for different tasks, with the understanding of our result. for researchers in the area, running on a variety of llms is prohibitive computationally. our senteval + makes experiments feasible on commodity machines. ## whitening transformations llm embeddings have the isotropy problem @xcite @xcite . whitening is a postprocessing technique that converts spatially correlated, anisotropic feature representations into uncorrelated, isotropic ones @xcite . for this purpose, whitening transforms the feature representations such that the mean is centred at the origin, covariances are eliminated, and the variance is normalized to an identity matrix. w in equation 1 varies as in equation 2. the most commonly used whitening operation is called pcawhitening, which is also the one used in the first a few papers on the performance gain of whitening on llms. since our initial result on pcawhitening shows the opposite for classification tasks, and @xcite reported different behaviour of zca-whitening, we exhaustively investigate all variations of whitening operations. in equation 2 , λ is the eigenvectors, and u is the eigenvalues of the covariance matrix, i.e., σ = u λu t . the matrix l corresponds to the cholesky decomposition of the inverse of σ, such that ll @xmath0. the matrices v and θ result from the eigen decomposition of the correlation matrix p , expressed as @xmath1, where v is the eigenvector matrix and θ contains the corresponding eigenvalues. ## experiments we experimented with 8 models on classification and sts tasks. the embeddings are extracted from the last layer of the bert and llama models, following the practice described in (reimers and gurevych, 2019). we also explored other pooling strategies and observed similar pattern. embeddings of sbert, angle, and simcse are generated using their provided frameworks. while angle and simcse typically use the cls pooling method to extract embeddings, which involves using the output of the 'cls' token from the model to represent the entire input sequence, simcse employs the mean pooling method instead. for all mentioned models, we used the original tokenizers. for generating chatgpt embeddings, we choose the recent text-small-3-embeddings. next, we employ the senteval setting to evaluate the embeddings. the classification setup involves using an mlp (multi-layer perceptron) classifier with no hidden layers, utilizing the rmsprop optimizer. we also experimented with other classifiers including logistic regression, svm, and random forests. although the accuracy of the classification varies, the overall conclusion remains the same. following the practice in senteval, we report accuracy instead of f1 because the datasets are balanced. ## conclusion we show that the performance of whitening is model-dependent and task-dependent. for classification tasks, we do not recommend to apply whitening. for sts tasks, the performance varies from model to model. we conjecture that it works only for llms before fine-tuning. also, the technical details of chatgpt remain to be a mystery. based on its reaction to the whitening operation, we can infer that it may be fine-tuned, probably using nli data. another contribution of our work is an embedding evaluation platform for llms.