Large Language Models Are Overparameterized Text Encoders (2025)

Abstract: Large language models (LLMs) demonstrate strong performance as text embedding models when finetuned with supervised contrastive training. However, their large size balloons inference time and memory requirements. In this paper, we show that by pruning the last % layers of an LLM before supervised training for only 1000 steps, we can achieve a proportional reduction in memory and inference time. We evaluate four different state-of-the-art LLMs on text embedding tasks and find that our method can prune up to 30% of layers with negligible impact on performance and up to 80% with only a modest drop. With only three lines of code, our method is easily implemented in any pipeline for transforming LLMs to text encoders. We also propose L3Prune, a novel layer-pruning strategy based on the model’s initial loss that provides two optimal pruning configurations: a large variant with negligible performance loss and a small variant for resource-constrained settings. On average, the large variant prunes 21% of the parameters with a performance drop, and the small variant only suffers from a decrease while pruning 74% of the model. We consider these results strong evidence that LLMs are overparameterized for text embedding tasks, and can be easily pruned.

## introduction in the past few years, the field of natural language processing (nlp) has seen a significant shift towards large-scale language models (llms). these models, due to a combination of their large size, extensive pre-training, and instruction-following ability, have achieved state-of-the-art performance on a wide range of nlp tasks, such as language modeling, text generation, and text understanding @xcite @xcite . despite their strong generative capabilities, decoder-only llms have seen comparatively little adoption for text embedding tasks until recently @xcite . text embedding, which involves mapping a text sequence of varying length to a fixed-dimensional vector representation, is a fundamental task in nlp and is used as a building block for a wide range of downstream tasks, such as semantic textual similarity, information retrieval, text classification, and retrieval-augmented generation @xcite . traditionally, text embedding models have been based on masked language models (mlms) and bidirectional encoders, such as bert @xcite and t5 @xcite , typically adapted for text embedding tasks by following a multi-step training pipeline consisting of weakly-and fullysupervised contrastive training @xcite @xcite . decoder-only llms, however, offer several advantages over their encoder-only counterparts. they are more sample-efficient during pre-training, leverage instruction-following capabilities for task generalization, and benefit from a rich and evolving research ecosystem @xcite @xcite . further, the availability of high-performing public pre-trained llms and their continual development make it appealing to explore their use for text embedding tasks. to this end, several studies have experimented with various pipelines, training methods, and architectural modifications, effectively converting llms into state-of-the-art text embedding models with small amounts of supervised contrastive training @xcite @xcite @xcite . on the other hand, the increasingly large size of llms, with parameters ranging up to 540b @xcite @xcite , stands in stark contrast to traditional small bidirectional encoders of sizes almost universally less than 1b parameters @xcite . even the smallest llms in use typically have 3-8b parameters @xcite . consequently, inference with llm-based text encoders is far more demanding in terms of computing and memory requirements. therefore, there are a variety of post-training techniques for reducing the cost of llms, such as pruning, quantization, and distillation @xcite . in particular, the recent work of @xcite showed that llms can be pruned to up to half their size with minimal impact on downstream performance (i.e. question answering) by dropping the last half of the model's layers, with the exception of the final layer, and applying a small amount of parameter-efficient finetuning. layer-dropping as a pruning strategy has particular benefits: it is straightforward to implement, with memory and inference time decreasing linearly with the number of layers dropped, and it can be combined with other efficiency methods such as quantization. in this work, we build on these findings and apply them in the context of text embedding, resulting in an easy-to-use and efficient approach to transform any pre-trained decoder-only llm into a much smaller text embedding model. by simply pruning the last n% layers of a model before supervised contrastive training, we reduce the final model size with a proportional decrease in memory and inference time. we experiment with four different decoder-only llms ranging from 3.8b to 8b parameters with a variety of pruning percentages and show that up to 30% of a model's layers may be pruned with almost no impact in performance and may even increase it. even intensive pruning of up to 80% still provides reasonably effective text embedding models, with a drop in performance on the downstream embedding task from 64.9 to 59.8 for our highest-performing model. further, we propose l 3 prune, a simple and novel heuristic that pinpoints particular layers to prune to based on the initial loss without requiring significant testing or experimentation. with no input, our method produces both a) a lightly-pruned model, 69-89% of the original size with minimal performance loss of -0.2 on average and even a performance improvement in one model, and b) a heavily pruned model, 16-36% of their original size with a modest performance drop of -4.4 to -6.9. our contributions can be summarized as follows: overall, our work demonstrates that decoderonly llms are generally overparameterized for text embedding tasks and that significant reductions in model size can be achieved with minimal impact on performance. we release the full code for l 3 prune foot_0 . ## pruning we borrow the intuition from @xcite , that the representations in a transformer can be thought of as a slowly changing function of the layer index. specifically, the representation can be formulated as the following iterative residual equation: with ϵ ≪ x ℓ as an approximation. they verify this hypothesis experimentally by calculating the distance between layer representations and using them for a pruning algorithm. their findings indicate that the earlier layers have a significantly larger impact on the representation compared to the later layers, with a particular caveat: the final layer also modifies the representation significantly. thus, they propose and verify a simpler pruning strategy, where the last n layers of the model, excluding the final layer, are dropped. this method requires a "healing" step, recovering the downstream performance with a few qlora finetuning steps @xcite . our hypothesis extends theirs and posits that for the text embedding task, the final layer is also not necessary. our pruning experiments are conducted with the percentage pruned p, between 0% (all layers intact) and 100% (all layers removed). given a pruning percentage and a total number of layers n, the new number of layers n * is calculated as given a model and its configuration, this straightforward procedure can be integrated with modern llm implementations with just three lines of code: @xmath0. num_hidden_layers * (1 -p )) model . @xmath1. layers [: n ] config . @xmath2 we then conduct supervised contrastive training, as with prior work on converting llms to text encoders. instead of an explicit healing step, we hypothesize that the aforementioned training acts as such. thus, no additional or separate training is necessary to execute our method. @xcite . these model families were chosen due to their widespread use in open-source communities and llm literature. as we are conducting pruning and are only concerned with its effects, we pick the smallest model available in each family, and we opt for no modification to the llm architecture itself. we use weighted mean pooling @xcite to generate embeddings from the outputs of the llm as it is straightforward to implement and outperforms other pooling measures when paired with causal attention @xcite . we also conduct supervised contrastive finetuning, known to outperform unsupervised finetuning and the zero-shot setting, and considered to be an integral part of effectively utilizing llms as embedding models @xcite @xcite . we use the replication of the public portion of the e5 dataset @xcite , curated by @xcite , as the training dataset. consisting of approximately 1.5 million samples, it is a multilingual compilation of various retrieval datasets meant for supervised contrastive training of embedding models. in accordance, we use contrastive loss with hard negatives and in-batch negatives @xcite . further details on the dataset and training are provided in appendix a. all experiments were conducted on a single a100 (80gb) gpu, reinforcing the accessible nature of our proposed procedure. ## conclusion in this work, we presented a simple and effective pruning approach to convert llms into lightweight, performant text embedding models. by dropping the last p% layers of the model, we achieved significant reductions in model size and inference time, with minimal impact on text embedding tasks. our procedure is straightforward to implement in pipelines converting llms to text encoders and requires no additional training, providing smaller models at no cost. based on the initial model loss, we also proposed l 3 prune, a heuristic to pinpoint optimal layers to prune to, providing an efficient strategy for pruning without extensive experimentation. we demonstrated that significant pruning-up to 31%-can be conducted with a negligible performance loss, and substantial pruning-up to 84%-can still produce effective models. overall, our results show that decoder-only llms are overparameterized for text embedding tasks and can be pruned with minimal performance loss.