Prompt Tuning Can Simply Adapt Large Language Models to Text Encoders (2025)

Abstract: Recently, many works have been attempting to adapt Large Language Models (LLMs) for sentence embedding, with most of them fine-tuning LLMs towards the contrastive objective and enabling bi-directional attention for better performance, using LoRA to address the large model scale.In this work, we suggest that this adaptation can also be simply and effectively achieved using causal attention and with even fewer trainable parameters through soft prompt tuning, as an alternative to fine-tuning with LoRA and other methods with extra post-training tasks.Our method only optimizes a few learnable tokens while keeping the rest of the model frozen.Through experiments on a diverse set of evaluation tasks, we find that simply tuning only a few tokens can achieve a competitive performance with that of fine-tuning with LoRA. The percentage of trainable parameters can be reduced to less than 0.001%. Moreover, we also demonstrate that turning causal attention to bi-directional attention with or without extra post-training tasks does not provide additional benefit when soft prompt tuning is applied, suggesting that causal attention can be naturally used in decoder-only LLMs for sentence embedding adaptation.

## introduction sentence embedding compresses the semantic meaning of sentences into fixed-size vectors in a shared space @xcite @xcite . conventional sentence embedding models are typically built on an encoder-only architecture trained with contrastive learning (cl) @xcite , where the distance between semantically similar sentences are pulled closer and dissimilar ones are pushed farther @xcite @xcite . on the other hand, scaled-up large language models (llms) in the decoder-only architecture have refer to to this end, numerous recent methods attempt to adapt llms for sentence embedding, e.g., cl-based fine-tuning @xcite , attention mechanism manipulation @xcite , instruction tuning @xcite , with some approaches employing the combinations thereof. among these efforts, llm2vec @xcite stands out as a promising method, employing a threestep approach: (1) enabling bi-directional attention, (2) using masked next token prediction (mntp) @xcite to effectively adapt llms to bi-directional attention, and (3) fine-tuning with cl, as shown in the upper part of figure 2 . step1: enable bidirectional attention step3: cl with lora step2: post-training with lora bi-directional attention llm bi-directional attention llm fine-tuned bi-directional attention llm cl with suffix prompt tuning causal attention llm our simple adaptation … … causal attention llm ## conclusion in this work, we first investigate a simple method to adapt llms for sentence embedding by tuning a few learnable tokens. we append trainable tokens to the inputs and utilize the output of the last one as the sentence embedding. our approach can achieve the adaptation with less than 0.001% trainable parameters, which is unattainable with lora. experimental results on various tasks demonstrate that only a few tokens with our approach can achieve competitive performance with fine-tuning with lora. moreover, we also find that directly using causal attention in decoder-only llms is capable of adapting them for sentence embedding. specifically, our simple method with causal attention outperforms bi-directional attention baselines with extra post-training tasks, offering insights on the adaptation of llms for sentence embedding.