Paper: Scaling Sentence Embeddings with Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
In Table 1, only the results based on OPT are presented. Why not also including the results based on LLaMA?

Author response:
In this work, the main focus is on scaling up sentence embeddings with various model sizes. While LLaMA only has four model sizes, we use OPT as the primary model to showcase the results.

To address the concern about the robustness of our method, we evaluated the performance of PromptEOL and PromptEOL+ICL on current popular LLMs, varying in size from 3B to 70B. The evaluated LLMs include LLaMA, LLaMA-2, open-LLaMA, MPT, and Mistral.
Our method shows significant improvement across above 13 models. PromptEOL achieves an average Spearman correlation score of 70.27 on STS tasks among the above models, while PromptEOL+ICL achieves an average score of 77.14. Comparing to the average pooling baseline, PromptEOL and PromptEOL+ICL achieve average improvement of 22.27 and 29.14, respectively. Furthermore, when compared to the previous prompt representation method, PromptEOL and PromptEOL+ICL exhibit an average improvement of 15.55 and 22.42. We have added results in Appendix F of our paper.

Reviewer comment (Experiments):
In Table 1, the best configuration PromptEOL+ICL+OPT (6.7B) does not show clear advantages than PromptRoBERTa (123M).

Author response:
PromptRoBERTa is fine-tuned using contrastive learning, whereas PromptEOL+ICL does not require any fine-tuning. It may not be fair to directly compare these two methods. However, PromptEOL+ICL also achieves similar performance to PromptRoBERTa, which demonstrates that a general language model can achieve such performance without relying on contrastive learning or fine-tuning.
We also introduce other advantages benefit from PromptEOL+ICL in **W5**.

Reviewer comment (Evaluation):
When the model size increases, the performance does always not increase. Especially, the 13B, 30B, and 60B models do not perform better than smaller models such as 1.3B and 6B models.

Author response:
In this work, we aim to investigate the capability of LLMs on sentence embeddings by scaling up the model size. As we mention in our paper, we find that continuing to scale up to over ten billion parameters does not always improve the performance of sentence embeddings for STS tasks. We believe that our work can provide valuable insights for future research on leveraging LLMs for sentence embeddings.

Reviewer comment (Experiments):
The overall method is a little bit heavy. It is worth to discuss whether we should improve the sentence embeddings using large language models.

Author response:
First, our work not only works for LLMs (7B, 13B, 60B models), but also smaller models like OPT 1.3B or 2.7B. Furthermore, we significantly reduce the computation cost of fine-tuning high-quality sentence embeddings compared to the previous SOTA methods like ST5. We can exceed the previous 4.8B ST5 with 2.7B OPT fine-tuned on less training data. Compared to 4.8B ST5,  it uses two-stage training with 2 billion extra question-answers pairs and 275k NLI datasets by fine-tuning all parameters, PromptEOL+CSE can leverage efficient fine-tuning method QLoRA on 275k NLI datasets with one epoch. We can train 2.7B OPT with PromptEOL+CSE for around one hour on four 3090 GPUs. However, it will take more than hundreds of hours with the same hardware environment to train 4.8B ST5.

Second, PromptEOL+ICL also provides a lightweight method that allows LLMs to generate sentence embeddings without the need of fine-tuning. We can use LLMs themselves to embedding sentence to perform retrieval augmented generation, without additional embedding models. Moreover, we can control the embedding behaviors by modifying prompt and ICL example to suit specific application, such as cross-lingual representation. For instance, we can utilize an ICL example with a Chinese sentence and an English summary word to project the semantics of Chinese sentences onto an English word. This can help bridge the language gap in sentence embeddings, which we leave for future work.

Reviewer comment (Experiments):
Do you also try LLaMA 2?

Author response:
Yes, we have tried LLaMA-2 with PromptEOL+CSE, it achieves slightly better results compared to LLaMA.

|                           | STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R |  Avg. |
| -------                   |  :--: |  :--: |  :--: |  :--: |  :--: |  :--: |   :--: |  :--: |
| PromptEOL+CSE LLaMA 7b    | 79.16 | 90.22 | 85.40 | 88.99 | 86.25 | 88.37 |  81.51 | 85.70 |
| PromptEOL+CSE LLaMA 13b   | 78.63 | 90.03 | 85.46 | 89.48 | 86.18 | 88.45 |  82.69 | 85.85 |
| PromptEOL+CSE LLaMA-2 7b  | 78.48 | 90.07 | 84.86 | 89.43 | 86.16 | 88.44 |  83.20 | 85.81 |
| PromptEOL+CSE LLaMA-2 13b | 78.84 | 90.35 | 85.88 | 89.72 | 86.68 | 88.91 |  82.64 | 86.15 |

Reviewer comment (Theory):
In Equation 1, why using the last token hidden state as the sentence representation instead of the representation vector of the last generated token using the explicit one word prompt?

Author response:
The last token hidden state is utilized to predict the next token in an autoregressive model. When using the explicit one-word prompt, the last token hidden state can capture the semantic meaning of the sentence.

For the last generated token, there are several problems with it. First, since LLMs are not instruction tuning, the generation can be uncontrollable. For example, LLMs may generate additional tokens even after providing a one-word summary of the sentence. Second, due to LLMs predicting token one by one, the generation process is slower compared to using the last token hidden state.

Reviewer comment (Experiments):
For the fine-tuning, do you also try including in-context demonstration for the fine-tuning?

Author response:
Yes, we did try including in-context demonstrations for the fine-tuning. However, it did not help improve the fine-tuning performance. For example, on OPT 6.7B, it achieved an average Spearman correlation of 85.56 on STS tasks, compared to the original results of 85.82.

Reviewer comment (Experiments):
While it is helpful to the reader to see the entire distribution of Spearman correlations, it may be relevant to give more details on how the two sources for ICL data impact the quality of downstream representations. The 1/3-2/3 mix of STS sentences vs Oxford definitions would benefit from an explicit ablation.

Author response:
We would like to clarify that in our method, only one example from the ICL data is used to perform sentence embeddings. To select this example, we evaluate the performance of each example on the STS-B development set and choose the one that performs the best. This process is described in detail in Section 3.2 of our paper. The proportion of STS-B data and the Oxford dictionary in our method is not a crucial factor; we use a 1/3-2/3 mix simply because obtaining data from the Oxford dictionary is much easier compared to generating words from ChatGPT.

Reviewer comment (Experiments):
The value of ICL and CSE is demonstrated only for OPT. Indeed, table 1 is missing results that would demonstrate the added value of ICL and CSE on Llama.

Author response:
In this work, we focus on scaling up sentence embeddings with various model sizes. Compared to OPT, LLaMA only has four model sizes. We use OPT as the main model to show the results. Due to the page limit, we only present the ICL results on OPT. However, we have included the CSE on LLaMA in Table 2.

To address the concern about the robustness of our method, we evaluated the performance of PromptEOL and PromptEOL+ICL on current popular LLMs, varying in size from 3B to 70B. The evaluated LLMs include LLaMA, LLaMA-2, open-LLaMA, MPT, and Mistral.
Our method shows significant improvement across above 13 models. PromptEOL achieves an average Spearman correlation score of 70.27 on STS tasks among the above models, while PromptEOL+ICL achieves an average score of 77.14. Comparing to the average pooling baseline, PromptEOL and PromptEOL+ICL achieve average improvement of 22.27 and 29.14, respectively. Furthermore, when compared to the previous prompt representation method, PromptEOL and PromptEOL+ICL exhibit an average improvement of 15.55 and 22.42. We have included detailed results in Appendix F of our paper.

Reviewer comment (Experiments):
The proposed methods (in-context learning and possibly fine-tuning) are performing worse than simple explicit-one-word-limit prompting on transfer tasks.

Author response:
Benefiting from our PromptEOL method, LLMs can capture the semantics of sentences well for transfer tasks. In Figure 4(c), we demonstrate that PromptEOL significantly outperforms other representation methods like prompt and average pooling on transfer tasks, achieving state-of-the-art results without fine-tuning.
For in-context learning, we can flexibility modify the prompt to generate embeddings that fit the task following [1]. For example, by using a sentiment classification example, we can improve the results of PromptEOL 6.7B OPT from 91.34 to 91.78 average accuracy on seven transfer tasks.
For fine-tuning, due to limited computational resources, we only use efficient fine-tuning LLMs with QLoRA, which may harm the original transfer task performance through 4-bit quantization.

**References:**

[1] Su H, Kasai J, Wang Y, et al. One embedder, any task: Instruction-finetuned text embeddings (ACL findings 2023)

Reviewer comment (Writing):
In section 5.2, the text mentions "in-context learning examples that were obtained from each model on the STS-B development set", while the table caption reads "In-context learning examples used in various model size". The paper states clearly in section 3.2 that the in-context learning examples (1) come from the STS-B training set and (2) are not generated / obtained from the model itself.

Author response:
We would like to clarify the misunderstandings regarding section 5.2 and our PromptEOL+ICL method.
* The text mentioning "in-context learning examples that were obtained from each model on the STS-B development set" is reference to the best performance example on STS-B development set as introduced in section 3.2. This example comes from STS-B training set and Oxford dictionary.

* The method used to sample the data is introduced in section 3.2, which we evaluate the performance of each example on STS-B development set and choose the best one. The "Improve" column is the average improvement of the PromptEOl+ICL compared to PromptEOL in the seven STS test set.

* For the conclusion that "related examples are usually more implicit," it is important to note that the samples in Table 4 were not randomly selected from the ICL examples. Instead, they were chosen based on the performance of the corresponding model on the STS-B development set. In other words, the samples listed in Table 4 represent the most suitable example among 300 ICL examples for each model to represent sentences as embeddings. For "appending 5-10 random samples of each category", sentence embeddings, as a task to represent the sentences as vectors, do not have categories like the classification task.

* For the relationship between section 5.2 and our methods, section 5.2 is to report that the ICL examples used in PromptEOL+ICL. It also show the improvement of PromptEOL+ICL compared to PromptEOL in the "Improve" column.

Reviewer comment (Experiments):
In section 5.2, the method used to sample the data from table 4 is not described, and the meaning of the "Improve" column is not clear: does it correspond to the improvement coming from one additional sentence in the prompt? Or from the addition of the 100s of sentences in the ICL prompt? In any case, it seems premature to draw generic conclusions such as "related examples are usually more implicit" from a sample size of 1 from each model. Appending 5-10 random samples of each category in the appendix would give more compelling evidence for this.

Author response:
We would like to clarify the misunderstandings regarding section 5.2 and our PromptEOL+ICL method.
* The text mentioning "in-context learning examples that were obtained from each model on the STS-B development set" is reference to the best performance example on STS-B development set as introduced in section 3.2. This example comes from STS-B training set and Oxford dictionary.

* The method used to sample the data is introduced in section 3.2, which we evaluate the performance of each example on STS-B development set and choose the best one. The "Improve" column is the average improvement of the PromptEOl+ICL compared to PromptEOL in the seven STS test set.

* For the conclusion that "related examples are usually more implicit," it is important to note that the samples in Table 4 were not randomly selected from the ICL examples. Instead, they were chosen based on the performance of the corresponding model on the STS-B development set. In other words, the samples listed in Table 4 represent the most suitable example among 300 ICL examples for each model to represent sentences as embeddings. For "appending 5-10 random samples of each category", sentence embeddings, as a task to represent the sentences as vectors, do not have categories like the classification task.

* For the relationship between section 5.2 and our methods, section 5.2 is to report that the ICL examples used in PromptEOL+ICL. It also show the improvement of PromptEOL+ICL compared to PromptEOL in the "Improve" column.

Reviewer comment (Evaluation):
It is not clear how to relate the findings of section 5.2 to the overall quality of the sentence embeddings introduced by this work.

Author response:
We would like to clarify the misunderstandings regarding section 5.2 and our PromptEOL+ICL method.
* The text mentioning "in-context learning examples that were obtained from each model on the STS-B development set" is reference to the best performance example on STS-B development set as introduced in section 3.2. This example comes from STS-B training set and Oxford dictionary.

* The method used to sample the data is introduced in section 3.2, which we evaluate the performance of each example on STS-B development set and choose the best one. The "Improve" column is the average improvement of the PromptEOl+ICL compared to PromptEOL in the seven STS test set.

* For the conclusion that "related examples are usually more implicit," it is important to note that the samples in Table 4 were not randomly selected from the ICL examples. Instead, they were chosen based on the performance of the corresponding model on the STS-B development set. In other words, the samples listed in Table 4 represent the most suitable example among 300 ICL examples for each model to represent sentences as embeddings. For "appending 5-10 random samples of each category", sentence embeddings, as a task to represent the sentences as vectors, do not have categories like the classification task.

* For the relationship between section 5.2 and our methods, section 5.2 is to report that the ICL examples used in PromptEOL+ICL. It also show the improvement of PromptEOL+ICL compared to PromptEOL in the "Improve" column.

Reviewer comment (Experiments):
It would seem that some experiments have been run only on OPT, while others have been run on OPT and Llama. It would be helpful to have all experiments run on both models to show that the conclusions are robust to the choice of base LLM.

Author response:
Please refer to **W2**.

Reviewer comment (Novelty):
While the methods proposed are sound, they consist of previously suggested and widely implemented techniques, which diminishes the novelty aspect of the work.

Author response:
The primary focus of our research is the scaling up of sentence embeddings using LLMs, which remains unexplored. We also proposed three methods to leverage the capacity of LLMs for sentence embeddings: PromptEOL, PromptEOL+ICL, and PromptEOL+CSE.

While our work is inspired by earlier studies such as PromptBERT, we have identified that PromptBERT does not extend its applicability to LLMs for sentence embeddings. To address this shortcoming, we have proposed a solution named PromptEOL. This method introduces an implicit single-word limitation to the prompt, thereby facilitating the generation of sentence embeddings. It is necessary to mention that, as the output of sentence embeddings results in a singular vector, in-context learning cannot be applied directly. To overcome this challenge, we propose PromptEOL+ICL. To the best of our knowledge, we are the first to show that in-context learning can be applied to sentence embeddings without fine-tuning.

Reviewer comment (Experiments):
Contrary to SimCSE, the in-context learning approach depends on the use of the STS-B dataset, including its training and validation components, which could potentially confer an unfair advantage to the method.

Author response:
We think the comparison with SimCSE and other similar methods is fair. First, SimCSE also uses STS-B validation set for selecting the best checkpoint during training. Second, we do not use any label information from the STS-B training set. Instead, we only utilize 100 sentences from the STS-B training set, ensuring that none of these sentences come from the same sentence pairs. In fact, our approach outperforms SimCSE even when solely leveraging the Oxford dictionary. Finally, our ICL method does not rely on fine-tuning or contrastive learning, which are considered essential for achieving better representation in current sentence embeddings methods[1,2].

Reviewer comment (Presentation):
There appears to be no direct link between the two proposed methods; that is, the approach based on in-context learning and the one utilizing contrastive learning.

Author response:
Both in-context learning and contrastive learning methods are built upon our proposed representation method, namely PromptEOL. This method focuses on two settings in sentence embeddings: with and without fine-tuning. In the case of in-context learning, we demonstrate that combining PromptEOL with in-context learning yields high-quality sentence representations without the need for fine-tuning. Additionally, in the context of contrastive learning, we provide evidence in Figure 4(b) that PromptEOL can benefit from contrastive learning, outperforming other representation methods.

Reviewer comment (Evaluation):
I'm curious whether the authors have any insights or hypotheses as to why (much) larger models (over 10B) do not excel as expected in computing sentence representations, which contrasts with their effectiveness in other standard applications.

Author response:
We have also observed that the scaling law performs well in transfer learning tasks, as shown in Table 3, but does not yield the same success in STS tasks, as shown in Table 1. We think that the main reason for this discrepancy may be related to the issue of anisotropy in sentence representation [1,3], which harms the performance on STS tasks. To validate this, we measure the anisotropy of sentence embeddings across different model sizes following [4].

|                    | sentence anisotropy |
| ---------- | :--------: |
| PromptEOL OPT-1.3b |               0.565 |
| PromptEOL OPT-2.7b |               0.589 |
| PromptEOL OPT-6.7b |               0.694 |
| PromptEOL OPT-13b  |               0.719 |
| PromptEOL OPT-30b  |               0.751 |
| PromptEOL OPT-66b  |               0.715 |

Through our analysis, we have observed that the anisotropy of sentence embeddings increases as the model size increases. While larger models may possess greater overall capacity, the issue of anisotropy may limit their performance.