Improving Sentence Embeddings with Automatic Generation of Training Data Using Few-shot Examples (2024)

Abstract: Decoder-based large language models (LLMs) have shown high performance on many tasks in natural language processing. This is also true for sentence embedding learning, where a decoder-based model, PromptEOL, has achieved the best performance on semantic textual similarity (STS) tasks. However, PromptEOL requires a manually annotated natural language inference (NLI) dataset for fine-tuning.We aim to improve sentence embeddings without using large manually annotated datasets by automatically generating an NLI dataset with an LLM and using it for fine-tuning of PromptEOL. To achieve this, we explore methods of data generation suitable for sentence embedding learning in this study. Specifically, we will focus on automatic dataset generation through few-shot learning and explore the appropriate methods to leverage few-shot examples. Experimental results on the STS tasks demonstrate that our approach outperforms existing models in settings without large manually annotated datasets.

## introduction sentence embeddings are widely studied as they can be used for many tasks such as text search, entailment recognition, and information extraction @xcite @xcite @xcite . among these, methods based on decoder-based large language models (llms) have shown high performance in recent years. for example, sgpt @xcite , which uses decoder-based llms to generate embeddings, and prompteol @xcite , which generates sentence embeddings using a prompt-based method focusing on a single word, have been proposed. prompteol achieves the highest performance in sts in a setting using manually annotated data. however, when not using manually annotated nli datasets, its performance is much lower. since the advent of high-performance decoderbased llms like gpt-4, foot_0 many efforts have been made to use data generated by decoder-based llms as a substitute for training data in various tasks, and their effectiveness has been reported @xcite . similarly, for sentence embedding learning, there are approaches such as gense @xcite , which automatically generates datasets using llms to augment sentence embedding datasets, and sts-dino @xcite , which is an automatically generated dataset for training sentence embedding models using llms. however, there has not been sufficient investigation on how to generate data using llms for sentence embedding learning. it is known that when generating datasets automatically via few-shot learning, the generated datasets are heavily dependent on the few-shot examples @xcite , and if all the data is generated by using the same few-shot examples, the diversity of the generated datasets may be limited. in this study, we explore how few-shot examples should be leveraged to automatically generate training data to obtain better sentence embeddings in a framework where nli datasets generated by an llm are used for fine-tuning of prompteol. specifically, we examine how the quality of the final sentence embeddings varies when the number of few-shot examples used to generate training data is varied or when multiple sets of few-shot examples are used, and we reveal the optimal way to leverage few-shot examples. our contributions are two-fold. first, we explored the optimal ways to leverage few-shot examples when using llms to generate nli datasets for sentence embedding learning. second, we achieved the highest score in the sts tasks in a setting without large manually annotated datasets. ## related work this section introduces promptroberta @xcite and prompeol @xcite , which successfully generate high-performance sentence embeddings by devising prompts. promptroberta promptroberta introduces a new contrastive learning method to improve sentence embedding performance of roberta. specifically, it takes a sentence like "i have a dog." as input and transforms it using templates as follows: "this sentence: "i have a dog." means @xcite ." and "the sentence: "i have a dog." means @xcite ." . by using the embeddings of the "[mask]", it can represent the same sentence from diverse perspectives using different templates, resulting in reasonable positive pairs of sentence embeddings. by learning to bring these positive pairs of sentence embeddings closer together, promptroberta significantly reduces the performance gap between supervised and unsupervised settings, achieving better sentence embedding performance compared to traditional methods. prompteol prompteol introduces a constraint called the "one-word limitation" and inputs the target sentence into llms along with a template. for example, to obtain the embedding of the sentence "i have a dog.", it inputs the prompt "this sentence: "i have a dog." means in one word: "" into a decoder-based llm. the hidden vector after "in one word: "" is then used as the sentence embedding. since the decoder-based llm is pretrained on the next-token prediction task, it can obtain a sentence embedding that captures the meaning of the whole sentence by using the prompt to predict a word that paraphrases the entire sentence. although prompteol demonstrates relatively high performance in an unsupervised setting, it can produce higher quality embeddings through supervised learning. prompteol achieved the best performance in the sts tasks by fine-tuning the llm via contrastive learning on nli datasets similar to supervised simcse @xcite . ## automatic nli dataset generation in this study, we explore how to automatically construct datasets for sentence embedding learning using llms. in this section, we explain the process of generating nli datasets with llms. ## experiments we first evaluate automatically generated nli datasets using nli classifiers. next, we evaluate sentence embedding models fine-tuned with automatically generated nli datasets. we explore how to use few-shot examples specifically for sentence ## conclusion and future work in this study, we explore optimal ways to leverage few-shot examples when using llms to generate nli datasets for sentence embedding learning. through experiments, we found that the performance could be enhanced by dividing the few-shot examples, as seen with the 5-shot×4 setting, since it improves dataset diversity. furthermore, models trained with automatically generated nli datasets outperformed existing unsupervised methods. in future work, we will explore more sophisticated ways to generate a diverse and high-quality dataset. for example, instead of just dividing fewshot examples, a set of various overlapping fewshot examples could be generated and used in fewshot learning. it is also future work to apply our data generation procedure, which generates data by dividing few-shot examples, to data generation other than nli datasets for sentence embedding learning.