Little Giants: Synthesizing High-Quality Embedding Data at Scale (2025)

Abstract: Synthetic data generation has become an increasingly popular way of training models without the need for large, manually labeled datasets. For tasks like text embedding, synthetic data offers diverse and scalable training examples, significantly reducing the cost of human annotation. However, most current approaches rely heavily on proprietary models like GPT-4, which are expensive and inefficient for generating large-scale embedding data. In this paper, we introduce SPEED, a framework that aligns open-source small models (8B) to efficiently generate large-scale synthetic embedding data. Through supervised fine-tuning, preference optimization, and self-improvement, SPEED enables small open-source models to produce high-quality data. Remarkably, SPEED uses only less than 1/10 of the GPT API calls, outperforming the state-of-the-art embedding model E5_mistral when both are trained solely on their synthetic data. Using this efficient generator, we conduct a comprehensive study on how various factors within the alignment pipeline impact data quality and reveal the scaling law for synthetic embedding data. Our codes and models are released in https://github.com/haon-chen/SPEED.

## introduction text embedding models encode natural language texts into latent vectors. they are widely used in downstream tasks such as classification, clustering, retrieval, and summarization. many researchers have trained general embedding models that can support various tasks @xcite @xcite . most of these models require large-scale weakly-supervised data and high-quality labeled data for multi-stage training, which requires careful data curation and costly human effort. thanks to the powerful language modeling ability and vast knowledge of large language models (llms), some works attempt to utilize llms to generate synthetic data for training embedding models @xcite @xcite . however, most of these works solely use proprietary llm like gpt-4 for data synthesis @xcite . for example, e5 mistral generates triplets of (query, positive document, hard negative document) for various embedding tasks from scratch. while synthesizing embedding data without relying on existing corpora can yield more diverse examples, using black-box models can be extremely costly, especially given that this data often includes long documents. a straightforward approach to reduce costs is to use small models to synthesize embedding data instead, which have proven effective for tasks such as mathematical reasoning @xcite @xcite . however, synthesizing embedding data often requires the generation of hard negatives -documents that are similar to positive ones and are essential for learning nuanced embedding representations. these hard negatives are challenging for small models to synthesize, as they are difficult for language models to distinguish. an early work explores the ability of small models for synthesizing embedding data @xcite , but it uses small models to generate data directly without special tailoring for data synthesis, resulting in poor performance. in this work, we propose to align open-source small models (8b) to synthesize large-scale highquality embedding data. compared to existing methods that rely solely on expensive gpt-4, our approach can generate more data at a much lower cost. our primary goal is to study the alignment of small models for synthesizing embedding data, which has been neglected by existing works. specifically, we aim to address the following research questions in this paper: rq1: how to align small models for synthesizing high-quality embedding data at scale? rq2: how do factors within the alignment framework affect the quality of synthetic data? rq3: synthetic data is theoretically infinite. what is the scaling law for synthetic embedding data? to shed light on rq1, we design an alignment framework that trains small llms to efficiently synthesize large-scale superior embedding data (speed). as illustrated in figure 1 , our framework consists of three key models: a junior generator for initial data synthesis, a senior generator for advanced data generation, and a data revisor for self-improvement. the goal is to distill knowledge from gpt-4 into these smaller models. we first use gpt-4 to brainstorm task descriptions. however, since gpt-4 often generates hallucinations and data of specific domains (e.g., climate change) (chang, 2023), we sample topics from the open directory project to ensure diverse and balanced tasks. 1 based on these tasks, gpt-4 produces a small set of seed data, which we use to finetune the junior generator via supervised fine-tuning (sft). the junior generator produces root data, which is further evaluated by gpt-4 to produce signals that guide the preference optimization process, resulting in a senior generator. the root data is also revised by gpt-4 to produce revision signals for training a data revisor. inspired by the idea of scaling inference compute for llms @xcite , the revisor refines the synthetic data with minimal additional inference cost, enabling self-improvement. as for rq2, with these low-cost yet powerful data synthesis models ready, we are able to conduct extensive experiments to study the factors affecting the alignment. we find that settings such as the base model used for alignment, the diversity of tasks, and the number of training samples can 1 http://odp.org : open-source collection of web topics. in summary, our contributions are as follows: text embedding text embedding models have gained much attention in the era of deep learning. some existing models, such as sbert @xcite , e5 @xcite , and bge @xcite , attempt to produce general text embeddings for various tasks. however, most of them require lots of labeled data. in this work, we attempt to train a model with synthetic data. large language models though proprietary llms (openai, 2023; anthropic, 2024) are very powerful, invoking their apis can be quite expensive and unaffordable for common usage. many open-source llms have been released for more efficient language modeling, such as llama @xcite and mistral @xcite . some works attempt to improve the ability of llms for text embedding tasks, such as ad-hoc retrieval @xcite , conversational retrieval @xcite , and multilingual text embedding @xcite . our work aims to use synthetic data to improve the llm's ability of text embedding. ## methodology: speed in this section, we aim to answer rq1 using our alignment framework, speed. as shown in figure 2, speed consists of four stages: (1) gpt-4 is first used to generate diverse task descriptions based on multi-grained topics sampled from the odp. a junior generator then distills knowledge from gpt-4 by training on a small set of seed data. ## conclusion in this work, we propose a framework speed that aligns small models for the efficient and effective synthesis of embedding data. through supervised finetuning, preference optimization, and selfimprovement, small models can also synthesize high-quality embedding data at scale. additionally, we comprehensively investigate how various factors within the alignment pipeline influence data quality. we reveal the scaling law of synthetic embedding data, demonstrating a log-linear relationship between the performance of the embedding model and the size of the synthetic data.