Understanding the Influence of Synthetic Data for Text Embedders (2025)

Abstract: Recent progress in developing general purpose text embedders has been driven by training on ever-growing corpora of synthetic LLM-generated data. Nonetheless, no publicly available synthetic dataset exists, posing a barrier to studying its role for generalization. To address this issue, we first reproduce and publicly release the synthetic data proposed by Wang et al. (2024) (Mistral-E5). Our synthetic data is high quality and leads to consistent improvements in performance. Next, we critically examine where exactly synthetic data improves model generalization. Our analysis reveals that benefits from synthetic data are sparse and highly localized to individual datasets. Moreover, we observe trade-offs between the performance on different categories and data that benefits one task, degrades performance on another. Our findings highlight the limitations of current synthetic data approaches for building general-purpose embedders and challenge the notion that training on synthetic data leads to more robust embedding models across tasks.

## introduction mirroring the success of generative llms @xcite @xcite , the nlp community has invested in building general-purpose embedding models-single models capable of producing embeddings for a wide array of embedding tasks, spanning classification, clustering, retrieval, reranking, and textsimilarity estimation @xcite @xcite @xcite . beyond architectural innovations and progress in base models, much of the recent progress on general-purpose embedders can be attributed to training on synthetic training data-for example, 2024 ), models trained on synthetic data have led to considerable improvements on the mteb leaderboard. we exclude more recent models that were trained on in-domain data. by leveraging gpt-4 to produce synthetic data that expands existing training datasets for embeddings to new tasks (see figure 1 ) @xcite . leveraging synthetic data is based on the premise that llms generate more diverse and high-quality data compared to human-annotated datasets, which are often limited in size. in our work, we critically examine the implicit assumption of this paradigm that training on synthetic data will broadly improve the generalpurpose quality of the model. we approach this by training models on different compositions of synthetic data to estimate the influence of differ-ent commonly adopted types of synthetic data on the downstream performance of the model. strikingly, we find that training on synthetic examples designed for a particular task can degrade the performance of other tasks, challenging the notion that training on more diverse synthetic data is strictly better. moreover, we observe that synthetic data leads to sparse improvement across tasks, showing no statistically significant improvement on a majority of mteb tasks. to conduct our analysis, we reproduce the synthetic data from @xcite and release this data publicly. in our reproduction, we compare the effectiveness of llama-3.1-8b and llama-3.1-70b for generating synthetic data, both of which are more cost-efficient than the gpt-family models used by @xcite and are publicly available @xcite . our results show that synthetic data generated by llama-3.1-8b performs nearly as well as data from the larger 70b model, while costing 5× less than the 70b model and over 50× less than gpt-4o. in total, our results underscore the need to develop broad and robust methods for generating and training on synthetic data that do not exhibit tradeoffs, and we hope that our public release of synthetic training data will accelerate open research into the development and understanding of generalpurpose embedding models. however, the synthetic datasets used to train these models are typically generated using proprietary llms and remain unavailable to the scientific community, making it difficult to understand their role for the generalization of general-purpose embedding models. we address this by reproducing and publicly releasing the synthetic data from our replication of @xcite . we describe our approach for generating synthetic data as follows. ## investigating task influence having replicated the findings of wang et al. (2024), i.e, that synthetic data improves mteb performance on average, we now turn to studying to what extent the different task types of the synthetic data influences performance of each category of mteb. more specifically, does each type of synthetic data broadly improve performance, or are improvements localized to particular tasks? even more importantly, are there inherent performance trade-offs between different tasks based upon the exact synthetic data composition? to these questions is crucial for our understanding on how synthetic data impacts the overall generalization of text embedding models. ## related work general-purpose text encoders. a major challenge when training embedding models is that the notion of similarity these models learn is corpusdriven and hence, models often fail to generalize beyond the similarity definitions they saw during contrastive training @xcite @xcite . driven by benchmarks such as beir @xcite and mteb @xcite , the community has shifted its focus on building text embedding methods that generalize to multiple tasks and domains. while prior bert-based approaches relied on complex multi-stage pipelines to achieve this goal @xcite , recent approaches which are based on decoder-only llms have shown superior performance @xcite @xcite @xcite . leveraging advances in instruction-following capabilities of llms, these models achieve improved generalization to novel tasks and domains by using natural language instruction combined with multi-task contrastive learning. synthetic data for text embeddings. synthetic data has been previously used in the information retrieval literature to generate pseudo queries or hypothetical documents @xcite @xcite . @xcite extended this paradigm to general-purpose text embedding methods, by using llms to generate high-quality training data for diverse embedding tasks. since then, synthetic data generation has become a widely adopted strategy for improving text embedding models, particularly for models competing on the mteb leaderboard. ## conclusion in this work, we investigate how training on synthetic data influences downstream text embedding tasks. our experiments confirm previous findings that synthetic data can significantly boost overall performance on mteb, but offer a more nuanced perspective: we find that training on certain synthetic data categories can exhibit trade-offs in the task-specific performances. for example, while some synthetic data categories improve classification or clustering, they may degrade retrieval performance. our results highlight that the addition of synthetic data does not always strictly improve text embedding models. instead, its benefits are nuanced, requiring refined generation and training strategies to balance trade-offs. beyond our analysis, we contribute a high quality reproduction of the synthetic data of wang et al. (2024) which we release publicly, along with code for our reproduction. by releasing our data and code, we aim to support further research into optimizing synthetic training for general-purpose text embeddings.