Adapting General-Purpose Embedding Models to Private Datasets Using Keyword-based Retrieval (2025)

Abstract: Text embedding models play a cornerstone role in AI applications, such as retrieval-augmented generation (RAG). While general-purpose text embedding models demonstrate strong performance on generic retrieval benchmarks, their effectiveness diminishes when applied to private datasets (e.g., company-specific proprietary data), which often contain specialized terminology and lingo. In this work, we introduce BMEmbed, a novel method for adapting general-purpose text embedding models to private datasets. By leveraging the well-established keyword-based retrieval technique (BM25), we construct supervisory signals from the ranking of keyword-based retrieval results to facilitate model adaptation. We evaluate BMEmbed across a range of domains, datasets, and models, showing consistent improvements in retrieval performance. Moreover, we provide empirical insights into how BM25-based signals contribute to improving embeddings by fostering alignment and uniformity, highlighting the value of this approach in adapting models to domain-specific data. We release the source code for the research community.

## introduction text embeddings serve as a cornerstone for various ai applications, particularly in information retrieval and retrieval-augmented generation (rag) systems @xcite . with the widespread adoption of ai, companies like openai and cohere now provide generalpurpose text embedding apis, enabling organizations to quickly integrate ai into their rag systems. however, while these general-purpose embedding models show impressive performance on generic benchmarks, they often face significant challenges when applied to private datasets, such as domain-specific or company-specific proprietary 1 the code is available at: https://github.com/baile ywei/bmembed . data, which often contain specialized terminology and jargon @xcite . for instance, consider a pharmaceutical company that seeks to build a rag system over its vast internal dataset. the company's employees may query the system for information about an internal product code (e.g., product code: phx-121). however, general-purpose models, not trained on this proprietary dataset, may fail to properly interpret or retrieve relevant documents containing such specific terms, leading to suboptimal answers. current practices in rag systems often attempt to address this issue by combining traditional keyword-based retrieval with embedding-based retrieval. one popular hybrid approach is reciprocal rank fusion (rrf), which reranks results based on a mathematical formula without fine-tuning the underlying embedding model @xcite . while simple and effective, rrf remains heuristic, with its effectiveness potentially limited by the lack of fine-tuning to the private dataset. this leads us to the following question: can we fine-tune general-purpose embedding models to better align with private datasets? one of the key challenges in adapting embedding models to domain-specific datasets is the lack of available tuning signals. while general-purpose embedding models are often trained on large, curated qa datasets using contrastive learning @xcite @xcite , private datasets, which often consist of free-text data without annotations, pose a particular challenge. this leads to an important sub-question: how can we generate supervisory signals for adapting general-purpose embedding models to private, unlabeled datasets? in this work, we introduce bmembed, an automated framework designed to adapt generalpurpose text embedding models to private datasets. our method leverages bm25 @xcite , a well-established keyword-based retrieval function based on tf-idf, to generate supervisory signals from the ranking of keywordbased retrieval results. the bmembed framework consists of three main components: @xcite domain query generation, where a large language model generates synthetic queries based on domainspecific events extracted from the private corpus; we evaluate bmembed across multiple domains and datasets, using two general-purpose embedding models with varying scales. compared to base embedding models, bmembed consistently achieves substantial improvements in retrieval accuracy. our experiments further show that bmembed outperforms or achieves competitive performance compared to two commonly used techniques in current rag systems: (1) fine-tuning with in-batch negative contrastive learning, and (2) the rrf hybrid approach. to better understand the inner workings of bmembed, we investigate the alignment and uniformity properties of the adapted embeddings @xcite . we find that bmembed successfully improves embedding uniformity while maintaining good alignment, leading to improved retrieval performance. in summary, this paper introduces a simple yet effective method for adapting general-purpose text embedding models to private datasets. given the increasing adoption of rag systems across industries, we believe our method provides a practical solution to enhance domain specificity, leading to more accurate and contextually relevant retrieval results in real-world applications. ## bmembed: domain adaptation for general-purpose embeddings in this section, we present bmembed, an automated framework designed to tailor generalpurpose embedding models to private datasets consisting of unannotated text. the method contains three steps, and the overall process is illustrated in figure 2 . ## why bmembed enhances embedding adaptation? an investigation from uniformity and alignment in this section, we further investigate why bmembed leads to improvements. we conduct ablation experiments to study how our samplers and temperature interact with retrieval performance. moreover, we introduce the alignment and uniformity properties, which reflect the quality of the embedding, to gain a deeper theoretical understanding. the reported experiments are based on the multihop-rag dataset and the qwen2-1.5b model by default. the complete ablation study setup and results are presented in appendix d. as observed in the ablation study, our experiments empirically reveal a strong agreement between embedding properties and retrieval performance, suggesting that the enhancement from bmembed results from the optimized embedding properties. here, we discuss our key observations and conclusions. ## how does bm25 signal boost embedding? integrating lexical sensitivity with semantic generalization while bmembed is fine-tuned using weak supervision signals derived from bm25, it remains unclear what specific capabilities this adaptation imparts to the embedding model. to investigate this, we design a set of controlled experiments and introduce a two-part decomposition of model behavior: semantic generalization, defined as the ability to capture general semantic patterns, and lexical sensitivity, defined as the sensitivity to domainspecific lexical cues. to assess how bmembed balances these two aspects, we conduct three groups of query perturbation experiments: (1) retrieval using original queries, (2) retrieval with domainspecific keywords masked, and (3) retrieval with query keywords substituted by synonyms. the experimental pipeline involves three stages. first, we use an llm to extract domain-specific keywords and generate semantically appropriate synonyms for each query (see prompt details in appendix f). next, we create two perturbed versions these results suggest that bmembed effectively combines the strengths of both lexical and semantic information. this dual capability makes it particularly well-suited for domains that require adaptation to specialized terminology, or for proprietary enterprise datasets. ## conclusion with the growing adoption of ai in real-world applications, particularly rag systems, adapting general-purpose models to domain-specific data remains a critical challenge. in this paper, we present bmembed, a novel method for adapting text embedding models to private datasets (e.g., company-specific proprietary data). since private datasets often contain specialized terminology and domain-specific language, we leverage keywordbased retrieval as a supervisory signal to fine-tune general-purpose embedding models. experimental results demonstrate that bmembed effectively enhances retrieval performance, producing more accurate query results on private datasets. as ai continues to transform industries, we hope that our proposed method can further advance the adoption and adaptation of ai in domain-specific applications, ensuring more effective and contextually relevant retrieval.