J ina Embeddings: A Novel Set of High-Performance Sentence Embedding Models (2023)

Abstract: Jina Embeddings constitutes a set of high-performance sentence embedding models adept at translating textual inputs into numerical representations, capturing the semantics of the text. These models excel in applications like dense retrieval and semantic textual similarity. This paper details the development of Jina Embeddings, starting with the creation of high-quality pairwise and triplet datasets.It underlines the crucial role of data cleaning in dataset preparation, offers in-depth insights into the model training process, and concludes with a comprehensive performance evaluation using the Massive Text Embedding Benchmark (MTEB). Furthermore, to increase the model’s awareness of grammatical negation, we construct a novel training and evaluation dataset of negated and non-negated statements, which we make publicly available to the community.

## introduction sentence embedding models are an effective instrument for encoding the semantic nuances of words, phrases, and larger textual units into a continuous vector space. they encapsulate the complexities of contexts and lexical and grammatical interrelationships within a text, facilitating downstream tasks like information retrieval, semantic similarity evaluation, and text classification. despite the potential of these models, questions remain about the effectiveness of different data preprocessing strategies, the optimal loss function for training sentence embedding models, and the impact on performance of increasing the number of model parameters. this paper addresses these challenges. we have develop a novel dataset specifically to train our sentence embedding models. furthermore, we design a dataset specifically to sensitize our models to distinguish negations of statements from confirming statements. this paper also presents jina embeddings, a set of high-performance sentence embedding models trained on these datasets. the jina embeddings set is expected to comprise five distinct models, ranging in size from 35 million to 6 billion parameters. three of those models are already trained and published. 1 the jina embeddings models employ contrastive training on the t5 architecture @xcite ]. it's important to note that we opt to use the t5 model as our base due to its pre-training on a mixed set of downstream tasks. we argue that incorporating this approach can potentially enhance our ability to accurately gauge the effectiveness of our training strategy. our large-scale contrastive fine-tuning approach surpasses zero-shot t5 and delivers a performance level on par with other leading t5-based sentence embedding models such as sentence-t5 @xcite and gtr @xcite . consequently, this work demonstrates that high-quality sentence embeddings can be achieved with the judicious use of resources and innovative training methodologies. ## dataset preparation in order to develop models that excel across a wide range of tasks, we collate a comprehensive set of both public and custom datasets. these datasets target various retrieval objectives, such as e-commerce search, duplicate detection, web retrieval, article retrieval for question-answering, and text classification. consolidating these datasets into a unified format facilitates concurrent model training for all tasks. definition of format: given the lack of nonrelevance information in many of the datasets, we reformat each training item into pairs, designated as (q, p) ∈ d pairs . each pair includes a query string q and an associated target string p. to leverage explicit non-relevance judgments, we create an auxiliary set of triplets (q, p, n) ∈ d triplets , which pair a query string q with a match p (positive) and a non-matching string n (negative). ## training training takes place in two distinct phases. the first phase centers on training the model using the voluminous quantity of text pairs, consolidating the semantics of an entire text phrase into a single representative embedding. the second phase uses the relatively small triplet dataset, comprising an anchor, an entailment, and a hard-negative, teaching it to differentiate between similar and dissimilar text phrases. ## evaluation we conduct a comprehensive evaluation to compare our models against other state-of-the-art models (section 4.1), investigate the impact of our filtering pipeline (section 4.2), and evaluate the models' sensitivity to negation of statements (section 4.3). section 6 mentions details about the training. to provide comprehensive results on the performance of models on various downstream tasks applicable to embeddings, we rely on the mteb benchmark frameworks introduced by @xcite . this also compromises all the retrieval tasks included in the beir @xcite benchmark. we also publish the code for executing it on our models on the hugging face pages of our model foot_10 . for evaluating models on the negation dataset, we use our own separate evaluation tool foot_11 . ## training details for training, we employ a100 gpus and leverage the deepspeed stage 2 distributed training strategy @xcite for effective multi-device management. for training our models we use the adamw optimizer, coupled with a learning rate scheduler that adjusts the learning rate during the initial stages of training. the hyperparameters used across all three models throughout the training process are listed in ## conclusion this paper introduces the jina embeddings set of embedding models, demonstrating that competitive performance on various tasks can be achieved while substantially reducing the amount of training data, when compared to other models with comparable backbones. through an extensive evaluation on the mteb benchmark, we show that employing judicious data filtering techniques can lead to enhanced performance in comparison to training with a larger, yet lower-quality dataset. these findings significantly shift the paradigm, indicating that training large language models for embedding tasks can be conducted with less data than previously as- additionally, the jina embeddings set fell short on some tasks. for instance, calculating sentence similarity on our negation dataset (as described in section 4.3) didn't meet our expectations (see looking ahead, we aim to refine our training processes to deliver models with improved performance and greater sequence length. our future endeavors also include generating bilingual training data and training an embedding model capable of understanding and translating between two languages, thereby expanding the utility and versatility of the jina embeddings set.