nomic-embed-text-v1-ablated: A Reproducible Long Context (8192) Text Embedder
nomic-embed-text-v1-ablated
is 8192 context length text encoder that surpasses OpenAI text-embedding-ada-002 performance on short and long tasks.
.
Name | SeqLen | MTEB | LoCo | Jina Long Context | Open Weights | Open Training Code | Open Data |
---|---|---|---|---|---|---|---|
nomic-embed-text-v1 | 8192 | 62.39 | 85.53 | 54.16 | ✅ | ✅ | ✅ |
jina-embeddings-v2-base-en | 8192 | 60.39 | 85.45 | 51.90 | ✅ | ❌ | ❌ |
text-embedding-3-small | 8191 | 62.26 | 82.40 | 58.20 | ❌ | ❌ | ❌ |
text-embedding-ada-002 | 8191 | 60.99 | 52.7 | 55.25 | ❌ | ❌ | ❌ |
If you would like to finetune a model on more data, you can use this model as an initialization
Hosted Inference API
The easiest way to get started with Nomic Embed is through the Nomic Embedding API.
Generating embeddings with the nomic
Python client is as easy as
from nomic import embed
output = embed.text(
texts=['Nomic Embedding API', '#keepAIOpen'],
model='nomic-embed-text-v1',
task_type='search_document'
)
print(output)
For more information, see the API reference
Data Visualization
Click the Nomic Atlas map below to visualize a 5M sample of our contrastive pretraining data!
Training Details
We train our embedder using a multi-stage training pipeline. Starting from a long-context BERT model, the first unsupervised contrastive stage trains on a dataset generated from weakly related text pairs, such as question-answer pairs from forums like StackExchange and Quora, title-body pairs from Amazon reviews, and summarizations from news articles.
In the second finetuning stage, higher quality labeled datasets such as search queries and answers from web searches are leveraged. Data curation and hard-example mining is crucial in this stage.
For more details, see the Nomic Embed Technical Report and corresponding blog post.
Training data to train the models is released in its entirety. For more details, see the contrastors
repository
Usage
Note nomic-embed-text
requires prefixes! We support the prefixes [search_query, search_document, classification, clustering]
.
For retrieval applications, you should prepend search_document
for all your documents and search_query
for your queries.
Sentence Transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("nomic-ai/nomic-embed-text-v1-ablated", trust_remote_code=True)
sentences = ['search_query: What is TSNE?', 'search_query Who is Laurens van der Maaten?']
embeddings = model.encode(sentences)
print(embeddings)
Transformers
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel
def mean_pooling(model_output, attention_mask):
token_embeddings = model_output[0]
input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
sentences = ['search_query: What is TSNE?', 'search_query: Who is Laurens van der Maaten?']
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('nomic-ai/nomic-embed-text-v1-ablated', trust_remote_code=True)
model.eval()
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
with torch.no_grad():
model_output = model(**encoded_input)
embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
embeddings = F.normalize(embeddings, p=2, dim=1)
print(embeddings)
The model natively supports scaling of the sequence length past 2048 tokens. To do so,
- tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
+ tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', model_max_length=8192)
- model = AutoModel.from_pretrained('nomic-ai/nomic-embed-text-v1-ablated', trust_remote_code=True)
+ model = AutoModel.from_pretrained('nomic-ai/nomic-embed-text-v1-ablated', trust_remote_code=True, rotary_scaling_factor=2)
Join the Nomic Community
- Nomic: https://nomic.ai
- Discord: https://discord.gg/myY5YDR8z8
- Twitter: https://twitter.com/nomic_ai
- Downloads last month
- 5,187
Collection including nomic-ai/nomic-embed-text-v1-ablated
Evaluation results
- accuracy on MTEB AmazonCounterfactualClassification (en)test set self-reported78.672
- ap on MTEB AmazonCounterfactualClassification (en)test set self-reported42.738
- f1 on MTEB AmazonCounterfactualClassification (en)test set self-reported72.800
- accuracy on MTEB AmazonPolarityClassificationtest set self-reported90.414
- ap on MTEB AmazonPolarityClassificationtest set self-reported87.088
- f1 on MTEB AmazonPolarityClassificationtest set self-reported90.392
- accuracy on MTEB AmazonReviewsClassification (en)test set self-reported47.808
- f1 on MTEB AmazonReviewsClassification (en)test set self-reported47.257
- map_at_1 on MTEB ArguAnatest set self-reported30.370
- map_at_10 on MTEB ArguAnatest set self-reported45.748
- map_at_100 on MTEB ArguAnatest set self-reported46.617
- map_at_1000 on MTEB ArguAnatest set self-reported46.622
- map_at_3 on MTEB ArguAnatest set self-reported40.564
- map_at_5 on MTEB ArguAnatest set self-reported43.690
- mrr_at_1 on MTEB ArguAnatest set self-reported30.868
- mrr_at_10 on MTEB ArguAnatest set self-reported45.905
- mrr_at_100 on MTEB ArguAnatest set self-reported46.787
- mrr_at_1000 on MTEB ArguAnatest set self-reported46.792
- mrr_at_3 on MTEB ArguAnatest set self-reported40.718
- mrr_at_5 on MTEB ArguAnatest set self-reported43.851
- ndcg_at_1 on MTEB ArguAnatest set self-reported30.370
- ndcg_at_10 on MTEB ArguAnatest set self-reported54.662
- ndcg_at_100 on MTEB ArguAnatest set self-reported58.237
- ndcg_at_1000 on MTEB ArguAnatest set self-reported58.373
- ndcg_at_3 on MTEB ArguAnatest set self-reported44.069
- ndcg_at_5 on MTEB ArguAnatest set self-reported49.728
- precision_at_1 on MTEB ArguAnatest set self-reported30.370
- precision_at_10 on MTEB ArguAnatest set self-reported8.321
- precision_at_100 on MTEB ArguAnatest set self-reported0.985
- precision_at_1000 on MTEB ArguAnatest set self-reported0.100
- precision_at_3 on MTEB ArguAnatest set self-reported18.089
- precision_at_5 on MTEB ArguAnatest set self-reported13.613
- recall_at_1 on MTEB ArguAnatest set self-reported30.370
- recall_at_10 on MTEB ArguAnatest set self-reported83.215
- recall_at_100 on MTEB ArguAnatest set self-reported98.506
- recall_at_1000 on MTEB ArguAnatest set self-reported99.573
- recall_at_3 on MTEB ArguAnatest set self-reported54.267
- recall_at_5 on MTEB ArguAnatest set self-reported68.065
- v_measure on MTEB ArxivClusteringP2Ptest set self-reported45.853
- v_measure on MTEB ArxivClusteringS2Stest set self-reported36.127
- map on MTEB AskUbuntuDupQuestionstest set self-reported57.588
- mrr on MTEB AskUbuntuDupQuestionstest set self-reported71.841
- cos_sim_pearson on MTEB BIOSSEStest set self-reported87.925
- cos_sim_spearman on MTEB BIOSSEStest set self-reported85.371
- euclidean_pearson on MTEB BIOSSEStest set self-reported86.193
- euclidean_spearman on MTEB BIOSSEStest set self-reported85.371
- manhattan_pearson on MTEB BIOSSEStest set self-reported86.469
- manhattan_spearman on MTEB BIOSSEStest set self-reported85.914
- accuracy on MTEB Banking77Classificationtest set self-reported83.818
- f1 on MTEB Banking77Classificationtest set self-reported83.762
- v_measure on MTEB BiorxivClusteringP2Ptest set self-reported38.464