pipeline_tag: sentence-similarity
tags:
- finetuner
- mteb
- sentence-transformers
- feature-extraction
- sentence-similarity
- alibi
datasets:
- allenai/c4
language: en
license: apache-2.0
model-index:
- name: jina-embedding-s-en-v2
results: []
The text embedding set trained by Jina AI, Finetuner team.
Intended Usage & Model Info
jina-embedding-s-en-v2 is an English, monolingual embedding model supporting 8k sequence length.
It is based on a Bert architecture that supports the symmetric bidirectional variant of ALiBi to support longer sequence length.
The backbone Jina Bert Small model is pretrained on the C4 dataset.
The model is further trained on Jina AI's collection of more than 40 datasets of sentence pairs and hard negatives.
These pairs were obtained from various domains and were carefully selected through a thorough cleaning process.
The embedding model was trained using 512 sequence length, but extrapolates to 8k sequence length thanks to ALiBi. This makes our model useful for a range of use cases, especially when processing long documents is needed, including long document retrieval, semantic textual similarity, text reranking, recommendation, RAG and LLM-based generative search,...
This model has 33 million parameters, which enables lightning-fast and memory efficient inference on long documents, while still delivering impressive performance. Additionally, we provide the following embedding models, supporting 8k sequence length as well:
jina-embedding-s-en-v2: 33 million parameters (you are here).jina-embedding-b-en-v2: 137 million parameters.jina-embedding-l-en-v2: 435 million parameters.
Data & Parameters
Please checkout our technical blog.
Metrics
We compared the model against all-minilm-l6-v2/all-mpnet-base-v2 from sbert and text-embeddings-ada-002 from OpenAI:
Usage
You can use Jina Embedding models directly from transformers package:
!pip install transformers
from transformers import AutoModel
from numpy.linalg import norm
cos_sim = lambda a,b: (a @ b.T) / (norm(a)*norm(b))
model = AutoModel.from_pretrained('jinaai/jina-embedding-s-en-v2', trust_remote_code=True) # trust_remote_code is needed to use the encode method
embeddings = model.encode(['How is the weather today?', 'What is the current weather like today?'])
print(cos_sim(embeddings[0], embeddings[1]))
For long sequences, it's recommended to perform inference using Flash Attention. Using Flash Attention allows you to increase the batch size and throughput for long sequence length.
We include an experimental implementation for Flash Attention, shipped with the model.
Install the following triton version:
pip install triton==2.0.0.dev20221202.
Now run the same code above, but make sure to set the parameter with_flash to True when you load the model. You also have to use either fp16 or bf16:
from transformers import AutoModel
from numpy.linalg import norm
import torch
cos_sim = lambda a,b: (a @ b.T) / (norm(a)*norm(b))
model = AutoModel.from_pretrained('jinaai/jina-embedding-s-en-v2', trust_remote_code=True, with_flash=True, torch_dtype=torch.float16).cuda() # trust_remote_code is needed to use the encode method
embeddings = model.encode(['How is the weather today?', 'What is the current weather like today?'])
print(cos_sim(embeddings[0], embeddings[1]))
Fine-tuning
Please consider Finetuner.
Plans
The development of new multilingual models is currently underway. We will be targeting mainly the German and Spanish languages. The upcoming models will be called jina-embedding-s/b/l-de/es-v2.
Contact
Join our Discord community and chat with other community members about ideas.
Citation
If you find Jina Embeddings useful in your research, please cite the following paper:
@misc{günther2023jina,
title={Beyond the 512-Token Barrier: Training General-Purpose Text
Embeddings for Large Documents},
author={Michael Günther and Jackmin Ong and Isabelle Mohr and Alaeddine Abdessalem and Tanguy Abel and Mohammad Kalim Akram and Susana Guzman and Georgios Mastrapas and Saba Sturua and Bo Wang},
year={2023},
eprint={2307.11224},
archivePrefix={arXiv},
primaryClass={cs.CL}
}