Update model card slightly
#2
by
tomaarsen
HF staff
- opened
README.md
CHANGED
@@ -114,7 +114,7 @@ language:
|
|
114 |
# nomic-embed-text-v2-moe: Multilingual Mixture of Experts Text Embeddings
|
115 |
|
116 |
## Model Overview
|
117 |
-
nomic-embed-text-v2-moe is SoTA multilingual MoE text embedding model:
|
118 |
|
119 |
- **High Performance**: SoTA Multilingual performance compared to ~300M parameter models, competitive with models 2x in size
|
120 |
- **Multilinguality**: Supports ~100 languages and trained over 1.6B pairs
|
@@ -157,14 +157,15 @@ For best performance on GPU, please install
|
|
157 |
pip install torch transformers einops git+https://github.com/nomic-ai/megablocks.git
|
158 |
```
|
159 |
|
160 |
-
|
|
|
|
|
161 |
|
162 |
-
|
163 |
|
164 |
-
|
165 |
|
166 |
-
|
167 |
-
If using Transformers, **make sure to prepend the task instruction prefix**
|
168 |
|
169 |
```python
|
170 |
import torch
|
@@ -187,11 +188,17 @@ with torch.no_grad():
|
|
187 |
model_output = model(**encoded_input)
|
188 |
embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
|
189 |
embeddings = F.normalize(embeddings, p=2, dim=1)
|
|
|
|
|
|
|
|
|
|
|
|
|
190 |
```
|
191 |
|
192 |
-
|
193 |
|
194 |
-
With SentenceTransformers, you can specify the prompt_name
|
195 |
|
196 |
```python
|
197 |
from sentence_transformers import SentenceTransformer
|
@@ -199,6 +206,12 @@ from sentence_transformers import SentenceTransformer
|
|
199 |
model = SentenceTransformer("nomic-ai/nomic-embed-text-v2-moe", trust_remote_code=True)
|
200 |
sentences = ["Hello!", "¡Hola!"]
|
201 |
embeddings = model.encode(sentences, prompt_name="passage")
|
|
|
|
|
|
|
|
|
|
|
|
|
202 |
```
|
203 |
|
204 |
## Performance
|
@@ -221,7 +234,7 @@ nomic-embed-text-v2-moe performance on BEIR at 768 dimension and truncated to 25
|
|
221 |
## Limitations
|
222 |
- Performance may vary across different languages
|
223 |
- Resource requirements may be higher than traditional dense models due to MoE architecture
|
224 |
-
- Must
|
225 |
|
226 |
## Training Details
|
227 |
|
|
|
114 |
# nomic-embed-text-v2-moe: Multilingual Mixture of Experts Text Embeddings
|
115 |
|
116 |
## Model Overview
|
117 |
+
`nomic-embed-text-v2-moe` is SoTA multilingual MoE text embedding model:
|
118 |
|
119 |
- **High Performance**: SoTA Multilingual performance compared to ~300M parameter models, competitive with models 2x in size
|
120 |
- **Multilinguality**: Supports ~100 languages and trained over 1.6B pairs
|
|
|
157 |
pip install torch transformers einops git+https://github.com/nomic-ai/megablocks.git
|
158 |
```
|
159 |
|
160 |
+
> [!IMPORTANT]
|
161 |
+
> **Important!**
|
162 |
+
> The text prompt *must* include a *task instruction prefix*, instructing the model which task is being performed.
|
163 |
|
164 |
+
Please use `search_query: ` before your queries/questions, and `search_document: ` before your documents.
|
165 |
|
166 |
+
### Transformers
|
167 |
|
168 |
+
If using Transformers, **make sure to prepend the task instruction prefix**.
|
|
|
169 |
|
170 |
```python
|
171 |
import torch
|
|
|
188 |
model_output = model(**encoded_input)
|
189 |
embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
|
190 |
embeddings = F.normalize(embeddings, p=2, dim=1)
|
191 |
+
print(embeddings.shape)
|
192 |
+
# torch.Size([2, 768])
|
193 |
+
|
194 |
+
similarity = F.cosine_similarity(embeddings[0], embeddings[1])
|
195 |
+
print(similarity)
|
196 |
+
# tensor(0.9118)
|
197 |
```
|
198 |
|
199 |
+
### SentenceTransformers
|
200 |
|
201 |
+
With SentenceTransformers, you can specify the `prompt_name` as either `"query"` or `"passage"`, and the task instruction will be included automatically.
|
202 |
|
203 |
```python
|
204 |
from sentence_transformers import SentenceTransformer
|
|
|
206 |
model = SentenceTransformer("nomic-ai/nomic-embed-text-v2-moe", trust_remote_code=True)
|
207 |
sentences = ["Hello!", "¡Hola!"]
|
208 |
embeddings = model.encode(sentences, prompt_name="passage")
|
209 |
+
print(embeddings.shape)
|
210 |
+
# (2, 768)
|
211 |
+
|
212 |
+
similarity = model.similarity(embeddings[0], embeddings[1])
|
213 |
+
print(similarity)
|
214 |
+
# tensor([[0.9118]])
|
215 |
```
|
216 |
|
217 |
## Performance
|
|
|
234 |
## Limitations
|
235 |
- Performance may vary across different languages
|
236 |
- Resource requirements may be higher than traditional dense models due to MoE architecture
|
237 |
+
- Must use `trust_remote_code=True` when loading the model to use our custom architecture implementation
|
238 |
|
239 |
## Training Details
|
240 |
|