Files changed (1) hide show
  1. README.md +2707 -0
README.md ADDED
@@ -0,0 +1,2707 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: sentence-similarity
3
+ tags:
4
+ - finetuner
5
+ - mteb
6
+ - sentence-transformers
7
+ - feature-extraction
8
+ - sentence-similarity
9
+ - alibi
10
+ datasets:
11
+ - allenai/c4
12
+ language: en
13
+ license: apache-2.0
14
+ model-index:
15
+ - name: jina-embedding-b-en-v2
16
+ results:
17
+ - task:
18
+ type: Classification
19
+ dataset:
20
+ type: mteb/amazon_counterfactual
21
+ name: MTEB AmazonCounterfactualClassification (en)
22
+ config: en
23
+ split: test
24
+ revision: e8379541af4e31359cca9fbcf4b00f2671dba205
25
+ metrics:
26
+ - type: accuracy
27
+ value: 73.4179104477612
28
+ - type: ap
29
+ value: 35.798378234524705
30
+ - type: f1
31
+ value: 67.27708504551819
32
+ - task:
33
+ type: Classification
34
+ dataset:
35
+ type: mteb/amazon_polarity
36
+ name: MTEB AmazonPolarityClassification
37
+ config: default
38
+ split: test
39
+ revision: e2d317d38cd51312af73b3d32a06d1a08b442046
40
+ metrics:
41
+ - type: accuracy
42
+ value: 88.977575
43
+ - type: ap
44
+ value: 85.00359027707599
45
+ - type: f1
46
+ value: 88.9585285941142
47
+ - task:
48
+ type: Classification
49
+ dataset:
50
+ type: mteb/amazon_reviews_multi
51
+ name: MTEB AmazonReviewsClassification (en)
52
+ config: en
53
+ split: test
54
+ revision: 1399c76144fd37290681b995c656ef9b2e06e26d
55
+ metrics:
56
+ - type: accuracy
57
+ value: 44.455999999999996
58
+ - type: f1
59
+ value: 42.80615676169829
60
+ - task:
61
+ type: Retrieval
62
+ dataset:
63
+ type: arguana
64
+ name: MTEB ArguAna
65
+ config: default
66
+ split: test
67
+ revision: None
68
+ metrics:
69
+ - type: map_at_1
70
+ value: 18.919
71
+ - type: map_at_10
72
+ value: 33.272
73
+ - type: map_at_100
74
+ value: 34.669
75
+ - type: map_at_1000
76
+ value: 34.68
77
+ - type: map_at_3
78
+ value: 28.011000000000003
79
+ - type: map_at_5
80
+ value: 30.767
81
+ - type: mrr_at_1
82
+ value: 19.061
83
+ - type: mrr_at_10
84
+ value: 33.352
85
+ - type: mrr_at_100
86
+ value: 34.75
87
+ - type: mrr_at_1000
88
+ value: 34.760999999999996
89
+ - type: mrr_at_3
90
+ value: 28.07
91
+ - type: mrr_at_5
92
+ value: 30.848
93
+ - type: ndcg_at_1
94
+ value: 18.919
95
+ - type: ndcg_at_10
96
+ value: 42.138
97
+ - type: ndcg_at_100
98
+ value: 48.165
99
+ - type: ndcg_at_1000
100
+ value: 48.435
101
+ - type: ndcg_at_3
102
+ value: 31.041
103
+ - type: ndcg_at_5
104
+ value: 36.015
105
+ - type: precision_at_1
106
+ value: 18.919
107
+ - type: precision_at_10
108
+ value: 7.098
109
+ - type: precision_at_100
110
+ value: 0.9740000000000001
111
+ - type: precision_at_1000
112
+ value: 0.1
113
+ - type: precision_at_3
114
+ value: 13.276
115
+ - type: precision_at_5
116
+ value: 10.384
117
+ - type: recall_at_1
118
+ value: 18.919
119
+ - type: recall_at_10
120
+ value: 70.982
121
+ - type: recall_at_100
122
+ value: 97.44
123
+ - type: recall_at_1000
124
+ value: 99.502
125
+ - type: recall_at_3
126
+ value: 39.829
127
+ - type: recall_at_5
128
+ value: 51.92
129
+ - task:
130
+ type: Clustering
131
+ dataset:
132
+ type: mteb/arxiv-clustering-p2p
133
+ name: MTEB ArxivClusteringP2P
134
+ config: default
135
+ split: test
136
+ revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d
137
+ metrics:
138
+ - type: v_measure
139
+ value: 45.38238451470738
140
+ - task:
141
+ type: Clustering
142
+ dataset:
143
+ type: mteb/arxiv-clustering-s2s
144
+ name: MTEB ArxivClusteringS2S
145
+ config: default
146
+ split: test
147
+ revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53
148
+ metrics:
149
+ - type: v_measure
150
+ value: 37.12265635737745
151
+ - task:
152
+ type: Reranking
153
+ dataset:
154
+ type: mteb/askubuntudupquestions-reranking
155
+ name: MTEB AskUbuntuDupQuestions
156
+ config: default
157
+ split: test
158
+ revision: 2000358ca161889fa9c082cb41daa8dcfb161a54
159
+ metrics:
160
+ - type: map
161
+ value: 62.473921100678695
162
+ - type: mrr
163
+ value: 75.28195488721803
164
+ - task:
165
+ type: STS
166
+ dataset:
167
+ type: mteb/biosses-sts
168
+ name: MTEB BIOSSES
169
+ config: default
170
+ split: test
171
+ revision: d3fb88f8f02e40887cd149695127462bbcf29b4a
172
+ metrics:
173
+ - type: cos_sim_pearson
174
+ value: 84.46030780641742
175
+ - type: cos_sim_spearman
176
+ value: 83.29647627997147
177
+ - type: euclidean_pearson
178
+ value: 83.63127685751004
179
+ - type: euclidean_spearman
180
+ value: 83.29647627997147
181
+ - type: manhattan_pearson
182
+ value: 83.29505322210208
183
+ - type: manhattan_spearman
184
+ value: 82.8398393691656
185
+ - task:
186
+ type: Classification
187
+ dataset:
188
+ type: mteb/banking77
189
+ name: MTEB Banking77Classification
190
+ config: default
191
+ split: test
192
+ revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
193
+ metrics:
194
+ - type: accuracy
195
+ value: 83.94480519480521
196
+ - type: f1
197
+ value: 83.26406143364741
198
+ - task:
199
+ type: Clustering
200
+ dataset:
201
+ type: mteb/biorxiv-clustering-p2p
202
+ name: MTEB BiorxivClusteringP2P
203
+ config: default
204
+ split: test
205
+ revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40
206
+ metrics:
207
+ - type: v_measure
208
+ value: 37.15926312173139
209
+ - task:
210
+ type: Clustering
211
+ dataset:
212
+ type: mteb/biorxiv-clustering-s2s
213
+ name: MTEB BiorxivClusteringS2S
214
+ config: default
215
+ split: test
216
+ revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908
217
+ metrics:
218
+ - type: v_measure
219
+ value: 31.20469085642121
220
+ - task:
221
+ type: Retrieval
222
+ dataset:
223
+ type: BeIR/cqadupstack
224
+ name: MTEB CQADupstackAndroidRetrieval
225
+ config: default
226
+ split: test
227
+ revision: None
228
+ metrics:
229
+ - type: map_at_1
230
+ value: 28.462
231
+ - type: map_at_10
232
+ value: 39.834
233
+ - type: map_at_100
234
+ value: 41.329
235
+ - type: map_at_1000
236
+ value: 41.465
237
+ - type: map_at_3
238
+ value: 36.586999999999996
239
+ - type: map_at_5
240
+ value: 38.239000000000004
241
+ - type: mrr_at_1
242
+ value: 34.335
243
+ - type: mrr_at_10
244
+ value: 45.493
245
+ - type: mrr_at_100
246
+ value: 46.323
247
+ - type: mrr_at_1000
248
+ value: 46.37
249
+ - type: mrr_at_3
250
+ value: 42.870999999999995
251
+ - type: mrr_at_5
252
+ value: 44.502
253
+ - type: ndcg_at_1
254
+ value: 34.335
255
+ - type: ndcg_at_10
256
+ value: 46.434
257
+ - type: ndcg_at_100
258
+ value: 52.013
259
+ - type: ndcg_at_1000
260
+ value: 54.079
261
+ - type: ndcg_at_3
262
+ value: 41.408
263
+ - type: ndcg_at_5
264
+ value: 43.562
265
+ - type: precision_at_1
266
+ value: 34.335
267
+ - type: precision_at_10
268
+ value: 8.913
269
+ - type: precision_at_100
270
+ value: 1.439
271
+ - type: precision_at_1000
272
+ value: 0.197
273
+ - type: precision_at_3
274
+ value: 20.029
275
+ - type: precision_at_5
276
+ value: 14.335
277
+ - type: recall_at_1
278
+ value: 28.462
279
+ - type: recall_at_10
280
+ value: 59.574000000000005
281
+ - type: recall_at_100
282
+ value: 82.631
283
+ - type: recall_at_1000
284
+ value: 95.45700000000001
285
+ - type: recall_at_3
286
+ value: 45.381
287
+ - type: recall_at_5
288
+ value: 51.18000000000001
289
+ - task:
290
+ type: Retrieval
291
+ dataset:
292
+ type: BeIR/cqadupstack
293
+ name: MTEB CQADupstackEnglishRetrieval
294
+ config: default
295
+ split: test
296
+ revision: None
297
+ metrics:
298
+ - type: map_at_1
299
+ value: 27.245
300
+ - type: map_at_10
301
+ value: 37.156
302
+ - type: map_at_100
303
+ value: 38.464999999999996
304
+ - type: map_at_1000
305
+ value: 38.607
306
+ - type: map_at_3
307
+ value: 34.613
308
+ - type: map_at_5
309
+ value: 35.924
310
+ - type: mrr_at_1
311
+ value: 34.777
312
+ - type: mrr_at_10
313
+ value: 43.425000000000004
314
+ - type: mrr_at_100
315
+ value: 44.163000000000004
316
+ - type: mrr_at_1000
317
+ value: 44.211
318
+ - type: mrr_at_3
319
+ value: 41.391
320
+ - type: mrr_at_5
321
+ value: 42.461
322
+ - type: ndcg_at_1
323
+ value: 34.777
324
+ - type: ndcg_at_10
325
+ value: 42.807
326
+ - type: ndcg_at_100
327
+ value: 47.629
328
+ - type: ndcg_at_1000
329
+ value: 49.84
330
+ - type: ndcg_at_3
331
+ value: 39.28
332
+ - type: ndcg_at_5
333
+ value: 40.671
334
+ - type: precision_at_1
335
+ value: 34.777
336
+ - type: precision_at_10
337
+ value: 8.134
338
+ - type: precision_at_100
339
+ value: 1.3599999999999999
340
+ - type: precision_at_1000
341
+ value: 0.186
342
+ - type: precision_at_3
343
+ value: 19.320999999999998
344
+ - type: precision_at_5
345
+ value: 13.286999999999999
346
+ - type: recall_at_1
347
+ value: 27.245
348
+ - type: recall_at_10
349
+ value: 52.491
350
+ - type: recall_at_100
351
+ value: 73.065
352
+ - type: recall_at_1000
353
+ value: 86.931
354
+ - type: recall_at_3
355
+ value: 41.257
356
+ - type: recall_at_5
357
+ value: 45.811
358
+ - task:
359
+ type: Retrieval
360
+ dataset:
361
+ type: BeIR/cqadupstack
362
+ name: MTEB CQADupstackGamingRetrieval
363
+ config: default
364
+ split: test
365
+ revision: None
366
+ metrics:
367
+ - type: map_at_1
368
+ value: 37.088
369
+ - type: map_at_10
370
+ value: 49.003
371
+ - type: map_at_100
372
+ value: 50.017999999999994
373
+ - type: map_at_1000
374
+ value: 50.07899999999999
375
+ - type: map_at_3
376
+ value: 45.846
377
+ - type: map_at_5
378
+ value: 47.733
379
+ - type: mrr_at_1
380
+ value: 42.193999999999996
381
+ - type: mrr_at_10
382
+ value: 52.522999999999996
383
+ - type: mrr_at_100
384
+ value: 53.177
385
+ - type: mrr_at_1000
386
+ value: 53.205999999999996
387
+ - type: mrr_at_3
388
+ value: 49.916
389
+ - type: mrr_at_5
390
+ value: 51.50900000000001
391
+ - type: ndcg_at_1
392
+ value: 42.193999999999996
393
+ - type: ndcg_at_10
394
+ value: 54.99699999999999
395
+ - type: ndcg_at_100
396
+ value: 59.058
397
+ - type: ndcg_at_1000
398
+ value: 60.355000000000004
399
+ - type: ndcg_at_3
400
+ value: 49.515
401
+ - type: ndcg_at_5
402
+ value: 52.412000000000006
403
+ - type: precision_at_1
404
+ value: 42.193999999999996
405
+ - type: precision_at_10
406
+ value: 8.84
407
+ - type: precision_at_100
408
+ value: 1.1820000000000002
409
+ - type: precision_at_1000
410
+ value: 0.134
411
+ - type: precision_at_3
412
+ value: 21.944
413
+ - type: precision_at_5
414
+ value: 15.197
415
+ - type: recall_at_1
416
+ value: 37.088
417
+ - type: recall_at_10
418
+ value: 69.13
419
+ - type: recall_at_100
420
+ value: 86.612
421
+ - type: recall_at_1000
422
+ value: 95.946
423
+ - type: recall_at_3
424
+ value: 54.76
425
+ - type: recall_at_5
426
+ value: 61.76199999999999
427
+ - task:
428
+ type: Retrieval
429
+ dataset:
430
+ type: BeIR/cqadupstack
431
+ name: MTEB CQADupstackGisRetrieval
432
+ config: default
433
+ split: test
434
+ revision: None
435
+ metrics:
436
+ - type: map_at_1
437
+ value: 21.816
438
+ - type: map_at_10
439
+ value: 30.630000000000003
440
+ - type: map_at_100
441
+ value: 31.641000000000002
442
+ - type: map_at_1000
443
+ value: 31.730999999999998
444
+ - type: map_at_3
445
+ value: 28.153
446
+ - type: map_at_5
447
+ value: 29.433
448
+ - type: mrr_at_1
449
+ value: 23.842
450
+ - type: mrr_at_10
451
+ value: 32.432
452
+ - type: mrr_at_100
453
+ value: 33.354
454
+ - type: mrr_at_1000
455
+ value: 33.421
456
+ - type: mrr_at_3
457
+ value: 30.131999999999998
458
+ - type: mrr_at_5
459
+ value: 31.358000000000004
460
+ - type: ndcg_at_1
461
+ value: 23.842
462
+ - type: ndcg_at_10
463
+ value: 35.626000000000005
464
+ - type: ndcg_at_100
465
+ value: 40.855999999999995
466
+ - type: ndcg_at_1000
467
+ value: 43.111
468
+ - type: ndcg_at_3
469
+ value: 30.712
470
+ - type: ndcg_at_5
471
+ value: 32.912
472
+ - type: precision_at_1
473
+ value: 23.842
474
+ - type: precision_at_10
475
+ value: 5.627
476
+ - type: precision_at_100
477
+ value: 0.873
478
+ - type: precision_at_1000
479
+ value: 0.11100000000000002
480
+ - type: precision_at_3
481
+ value: 13.333
482
+ - type: precision_at_5
483
+ value: 9.266
484
+ - type: recall_at_1
485
+ value: 21.816
486
+ - type: recall_at_10
487
+ value: 49.370000000000005
488
+ - type: recall_at_100
489
+ value: 73.855
490
+ - type: recall_at_1000
491
+ value: 90.67399999999999
492
+ - type: recall_at_3
493
+ value: 35.85
494
+ - type: recall_at_5
495
+ value: 41.282000000000004
496
+ - task:
497
+ type: Retrieval
498
+ dataset:
499
+ type: BeIR/cqadupstack
500
+ name: MTEB CQADupstackMathematicaRetrieval
501
+ config: default
502
+ split: test
503
+ revision: None
504
+ metrics:
505
+ - type: map_at_1
506
+ value: 14.402000000000001
507
+ - type: map_at_10
508
+ value: 21.401999999999997
509
+ - type: map_at_100
510
+ value: 22.425
511
+ - type: map_at_1000
512
+ value: 22.561
513
+ - type: map_at_3
514
+ value: 19.238
515
+ - type: map_at_5
516
+ value: 20.213
517
+ - type: mrr_at_1
518
+ value: 17.91
519
+ - type: mrr_at_10
520
+ value: 25.629999999999995
521
+ - type: mrr_at_100
522
+ value: 26.529999999999998
523
+ - type: mrr_at_1000
524
+ value: 26.616
525
+ - type: mrr_at_3
526
+ value: 23.362
527
+ - type: mrr_at_5
528
+ value: 24.438
529
+ - type: ndcg_at_1
530
+ value: 17.91
531
+ - type: ndcg_at_10
532
+ value: 26.161
533
+ - type: ndcg_at_100
534
+ value: 31.474000000000004
535
+ - type: ndcg_at_1000
536
+ value: 34.802
537
+ - type: ndcg_at_3
538
+ value: 21.965
539
+ - type: ndcg_at_5
540
+ value: 23.511000000000003
541
+ - type: precision_at_1
542
+ value: 17.91
543
+ - type: precision_at_10
544
+ value: 4.8629999999999995
545
+ - type: precision_at_100
546
+ value: 0.869
547
+ - type: precision_at_1000
548
+ value: 0.129
549
+ - type: precision_at_3
550
+ value: 10.655000000000001
551
+ - type: precision_at_5
552
+ value: 7.5120000000000005
553
+ - type: recall_at_1
554
+ value: 14.402000000000001
555
+ - type: recall_at_10
556
+ value: 36.760999999999996
557
+ - type: recall_at_100
558
+ value: 60.549
559
+ - type: recall_at_1000
560
+ value: 84.414
561
+ - type: recall_at_3
562
+ value: 25.130000000000003
563
+ - type: recall_at_5
564
+ value: 29.079
565
+ - task:
566
+ type: Retrieval
567
+ dataset:
568
+ type: BeIR/cqadupstack
569
+ name: MTEB CQADupstackPhysicsRetrieval
570
+ config: default
571
+ split: test
572
+ revision: None
573
+ metrics:
574
+ - type: map_at_1
575
+ value: 26.176
576
+ - type: map_at_10
577
+ value: 35.789
578
+ - type: map_at_100
579
+ value: 37.092000000000006
580
+ - type: map_at_1000
581
+ value: 37.206
582
+ - type: map_at_3
583
+ value: 33.207
584
+ - type: map_at_5
585
+ value: 34.436
586
+ - type: mrr_at_1
587
+ value: 31.569000000000003
588
+ - type: mrr_at_10
589
+ value: 41.219
590
+ - type: mrr_at_100
591
+ value: 42.016999999999996
592
+ - type: mrr_at_1000
593
+ value: 42.065000000000005
594
+ - type: mrr_at_3
595
+ value: 39.012
596
+ - type: mrr_at_5
597
+ value: 40.22
598
+ - type: ndcg_at_1
599
+ value: 31.569000000000003
600
+ - type: ndcg_at_10
601
+ value: 41.515
602
+ - type: ndcg_at_100
603
+ value: 47.125
604
+ - type: ndcg_at_1000
605
+ value: 49.314
606
+ - type: ndcg_at_3
607
+ value: 37.201
608
+ - type: ndcg_at_5
609
+ value: 38.906
610
+ - type: precision_at_1
611
+ value: 31.569000000000003
612
+ - type: precision_at_10
613
+ value: 7.517
614
+ - type: precision_at_100
615
+ value: 1.225
616
+ - type: precision_at_1000
617
+ value: 0.161
618
+ - type: precision_at_3
619
+ value: 17.485
620
+ - type: precision_at_5
621
+ value: 12.089
622
+ - type: recall_at_1
623
+ value: 26.176
624
+ - type: recall_at_10
625
+ value: 53.076
626
+ - type: recall_at_100
627
+ value: 77.049
628
+ - type: recall_at_1000
629
+ value: 91.51
630
+ - type: recall_at_3
631
+ value: 40.82
632
+ - type: recall_at_5
633
+ value: 45.479
634
+ - task:
635
+ type: Retrieval
636
+ dataset:
637
+ type: BeIR/cqadupstack
638
+ name: MTEB CQADupstackProgrammersRetrieval
639
+ config: default
640
+ split: test
641
+ revision: None
642
+ metrics:
643
+ - type: map_at_1
644
+ value: 22.675
645
+ - type: map_at_10
646
+ value: 31.752999999999997
647
+ - type: map_at_100
648
+ value: 33.19
649
+ - type: map_at_1000
650
+ value: 33.303
651
+ - type: map_at_3
652
+ value: 28.89
653
+ - type: map_at_5
654
+ value: 30.451
655
+ - type: mrr_at_1
656
+ value: 27.854
657
+ - type: mrr_at_10
658
+ value: 36.736999999999995
659
+ - type: mrr_at_100
660
+ value: 37.783
661
+ - type: mrr_at_1000
662
+ value: 37.836
663
+ - type: mrr_at_3
664
+ value: 34.266000000000005
665
+ - type: mrr_at_5
666
+ value: 35.577999999999996
667
+ - type: ndcg_at_1
668
+ value: 27.854
669
+ - type: ndcg_at_10
670
+ value: 37.391999999999996
671
+ - type: ndcg_at_100
672
+ value: 43.682
673
+ - type: ndcg_at_1000
674
+ value: 46.005
675
+ - type: ndcg_at_3
676
+ value: 32.66
677
+ - type: ndcg_at_5
678
+ value: 34.73
679
+ - type: precision_at_1
680
+ value: 27.854
681
+ - type: precision_at_10
682
+ value: 6.963
683
+ - type: precision_at_100
684
+ value: 1.184
685
+ - type: precision_at_1000
686
+ value: 0.159
687
+ - type: precision_at_3
688
+ value: 15.715000000000002
689
+ - type: precision_at_5
690
+ value: 11.256
691
+ - type: recall_at_1
692
+ value: 22.675
693
+ - type: recall_at_10
694
+ value: 49.15
695
+ - type: recall_at_100
696
+ value: 76.542
697
+ - type: recall_at_1000
698
+ value: 92.19000000000001
699
+ - type: recall_at_3
700
+ value: 35.607
701
+ - type: recall_at_5
702
+ value: 41.288000000000004
703
+ - task:
704
+ type: Retrieval
705
+ dataset:
706
+ type: BeIR/cqadupstack
707
+ name: MTEB CQADupstackRetrieval
708
+ config: default
709
+ split: test
710
+ revision: None
711
+ metrics:
712
+ - type: map_at_1
713
+ value: 23.214499999999997
714
+ - type: map_at_10
715
+ value: 31.979833333333335
716
+ - type: map_at_100
717
+ value: 33.20666666666666
718
+ - type: map_at_1000
719
+ value: 33.328583333333334
720
+ - type: map_at_3
721
+ value: 29.341416666666664
722
+ - type: map_at_5
723
+ value: 30.718083333333336
724
+ - type: mrr_at_1
725
+ value: 27.328583333333338
726
+ - type: mrr_at_10
727
+ value: 35.88433333333333
728
+ - type: mrr_at_100
729
+ value: 36.80075000000001
730
+ - type: mrr_at_1000
731
+ value: 36.86175
732
+ - type: mrr_at_3
733
+ value: 33.51625
734
+ - type: mrr_at_5
735
+ value: 34.821416666666664
736
+ - type: ndcg_at_1
737
+ value: 27.328583333333338
738
+ - type: ndcg_at_10
739
+ value: 37.24475
740
+ - type: ndcg_at_100
741
+ value: 42.63825
742
+ - type: ndcg_at_1000
743
+ value: 45.08266666666667
744
+ - type: ndcg_at_3
745
+ value: 32.61783333333334
746
+ - type: ndcg_at_5
747
+ value: 34.631249999999994
748
+ - type: precision_at_1
749
+ value: 27.328583333333338
750
+ - type: precision_at_10
751
+ value: 6.5873333333333335
752
+ - type: precision_at_100
753
+ value: 1.094916666666667
754
+ - type: precision_at_1000
755
+ value: 0.15091666666666664
756
+ - type: precision_at_3
757
+ value: 15.073499999999997
758
+ - type: precision_at_5
759
+ value: 10.651916666666667
760
+ - type: recall_at_1
761
+ value: 23.214499999999997
762
+ - type: recall_at_10
763
+ value: 49.010250000000006
764
+ - type: recall_at_100
765
+ value: 72.70374999999999
766
+ - type: recall_at_1000
767
+ value: 89.66041666666666
768
+ - type: recall_at_3
769
+ value: 36.06008333333334
770
+ - type: recall_at_5
771
+ value: 41.289166666666674
772
+ - task:
773
+ type: Retrieval
774
+ dataset:
775
+ type: BeIR/cqadupstack
776
+ name: MTEB CQADupstackStatsRetrieval
777
+ config: default
778
+ split: test
779
+ revision: None
780
+ metrics:
781
+ - type: map_at_1
782
+ value: 23.497
783
+ - type: map_at_10
784
+ value: 29.176000000000002
785
+ - type: map_at_100
786
+ value: 30.218
787
+ - type: map_at_1000
788
+ value: 30.317
789
+ - type: map_at_3
790
+ value: 27.072000000000003
791
+ - type: map_at_5
792
+ value: 28.162
793
+ - type: mrr_at_1
794
+ value: 25.919999999999998
795
+ - type: mrr_at_10
796
+ value: 31.513
797
+ - type: mrr_at_100
798
+ value: 32.434000000000005
799
+ - type: mrr_at_1000
800
+ value: 32.507000000000005
801
+ - type: mrr_at_3
802
+ value: 29.576
803
+ - type: mrr_at_5
804
+ value: 30.45
805
+ - type: ndcg_at_1
806
+ value: 25.919999999999998
807
+ - type: ndcg_at_10
808
+ value: 32.958999999999996
809
+ - type: ndcg_at_100
810
+ value: 37.937
811
+ - type: ndcg_at_1000
812
+ value: 40.455000000000005
813
+ - type: ndcg_at_3
814
+ value: 28.969
815
+ - type: ndcg_at_5
816
+ value: 30.552
817
+ - type: precision_at_1
818
+ value: 25.919999999999998
819
+ - type: precision_at_10
820
+ value: 5.106999999999999
821
+ - type: precision_at_100
822
+ value: 0.8170000000000001
823
+ - type: precision_at_1000
824
+ value: 0.11100000000000002
825
+ - type: precision_at_3
826
+ value: 12.117
827
+ - type: precision_at_5
828
+ value: 8.373999999999999
829
+ - type: recall_at_1
830
+ value: 23.497
831
+ - type: recall_at_10
832
+ value: 42.506
833
+ - type: recall_at_100
834
+ value: 65.048
835
+ - type: recall_at_1000
836
+ value: 83.545
837
+ - type: recall_at_3
838
+ value: 31.078
839
+ - type: recall_at_5
840
+ value: 35.018
841
+ - task:
842
+ type: Retrieval
843
+ dataset:
844
+ type: BeIR/cqadupstack
845
+ name: MTEB CQADupstackTexRetrieval
846
+ config: default
847
+ split: test
848
+ revision: None
849
+ metrics:
850
+ - type: map_at_1
851
+ value: 15.267
852
+ - type: map_at_10
853
+ value: 22.292
854
+ - type: map_at_100
855
+ value: 23.412
856
+ - type: map_at_1000
857
+ value: 23.543
858
+ - type: map_at_3
859
+ value: 19.993
860
+ - type: map_at_5
861
+ value: 21.256
862
+ - type: mrr_at_1
863
+ value: 18.445
864
+ - type: mrr_at_10
865
+ value: 25.698999999999998
866
+ - type: mrr_at_100
867
+ value: 26.682
868
+ - type: mrr_at_1000
869
+ value: 26.764
870
+ - type: mrr_at_3
871
+ value: 23.446
872
+ - type: mrr_at_5
873
+ value: 24.757
874
+ - type: ndcg_at_1
875
+ value: 18.445
876
+ - type: ndcg_at_10
877
+ value: 26.833000000000002
878
+ - type: ndcg_at_100
879
+ value: 32.151999999999994
880
+ - type: ndcg_at_1000
881
+ value: 35.235
882
+ - type: ndcg_at_3
883
+ value: 22.597
884
+ - type: ndcg_at_5
885
+ value: 24.585
886
+ - type: precision_at_1
887
+ value: 18.445
888
+ - type: precision_at_10
889
+ value: 4.942
890
+ - type: precision_at_100
891
+ value: 0.894
892
+ - type: precision_at_1000
893
+ value: 0.135
894
+ - type: precision_at_3
895
+ value: 10.735999999999999
896
+ - type: precision_at_5
897
+ value: 7.915
898
+ - type: recall_at_1
899
+ value: 15.267
900
+ - type: recall_at_10
901
+ value: 37.198
902
+ - type: recall_at_100
903
+ value: 60.748999999999995
904
+ - type: recall_at_1000
905
+ value: 82.72699999999999
906
+ - type: recall_at_3
907
+ value: 25.419000000000004
908
+ - type: recall_at_5
909
+ value: 30.416999999999998
910
+ - task:
911
+ type: Retrieval
912
+ dataset:
913
+ type: BeIR/cqadupstack
914
+ name: MTEB CQADupstackUnixRetrieval
915
+ config: default
916
+ split: test
917
+ revision: None
918
+ metrics:
919
+ - type: map_at_1
920
+ value: 22.839000000000002
921
+ - type: map_at_10
922
+ value: 31.287
923
+ - type: map_at_100
924
+ value: 32.474
925
+ - type: map_at_1000
926
+ value: 32.586
927
+ - type: map_at_3
928
+ value: 28.735
929
+ - type: map_at_5
930
+ value: 30.11
931
+ - type: mrr_at_1
932
+ value: 26.959
933
+ - type: mrr_at_10
934
+ value: 34.943000000000005
935
+ - type: mrr_at_100
936
+ value: 35.957
937
+ - type: mrr_at_1000
938
+ value: 36.022
939
+ - type: mrr_at_3
940
+ value: 32.572
941
+ - type: mrr_at_5
942
+ value: 33.952
943
+ - type: ndcg_at_1
944
+ value: 26.959
945
+ - type: ndcg_at_10
946
+ value: 36.252
947
+ - type: ndcg_at_100
948
+ value: 41.915
949
+ - type: ndcg_at_1000
950
+ value: 44.461
951
+ - type: ndcg_at_3
952
+ value: 31.532
953
+ - type: ndcg_at_5
954
+ value: 33.674
955
+ - type: precision_at_1
956
+ value: 26.959
957
+ - type: precision_at_10
958
+ value: 6.166
959
+ - type: precision_at_100
960
+ value: 1.01
961
+ - type: precision_at_1000
962
+ value: 0.134
963
+ - type: precision_at_3
964
+ value: 14.302999999999999
965
+ - type: precision_at_5
966
+ value: 10.131
967
+ - type: recall_at_1
968
+ value: 22.839000000000002
969
+ - type: recall_at_10
970
+ value: 47.796
971
+ - type: recall_at_100
972
+ value: 72.68
973
+ - type: recall_at_1000
974
+ value: 90.556
975
+ - type: recall_at_3
976
+ value: 34.955000000000005
977
+ - type: recall_at_5
978
+ value: 40.293
979
+ - task:
980
+ type: Retrieval
981
+ dataset:
982
+ type: BeIR/cqadupstack
983
+ name: MTEB CQADupstackWebmastersRetrieval
984
+ config: default
985
+ split: test
986
+ revision: None
987
+ metrics:
988
+ - type: map_at_1
989
+ value: 21.676000000000002
990
+ - type: map_at_10
991
+ value: 30.742000000000004
992
+ - type: map_at_100
993
+ value: 32.332
994
+ - type: map_at_1000
995
+ value: 32.548
996
+ - type: map_at_3
997
+ value: 27.560000000000002
998
+ - type: map_at_5
999
+ value: 29.331000000000003
1000
+ - type: mrr_at_1
1001
+ value: 25.099
1002
+ - type: mrr_at_10
1003
+ value: 34.538999999999994
1004
+ - type: mrr_at_100
1005
+ value: 35.629
1006
+ - type: mrr_at_1000
1007
+ value: 35.687000000000005
1008
+ - type: mrr_at_3
1009
+ value: 31.621
1010
+ - type: mrr_at_5
1011
+ value: 33.419
1012
+ - type: ndcg_at_1
1013
+ value: 25.099
1014
+ - type: ndcg_at_10
1015
+ value: 36.741
1016
+ - type: ndcg_at_100
1017
+ value: 42.964
1018
+ - type: ndcg_at_1000
1019
+ value: 45.754
1020
+ - type: ndcg_at_3
1021
+ value: 31.356
1022
+ - type: ndcg_at_5
1023
+ value: 33.934999999999995
1024
+ - type: precision_at_1
1025
+ value: 25.099
1026
+ - type: precision_at_10
1027
+ value: 7.115
1028
+ - type: precision_at_100
1029
+ value: 1.46
1030
+ - type: precision_at_1000
1031
+ value: 0.23800000000000002
1032
+ - type: precision_at_3
1033
+ value: 14.954
1034
+ - type: precision_at_5
1035
+ value: 11.067
1036
+ - type: recall_at_1
1037
+ value: 21.676000000000002
1038
+ - type: recall_at_10
1039
+ value: 49.546
1040
+ - type: recall_at_100
1041
+ value: 76.544
1042
+ - type: recall_at_1000
1043
+ value: 94.39999999999999
1044
+ - type: recall_at_3
1045
+ value: 34.67
1046
+ - type: recall_at_5
1047
+ value: 41.528999999999996
1048
+ - task:
1049
+ type: Retrieval
1050
+ dataset:
1051
+ type: BeIR/cqadupstack
1052
+ name: MTEB CQADupstackWordpressRetrieval
1053
+ config: default
1054
+ split: test
1055
+ revision: None
1056
+ metrics:
1057
+ - type: map_at_1
1058
+ value: 17.431
1059
+ - type: map_at_10
1060
+ value: 24.694
1061
+ - type: map_at_100
1062
+ value: 25.884
1063
+ - type: map_at_1000
1064
+ value: 25.996999999999996
1065
+ - type: map_at_3
1066
+ value: 22.203
1067
+ - type: map_at_5
1068
+ value: 23.329
1069
+ - type: mrr_at_1
1070
+ value: 19.039
1071
+ - type: mrr_at_10
1072
+ value: 26.459
1073
+ - type: mrr_at_100
1074
+ value: 27.560000000000002
1075
+ - type: mrr_at_1000
1076
+ value: 27.636
1077
+ - type: mrr_at_3
1078
+ value: 24.03
1079
+ - type: mrr_at_5
1080
+ value: 25.213
1081
+ - type: ndcg_at_1
1082
+ value: 19.039
1083
+ - type: ndcg_at_10
1084
+ value: 29.220000000000002
1085
+ - type: ndcg_at_100
1086
+ value: 34.854
1087
+ - type: ndcg_at_1000
1088
+ value: 37.580999999999996
1089
+ - type: ndcg_at_3
1090
+ value: 24.218999999999998
1091
+ - type: ndcg_at_5
1092
+ value: 26.125
1093
+ - type: precision_at_1
1094
+ value: 19.039
1095
+ - type: precision_at_10
1096
+ value: 4.861
1097
+ - type: precision_at_100
1098
+ value: 0.826
1099
+ - type: precision_at_1000
1100
+ value: 0.116
1101
+ - type: precision_at_3
1102
+ value: 10.290000000000001
1103
+ - type: precision_at_5
1104
+ value: 7.394
1105
+ - type: recall_at_1
1106
+ value: 17.431
1107
+ - type: recall_at_10
1108
+ value: 41.525
1109
+ - type: recall_at_100
1110
+ value: 67.121
1111
+ - type: recall_at_1000
1112
+ value: 87.575
1113
+ - type: recall_at_3
1114
+ value: 27.794
1115
+ - type: recall_at_5
1116
+ value: 32.332
1117
+ - task:
1118
+ type: Retrieval
1119
+ dataset:
1120
+ type: climate-fever
1121
+ name: MTEB ClimateFEVER
1122
+ config: default
1123
+ split: test
1124
+ revision: None
1125
+ metrics:
1126
+ - type: map_at_1
1127
+ value: 10.767
1128
+ - type: map_at_10
1129
+ value: 17.456
1130
+ - type: map_at_100
1131
+ value: 19.097
1132
+ - type: map_at_1000
1133
+ value: 19.272
1134
+ - type: map_at_3
1135
+ value: 14.530000000000001
1136
+ - type: map_at_5
1137
+ value: 15.943999999999999
1138
+ - type: mrr_at_1
1139
+ value: 23.583000000000002
1140
+ - type: mrr_at_10
1141
+ value: 33.391
1142
+ - type: mrr_at_100
1143
+ value: 34.43
1144
+ - type: mrr_at_1000
1145
+ value: 34.479
1146
+ - type: mrr_at_3
1147
+ value: 30.239
1148
+ - type: mrr_at_5
1149
+ value: 31.923000000000002
1150
+ - type: ndcg_at_1
1151
+ value: 23.583000000000002
1152
+ - type: ndcg_at_10
1153
+ value: 24.84
1154
+ - type: ndcg_at_100
1155
+ value: 31.749
1156
+ - type: ndcg_at_1000
1157
+ value: 35.161
1158
+ - type: ndcg_at_3
1159
+ value: 19.906
1160
+ - type: ndcg_at_5
1161
+ value: 21.543
1162
+ - type: precision_at_1
1163
+ value: 23.583000000000002
1164
+ - type: precision_at_10
1165
+ value: 7.739
1166
+ - type: precision_at_100
1167
+ value: 1.5110000000000001
1168
+ - type: precision_at_1000
1169
+ value: 0.215
1170
+ - type: precision_at_3
1171
+ value: 14.506
1172
+ - type: precision_at_5
1173
+ value: 11.179
1174
+ - type: recall_at_1
1175
+ value: 10.767
1176
+ - type: recall_at_10
1177
+ value: 30.270000000000003
1178
+ - type: recall_at_100
1179
+ value: 54.467
1180
+ - type: recall_at_1000
1181
+ value: 73.71799999999999
1182
+ - type: recall_at_3
1183
+ value: 18.251
1184
+ - type: recall_at_5
1185
+ value: 22.831000000000003
1186
+ - task:
1187
+ type: Retrieval
1188
+ dataset:
1189
+ type: dbpedia-entity
1190
+ name: MTEB DBPedia
1191
+ config: default
1192
+ split: test
1193
+ revision: None
1194
+ metrics:
1195
+ - type: map_at_1
1196
+ value: 6.493
1197
+ - type: map_at_10
1198
+ value: 15.290999999999999
1199
+ - type: map_at_100
1200
+ value: 21.523999999999997
1201
+ - type: map_at_1000
1202
+ value: 22.980999999999998
1203
+ - type: map_at_3
1204
+ value: 11.015
1205
+ - type: map_at_5
1206
+ value: 12.631
1207
+ - type: mrr_at_1
1208
+ value: 55.50000000000001
1209
+ - type: mrr_at_10
1210
+ value: 65.068
1211
+ - type: mrr_at_100
1212
+ value: 65.608
1213
+ - type: mrr_at_1000
1214
+ value: 65.622
1215
+ - type: mrr_at_3
1216
+ value: 62.625
1217
+ - type: mrr_at_5
1218
+ value: 64.2
1219
+ - type: ndcg_at_1
1220
+ value: 44.875
1221
+ - type: ndcg_at_10
1222
+ value: 35.046
1223
+ - type: ndcg_at_100
1224
+ value: 38.662
1225
+ - type: ndcg_at_1000
1226
+ value: 45.916000000000004
1227
+ - type: ndcg_at_3
1228
+ value: 38.888
1229
+ - type: ndcg_at_5
1230
+ value: 36.411
1231
+ - type: precision_at_1
1232
+ value: 55.50000000000001
1233
+ - type: precision_at_10
1234
+ value: 28.175
1235
+ - type: precision_at_100
1236
+ value: 8.938
1237
+ - type: precision_at_1000
1238
+ value: 1.894
1239
+ - type: precision_at_3
1240
+ value: 41.917
1241
+ - type: precision_at_5
1242
+ value: 34.949999999999996
1243
+ - type: recall_at_1
1244
+ value: 6.493
1245
+ - type: recall_at_10
1246
+ value: 20.992
1247
+ - type: recall_at_100
1248
+ value: 44.138
1249
+ - type: recall_at_1000
1250
+ value: 67.181
1251
+ - type: recall_at_3
1252
+ value: 12.546
1253
+ - type: recall_at_5
1254
+ value: 15.552
1255
+ - task:
1256
+ type: Classification
1257
+ dataset:
1258
+ type: mteb/emotion
1259
+ name: MTEB EmotionClassification
1260
+ config: default
1261
+ split: test
1262
+ revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37
1263
+ metrics:
1264
+ - type: accuracy
1265
+ value: 45.955
1266
+ - type: f1
1267
+ value: 40.97084067876041
1268
+ - task:
1269
+ type: Retrieval
1270
+ dataset:
1271
+ type: fever
1272
+ name: MTEB FEVER
1273
+ config: default
1274
+ split: test
1275
+ revision: None
1276
+ metrics:
1277
+ - type: map_at_1
1278
+ value: 43.765
1279
+ - type: map_at_10
1280
+ value: 56.566
1281
+ - type: map_at_100
1282
+ value: 57.154
1283
+ - type: map_at_1000
1284
+ value: 57.181000000000004
1285
+ - type: map_at_3
1286
+ value: 53.637
1287
+ - type: map_at_5
1288
+ value: 55.457
1289
+ - type: mrr_at_1
1290
+ value: 47.03
1291
+ - type: mrr_at_10
1292
+ value: 59.938
1293
+ - type: mrr_at_100
1294
+ value: 60.44500000000001
1295
+ - type: mrr_at_1000
1296
+ value: 60.458999999999996
1297
+ - type: mrr_at_3
1298
+ value: 57.141
1299
+ - type: mrr_at_5
1300
+ value: 58.862
1301
+ - type: ndcg_at_1
1302
+ value: 47.03
1303
+ - type: ndcg_at_10
1304
+ value: 63.227
1305
+ - type: ndcg_at_100
1306
+ value: 65.846
1307
+ - type: ndcg_at_1000
1308
+ value: 66.412
1309
+ - type: ndcg_at_3
1310
+ value: 57.546
1311
+ - type: ndcg_at_5
1312
+ value: 60.638000000000005
1313
+ - type: precision_at_1
1314
+ value: 47.03
1315
+ - type: precision_at_10
1316
+ value: 8.831
1317
+ - type: precision_at_100
1318
+ value: 1.027
1319
+ - type: precision_at_1000
1320
+ value: 0.109
1321
+ - type: precision_at_3
1322
+ value: 23.642
1323
+ - type: precision_at_5
1324
+ value: 15.884
1325
+ - type: recall_at_1
1326
+ value: 43.765
1327
+ - type: recall_at_10
1328
+ value: 80.537
1329
+ - type: recall_at_100
1330
+ value: 92.06400000000001
1331
+ - type: recall_at_1000
1332
+ value: 96.054
1333
+ - type: recall_at_3
1334
+ value: 65.27199999999999
1335
+ - type: recall_at_5
1336
+ value: 72.71
1337
+ - task:
1338
+ type: Retrieval
1339
+ dataset:
1340
+ type: fiqa
1341
+ name: MTEB FiQA2018
1342
+ config: default
1343
+ split: test
1344
+ revision: None
1345
+ metrics:
1346
+ - type: map_at_1
1347
+ value: 20.684
1348
+ - type: map_at_10
1349
+ value: 33.393
1350
+ - type: map_at_100
1351
+ value: 35.370000000000005
1352
+ - type: map_at_1000
1353
+ value: 35.539
1354
+ - type: map_at_3
1355
+ value: 28.810000000000002
1356
+ - type: map_at_5
1357
+ value: 31.484
1358
+ - type: mrr_at_1
1359
+ value: 41.049
1360
+ - type: mrr_at_10
1361
+ value: 49.736999999999995
1362
+ - type: mrr_at_100
1363
+ value: 50.541000000000004
1364
+ - type: mrr_at_1000
1365
+ value: 50.575
1366
+ - type: mrr_at_3
1367
+ value: 47.094
1368
+ - type: mrr_at_5
1369
+ value: 48.768
1370
+ - type: ndcg_at_1
1371
+ value: 41.049
1372
+ - type: ndcg_at_10
1373
+ value: 41.338
1374
+ - type: ndcg_at_100
1375
+ value: 48.386
1376
+ - type: ndcg_at_1000
1377
+ value: 51.209
1378
+ - type: ndcg_at_3
1379
+ value: 37.208000000000006
1380
+ - type: ndcg_at_5
1381
+ value: 38.788
1382
+ - type: precision_at_1
1383
+ value: 41.049
1384
+ - type: precision_at_10
1385
+ value: 11.466
1386
+ - type: precision_at_100
1387
+ value: 1.8769999999999998
1388
+ - type: precision_at_1000
1389
+ value: 0.23800000000000002
1390
+ - type: precision_at_3
1391
+ value: 24.691
1392
+ - type: precision_at_5
1393
+ value: 18.519
1394
+ - type: recall_at_1
1395
+ value: 20.684
1396
+ - type: recall_at_10
1397
+ value: 48.431000000000004
1398
+ - type: recall_at_100
1399
+ value: 74.331
1400
+ - type: recall_at_1000
1401
+ value: 91.268
1402
+ - type: recall_at_3
1403
+ value: 33.267
1404
+ - type: recall_at_5
1405
+ value: 40.313
1406
+ - task:
1407
+ type: Retrieval
1408
+ dataset:
1409
+ type: hotpotqa
1410
+ name: MTEB HotpotQA
1411
+ config: default
1412
+ split: test
1413
+ revision: None
1414
+ metrics:
1415
+ - type: map_at_1
1416
+ value: 32.242
1417
+ - type: map_at_10
1418
+ value: 47.49
1419
+ - type: map_at_100
1420
+ value: 48.409
1421
+ - type: map_at_1000
1422
+ value: 48.489
1423
+ - type: map_at_3
1424
+ value: 44.519
1425
+ - type: map_at_5
1426
+ value: 46.298
1427
+ - type: mrr_at_1
1428
+ value: 64.483
1429
+ - type: mrr_at_10
1430
+ value: 71.364
1431
+ - type: mrr_at_100
1432
+ value: 71.734
1433
+ - type: mrr_at_1000
1434
+ value: 71.751
1435
+ - type: mrr_at_3
1436
+ value: 69.899
1437
+ - type: mrr_at_5
1438
+ value: 70.791
1439
+ - type: ndcg_at_1
1440
+ value: 64.483
1441
+ - type: ndcg_at_10
1442
+ value: 56.274
1443
+ - type: ndcg_at_100
1444
+ value: 59.855999999999995
1445
+ - type: ndcg_at_1000
1446
+ value: 61.538000000000004
1447
+ - type: ndcg_at_3
1448
+ value: 51.636
1449
+ - type: ndcg_at_5
1450
+ value: 54.089
1451
+ - type: precision_at_1
1452
+ value: 64.483
1453
+ - type: precision_at_10
1454
+ value: 11.858
1455
+ - type: precision_at_100
1456
+ value: 1.47
1457
+ - type: precision_at_1000
1458
+ value: 0.169
1459
+ - type: precision_at_3
1460
+ value: 32.635999999999996
1461
+ - type: precision_at_5
1462
+ value: 21.521
1463
+ - type: recall_at_1
1464
+ value: 32.242
1465
+ - type: recall_at_10
1466
+ value: 59.291000000000004
1467
+ - type: recall_at_100
1468
+ value: 73.518
1469
+ - type: recall_at_1000
1470
+ value: 84.747
1471
+ - type: recall_at_3
1472
+ value: 48.953
1473
+ - type: recall_at_5
1474
+ value: 53.801
1475
+ - task:
1476
+ type: Classification
1477
+ dataset:
1478
+ type: mteb/imdb
1479
+ name: MTEB ImdbClassification
1480
+ config: default
1481
+ split: test
1482
+ revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
1483
+ metrics:
1484
+ - type: accuracy
1485
+ value: 80.9492
1486
+ - type: ap
1487
+ value: 75.30846930618502
1488
+ - type: f1
1489
+ value: 80.89150705991759
1490
+ - task:
1491
+ type: Retrieval
1492
+ dataset:
1493
+ type: msmarco
1494
+ name: MTEB MSMARCO
1495
+ config: default
1496
+ split: dev
1497
+ revision: None
1498
+ metrics:
1499
+ - type: map_at_1
1500
+ value: 22.033
1501
+ - type: map_at_10
1502
+ value: 34.331
1503
+ - type: map_at_100
1504
+ value: 35.536
1505
+ - type: map_at_1000
1506
+ value: 35.583
1507
+ - type: map_at_3
1508
+ value: 30.562
1509
+ - type: map_at_5
1510
+ value: 32.667
1511
+ - type: mrr_at_1
1512
+ value: 22.708000000000002
1513
+ - type: mrr_at_10
1514
+ value: 34.967999999999996
1515
+ - type: mrr_at_100
1516
+ value: 36.105
1517
+ - type: mrr_at_1000
1518
+ value: 36.147
1519
+ - type: mrr_at_3
1520
+ value: 31.256
1521
+ - type: mrr_at_5
1522
+ value: 33.322
1523
+ - type: ndcg_at_1
1524
+ value: 22.708000000000002
1525
+ - type: ndcg_at_10
1526
+ value: 41.211999999999996
1527
+ - type: ndcg_at_100
1528
+ value: 46.952
1529
+ - type: ndcg_at_1000
1530
+ value: 48.131
1531
+ - type: ndcg_at_3
1532
+ value: 33.501
1533
+ - type: ndcg_at_5
1534
+ value: 37.248999999999995
1535
+ - type: precision_at_1
1536
+ value: 22.708000000000002
1537
+ - type: precision_at_10
1538
+ value: 6.519
1539
+ - type: precision_at_100
1540
+ value: 0.9390000000000001
1541
+ - type: precision_at_1000
1542
+ value: 0.104
1543
+ - type: precision_at_3
1544
+ value: 14.302999999999999
1545
+ - type: precision_at_5
1546
+ value: 10.481
1547
+ - type: recall_at_1
1548
+ value: 22.033
1549
+ - type: recall_at_10
1550
+ value: 62.348000000000006
1551
+ - type: recall_at_100
1552
+ value: 88.771
1553
+ - type: recall_at_1000
1554
+ value: 97.782
1555
+ - type: recall_at_3
1556
+ value: 41.331
1557
+ - type: recall_at_5
1558
+ value: 50.32600000000001
1559
+ - task:
1560
+ type: Classification
1561
+ dataset:
1562
+ type: mteb/mtop_domain
1563
+ name: MTEB MTOPDomainClassification (en)
1564
+ config: en
1565
+ split: test
1566
+ revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
1567
+ metrics:
1568
+ - type: accuracy
1569
+ value: 92.69037847697219
1570
+ - type: f1
1571
+ value: 92.20814766144707
1572
+ - task:
1573
+ type: Classification
1574
+ dataset:
1575
+ type: mteb/mtop_intent
1576
+ name: MTEB MTOPIntentClassification (en)
1577
+ config: en
1578
+ split: test
1579
+ revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
1580
+ metrics:
1581
+ - type: accuracy
1582
+ value: 61.12859097127223
1583
+ - type: f1
1584
+ value: 44.859837744275346
1585
+ - task:
1586
+ type: Classification
1587
+ dataset:
1588
+ type: mteb/amazon_massive_intent
1589
+ name: MTEB MassiveIntentClassification (en)
1590
+ config: en
1591
+ split: test
1592
+ revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
1593
+ metrics:
1594
+ - type: accuracy
1595
+ value: 67.59246805648958
1596
+ - type: f1
1597
+ value: 65.35653843975764
1598
+ - task:
1599
+ type: Classification
1600
+ dataset:
1601
+ type: mteb/amazon_massive_scenario
1602
+ name: MTEB MassiveScenarioClassification (en)
1603
+ config: en
1604
+ split: test
1605
+ revision: 7d571f92784cd94a019292a1f45445077d0ef634
1606
+ metrics:
1607
+ - type: accuracy
1608
+ value: 72.82447881640888
1609
+ - type: f1
1610
+ value: 71.74294810351809
1611
+ - task:
1612
+ type: Clustering
1613
+ dataset:
1614
+ type: mteb/medrxiv-clustering-p2p
1615
+ name: MTEB MedrxivClusteringP2P
1616
+ config: default
1617
+ split: test
1618
+ revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73
1619
+ metrics:
1620
+ - type: v_measure
1621
+ value: 32.623627054114884
1622
+ - task:
1623
+ type: Clustering
1624
+ dataset:
1625
+ type: mteb/medrxiv-clustering-s2s
1626
+ name: MTEB MedrxivClusteringS2S
1627
+ config: default
1628
+ split: test
1629
+ revision: 35191c8c0dca72d8ff3efcd72aa802307d469663
1630
+ metrics:
1631
+ - type: v_measure
1632
+ value: 28.715250618201516
1633
+ - task:
1634
+ type: Reranking
1635
+ dataset:
1636
+ type: mteb/mind_small
1637
+ name: MTEB MindSmallReranking
1638
+ config: default
1639
+ split: test
1640
+ revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69
1641
+ metrics:
1642
+ - type: map
1643
+ value: 31.268319417897434
1644
+ - type: mrr
1645
+ value: 32.363138927039806
1646
+ - task:
1647
+ type: Retrieval
1648
+ dataset:
1649
+ type: nfcorpus
1650
+ name: MTEB NFCorpus
1651
+ config: default
1652
+ split: test
1653
+ revision: None
1654
+ metrics:
1655
+ - type: map_at_1
1656
+ value: 5.702
1657
+ - type: map_at_10
1658
+ value: 11.838999999999999
1659
+ - type: map_at_100
1660
+ value: 14.879999999999999
1661
+ - type: map_at_1000
1662
+ value: 16.277
1663
+ - type: map_at_3
1664
+ value: 8.912
1665
+ - type: map_at_5
1666
+ value: 10.213999999999999
1667
+ - type: mrr_at_1
1668
+ value: 44.891999999999996
1669
+ - type: mrr_at_10
1670
+ value: 53.15800000000001
1671
+ - type: mrr_at_100
1672
+ value: 53.830999999999996
1673
+ - type: mrr_at_1000
1674
+ value: 53.882
1675
+ - type: mrr_at_3
1676
+ value: 51.135
1677
+ - type: mrr_at_5
1678
+ value: 52.234
1679
+ - type: ndcg_at_1
1680
+ value: 43.808
1681
+ - type: ndcg_at_10
1682
+ value: 32.179
1683
+ - type: ndcg_at_100
1684
+ value: 29.842000000000002
1685
+ - type: ndcg_at_1000
1686
+ value: 38.858
1687
+ - type: ndcg_at_3
1688
+ value: 38.015
1689
+ - type: ndcg_at_5
1690
+ value: 35.574
1691
+ - type: precision_at_1
1692
+ value: 44.891999999999996
1693
+ - type: precision_at_10
1694
+ value: 23.375
1695
+ - type: precision_at_100
1696
+ value: 7.545
1697
+ - type: precision_at_1000
1698
+ value: 2.052
1699
+ - type: precision_at_3
1700
+ value: 35.088
1701
+ - type: precision_at_5
1702
+ value: 30.154999999999998
1703
+ - type: recall_at_1
1704
+ value: 5.702
1705
+ - type: recall_at_10
1706
+ value: 15.421000000000001
1707
+ - type: recall_at_100
1708
+ value: 30.708999999999996
1709
+ - type: recall_at_1000
1710
+ value: 62.487
1711
+ - type: recall_at_3
1712
+ value: 9.966999999999999
1713
+ - type: recall_at_5
1714
+ value: 12.059000000000001
1715
+ - task:
1716
+ type: Retrieval
1717
+ dataset:
1718
+ type: nq
1719
+ name: MTEB NQ
1720
+ config: default
1721
+ split: test
1722
+ revision: None
1723
+ metrics:
1724
+ - type: map_at_1
1725
+ value: 39.117000000000004
1726
+ - type: map_at_10
1727
+ value: 54.041
1728
+ - type: map_at_100
1729
+ value: 54.845
1730
+ - type: map_at_1000
1731
+ value: 54.876999999999995
1732
+ - type: map_at_3
1733
+ value: 50.339999999999996
1734
+ - type: map_at_5
1735
+ value: 52.678999999999995
1736
+ - type: mrr_at_1
1737
+ value: 43.627
1738
+ - type: mrr_at_10
1739
+ value: 56.752
1740
+ - type: mrr_at_100
1741
+ value: 57.32899999999999
1742
+ - type: mrr_at_1000
1743
+ value: 57.35
1744
+ - type: mrr_at_3
1745
+ value: 53.818999999999996
1746
+ - type: mrr_at_5
1747
+ value: 55.684999999999995
1748
+ - type: ndcg_at_1
1749
+ value: 43.627
1750
+ - type: ndcg_at_10
1751
+ value: 60.934
1752
+ - type: ndcg_at_100
1753
+ value: 64.277
1754
+ - type: ndcg_at_1000
1755
+ value: 64.97
1756
+ - type: ndcg_at_3
1757
+ value: 54.164
1758
+ - type: ndcg_at_5
1759
+ value: 57.994
1760
+ - type: precision_at_1
1761
+ value: 43.627
1762
+ - type: precision_at_10
1763
+ value: 9.383
1764
+ - type: precision_at_100
1765
+ value: 1.131
1766
+ - type: precision_at_1000
1767
+ value: 0.12
1768
+ - type: precision_at_3
1769
+ value: 23.919
1770
+ - type: precision_at_5
1771
+ value: 16.541
1772
+ - type: recall_at_1
1773
+ value: 39.117000000000004
1774
+ - type: recall_at_10
1775
+ value: 79.012
1776
+ - type: recall_at_100
1777
+ value: 93.395
1778
+ - type: recall_at_1000
1779
+ value: 98.494
1780
+ - type: recall_at_3
1781
+ value: 61.714999999999996
1782
+ - type: recall_at_5
1783
+ value: 70.55799999999999
1784
+ - task:
1785
+ type: Retrieval
1786
+ dataset:
1787
+ type: quora
1788
+ name: MTEB QuoraRetrieval
1789
+ config: default
1790
+ split: test
1791
+ revision: None
1792
+ metrics:
1793
+ - type: map_at_1
1794
+ value: 70.832
1795
+ - type: map_at_10
1796
+ value: 84.82300000000001
1797
+ - type: map_at_100
1798
+ value: 85.44500000000001
1799
+ - type: map_at_1000
1800
+ value: 85.461
1801
+ - type: map_at_3
1802
+ value: 81.917
1803
+ - type: map_at_5
1804
+ value: 83.734
1805
+ - type: mrr_at_1
1806
+ value: 81.61
1807
+ - type: mrr_at_10
1808
+ value: 87.75500000000001
1809
+ - type: mrr_at_100
1810
+ value: 87.85300000000001
1811
+ - type: mrr_at_1000
1812
+ value: 87.854
1813
+ - type: mrr_at_3
1814
+ value: 86.855
1815
+ - type: mrr_at_5
1816
+ value: 87.465
1817
+ - type: ndcg_at_1
1818
+ value: 81.58999999999999
1819
+ - type: ndcg_at_10
1820
+ value: 88.536
1821
+ - type: ndcg_at_100
1822
+ value: 89.714
1823
+ - type: ndcg_at_1000
1824
+ value: 89.80799999999999
1825
+ - type: ndcg_at_3
1826
+ value: 85.8
1827
+ - type: ndcg_at_5
1828
+ value: 87.286
1829
+ - type: precision_at_1
1830
+ value: 81.58999999999999
1831
+ - type: precision_at_10
1832
+ value: 13.438
1833
+ - type: precision_at_100
1834
+ value: 1.5310000000000001
1835
+ - type: precision_at_1000
1836
+ value: 0.157
1837
+ - type: precision_at_3
1838
+ value: 37.563
1839
+ - type: precision_at_5
1840
+ value: 24.65
1841
+ - type: recall_at_1
1842
+ value: 70.832
1843
+ - type: recall_at_10
1844
+ value: 95.574
1845
+ - type: recall_at_100
1846
+ value: 99.575
1847
+ - type: recall_at_1000
1848
+ value: 99.99
1849
+ - type: recall_at_3
1850
+ value: 87.61
1851
+ - type: recall_at_5
1852
+ value: 91.9
1853
+ - task:
1854
+ type: Clustering
1855
+ dataset:
1856
+ type: mteb/reddit-clustering
1857
+ name: MTEB RedditClustering
1858
+ config: default
1859
+ split: test
1860
+ revision: 24640382cdbf8abc73003fb0fa6d111a705499eb
1861
+ metrics:
1862
+ - type: v_measure
1863
+ value: 54.4131741738767
1864
+ - task:
1865
+ type: Clustering
1866
+ dataset:
1867
+ type: mteb/reddit-clustering-p2p
1868
+ name: MTEB RedditClusteringP2P
1869
+ config: default
1870
+ split: test
1871
+ revision: 282350215ef01743dc01b456c7f5241fa8937f16
1872
+ metrics:
1873
+ - type: v_measure
1874
+ value: 59.816632341901865
1875
+ - task:
1876
+ type: Retrieval
1877
+ dataset:
1878
+ type: scidocs
1879
+ name: MTEB SCIDOCS
1880
+ config: default
1881
+ split: test
1882
+ revision: None
1883
+ metrics:
1884
+ - type: map_at_1
1885
+ value: 4.857
1886
+ - type: map_at_10
1887
+ value: 11.937000000000001
1888
+ - type: map_at_100
1889
+ value: 14.143
1890
+ - type: map_at_1000
1891
+ value: 14.451
1892
+ - type: map_at_3
1893
+ value: 8.376999999999999
1894
+ - type: map_at_5
1895
+ value: 10.172
1896
+ - type: mrr_at_1
1897
+ value: 23.799999999999997
1898
+ - type: mrr_at_10
1899
+ value: 34.134
1900
+ - type: mrr_at_100
1901
+ value: 35.285
1902
+ - type: mrr_at_1000
1903
+ value: 35.33
1904
+ - type: mrr_at_3
1905
+ value: 30.833
1906
+ - type: mrr_at_5
1907
+ value: 32.828
1908
+ - type: ndcg_at_1
1909
+ value: 23.799999999999997
1910
+ - type: ndcg_at_10
1911
+ value: 20.0
1912
+ - type: ndcg_at_100
1913
+ value: 28.486
1914
+ - type: ndcg_at_1000
1915
+ value: 33.781
1916
+ - type: ndcg_at_3
1917
+ value: 18.726000000000003
1918
+ - type: ndcg_at_5
1919
+ value: 16.587
1920
+ - type: precision_at_1
1921
+ value: 23.799999999999997
1922
+ - type: precision_at_10
1923
+ value: 10.39
1924
+ - type: precision_at_100
1925
+ value: 2.263
1926
+ - type: precision_at_1000
1927
+ value: 0.35300000000000004
1928
+ - type: precision_at_3
1929
+ value: 17.333000000000002
1930
+ - type: precision_at_5
1931
+ value: 14.56
1932
+ - type: recall_at_1
1933
+ value: 4.857
1934
+ - type: recall_at_10
1935
+ value: 21.02
1936
+ - type: recall_at_100
1937
+ value: 45.932
1938
+ - type: recall_at_1000
1939
+ value: 71.693
1940
+ - type: recall_at_3
1941
+ value: 10.552
1942
+ - type: recall_at_5
1943
+ value: 14.760000000000002
1944
+ - task:
1945
+ type: STS
1946
+ dataset:
1947
+ type: mteb/sickr-sts
1948
+ name: MTEB SICK-R
1949
+ config: default
1950
+ split: test
1951
+ revision: a6ea5a8cab320b040a23452cc28066d9beae2cee
1952
+ metrics:
1953
+ - type: cos_sim_pearson
1954
+ value: 85.00513539036214
1955
+ - type: cos_sim_spearman
1956
+ value: 79.19581558052613
1957
+ - type: euclidean_pearson
1958
+ value: 82.46689229301268
1959
+ - type: euclidean_spearman
1960
+ value: 79.19581263972574
1961
+ - type: manhattan_pearson
1962
+ value: 82.46839559537645
1963
+ - type: manhattan_spearman
1964
+ value: 79.19301791744469
1965
+ - task:
1966
+ type: STS
1967
+ dataset:
1968
+ type: mteb/sts12-sts
1969
+ name: MTEB STS12
1970
+ config: default
1971
+ split: test
1972
+ revision: a0d554a64d88156834ff5ae9920b964011b16384
1973
+ metrics:
1974
+ - type: cos_sim_pearson
1975
+ value: 82.44111721768361
1976
+ - type: cos_sim_spearman
1977
+ value: 73.14524004507561
1978
+ - type: euclidean_pearson
1979
+ value: 78.70346379990235
1980
+ - type: euclidean_spearman
1981
+ value: 73.14518679640568
1982
+ - type: manhattan_pearson
1983
+ value: 78.68478215009414
1984
+ - type: manhattan_spearman
1985
+ value: 73.10912398034866
1986
+ - task:
1987
+ type: STS
1988
+ dataset:
1989
+ type: mteb/sts13-sts
1990
+ name: MTEB STS13
1991
+ config: default
1992
+ split: test
1993
+ revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca
1994
+ metrics:
1995
+ - type: cos_sim_pearson
1996
+ value: 82.17030364533524
1997
+ - type: cos_sim_spearman
1998
+ value: 82.88382996129783
1999
+ - type: euclidean_pearson
2000
+ value: 82.25266887145027
2001
+ - type: euclidean_spearman
2002
+ value: 82.88382996129783
2003
+ - type: manhattan_pearson
2004
+ value: 82.21831434263969
2005
+ - type: manhattan_spearman
2006
+ value: 82.83144970048046
2007
+ - task:
2008
+ type: STS
2009
+ dataset:
2010
+ type: mteb/sts14-sts
2011
+ name: MTEB STS14
2012
+ config: default
2013
+ split: test
2014
+ revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375
2015
+ metrics:
2016
+ - type: cos_sim_pearson
2017
+ value: 80.73413303490618
2018
+ - type: cos_sim_spearman
2019
+ value: 76.95203008005365
2020
+ - type: euclidean_pearson
2021
+ value: 79.09169854088067
2022
+ - type: euclidean_spearman
2023
+ value: 76.95202489005659
2024
+ - type: manhattan_pearson
2025
+ value: 79.04289364751341
2026
+ - type: manhattan_spearman
2027
+ value: 76.89976809512328
2028
+ - task:
2029
+ type: STS
2030
+ dataset:
2031
+ type: mteb/sts15-sts
2032
+ name: MTEB STS15
2033
+ config: default
2034
+ split: test
2035
+ revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3
2036
+ metrics:
2037
+ - type: cos_sim_pearson
2038
+ value: 86.84421416279349
2039
+ - type: cos_sim_spearman
2040
+ value: 87.67393507190887
2041
+ - type: euclidean_pearson
2042
+ value: 86.81662915280972
2043
+ - type: euclidean_spearman
2044
+ value: 87.67395576051472
2045
+ - type: manhattan_pearson
2046
+ value: 86.76502179645067
2047
+ - type: manhattan_spearman
2048
+ value: 87.60931601838358
2049
+ - task:
2050
+ type: STS
2051
+ dataset:
2052
+ type: mteb/sts16-sts
2053
+ name: MTEB STS16
2054
+ config: default
2055
+ split: test
2056
+ revision: 4d8694f8f0e0100860b497b999b3dbed754a0513
2057
+ metrics:
2058
+ - type: cos_sim_pearson
2059
+ value: 83.47603001840406
2060
+ - type: cos_sim_spearman
2061
+ value: 84.57363689562743
2062
+ - type: euclidean_pearson
2063
+ value: 83.62746191773213
2064
+ - type: euclidean_spearman
2065
+ value: 84.57363689562743
2066
+ - type: manhattan_pearson
2067
+ value: 83.5049257196953
2068
+ - type: manhattan_spearman
2069
+ value: 84.43576972291818
2070
+ - task:
2071
+ type: STS
2072
+ dataset:
2073
+ type: mteb/sts17-crosslingual-sts
2074
+ name: MTEB STS17 (en-en)
2075
+ config: en-en
2076
+ split: test
2077
+ revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
2078
+ metrics:
2079
+ - type: cos_sim_pearson
2080
+ value: 89.17222804445805
2081
+ - type: cos_sim_spearman
2082
+ value: 89.04642204765032
2083
+ - type: euclidean_pearson
2084
+ value: 88.93412366747594
2085
+ - type: euclidean_spearman
2086
+ value: 89.04642204765032
2087
+ - type: manhattan_pearson
2088
+ value: 88.88891722217033
2089
+ - type: manhattan_spearman
2090
+ value: 88.95405155642727
2091
+ - task:
2092
+ type: STS
2093
+ dataset:
2094
+ type: mteb/sts22-crosslingual-sts
2095
+ name: MTEB STS22 (en)
2096
+ config: en
2097
+ split: test
2098
+ revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
2099
+ metrics:
2100
+ - type: cos_sim_pearson
2101
+ value: 63.4232873899918
2102
+ - type: cos_sim_spearman
2103
+ value: 62.53261852485254
2104
+ - type: euclidean_pearson
2105
+ value: 63.95808586267597
2106
+ - type: euclidean_spearman
2107
+ value: 62.53261852485254
2108
+ - type: manhattan_pearson
2109
+ value: 64.07446205165546
2110
+ - type: manhattan_spearman
2111
+ value: 62.86514483815617
2112
+ - task:
2113
+ type: STS
2114
+ dataset:
2115
+ type: mteb/stsbenchmark-sts
2116
+ name: MTEB STSBenchmark
2117
+ config: default
2118
+ split: test
2119
+ revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831
2120
+ metrics:
2121
+ - type: cos_sim_pearson
2122
+ value: 84.324835033109
2123
+ - type: cos_sim_spearman
2124
+ value: 84.75551248417419
2125
+ - type: euclidean_pearson
2126
+ value: 84.98725144123726
2127
+ - type: euclidean_spearman
2128
+ value: 84.75551248417419
2129
+ - type: manhattan_pearson
2130
+ value: 84.9546533100131
2131
+ - type: manhattan_spearman
2132
+ value: 84.73671830914728
2133
+ - task:
2134
+ type: Reranking
2135
+ dataset:
2136
+ type: mteb/scidocs-reranking
2137
+ name: MTEB SciDocsRR
2138
+ config: default
2139
+ split: test
2140
+ revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab
2141
+ metrics:
2142
+ - type: map
2143
+ value: 83.62940531539546
2144
+ - type: mrr
2145
+ value: 95.50283503714876
2146
+ - task:
2147
+ type: Retrieval
2148
+ dataset:
2149
+ type: scifact
2150
+ name: MTEB SciFact
2151
+ config: default
2152
+ split: test
2153
+ revision: None
2154
+ metrics:
2155
+ - type: map_at_1
2156
+ value: 52.428
2157
+ - type: map_at_10
2158
+ value: 62.731
2159
+ - type: map_at_100
2160
+ value: 63.327
2161
+ - type: map_at_1000
2162
+ value: 63.356
2163
+ - type: map_at_3
2164
+ value: 60.17400000000001
2165
+ - type: map_at_5
2166
+ value: 61.461
2167
+ - type: mrr_at_1
2168
+ value: 55.333
2169
+ - type: mrr_at_10
2170
+ value: 63.788999999999994
2171
+ - type: mrr_at_100
2172
+ value: 64.27000000000001
2173
+ - type: mrr_at_1000
2174
+ value: 64.298
2175
+ - type: mrr_at_3
2176
+ value: 61.944
2177
+ - type: mrr_at_5
2178
+ value: 62.861
2179
+ - type: ndcg_at_1
2180
+ value: 55.333
2181
+ - type: ndcg_at_10
2182
+ value: 67.309
2183
+ - type: ndcg_at_100
2184
+ value: 70.033
2185
+ - type: ndcg_at_1000
2186
+ value: 70.842
2187
+ - type: ndcg_at_3
2188
+ value: 63.05500000000001
2189
+ - type: ndcg_at_5
2190
+ value: 64.8
2191
+ - type: precision_at_1
2192
+ value: 55.333
2193
+ - type: precision_at_10
2194
+ value: 9.1
2195
+ - type: precision_at_100
2196
+ value: 1.057
2197
+ - type: precision_at_1000
2198
+ value: 0.11199999999999999
2199
+ - type: precision_at_3
2200
+ value: 25.111
2201
+ - type: precision_at_5
2202
+ value: 16.333000000000002
2203
+ - type: recall_at_1
2204
+ value: 52.428
2205
+ - type: recall_at_10
2206
+ value: 80.156
2207
+ - type: recall_at_100
2208
+ value: 92.833
2209
+ - type: recall_at_1000
2210
+ value: 99.333
2211
+ - type: recall_at_3
2212
+ value: 68.73899999999999
2213
+ - type: recall_at_5
2214
+ value: 73.13300000000001
2215
+ - task:
2216
+ type: PairClassification
2217
+ dataset:
2218
+ type: mteb/sprintduplicatequestions-pairclassification
2219
+ name: MTEB SprintDuplicateQuestions
2220
+ config: default
2221
+ split: test
2222
+ revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46
2223
+ metrics:
2224
+ - type: cos_sim_accuracy
2225
+ value: 99.8069306930693
2226
+ - type: cos_sim_ap
2227
+ value: 94.89496931806809
2228
+ - type: cos_sim_f1
2229
+ value: 90.0763358778626
2230
+ - type: cos_sim_precision
2231
+ value: 91.70984455958549
2232
+ - type: cos_sim_recall
2233
+ value: 88.5
2234
+ - type: dot_accuracy
2235
+ value: 99.8069306930693
2236
+ - type: dot_ap
2237
+ value: 94.89495820622456
2238
+ - type: dot_f1
2239
+ value: 90.0763358778626
2240
+ - type: dot_precision
2241
+ value: 91.70984455958549
2242
+ - type: dot_recall
2243
+ value: 88.5
2244
+ - type: euclidean_accuracy
2245
+ value: 99.8069306930693
2246
+ - type: euclidean_ap
2247
+ value: 94.8949693180681
2248
+ - type: euclidean_f1
2249
+ value: 90.0763358778626
2250
+ - type: euclidean_precision
2251
+ value: 91.70984455958549
2252
+ - type: euclidean_recall
2253
+ value: 88.5
2254
+ - type: manhattan_accuracy
2255
+ value: 99.8009900990099
2256
+ - type: manhattan_ap
2257
+ value: 94.81699021810266
2258
+ - type: manhattan_f1
2259
+ value: 89.82278481012658
2260
+ - type: manhattan_precision
2261
+ value: 90.97435897435898
2262
+ - type: manhattan_recall
2263
+ value: 88.7
2264
+ - type: max_accuracy
2265
+ value: 99.8069306930693
2266
+ - type: max_ap
2267
+ value: 94.8949693180681
2268
+ - type: max_f1
2269
+ value: 90.0763358778626
2270
+ - task:
2271
+ type: Clustering
2272
+ dataset:
2273
+ type: mteb/stackexchange-clustering
2274
+ name: MTEB StackExchangeClustering
2275
+ config: default
2276
+ split: test
2277
+ revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259
2278
+ metrics:
2279
+ - type: v_measure
2280
+ value: 58.95255708336027
2281
+ - task:
2282
+ type: Clustering
2283
+ dataset:
2284
+ type: mteb/stackexchange-clustering-p2p
2285
+ name: MTEB StackExchangeClusteringP2P
2286
+ config: default
2287
+ split: test
2288
+ revision: 815ca46b2622cec33ccafc3735d572c266efdb44
2289
+ metrics:
2290
+ - type: v_measure
2291
+ value: 34.26328409998647
2292
+ - task:
2293
+ type: Reranking
2294
+ dataset:
2295
+ type: mteb/stackoverflowdupquestions-reranking
2296
+ name: MTEB StackOverflowDupQuestions
2297
+ config: default
2298
+ split: test
2299
+ revision: e185fbe320c72810689fc5848eb6114e1ef5ec69
2300
+ metrics:
2301
+ - type: map
2302
+ value: 52.324949351182134
2303
+ - type: mrr
2304
+ value: 53.08798329938036
2305
+ - task:
2306
+ type: Summarization
2307
+ dataset:
2308
+ type: mteb/summeval
2309
+ name: MTEB SummEval
2310
+ config: default
2311
+ split: test
2312
+ revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c
2313
+ metrics:
2314
+ - type: cos_sim_pearson
2315
+ value: 30.286127875761963
2316
+ - type: cos_sim_spearman
2317
+ value: 30.85723241148158
2318
+ - type: dot_pearson
2319
+ value: 30.28613033184199
2320
+ - type: dot_spearman
2321
+ value: 30.85723241148158
2322
+ - task:
2323
+ type: Retrieval
2324
+ dataset:
2325
+ type: trec-covid
2326
+ name: MTEB TRECCOVID
2327
+ config: default
2328
+ split: test
2329
+ revision: None
2330
+ metrics:
2331
+ - type: map_at_1
2332
+ value: 0.199
2333
+ - type: map_at_10
2334
+ value: 1.633
2335
+ - type: map_at_100
2336
+ value: 8.813
2337
+ - type: map_at_1000
2338
+ value: 21.015
2339
+ - type: map_at_3
2340
+ value: 0.577
2341
+ - type: map_at_5
2342
+ value: 0.907
2343
+ - type: mrr_at_1
2344
+ value: 72.0
2345
+ - type: mrr_at_10
2346
+ value: 82.667
2347
+ - type: mrr_at_100
2348
+ value: 82.667
2349
+ - type: mrr_at_1000
2350
+ value: 82.667
2351
+ - type: mrr_at_3
2352
+ value: 80.667
2353
+ - type: mrr_at_5
2354
+ value: 82.667
2355
+ - type: ndcg_at_1
2356
+ value: 67.0
2357
+ - type: ndcg_at_10
2358
+ value: 65.377
2359
+ - type: ndcg_at_100
2360
+ value: 50.693
2361
+ - type: ndcg_at_1000
2362
+ value: 45.449
2363
+ - type: ndcg_at_3
2364
+ value: 67.78800000000001
2365
+ - type: ndcg_at_5
2366
+ value: 67.19000000000001
2367
+ - type: precision_at_1
2368
+ value: 72.0
2369
+ - type: precision_at_10
2370
+ value: 70.6
2371
+ - type: precision_at_100
2372
+ value: 52.0
2373
+ - type: precision_at_1000
2374
+ value: 20.316000000000003
2375
+ - type: precision_at_3
2376
+ value: 72.667
2377
+ - type: precision_at_5
2378
+ value: 72.39999999999999
2379
+ - type: recall_at_1
2380
+ value: 0.199
2381
+ - type: recall_at_10
2382
+ value: 1.8800000000000001
2383
+ - type: recall_at_100
2384
+ value: 12.195
2385
+ - type: recall_at_1000
2386
+ value: 42.612
2387
+ - type: recall_at_3
2388
+ value: 0.608
2389
+ - type: recall_at_5
2390
+ value: 1.004
2391
+ - task:
2392
+ type: Retrieval
2393
+ dataset:
2394
+ type: webis-touche2020
2395
+ name: MTEB Touche2020
2396
+ config: default
2397
+ split: test
2398
+ revision: None
2399
+ metrics:
2400
+ - type: map_at_1
2401
+ value: 2.34
2402
+ - type: map_at_10
2403
+ value: 7.983
2404
+ - type: map_at_100
2405
+ value: 14.488999999999999
2406
+ - type: map_at_1000
2407
+ value: 16.133
2408
+ - type: map_at_3
2409
+ value: 4.312
2410
+ - type: map_at_5
2411
+ value: 6.3420000000000005
2412
+ - type: mrr_at_1
2413
+ value: 26.531
2414
+ - type: mrr_at_10
2415
+ value: 41.558
2416
+ - type: mrr_at_100
2417
+ value: 42.211999999999996
2418
+ - type: mrr_at_1000
2419
+ value: 42.211999999999996
2420
+ - type: mrr_at_3
2421
+ value: 36.054
2422
+ - type: mrr_at_5
2423
+ value: 39.217999999999996
2424
+ - type: ndcg_at_1
2425
+ value: 23.469
2426
+ - type: ndcg_at_10
2427
+ value: 21.077
2428
+ - type: ndcg_at_100
2429
+ value: 35.497
2430
+ - type: ndcg_at_1000
2431
+ value: 47.282000000000004
2432
+ - type: ndcg_at_3
2433
+ value: 20.906
2434
+ - type: ndcg_at_5
2435
+ value: 21.78
2436
+ - type: precision_at_1
2437
+ value: 26.531
2438
+ - type: precision_at_10
2439
+ value: 18.570999999999998
2440
+ - type: precision_at_100
2441
+ value: 7.673000000000001
2442
+ - type: precision_at_1000
2443
+ value: 1.551
2444
+ - type: precision_at_3
2445
+ value: 21.769
2446
+ - type: precision_at_5
2447
+ value: 22.448999999999998
2448
+ - type: recall_at_1
2449
+ value: 2.34
2450
+ - type: recall_at_10
2451
+ value: 14.154
2452
+ - type: recall_at_100
2453
+ value: 48.355
2454
+ - type: recall_at_1000
2455
+ value: 84.872
2456
+ - type: recall_at_3
2457
+ value: 5.19
2458
+ - type: recall_at_5
2459
+ value: 9.211
2460
+ - task:
2461
+ type: Classification
2462
+ dataset:
2463
+ type: mteb/toxic_conversations_50k
2464
+ name: MTEB ToxicConversationsClassification
2465
+ config: default
2466
+ split: test
2467
+ revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c
2468
+ metrics:
2469
+ - type: accuracy
2470
+ value: 71.9318
2471
+ - type: ap
2472
+ value: 14.755439516631267
2473
+ - type: f1
2474
+ value: 55.39101096477449
2475
+ - task:
2476
+ type: Classification
2477
+ dataset:
2478
+ type: mteb/tweet_sentiment_extraction
2479
+ name: MTEB TweetSentimentExtractionClassification
2480
+ config: default
2481
+ split: test
2482
+ revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
2483
+ metrics:
2484
+ - type: accuracy
2485
+ value: 61.06395019807584
2486
+ - type: f1
2487
+ value: 61.18513886850968
2488
+ - task:
2489
+ type: Clustering
2490
+ dataset:
2491
+ type: mteb/twentynewsgroups-clustering
2492
+ name: MTEB TwentyNewsgroupsClustering
2493
+ config: default
2494
+ split: test
2495
+ revision: 6125ec4e24fa026cec8a478383ee943acfbd5449
2496
+ metrics:
2497
+ - type: v_measure
2498
+ value: 43.68814723462553
2499
+ - task:
2500
+ type: PairClassification
2501
+ dataset:
2502
+ type: mteb/twittersemeval2015-pairclassification
2503
+ name: MTEB TwitterSemEval2015
2504
+ config: default
2505
+ split: test
2506
+ revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
2507
+ metrics:
2508
+ - type: cos_sim_accuracy
2509
+ value: 85.8258329856351
2510
+ - type: cos_sim_ap
2511
+ value: 73.51953909054856
2512
+ - type: cos_sim_f1
2513
+ value: 68.17958783120707
2514
+ - type: cos_sim_precision
2515
+ value: 63.70930765703806
2516
+ - type: cos_sim_recall
2517
+ value: 73.3245382585752
2518
+ - type: dot_accuracy
2519
+ value: 85.8258329856351
2520
+ - type: dot_ap
2521
+ value: 73.51954936569123
2522
+ - type: dot_f1
2523
+ value: 68.17958783120707
2524
+ - type: dot_precision
2525
+ value: 63.70930765703806
2526
+ - type: dot_recall
2527
+ value: 73.3245382585752
2528
+ - type: euclidean_accuracy
2529
+ value: 85.8258329856351
2530
+ - type: euclidean_ap
2531
+ value: 73.51954390509214
2532
+ - type: euclidean_f1
2533
+ value: 68.17958783120707
2534
+ - type: euclidean_precision
2535
+ value: 63.70930765703806
2536
+ - type: euclidean_recall
2537
+ value: 73.3245382585752
2538
+ - type: manhattan_accuracy
2539
+ value: 85.8258329856351
2540
+ - type: manhattan_ap
2541
+ value: 73.44954175022839
2542
+ - type: manhattan_f1
2543
+ value: 68.08816482989938
2544
+ - type: manhattan_precision
2545
+ value: 62.351908731899954
2546
+ - type: manhattan_recall
2547
+ value: 74.9868073878628
2548
+ - type: max_accuracy
2549
+ value: 85.8258329856351
2550
+ - type: max_ap
2551
+ value: 73.51954936569123
2552
+ - type: max_f1
2553
+ value: 68.17958783120707
2554
+ - task:
2555
+ type: PairClassification
2556
+ dataset:
2557
+ type: mteb/twitterurlcorpus-pairclassification
2558
+ name: MTEB TwitterURLCorpus
2559
+ config: default
2560
+ split: test
2561
+ revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
2562
+ metrics:
2563
+ - type: cos_sim_accuracy
2564
+ value: 88.6094617145962
2565
+ - type: cos_sim_ap
2566
+ value: 85.4121913477208
2567
+ - type: cos_sim_f1
2568
+ value: 77.61548157484985
2569
+ - type: cos_sim_precision
2570
+ value: 74.84627484627485
2571
+ - type: cos_sim_recall
2572
+ value: 80.59747459193102
2573
+ - type: dot_accuracy
2574
+ value: 88.6094617145962
2575
+ - type: dot_ap
2576
+ value: 85.41219830675979
2577
+ - type: dot_f1
2578
+ value: 77.61548157484985
2579
+ - type: dot_precision
2580
+ value: 74.84627484627485
2581
+ - type: dot_recall
2582
+ value: 80.59747459193102
2583
+ - type: euclidean_accuracy
2584
+ value: 88.6094617145962
2585
+ - type: euclidean_ap
2586
+ value: 85.41219328124808
2587
+ - type: euclidean_f1
2588
+ value: 77.61548157484985
2589
+ - type: euclidean_precision
2590
+ value: 74.84627484627485
2591
+ - type: euclidean_recall
2592
+ value: 80.59747459193102
2593
+ - type: manhattan_accuracy
2594
+ value: 88.53960492102301
2595
+ - type: manhattan_ap
2596
+ value: 85.35022078482446
2597
+ - type: manhattan_f1
2598
+ value: 77.56588974387569
2599
+ - type: manhattan_precision
2600
+ value: 74.98742183569324
2601
+ - type: manhattan_recall
2602
+ value: 80.3279950723745
2603
+ - type: max_accuracy
2604
+ value: 88.6094617145962
2605
+ - type: max_ap
2606
+ value: 85.41219830675979
2607
+ - type: max_f1
2608
+ value: 77.61548157484985
2609
+ ---
2610
+ <!-- TODO: add evaluation results here -->
2611
+ <br><br>
2612
+
2613
+ <p align="center">
2614
+ <img src="https://github.com/jina-ai/finetuner/blob/main/docs/_static/finetuner-logo-ani.svg?raw=true" alt="Finetuner logo: Finetuner helps you to create experiments in order to improve embeddings on search tasks. It accompanies you to deliver the last mile of performance-tuning for neural search applications." width="150px">
2615
+ </p>
2616
+
2617
+
2618
+ <p align="center">
2619
+ <b>The text embedding set trained by <a href="https://jina.ai/"><b>Jina AI</b></a>, <a href="https://github.com/jina-ai/finetuner"><b>Finetuner</b></a> team.</b>
2620
+ </p>
2621
+
2622
+
2623
+ ## Intended Usage & Model Info
2624
+
2625
+ `jina-embedding-b-en-v2` is an English, monolingual embedding model supporting 8k sequence length.
2626
+ It is based on a Bert architecture that supports the symmetric bidirectional variant of ALiBi to support longer sequence length.
2627
+ The backbone Jina Bert Small model is pretrained on the C4 dataset.
2628
+ The model is further trained on Jina AI's collection of more than 40 datasets of sentence pairs and hard negatives.
2629
+ These pairs were obtained from various domains and were carefully selected through a thorough cleaning process.
2630
+
2631
+ The embedding model was trained using 512 sequence length, but extrapolates to 8k sequence length thanks to ALiBi.
2632
+ This makes our model useful for a range of use cases, especially when processing long documents is needed, including long document retrieval, semantic textual similarity, text reranking, recommendation, RAG and LLM-based generative search,...
2633
+
2634
+ This model has 33 million parameters, which enables lightning-fast and memory efficient inference on long documents, while still delivering impressive performance.
2635
+ Additionally, we provide the following embedding models, supporting 8k sequence length as well:
2636
+
2637
+ - [`jina-embedding-s-en-v2`](https://huggingface.co/jinaai/jina-embedding-s-en-v2): 33 million parameters.
2638
+ - [`jina-embedding-b-en-v2`](https://huggingface.co/jinaai/jina-embedding-b-en-v2): 137 million parameters **(you are here)**.
2639
+ - [`jina-embedding-l-en-v2`](https://huggingface.co/jinaai/jina-embedding-l-en-v2): 435 million parameters.
2640
+
2641
+ ## Data & Parameters
2642
+ <!-- TODO: update the paper ID once it is published on arxiv -->
2643
+ Please checkout our [technical blog](https://arxiv.org/abs/2307.11224).
2644
+
2645
+ ## Metrics
2646
+
2647
+ We compared the model against `all-minilm-l6-v2`/`all-mpnet-base-v2` from sbert and `text-embeddings-ada-002` from OpenAI:
2648
+
2649
+ <!-- TODO: add evaluation table here -->
2650
+
2651
+ ## Usage
2652
+
2653
+ You can use Jina Embedding models directly from transformers package:
2654
+ ```python
2655
+ !pip install transformers
2656
+ from transformers import AutoModel
2657
+ from numpy.linalg import norm
2658
+
2659
+ cos_sim = lambda a,b: (a @ b.T) / (norm(a)*norm(b))
2660
+ model = AutoModel.from_pretrained('jinaai/jina-embedding-b-en-v2', trust_remote_code=True) # trust_remote_code is needed to use the encode method
2661
+ embeddings = model.encode(['How is the weather today?', 'What is the current weather like today?'])
2662
+ print(cos_sim(embeddings[0], embeddings[1]))
2663
+ ```
2664
+
2665
+ For long sequences, it's recommended to perform inference using Flash Attention. Using Flash Attention allows you to increase the batch size and throughput for long sequence length.
2666
+ We include an experimental implementation for Flash Attention, shipped with the model.
2667
+ Install the following triton version:
2668
+ `pip install triton==2.0.0.dev20221202`.
2669
+ Now run the same code above, but make sure to set the parameter `with_flash` to `True` when you load the model. You also have to use either `fp16` or `bf16`:
2670
+ ```python
2671
+ from transformers import AutoModel
2672
+ from numpy.linalg import norm
2673
+ import torch
2674
+
2675
+ cos_sim = lambda a,b: (a @ b.T) / (norm(a)*norm(b))
2676
+ model = AutoModel.from_pretrained('jinaai/jina-embedding-b-en-v2', trust_remote_code=True, with_flash=True, torch_dtype=torch.float16).cuda() # trust_remote_code is needed to use the encode method
2677
+ embeddings = model.encode(['How is the weather today?', 'What is the current weather like today?'])
2678
+ print(cos_sim(embeddings[0], embeddings[1]))
2679
+ ```
2680
+
2681
+ ## Fine-tuning
2682
+
2683
+ Please consider [Finetuner](https://github.com/jina-ai/finetuner).
2684
+
2685
+ ## Plans
2686
+ The development of new multilingual models is currently underway. We will be targeting mainly the German and Spanish languages. The upcoming models will be called `jina-embedding-s/b/l-de/es-v2`.
2687
+
2688
+ ## Contact
2689
+
2690
+ Join our [Discord community](https://discord.jina.ai) and chat with other community members about ideas.
2691
+
2692
+ ## Citation
2693
+
2694
+ If you find Jina Embeddings useful in your research, please cite the following paper:
2695
+
2696
+ <!-- TODO: update the paper ID once it is published on arxiv -->
2697
+ ``` latex
2698
+ @misc{günther2023jina,
2699
+ title={Beyond the 512-Token Barrier: Training General-Purpose Text
2700
+ Embeddings for Large Documents},
2701
+ author={Michael Günther and Jackmin Ong and Isabelle Mohr and Alaeddine Abdessalem and Tanguy Abel and Mohammad Kalim Akram and Susana Guzman and Georgios Mastrapas and Saba Sturua and Bo Wang},
2702
+ year={2023},
2703
+ eprint={2307.11224},
2704
+ archivePrefix={arXiv},
2705
+ primaryClass={cs.CL}
2706
+ }
2707
+ ```