|
--- |
|
tags: |
|
- mteb |
|
model-index: |
|
- name: sf_model_e5 |
|
results: |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (en) |
|
config: en |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 70.85074626865672 |
|
- type: ap |
|
value: 33.779217850079206 |
|
- type: f1 |
|
value: 64.96977487239377 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_polarity |
|
name: MTEB AmazonPolarityClassification |
|
config: default |
|
split: test |
|
revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
|
metrics: |
|
- type: accuracy |
|
value: 91.80945 |
|
- type: ap |
|
value: 88.22978189506895 |
|
- type: f1 |
|
value: 91.7858219911604 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (en) |
|
config: en |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 48.94200000000001 |
|
- type: f1 |
|
value: 47.911934405973895 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana |
|
name: MTEB ArguAna |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 39.616 |
|
- type: map_at_10 |
|
value: 55.938 |
|
- type: map_at_100 |
|
value: 56.552 |
|
- type: map_at_1000 |
|
value: 56.556 |
|
- type: map_at_3 |
|
value: 51.754 |
|
- type: map_at_5 |
|
value: 54.623999999999995 |
|
- type: mrr_at_1 |
|
value: 40.967 |
|
- type: mrr_at_10 |
|
value: 56.452999999999996 |
|
- type: mrr_at_100 |
|
value: 57.053 |
|
- type: mrr_at_1000 |
|
value: 57.057 |
|
- type: mrr_at_3 |
|
value: 52.312000000000005 |
|
- type: mrr_at_5 |
|
value: 55.1 |
|
- type: ndcg_at_1 |
|
value: 39.616 |
|
- type: ndcg_at_10 |
|
value: 64.067 |
|
- type: ndcg_at_100 |
|
value: 66.384 |
|
- type: ndcg_at_1000 |
|
value: 66.468 |
|
- type: ndcg_at_3 |
|
value: 55.74 |
|
- type: ndcg_at_5 |
|
value: 60.889 |
|
- type: precision_at_1 |
|
value: 39.616 |
|
- type: precision_at_10 |
|
value: 8.953999999999999 |
|
- type: precision_at_100 |
|
value: 0.9900000000000001 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 22.428 |
|
- type: precision_at_5 |
|
value: 15.946 |
|
- type: recall_at_1 |
|
value: 39.616 |
|
- type: recall_at_10 |
|
value: 89.545 |
|
- type: recall_at_100 |
|
value: 99.004 |
|
- type: recall_at_1000 |
|
value: 99.644 |
|
- type: recall_at_3 |
|
value: 67.283 |
|
- type: recall_at_5 |
|
value: 79.73 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-p2p |
|
name: MTEB ArxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
|
metrics: |
|
- type: v_measure |
|
value: 48.72923923743124 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-s2s |
|
name: MTEB ArxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 |
|
metrics: |
|
- type: v_measure |
|
value: 42.87449955203238 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/askubuntudupquestions-reranking |
|
name: MTEB AskUbuntuDupQuestions |
|
config: default |
|
split: test |
|
revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
|
metrics: |
|
- type: map |
|
value: 64.3214434754065 |
|
- type: mrr |
|
value: 77.87879787187265 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/biosses-sts |
|
name: MTEB BIOSSES |
|
config: default |
|
split: test |
|
revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 88.82418607751953 |
|
- type: cos_sim_spearman |
|
value: 86.74535004562274 |
|
- type: euclidean_pearson |
|
value: 86.58792166831103 |
|
- type: euclidean_spearman |
|
value: 86.74535004562274 |
|
- type: manhattan_pearson |
|
value: 86.23957813056677 |
|
- type: manhattan_spearman |
|
value: 86.41522204150452 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/banking77 |
|
name: MTEB Banking77Classification |
|
config: default |
|
split: test |
|
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
|
metrics: |
|
- type: accuracy |
|
value: 84.61363636363636 |
|
- type: f1 |
|
value: 83.98373241136187 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-p2p |
|
name: MTEB BiorxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 |
|
metrics: |
|
- type: v_measure |
|
value: 39.73148995791471 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-s2s |
|
name: MTEB BiorxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 |
|
metrics: |
|
- type: v_measure |
|
value: 37.23723038699733 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackAndroidRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 32.217 |
|
- type: map_at_10 |
|
value: 43.453 |
|
- type: map_at_100 |
|
value: 45.038 |
|
- type: map_at_1000 |
|
value: 45.162 |
|
- type: map_at_3 |
|
value: 39.589 |
|
- type: map_at_5 |
|
value: 41.697 |
|
- type: mrr_at_1 |
|
value: 39.628 |
|
- type: mrr_at_10 |
|
value: 49.698 |
|
- type: mrr_at_100 |
|
value: 50.44 |
|
- type: mrr_at_1000 |
|
value: 50.482000000000006 |
|
- type: mrr_at_3 |
|
value: 46.781 |
|
- type: mrr_at_5 |
|
value: 48.548 |
|
- type: ndcg_at_1 |
|
value: 39.628 |
|
- type: ndcg_at_10 |
|
value: 50.158 |
|
- type: ndcg_at_100 |
|
value: 55.687 |
|
- type: ndcg_at_1000 |
|
value: 57.499 |
|
- type: ndcg_at_3 |
|
value: 44.594 |
|
- type: ndcg_at_5 |
|
value: 47.198 |
|
- type: precision_at_1 |
|
value: 39.628 |
|
- type: precision_at_10 |
|
value: 9.828000000000001 |
|
- type: precision_at_100 |
|
value: 1.591 |
|
- type: precision_at_1000 |
|
value: 0.20600000000000002 |
|
- type: precision_at_3 |
|
value: 21.507 |
|
- type: precision_at_5 |
|
value: 15.765 |
|
- type: recall_at_1 |
|
value: 32.217 |
|
- type: recall_at_10 |
|
value: 62.717999999999996 |
|
- type: recall_at_100 |
|
value: 85.992 |
|
- type: recall_at_1000 |
|
value: 97.271 |
|
- type: recall_at_3 |
|
value: 46.694 |
|
- type: recall_at_5 |
|
value: 53.952 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackEnglishRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 30.862000000000002 |
|
- type: map_at_10 |
|
value: 41.287 |
|
- type: map_at_100 |
|
value: 42.526 |
|
- type: map_at_1000 |
|
value: 42.653999999999996 |
|
- type: map_at_3 |
|
value: 38.055 |
|
- type: map_at_5 |
|
value: 40.022000000000006 |
|
- type: mrr_at_1 |
|
value: 38.408 |
|
- type: mrr_at_10 |
|
value: 46.943 |
|
- type: mrr_at_100 |
|
value: 47.597 |
|
- type: mrr_at_1000 |
|
value: 47.64 |
|
- type: mrr_at_3 |
|
value: 44.607 |
|
- type: mrr_at_5 |
|
value: 46.079 |
|
- type: ndcg_at_1 |
|
value: 38.408 |
|
- type: ndcg_at_10 |
|
value: 46.936 |
|
- type: ndcg_at_100 |
|
value: 51.307 |
|
- type: ndcg_at_1000 |
|
value: 53.312000000000005 |
|
- type: ndcg_at_3 |
|
value: 42.579 |
|
- type: ndcg_at_5 |
|
value: 44.877 |
|
- type: precision_at_1 |
|
value: 38.408 |
|
- type: precision_at_10 |
|
value: 8.885 |
|
- type: precision_at_100 |
|
value: 1.4449999999999998 |
|
- type: precision_at_1000 |
|
value: 0.192 |
|
- type: precision_at_3 |
|
value: 20.616 |
|
- type: precision_at_5 |
|
value: 14.841 |
|
- type: recall_at_1 |
|
value: 30.862000000000002 |
|
- type: recall_at_10 |
|
value: 56.994 |
|
- type: recall_at_100 |
|
value: 75.347 |
|
- type: recall_at_1000 |
|
value: 87.911 |
|
- type: recall_at_3 |
|
value: 44.230000000000004 |
|
- type: recall_at_5 |
|
value: 50.625 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGamingRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 39.076 |
|
- type: map_at_10 |
|
value: 52.535 |
|
- type: map_at_100 |
|
value: 53.537 |
|
- type: map_at_1000 |
|
value: 53.591 |
|
- type: map_at_3 |
|
value: 48.961 |
|
- type: map_at_5 |
|
value: 50.96000000000001 |
|
- type: mrr_at_1 |
|
value: 44.765 |
|
- type: mrr_at_10 |
|
value: 55.615 |
|
- type: mrr_at_100 |
|
value: 56.24 |
|
- type: mrr_at_1000 |
|
value: 56.264 |
|
- type: mrr_at_3 |
|
value: 52.925999999999995 |
|
- type: mrr_at_5 |
|
value: 54.493 |
|
- type: ndcg_at_1 |
|
value: 44.765 |
|
- type: ndcg_at_10 |
|
value: 58.777 |
|
- type: ndcg_at_100 |
|
value: 62.574 |
|
- type: ndcg_at_1000 |
|
value: 63.624 |
|
- type: ndcg_at_3 |
|
value: 52.81 |
|
- type: ndcg_at_5 |
|
value: 55.657999999999994 |
|
- type: precision_at_1 |
|
value: 44.765 |
|
- type: precision_at_10 |
|
value: 9.693 |
|
- type: precision_at_100 |
|
value: 1.248 |
|
- type: precision_at_1000 |
|
value: 0.13799999999999998 |
|
- type: precision_at_3 |
|
value: 23.866 |
|
- type: precision_at_5 |
|
value: 16.489 |
|
- type: recall_at_1 |
|
value: 39.076 |
|
- type: recall_at_10 |
|
value: 74.01299999999999 |
|
- type: recall_at_100 |
|
value: 90.363 |
|
- type: recall_at_1000 |
|
value: 97.782 |
|
- type: recall_at_3 |
|
value: 58.056 |
|
- type: recall_at_5 |
|
value: 65.029 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGisRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 26.357000000000003 |
|
- type: map_at_10 |
|
value: 35.492000000000004 |
|
- type: map_at_100 |
|
value: 36.504999999999995 |
|
- type: map_at_1000 |
|
value: 36.578 |
|
- type: map_at_3 |
|
value: 32.696999999999996 |
|
- type: map_at_5 |
|
value: 34.388999999999996 |
|
- type: mrr_at_1 |
|
value: 28.136 |
|
- type: mrr_at_10 |
|
value: 37.383 |
|
- type: mrr_at_100 |
|
value: 38.271 |
|
- type: mrr_at_1000 |
|
value: 38.324999999999996 |
|
- type: mrr_at_3 |
|
value: 34.782999999999994 |
|
- type: mrr_at_5 |
|
value: 36.416 |
|
- type: ndcg_at_1 |
|
value: 28.136 |
|
- type: ndcg_at_10 |
|
value: 40.741 |
|
- type: ndcg_at_100 |
|
value: 45.803 |
|
- type: ndcg_at_1000 |
|
value: 47.637 |
|
- type: ndcg_at_3 |
|
value: 35.412 |
|
- type: ndcg_at_5 |
|
value: 38.251000000000005 |
|
- type: precision_at_1 |
|
value: 28.136 |
|
- type: precision_at_10 |
|
value: 6.315999999999999 |
|
- type: precision_at_100 |
|
value: 0.931 |
|
- type: precision_at_1000 |
|
value: 0.11299999999999999 |
|
- type: precision_at_3 |
|
value: 15.254000000000001 |
|
- type: precision_at_5 |
|
value: 10.757 |
|
- type: recall_at_1 |
|
value: 26.357000000000003 |
|
- type: recall_at_10 |
|
value: 55.021 |
|
- type: recall_at_100 |
|
value: 78.501 |
|
- type: recall_at_1000 |
|
value: 92.133 |
|
- type: recall_at_3 |
|
value: 40.798 |
|
- type: recall_at_5 |
|
value: 47.591 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackMathematicaRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 17.302 |
|
- type: map_at_10 |
|
value: 26.365 |
|
- type: map_at_100 |
|
value: 27.581 |
|
- type: map_at_1000 |
|
value: 27.705999999999996 |
|
- type: map_at_3 |
|
value: 23.682 |
|
- type: map_at_5 |
|
value: 25.304 |
|
- type: mrr_at_1 |
|
value: 21.891 |
|
- type: mrr_at_10 |
|
value: 31.227 |
|
- type: mrr_at_100 |
|
value: 32.22 |
|
- type: mrr_at_1000 |
|
value: 32.282 |
|
- type: mrr_at_3 |
|
value: 28.711 |
|
- type: mrr_at_5 |
|
value: 30.314999999999998 |
|
- type: ndcg_at_1 |
|
value: 21.891 |
|
- type: ndcg_at_10 |
|
value: 31.965 |
|
- type: ndcg_at_100 |
|
value: 37.869 |
|
- type: ndcg_at_1000 |
|
value: 40.642 |
|
- type: ndcg_at_3 |
|
value: 27.184 |
|
- type: ndcg_at_5 |
|
value: 29.686 |
|
- type: precision_at_1 |
|
value: 21.891 |
|
- type: precision_at_10 |
|
value: 5.9830000000000005 |
|
- type: precision_at_100 |
|
value: 1.0250000000000001 |
|
- type: precision_at_1000 |
|
value: 0.14100000000000001 |
|
- type: precision_at_3 |
|
value: 13.391 |
|
- type: precision_at_5 |
|
value: 9.801 |
|
- type: recall_at_1 |
|
value: 17.302 |
|
- type: recall_at_10 |
|
value: 44.312000000000005 |
|
- type: recall_at_100 |
|
value: 70.274 |
|
- type: recall_at_1000 |
|
value: 89.709 |
|
- type: recall_at_3 |
|
value: 31.117 |
|
- type: recall_at_5 |
|
value: 37.511 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackPhysicsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 29.404000000000003 |
|
- type: map_at_10 |
|
value: 40.571 |
|
- type: map_at_100 |
|
value: 42.049 |
|
- type: map_at_1000 |
|
value: 42.156 |
|
- type: map_at_3 |
|
value: 37.413000000000004 |
|
- type: map_at_5 |
|
value: 39.206 |
|
- type: mrr_at_1 |
|
value: 36.285000000000004 |
|
- type: mrr_at_10 |
|
value: 46.213 |
|
- type: mrr_at_100 |
|
value: 47.129 |
|
- type: mrr_at_1000 |
|
value: 47.168 |
|
- type: mrr_at_3 |
|
value: 43.84 |
|
- type: mrr_at_5 |
|
value: 45.226 |
|
- type: ndcg_at_1 |
|
value: 36.285000000000004 |
|
- type: ndcg_at_10 |
|
value: 46.809 |
|
- type: ndcg_at_100 |
|
value: 52.615 |
|
- type: ndcg_at_1000 |
|
value: 54.538 |
|
- type: ndcg_at_3 |
|
value: 41.91 |
|
- type: ndcg_at_5 |
|
value: 44.224999999999994 |
|
- type: precision_at_1 |
|
value: 36.285000000000004 |
|
- type: precision_at_10 |
|
value: 8.527 |
|
- type: precision_at_100 |
|
value: 1.3259999999999998 |
|
- type: precision_at_1000 |
|
value: 0.167 |
|
- type: precision_at_3 |
|
value: 20.083000000000002 |
|
- type: precision_at_5 |
|
value: 14.071 |
|
- type: recall_at_1 |
|
value: 29.404000000000003 |
|
- type: recall_at_10 |
|
value: 59.611999999999995 |
|
- type: recall_at_100 |
|
value: 83.383 |
|
- type: recall_at_1000 |
|
value: 95.703 |
|
- type: recall_at_3 |
|
value: 45.663 |
|
- type: recall_at_5 |
|
value: 51.971999999999994 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackProgrammersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 25.317 |
|
- type: map_at_10 |
|
value: 35.217999999999996 |
|
- type: map_at_100 |
|
value: 36.665 |
|
- type: map_at_1000 |
|
value: 36.768 |
|
- type: map_at_3 |
|
value: 31.924000000000003 |
|
- type: map_at_5 |
|
value: 33.591 |
|
- type: mrr_at_1 |
|
value: 31.507 |
|
- type: mrr_at_10 |
|
value: 40.671 |
|
- type: mrr_at_100 |
|
value: 41.609 |
|
- type: mrr_at_1000 |
|
value: 41.657 |
|
- type: mrr_at_3 |
|
value: 38.261 |
|
- type: mrr_at_5 |
|
value: 39.431 |
|
- type: ndcg_at_1 |
|
value: 31.507 |
|
- type: ndcg_at_10 |
|
value: 41.375 |
|
- type: ndcg_at_100 |
|
value: 47.426 |
|
- type: ndcg_at_1000 |
|
value: 49.504 |
|
- type: ndcg_at_3 |
|
value: 35.989 |
|
- type: ndcg_at_5 |
|
value: 38.068000000000005 |
|
- type: precision_at_1 |
|
value: 31.507 |
|
- type: precision_at_10 |
|
value: 7.8420000000000005 |
|
- type: precision_at_100 |
|
value: 1.257 |
|
- type: precision_at_1000 |
|
value: 0.16199999999999998 |
|
- type: precision_at_3 |
|
value: 17.352 |
|
- type: precision_at_5 |
|
value: 12.328999999999999 |
|
- type: recall_at_1 |
|
value: 25.317 |
|
- type: recall_at_10 |
|
value: 54.254999999999995 |
|
- type: recall_at_100 |
|
value: 80.184 |
|
- type: recall_at_1000 |
|
value: 94.07 |
|
- type: recall_at_3 |
|
value: 39.117000000000004 |
|
- type: recall_at_5 |
|
value: 44.711 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 25.813000000000002 |
|
- type: map_at_10 |
|
value: 35.47183333333334 |
|
- type: map_at_100 |
|
value: 36.71775 |
|
- type: map_at_1000 |
|
value: 36.833000000000006 |
|
- type: map_at_3 |
|
value: 32.449916666666674 |
|
- type: map_at_5 |
|
value: 34.1235 |
|
- type: mrr_at_1 |
|
value: 30.766750000000005 |
|
- type: mrr_at_10 |
|
value: 39.77508333333334 |
|
- type: mrr_at_100 |
|
value: 40.64233333333333 |
|
- type: mrr_at_1000 |
|
value: 40.69658333333333 |
|
- type: mrr_at_3 |
|
value: 37.27349999999999 |
|
- type: mrr_at_5 |
|
value: 38.723416666666665 |
|
- type: ndcg_at_1 |
|
value: 30.766750000000005 |
|
- type: ndcg_at_10 |
|
value: 41.141416666666665 |
|
- type: ndcg_at_100 |
|
value: 46.42016666666666 |
|
- type: ndcg_at_1000 |
|
value: 48.61916666666667 |
|
- type: ndcg_at_3 |
|
value: 36.06883333333333 |
|
- type: ndcg_at_5 |
|
value: 38.43966666666666 |
|
- type: precision_at_1 |
|
value: 30.766750000000005 |
|
- type: precision_at_10 |
|
value: 7.340000000000001 |
|
- type: precision_at_100 |
|
value: 1.1796666666666666 |
|
- type: precision_at_1000 |
|
value: 0.15625 |
|
- type: precision_at_3 |
|
value: 16.763833333333334 |
|
- type: precision_at_5 |
|
value: 11.972166666666666 |
|
- type: recall_at_1 |
|
value: 25.813000000000002 |
|
- type: recall_at_10 |
|
value: 53.62741666666667 |
|
- type: recall_at_100 |
|
value: 76.70125000000002 |
|
- type: recall_at_1000 |
|
value: 91.85566666666666 |
|
- type: recall_at_3 |
|
value: 39.55075 |
|
- type: recall_at_5 |
|
value: 45.645250000000004 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackStatsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.249 |
|
- type: map_at_10 |
|
value: 31.095 |
|
- type: map_at_100 |
|
value: 32.056000000000004 |
|
- type: map_at_1000 |
|
value: 32.163000000000004 |
|
- type: map_at_3 |
|
value: 29.275000000000002 |
|
- type: map_at_5 |
|
value: 30.333 |
|
- type: mrr_at_1 |
|
value: 26.687 |
|
- type: mrr_at_10 |
|
value: 34.122 |
|
- type: mrr_at_100 |
|
value: 34.958 |
|
- type: mrr_at_1000 |
|
value: 35.039 |
|
- type: mrr_at_3 |
|
value: 32.541 |
|
- type: mrr_at_5 |
|
value: 33.43 |
|
- type: ndcg_at_1 |
|
value: 26.687 |
|
- type: ndcg_at_10 |
|
value: 35.248000000000005 |
|
- type: ndcg_at_100 |
|
value: 39.933 |
|
- type: ndcg_at_1000 |
|
value: 42.616 |
|
- type: ndcg_at_3 |
|
value: 31.980999999999998 |
|
- type: ndcg_at_5 |
|
value: 33.583 |
|
- type: precision_at_1 |
|
value: 26.687 |
|
- type: precision_at_10 |
|
value: 5.445 |
|
- type: precision_at_100 |
|
value: 0.848 |
|
- type: precision_at_1000 |
|
value: 0.11499999999999999 |
|
- type: precision_at_3 |
|
value: 13.957 |
|
- type: precision_at_5 |
|
value: 9.479 |
|
- type: recall_at_1 |
|
value: 23.249 |
|
- type: recall_at_10 |
|
value: 45.005 |
|
- type: recall_at_100 |
|
value: 66.175 |
|
- type: recall_at_1000 |
|
value: 86.116 |
|
- type: recall_at_3 |
|
value: 36.03 |
|
- type: recall_at_5 |
|
value: 40.037 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackTexRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 17.592 |
|
- type: map_at_10 |
|
value: 25.003999999999998 |
|
- type: map_at_100 |
|
value: 26.208 |
|
- type: map_at_1000 |
|
value: 26.333000000000002 |
|
- type: map_at_3 |
|
value: 22.479 |
|
- type: map_at_5 |
|
value: 23.712 |
|
- type: mrr_at_1 |
|
value: 21.37 |
|
- type: mrr_at_10 |
|
value: 28.951999999999998 |
|
- type: mrr_at_100 |
|
value: 29.915999999999997 |
|
- type: mrr_at_1000 |
|
value: 29.99 |
|
- type: mrr_at_3 |
|
value: 26.503 |
|
- type: mrr_at_5 |
|
value: 27.728 |
|
- type: ndcg_at_1 |
|
value: 21.37 |
|
- type: ndcg_at_10 |
|
value: 29.944 |
|
- type: ndcg_at_100 |
|
value: 35.632000000000005 |
|
- type: ndcg_at_1000 |
|
value: 38.393 |
|
- type: ndcg_at_3 |
|
value: 25.263999999999996 |
|
- type: ndcg_at_5 |
|
value: 27.115000000000002 |
|
- type: precision_at_1 |
|
value: 21.37 |
|
- type: precision_at_10 |
|
value: 5.568 |
|
- type: precision_at_100 |
|
value: 0.992 |
|
- type: precision_at_1000 |
|
value: 0.13999999999999999 |
|
- type: precision_at_3 |
|
value: 11.895 |
|
- type: precision_at_5 |
|
value: 8.61 |
|
- type: recall_at_1 |
|
value: 17.592 |
|
- type: recall_at_10 |
|
value: 40.976 |
|
- type: recall_at_100 |
|
value: 66.487 |
|
- type: recall_at_1000 |
|
value: 85.954 |
|
- type: recall_at_3 |
|
value: 27.797 |
|
- type: recall_at_5 |
|
value: 32.553 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackUnixRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 25.173000000000002 |
|
- type: map_at_10 |
|
value: 34.611999999999995 |
|
- type: map_at_100 |
|
value: 35.735 |
|
- type: map_at_1000 |
|
value: 35.842 |
|
- type: map_at_3 |
|
value: 31.345 |
|
- type: map_at_5 |
|
value: 33.123000000000005 |
|
- type: mrr_at_1 |
|
value: 29.570999999999998 |
|
- type: mrr_at_10 |
|
value: 38.775999999999996 |
|
- type: mrr_at_100 |
|
value: 39.621 |
|
- type: mrr_at_1000 |
|
value: 39.684000000000005 |
|
- type: mrr_at_3 |
|
value: 35.992000000000004 |
|
- type: mrr_at_5 |
|
value: 37.586999999999996 |
|
- type: ndcg_at_1 |
|
value: 29.570999999999998 |
|
- type: ndcg_at_10 |
|
value: 40.388000000000005 |
|
- type: ndcg_at_100 |
|
value: 45.59 |
|
- type: ndcg_at_1000 |
|
value: 47.948 |
|
- type: ndcg_at_3 |
|
value: 34.497 |
|
- type: ndcg_at_5 |
|
value: 37.201 |
|
- type: precision_at_1 |
|
value: 29.570999999999998 |
|
- type: precision_at_10 |
|
value: 6.931 |
|
- type: precision_at_100 |
|
value: 1.082 |
|
- type: precision_at_1000 |
|
value: 0.13999999999999999 |
|
- type: precision_at_3 |
|
value: 15.609 |
|
- type: precision_at_5 |
|
value: 11.286999999999999 |
|
- type: recall_at_1 |
|
value: 25.173000000000002 |
|
- type: recall_at_10 |
|
value: 53.949000000000005 |
|
- type: recall_at_100 |
|
value: 76.536 |
|
- type: recall_at_1000 |
|
value: 92.979 |
|
- type: recall_at_3 |
|
value: 37.987 |
|
- type: recall_at_5 |
|
value: 44.689 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWebmastersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 24.224 |
|
- type: map_at_10 |
|
value: 32.903 |
|
- type: map_at_100 |
|
value: 34.65 |
|
- type: map_at_1000 |
|
value: 34.873 |
|
- type: map_at_3 |
|
value: 29.673 |
|
- type: map_at_5 |
|
value: 31.361 |
|
- type: mrr_at_1 |
|
value: 30.435000000000002 |
|
- type: mrr_at_10 |
|
value: 38.677 |
|
- type: mrr_at_100 |
|
value: 39.805 |
|
- type: mrr_at_1000 |
|
value: 39.851 |
|
- type: mrr_at_3 |
|
value: 35.935 |
|
- type: mrr_at_5 |
|
value: 37.566 |
|
- type: ndcg_at_1 |
|
value: 30.435000000000002 |
|
- type: ndcg_at_10 |
|
value: 39.012 |
|
- type: ndcg_at_100 |
|
value: 45.553 |
|
- type: ndcg_at_1000 |
|
value: 47.919 |
|
- type: ndcg_at_3 |
|
value: 33.809 |
|
- type: ndcg_at_5 |
|
value: 36.120999999999995 |
|
- type: precision_at_1 |
|
value: 30.435000000000002 |
|
- type: precision_at_10 |
|
value: 7.628 |
|
- type: precision_at_100 |
|
value: 1.5810000000000002 |
|
- type: precision_at_1000 |
|
value: 0.243 |
|
- type: precision_at_3 |
|
value: 15.744 |
|
- type: precision_at_5 |
|
value: 11.66 |
|
- type: recall_at_1 |
|
value: 24.224 |
|
- type: recall_at_10 |
|
value: 50.009 |
|
- type: recall_at_100 |
|
value: 78.839 |
|
- type: recall_at_1000 |
|
value: 93.71300000000001 |
|
- type: recall_at_3 |
|
value: 35.512 |
|
- type: recall_at_5 |
|
value: 41.541 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWordpressRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 18.983 |
|
- type: map_at_10 |
|
value: 27.127000000000002 |
|
- type: map_at_100 |
|
value: 28.063 |
|
- type: map_at_1000 |
|
value: 28.17 |
|
- type: map_at_3 |
|
value: 24.306 |
|
- type: map_at_5 |
|
value: 25.784000000000002 |
|
- type: mrr_at_1 |
|
value: 20.518 |
|
- type: mrr_at_10 |
|
value: 29.024 |
|
- type: mrr_at_100 |
|
value: 29.902 |
|
- type: mrr_at_1000 |
|
value: 29.976999999999997 |
|
- type: mrr_at_3 |
|
value: 26.401999999999997 |
|
- type: mrr_at_5 |
|
value: 27.862 |
|
- type: ndcg_at_1 |
|
value: 20.518 |
|
- type: ndcg_at_10 |
|
value: 32.344 |
|
- type: ndcg_at_100 |
|
value: 37.053000000000004 |
|
- type: ndcg_at_1000 |
|
value: 39.798 |
|
- type: ndcg_at_3 |
|
value: 26.796999999999997 |
|
- type: ndcg_at_5 |
|
value: 29.293000000000003 |
|
- type: precision_at_1 |
|
value: 20.518 |
|
- type: precision_at_10 |
|
value: 5.434 |
|
- type: precision_at_100 |
|
value: 0.83 |
|
- type: precision_at_1000 |
|
value: 0.11800000000000001 |
|
- type: precision_at_3 |
|
value: 11.892 |
|
- type: precision_at_5 |
|
value: 8.577 |
|
- type: recall_at_1 |
|
value: 18.983 |
|
- type: recall_at_10 |
|
value: 46.665 |
|
- type: recall_at_100 |
|
value: 68.33399999999999 |
|
- type: recall_at_1000 |
|
value: 88.927 |
|
- type: recall_at_3 |
|
value: 31.608000000000004 |
|
- type: recall_at_5 |
|
value: 37.532 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: climate-fever |
|
name: MTEB ClimateFEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 11.200000000000001 |
|
- type: map_at_10 |
|
value: 20.241999999999997 |
|
- type: map_at_100 |
|
value: 22.357 |
|
- type: map_at_1000 |
|
value: 22.556 |
|
- type: map_at_3 |
|
value: 16.564999999999998 |
|
- type: map_at_5 |
|
value: 18.443 |
|
- type: mrr_at_1 |
|
value: 25.277 |
|
- type: mrr_at_10 |
|
value: 37.582 |
|
- type: mrr_at_100 |
|
value: 38.525999999999996 |
|
- type: mrr_at_1000 |
|
value: 38.564 |
|
- type: mrr_at_3 |
|
value: 33.898 |
|
- type: mrr_at_5 |
|
value: 36.191 |
|
- type: ndcg_at_1 |
|
value: 25.277 |
|
- type: ndcg_at_10 |
|
value: 28.74 |
|
- type: ndcg_at_100 |
|
value: 36.665 |
|
- type: ndcg_at_1000 |
|
value: 40.08 |
|
- type: ndcg_at_3 |
|
value: 22.888 |
|
- type: ndcg_at_5 |
|
value: 25.081999999999997 |
|
- type: precision_at_1 |
|
value: 25.277 |
|
- type: precision_at_10 |
|
value: 9.251 |
|
- type: precision_at_100 |
|
value: 1.773 |
|
- type: precision_at_1000 |
|
value: 0.241 |
|
- type: precision_at_3 |
|
value: 17.329 |
|
- type: precision_at_5 |
|
value: 13.746 |
|
- type: recall_at_1 |
|
value: 11.200000000000001 |
|
- type: recall_at_10 |
|
value: 35.419 |
|
- type: recall_at_100 |
|
value: 62.41 |
|
- type: recall_at_1000 |
|
value: 81.467 |
|
- type: recall_at_3 |
|
value: 21.275 |
|
- type: recall_at_5 |
|
value: 27.201999999999998 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: dbpedia-entity |
|
name: MTEB DBPedia |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 9.396 |
|
- type: map_at_10 |
|
value: 20.735 |
|
- type: map_at_100 |
|
value: 30.098000000000003 |
|
- type: map_at_1000 |
|
value: 31.866 |
|
- type: map_at_3 |
|
value: 14.71 |
|
- type: map_at_5 |
|
value: 17.259 |
|
- type: mrr_at_1 |
|
value: 70.25 |
|
- type: mrr_at_10 |
|
value: 77.09700000000001 |
|
- type: mrr_at_100 |
|
value: 77.398 |
|
- type: mrr_at_1000 |
|
value: 77.40899999999999 |
|
- type: mrr_at_3 |
|
value: 75.542 |
|
- type: mrr_at_5 |
|
value: 76.354 |
|
- type: ndcg_at_1 |
|
value: 57.75 |
|
- type: ndcg_at_10 |
|
value: 42.509 |
|
- type: ndcg_at_100 |
|
value: 48.94 |
|
- type: ndcg_at_1000 |
|
value: 56.501000000000005 |
|
- type: ndcg_at_3 |
|
value: 46.827000000000005 |
|
- type: ndcg_at_5 |
|
value: 44.033 |
|
- type: precision_at_1 |
|
value: 70.25 |
|
- type: precision_at_10 |
|
value: 33.85 |
|
- type: precision_at_100 |
|
value: 11.373 |
|
- type: precision_at_1000 |
|
value: 2.136 |
|
- type: precision_at_3 |
|
value: 50.917 |
|
- type: precision_at_5 |
|
value: 42.8 |
|
- type: recall_at_1 |
|
value: 9.396 |
|
- type: recall_at_10 |
|
value: 26.472 |
|
- type: recall_at_100 |
|
value: 57.30800000000001 |
|
- type: recall_at_1000 |
|
value: 80.983 |
|
- type: recall_at_3 |
|
value: 15.859000000000002 |
|
- type: recall_at_5 |
|
value: 19.758 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/emotion |
|
name: MTEB EmotionClassification |
|
config: default |
|
split: test |
|
revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
|
metrics: |
|
- type: accuracy |
|
value: 54.900000000000006 |
|
- type: f1 |
|
value: 48.14707395235448 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fever |
|
name: MTEB FEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 66.369 |
|
- type: map_at_10 |
|
value: 76.708 |
|
- type: map_at_100 |
|
value: 76.981 |
|
- type: map_at_1000 |
|
value: 76.995 |
|
- type: map_at_3 |
|
value: 75.114 |
|
- type: map_at_5 |
|
value: 76.116 |
|
- type: mrr_at_1 |
|
value: 71.557 |
|
- type: mrr_at_10 |
|
value: 80.95 |
|
- type: mrr_at_100 |
|
value: 81.075 |
|
- type: mrr_at_1000 |
|
value: 81.07900000000001 |
|
- type: mrr_at_3 |
|
value: 79.728 |
|
- type: mrr_at_5 |
|
value: 80.522 |
|
- type: ndcg_at_1 |
|
value: 71.557 |
|
- type: ndcg_at_10 |
|
value: 81.381 |
|
- type: ndcg_at_100 |
|
value: 82.421 |
|
- type: ndcg_at_1000 |
|
value: 82.709 |
|
- type: ndcg_at_3 |
|
value: 78.671 |
|
- type: ndcg_at_5 |
|
value: 80.17 |
|
- type: precision_at_1 |
|
value: 71.557 |
|
- type: precision_at_10 |
|
value: 10.159 |
|
- type: precision_at_100 |
|
value: 1.089 |
|
- type: precision_at_1000 |
|
value: 0.11299999999999999 |
|
- type: precision_at_3 |
|
value: 30.668 |
|
- type: precision_at_5 |
|
value: 19.337 |
|
- type: recall_at_1 |
|
value: 66.369 |
|
- type: recall_at_10 |
|
value: 91.482 |
|
- type: recall_at_100 |
|
value: 95.848 |
|
- type: recall_at_1000 |
|
value: 97.749 |
|
- type: recall_at_3 |
|
value: 84.185 |
|
- type: recall_at_5 |
|
value: 87.908 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fiqa |
|
name: MTEB FiQA2018 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 20.902 |
|
- type: map_at_10 |
|
value: 34.554 |
|
- type: map_at_100 |
|
value: 36.632 |
|
- type: map_at_1000 |
|
value: 36.811 |
|
- type: map_at_3 |
|
value: 30.264000000000003 |
|
- type: map_at_5 |
|
value: 32.714999999999996 |
|
- type: mrr_at_1 |
|
value: 42.13 |
|
- type: mrr_at_10 |
|
value: 51.224000000000004 |
|
- type: mrr_at_100 |
|
value: 52.044999999999995 |
|
- type: mrr_at_1000 |
|
value: 52.075 |
|
- type: mrr_at_3 |
|
value: 48.842999999999996 |
|
- type: mrr_at_5 |
|
value: 50.108 |
|
- type: ndcg_at_1 |
|
value: 42.13 |
|
- type: ndcg_at_10 |
|
value: 42.643 |
|
- type: ndcg_at_100 |
|
value: 49.806 |
|
- type: ndcg_at_1000 |
|
value: 52.583 |
|
- type: ndcg_at_3 |
|
value: 38.927 |
|
- type: ndcg_at_5 |
|
value: 40.071 |
|
- type: precision_at_1 |
|
value: 42.13 |
|
- type: precision_at_10 |
|
value: 11.928999999999998 |
|
- type: precision_at_100 |
|
value: 1.931 |
|
- type: precision_at_1000 |
|
value: 0.243 |
|
- type: precision_at_3 |
|
value: 26.337 |
|
- type: precision_at_5 |
|
value: 19.29 |
|
- type: recall_at_1 |
|
value: 20.902 |
|
- type: recall_at_10 |
|
value: 49.527 |
|
- type: recall_at_100 |
|
value: 75.754 |
|
- type: recall_at_1000 |
|
value: 92.171 |
|
- type: recall_at_3 |
|
value: 35.024 |
|
- type: recall_at_5 |
|
value: 41.207 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: hotpotqa |
|
name: MTEB HotpotQA |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 39.831 |
|
- type: map_at_10 |
|
value: 63.958999999999996 |
|
- type: map_at_100 |
|
value: 64.869 |
|
- type: map_at_1000 |
|
value: 64.924 |
|
- type: map_at_3 |
|
value: 60.25 |
|
- type: map_at_5 |
|
value: 62.572 |
|
- type: mrr_at_1 |
|
value: 79.662 |
|
- type: mrr_at_10 |
|
value: 85.57900000000001 |
|
- type: mrr_at_100 |
|
value: 85.744 |
|
- type: mrr_at_1000 |
|
value: 85.748 |
|
- type: mrr_at_3 |
|
value: 84.718 |
|
- type: mrr_at_5 |
|
value: 85.312 |
|
- type: ndcg_at_1 |
|
value: 79.662 |
|
- type: ndcg_at_10 |
|
value: 72.366 |
|
- type: ndcg_at_100 |
|
value: 75.42999999999999 |
|
- type: ndcg_at_1000 |
|
value: 76.469 |
|
- type: ndcg_at_3 |
|
value: 67.258 |
|
- type: ndcg_at_5 |
|
value: 70.14099999999999 |
|
- type: precision_at_1 |
|
value: 79.662 |
|
- type: precision_at_10 |
|
value: 15.254999999999999 |
|
- type: precision_at_100 |
|
value: 1.763 |
|
- type: precision_at_1000 |
|
value: 0.19 |
|
- type: precision_at_3 |
|
value: 43.358000000000004 |
|
- type: precision_at_5 |
|
value: 28.288999999999998 |
|
- type: recall_at_1 |
|
value: 39.831 |
|
- type: recall_at_10 |
|
value: 76.273 |
|
- type: recall_at_100 |
|
value: 88.163 |
|
- type: recall_at_1000 |
|
value: 95.017 |
|
- type: recall_at_3 |
|
value: 65.037 |
|
- type: recall_at_5 |
|
value: 70.722 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/imdb |
|
name: MTEB ImdbClassification |
|
config: default |
|
split: test |
|
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
|
metrics: |
|
- type: accuracy |
|
value: 93.13879999999999 |
|
- type: ap |
|
value: 89.94638859649079 |
|
- type: f1 |
|
value: 93.13371537570421 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: msmarco |
|
name: MTEB MSMARCO |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 21.482 |
|
- type: map_at_10 |
|
value: 33.635999999999996 |
|
- type: map_at_100 |
|
value: 34.792 |
|
- type: map_at_1000 |
|
value: 34.839999999999996 |
|
- type: map_at_3 |
|
value: 29.553 |
|
- type: map_at_5 |
|
value: 31.892 |
|
- type: mrr_at_1 |
|
value: 22.076999999999998 |
|
- type: mrr_at_10 |
|
value: 34.247 |
|
- type: mrr_at_100 |
|
value: 35.337 |
|
- type: mrr_at_1000 |
|
value: 35.38 |
|
- type: mrr_at_3 |
|
value: 30.208000000000002 |
|
- type: mrr_at_5 |
|
value: 32.554 |
|
- type: ndcg_at_1 |
|
value: 22.092 |
|
- type: ndcg_at_10 |
|
value: 40.657 |
|
- type: ndcg_at_100 |
|
value: 46.251999999999995 |
|
- type: ndcg_at_1000 |
|
value: 47.466 |
|
- type: ndcg_at_3 |
|
value: 32.353 |
|
- type: ndcg_at_5 |
|
value: 36.532 |
|
- type: precision_at_1 |
|
value: 22.092 |
|
- type: precision_at_10 |
|
value: 6.5040000000000004 |
|
- type: precision_at_100 |
|
value: 0.9329999999999999 |
|
- type: precision_at_1000 |
|
value: 0.104 |
|
- type: precision_at_3 |
|
value: 13.719999999999999 |
|
- type: precision_at_5 |
|
value: 10.344000000000001 |
|
- type: recall_at_1 |
|
value: 21.482 |
|
- type: recall_at_10 |
|
value: 62.316 |
|
- type: recall_at_100 |
|
value: 88.283 |
|
- type: recall_at_1000 |
|
value: 97.554 |
|
- type: recall_at_3 |
|
value: 39.822 |
|
- type: recall_at_5 |
|
value: 49.805 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (en) |
|
config: en |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 93.63657090743274 |
|
- type: f1 |
|
value: 93.49355466580484 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 66.01459188326493 |
|
- type: f1 |
|
value: 48.48386472180784 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 73.49024882313383 |
|
- type: f1 |
|
value: 71.8750196914349 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (en) |
|
config: en |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 77.38063214525891 |
|
- type: f1 |
|
value: 76.87364042122763 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-p2p |
|
name: MTEB MedrxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
|
metrics: |
|
- type: v_measure |
|
value: 34.30572302322684 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-s2s |
|
name: MTEB MedrxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
|
metrics: |
|
- type: v_measure |
|
value: 32.18418556367587 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/mind_small |
|
name: MTEB MindSmallReranking |
|
config: default |
|
split: test |
|
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
|
metrics: |
|
- type: map |
|
value: 32.268707296386154 |
|
- type: mrr |
|
value: 33.481925531215055 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nfcorpus |
|
name: MTEB NFCorpus |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 6.586 |
|
- type: map_at_10 |
|
value: 14.954999999999998 |
|
- type: map_at_100 |
|
value: 19.03 |
|
- type: map_at_1000 |
|
value: 20.653 |
|
- type: map_at_3 |
|
value: 10.859 |
|
- type: map_at_5 |
|
value: 12.577 |
|
- type: mrr_at_1 |
|
value: 47.988 |
|
- type: mrr_at_10 |
|
value: 57.57 |
|
- type: mrr_at_100 |
|
value: 58.050000000000004 |
|
- type: mrr_at_1000 |
|
value: 58.083 |
|
- type: mrr_at_3 |
|
value: 55.212 |
|
- type: mrr_at_5 |
|
value: 56.713 |
|
- type: ndcg_at_1 |
|
value: 45.975 |
|
- type: ndcg_at_10 |
|
value: 38.432 |
|
- type: ndcg_at_100 |
|
value: 35.287 |
|
- type: ndcg_at_1000 |
|
value: 44.35 |
|
- type: ndcg_at_3 |
|
value: 43.077 |
|
- type: ndcg_at_5 |
|
value: 40.952 |
|
- type: precision_at_1 |
|
value: 47.368 |
|
- type: precision_at_10 |
|
value: 28.483000000000004 |
|
- type: precision_at_100 |
|
value: 8.882 |
|
- type: precision_at_1000 |
|
value: 2.217 |
|
- type: precision_at_3 |
|
value: 40.144000000000005 |
|
- type: precision_at_5 |
|
value: 35.17 |
|
- type: recall_at_1 |
|
value: 6.586 |
|
- type: recall_at_10 |
|
value: 19.688 |
|
- type: recall_at_100 |
|
value: 35.426 |
|
- type: recall_at_1000 |
|
value: 68.09100000000001 |
|
- type: recall_at_3 |
|
value: 12.234 |
|
- type: recall_at_5 |
|
value: 14.937000000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nq |
|
name: MTEB NQ |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 27.322000000000003 |
|
- type: map_at_10 |
|
value: 43.224000000000004 |
|
- type: map_at_100 |
|
value: 44.275999999999996 |
|
- type: map_at_1000 |
|
value: 44.308 |
|
- type: map_at_3 |
|
value: 38.239000000000004 |
|
- type: map_at_5 |
|
value: 41.244 |
|
- type: mrr_at_1 |
|
value: 31.025000000000002 |
|
- type: mrr_at_10 |
|
value: 45.635 |
|
- type: mrr_at_100 |
|
value: 46.425 |
|
- type: mrr_at_1000 |
|
value: 46.445 |
|
- type: mrr_at_3 |
|
value: 41.42 |
|
- type: mrr_at_5 |
|
value: 44.038 |
|
- type: ndcg_at_1 |
|
value: 30.997000000000003 |
|
- type: ndcg_at_10 |
|
value: 51.55499999999999 |
|
- type: ndcg_at_100 |
|
value: 55.964999999999996 |
|
- type: ndcg_at_1000 |
|
value: 56.657000000000004 |
|
- type: ndcg_at_3 |
|
value: 42.185 |
|
- type: ndcg_at_5 |
|
value: 47.229 |
|
- type: precision_at_1 |
|
value: 30.997000000000003 |
|
- type: precision_at_10 |
|
value: 8.885 |
|
- type: precision_at_100 |
|
value: 1.1360000000000001 |
|
- type: precision_at_1000 |
|
value: 0.12 |
|
- type: precision_at_3 |
|
value: 19.457 |
|
- type: precision_at_5 |
|
value: 14.554 |
|
- type: recall_at_1 |
|
value: 27.322000000000003 |
|
- type: recall_at_10 |
|
value: 74.59400000000001 |
|
- type: recall_at_100 |
|
value: 93.699 |
|
- type: recall_at_1000 |
|
value: 98.76599999999999 |
|
- type: recall_at_3 |
|
value: 50.43 |
|
- type: recall_at_5 |
|
value: 62.073 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: quora |
|
name: MTEB QuoraRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 71.109 |
|
- type: map_at_10 |
|
value: 85.137 |
|
- type: map_at_100 |
|
value: 85.759 |
|
- type: map_at_1000 |
|
value: 85.774 |
|
- type: map_at_3 |
|
value: 82.25200000000001 |
|
- type: map_at_5 |
|
value: 84.031 |
|
- type: mrr_at_1 |
|
value: 82.01 |
|
- type: mrr_at_10 |
|
value: 87.97 |
|
- type: mrr_at_100 |
|
value: 88.076 |
|
- type: mrr_at_1000 |
|
value: 88.076 |
|
- type: mrr_at_3 |
|
value: 87.06 |
|
- type: mrr_at_5 |
|
value: 87.694 |
|
- type: ndcg_at_1 |
|
value: 81.99 |
|
- type: ndcg_at_10 |
|
value: 88.738 |
|
- type: ndcg_at_100 |
|
value: 89.928 |
|
- type: ndcg_at_1000 |
|
value: 90.01400000000001 |
|
- type: ndcg_at_3 |
|
value: 86.042 |
|
- type: ndcg_at_5 |
|
value: 87.505 |
|
- type: precision_at_1 |
|
value: 81.99 |
|
- type: precision_at_10 |
|
value: 13.468 |
|
- type: precision_at_100 |
|
value: 1.534 |
|
- type: precision_at_1000 |
|
value: 0.157 |
|
- type: precision_at_3 |
|
value: 37.702999999999996 |
|
- type: precision_at_5 |
|
value: 24.706 |
|
- type: recall_at_1 |
|
value: 71.109 |
|
- type: recall_at_10 |
|
value: 95.58 |
|
- type: recall_at_100 |
|
value: 99.62299999999999 |
|
- type: recall_at_1000 |
|
value: 99.98899999999999 |
|
- type: recall_at_3 |
|
value: 87.69 |
|
- type: recall_at_5 |
|
value: 91.982 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering |
|
name: MTEB RedditClustering |
|
config: default |
|
split: test |
|
revision: 24640382cdbf8abc73003fb0fa6d111a705499eb |
|
metrics: |
|
- type: v_measure |
|
value: 59.43361510023748 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering-p2p |
|
name: MTEB RedditClusteringP2P |
|
config: default |
|
split: test |
|
revision: 282350215ef01743dc01b456c7f5241fa8937f16 |
|
metrics: |
|
- type: v_measure |
|
value: 64.53582642500159 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scidocs |
|
name: MTEB SCIDOCS |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 4.2299999999999995 |
|
- type: map_at_10 |
|
value: 11.802 |
|
- type: map_at_100 |
|
value: 14.454 |
|
- type: map_at_1000 |
|
value: 14.865 |
|
- type: map_at_3 |
|
value: 7.911 |
|
- type: map_at_5 |
|
value: 9.912 |
|
- type: mrr_at_1 |
|
value: 21.0 |
|
- type: mrr_at_10 |
|
value: 32.722 |
|
- type: mrr_at_100 |
|
value: 33.989000000000004 |
|
- type: mrr_at_1000 |
|
value: 34.026 |
|
- type: mrr_at_3 |
|
value: 28.65 |
|
- type: mrr_at_5 |
|
value: 31.075000000000003 |
|
- type: ndcg_at_1 |
|
value: 21.0 |
|
- type: ndcg_at_10 |
|
value: 20.161 |
|
- type: ndcg_at_100 |
|
value: 30.122 |
|
- type: ndcg_at_1000 |
|
value: 36.399 |
|
- type: ndcg_at_3 |
|
value: 17.881 |
|
- type: ndcg_at_5 |
|
value: 16.439999999999998 |
|
- type: precision_at_1 |
|
value: 21.0 |
|
- type: precision_at_10 |
|
value: 10.94 |
|
- type: precision_at_100 |
|
value: 2.5340000000000003 |
|
- type: precision_at_1000 |
|
value: 0.402 |
|
- type: precision_at_3 |
|
value: 17.067 |
|
- type: precision_at_5 |
|
value: 15.120000000000001 |
|
- type: recall_at_1 |
|
value: 4.2299999999999995 |
|
- type: recall_at_10 |
|
value: 22.163 |
|
- type: recall_at_100 |
|
value: 51.42 |
|
- type: recall_at_1000 |
|
value: 81.652 |
|
- type: recall_at_3 |
|
value: 10.353 |
|
- type: recall_at_5 |
|
value: 15.323 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sickr-sts |
|
name: MTEB SICK-R |
|
config: default |
|
split: test |
|
revision: a6ea5a8cab320b040a23452cc28066d9beae2cee |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 86.44056731476951 |
|
- type: cos_sim_spearman |
|
value: 82.32974396072802 |
|
- type: euclidean_pearson |
|
value: 83.63616080755894 |
|
- type: euclidean_spearman |
|
value: 82.32974071069209 |
|
- type: manhattan_pearson |
|
value: 83.64149958303744 |
|
- type: manhattan_spearman |
|
value: 82.32161014878858 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts12-sts |
|
name: MTEB STS12 |
|
config: default |
|
split: test |
|
revision: a0d554a64d88156834ff5ae9920b964011b16384 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 85.65083720426293 |
|
- type: cos_sim_spearman |
|
value: 77.60786500521749 |
|
- type: euclidean_pearson |
|
value: 81.8149634918642 |
|
- type: euclidean_spearman |
|
value: 77.60637450428892 |
|
- type: manhattan_pearson |
|
value: 81.83507575657566 |
|
- type: manhattan_spearman |
|
value: 77.613220311151 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts13-sts |
|
name: MTEB STS13 |
|
config: default |
|
split: test |
|
revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 87.35683624595698 |
|
- type: cos_sim_spearman |
|
value: 87.94550696434106 |
|
- type: euclidean_pearson |
|
value: 87.50272679030367 |
|
- type: euclidean_spearman |
|
value: 87.94550696434106 |
|
- type: manhattan_pearson |
|
value: 87.4759786099497 |
|
- type: manhattan_spearman |
|
value: 87.90226811166427 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts14-sts |
|
name: MTEB STS14 |
|
config: default |
|
split: test |
|
revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 86.27438743391316 |
|
- type: cos_sim_spearman |
|
value: 83.85378984594779 |
|
- type: euclidean_pearson |
|
value: 85.25840635223642 |
|
- type: euclidean_spearman |
|
value: 83.85378983163673 |
|
- type: manhattan_pearson |
|
value: 85.24936075631025 |
|
- type: manhattan_spearman |
|
value: 83.85052479958138 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts15-sts |
|
name: MTEB STS15 |
|
config: default |
|
split: test |
|
revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 87.4783814521557 |
|
- type: cos_sim_spearman |
|
value: 88.473284566453 |
|
- type: euclidean_pearson |
|
value: 87.94757741870404 |
|
- type: euclidean_spearman |
|
value: 88.47327698999878 |
|
- type: manhattan_pearson |
|
value: 87.93617414057984 |
|
- type: manhattan_spearman |
|
value: 88.45889274229359 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts16-sts |
|
name: MTEB STS16 |
|
config: default |
|
split: test |
|
revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 84.68359147631057 |
|
- type: cos_sim_spearman |
|
value: 86.46426572535646 |
|
- type: euclidean_pearson |
|
value: 85.98303971468599 |
|
- type: euclidean_spearman |
|
value: 86.46426572535646 |
|
- type: manhattan_pearson |
|
value: 85.95109710640726 |
|
- type: manhattan_spearman |
|
value: 86.43282632541583 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-en) |
|
config: en-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 88.88758959688604 |
|
- type: cos_sim_spearman |
|
value: 88.70384784133324 |
|
- type: euclidean_pearson |
|
value: 89.27293800474978 |
|
- type: euclidean_spearman |
|
value: 88.70384784133324 |
|
- type: manhattan_pearson |
|
value: 89.41494348093664 |
|
- type: manhattan_spearman |
|
value: 88.8330050824941 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (en) |
|
config: en |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 67.66759812551814 |
|
- type: cos_sim_spearman |
|
value: 68.02368115471576 |
|
- type: euclidean_pearson |
|
value: 69.52859542757353 |
|
- type: euclidean_spearman |
|
value: 68.02368115471576 |
|
- type: manhattan_pearson |
|
value: 69.50332399468952 |
|
- type: manhattan_spearman |
|
value: 67.91228681203849 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/stsbenchmark-sts |
|
name: MTEB STSBenchmark |
|
config: default |
|
split: test |
|
revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 87.75891320010409 |
|
- type: cos_sim_spearman |
|
value: 88.33063922402347 |
|
- type: euclidean_pearson |
|
value: 88.02964654543274 |
|
- type: euclidean_spearman |
|
value: 88.33063922402347 |
|
- type: manhattan_pearson |
|
value: 88.03029440701458 |
|
- type: manhattan_spearman |
|
value: 88.3158691488696 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/scidocs-reranking |
|
name: MTEB SciDocsRR |
|
config: default |
|
split: test |
|
revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
|
metrics: |
|
- type: map |
|
value: 87.46897310470844 |
|
- type: mrr |
|
value: 96.29042072669523 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scifact |
|
name: MTEB SciFact |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 62.261 |
|
- type: map_at_10 |
|
value: 71.023 |
|
- type: map_at_100 |
|
value: 71.5 |
|
- type: map_at_1000 |
|
value: 71.518 |
|
- type: map_at_3 |
|
value: 67.857 |
|
- type: map_at_5 |
|
value: 69.44500000000001 |
|
- type: mrr_at_1 |
|
value: 65.0 |
|
- type: mrr_at_10 |
|
value: 72.11 |
|
- type: mrr_at_100 |
|
value: 72.479 |
|
- type: mrr_at_1000 |
|
value: 72.49600000000001 |
|
- type: mrr_at_3 |
|
value: 69.722 |
|
- type: mrr_at_5 |
|
value: 71.02199999999999 |
|
- type: ndcg_at_1 |
|
value: 65.0 |
|
- type: ndcg_at_10 |
|
value: 75.40599999999999 |
|
- type: ndcg_at_100 |
|
value: 77.41 |
|
- type: ndcg_at_1000 |
|
value: 77.83200000000001 |
|
- type: ndcg_at_3 |
|
value: 69.95599999999999 |
|
- type: ndcg_at_5 |
|
value: 72.296 |
|
- type: precision_at_1 |
|
value: 65.0 |
|
- type: precision_at_10 |
|
value: 9.966999999999999 |
|
- type: precision_at_100 |
|
value: 1.097 |
|
- type: precision_at_1000 |
|
value: 0.11299999999999999 |
|
- type: precision_at_3 |
|
value: 26.667 |
|
- type: precision_at_5 |
|
value: 17.666999999999998 |
|
- type: recall_at_1 |
|
value: 62.261 |
|
- type: recall_at_10 |
|
value: 87.822 |
|
- type: recall_at_100 |
|
value: 96.833 |
|
- type: recall_at_1000 |
|
value: 100.0 |
|
- type: recall_at_3 |
|
value: 73.06099999999999 |
|
- type: recall_at_5 |
|
value: 78.88300000000001 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/sprintduplicatequestions-pairclassification |
|
name: MTEB SprintDuplicateQuestions |
|
config: default |
|
split: test |
|
revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 99.86138613861387 |
|
- type: cos_sim_ap |
|
value: 96.7851799601876 |
|
- type: cos_sim_f1 |
|
value: 92.94354838709677 |
|
- type: cos_sim_precision |
|
value: 93.69918699186992 |
|
- type: cos_sim_recall |
|
value: 92.2 |
|
- type: dot_accuracy |
|
value: 99.86138613861387 |
|
- type: dot_ap |
|
value: 96.78517996018759 |
|
- type: dot_f1 |
|
value: 92.94354838709677 |
|
- type: dot_precision |
|
value: 93.69918699186992 |
|
- type: dot_recall |
|
value: 92.2 |
|
- type: euclidean_accuracy |
|
value: 99.86138613861387 |
|
- type: euclidean_ap |
|
value: 96.78517996018759 |
|
- type: euclidean_f1 |
|
value: 92.94354838709677 |
|
- type: euclidean_precision |
|
value: 93.69918699186992 |
|
- type: euclidean_recall |
|
value: 92.2 |
|
- type: manhattan_accuracy |
|
value: 99.86336633663366 |
|
- type: manhattan_ap |
|
value: 96.79790073128503 |
|
- type: manhattan_f1 |
|
value: 93.0930930930931 |
|
- type: manhattan_precision |
|
value: 93.18637274549098 |
|
- type: manhattan_recall |
|
value: 93.0 |
|
- type: max_accuracy |
|
value: 99.86336633663366 |
|
- type: max_ap |
|
value: 96.79790073128503 |
|
- type: max_f1 |
|
value: 93.0930930930931 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering |
|
name: MTEB StackExchangeClustering |
|
config: default |
|
split: test |
|
revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
|
metrics: |
|
- type: v_measure |
|
value: 65.07696952556874 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering-p2p |
|
name: MTEB StackExchangeClusteringP2P |
|
config: default |
|
split: test |
|
revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
|
metrics: |
|
- type: v_measure |
|
value: 35.51701116515262 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/stackoverflowdupquestions-reranking |
|
name: MTEB StackOverflowDupQuestions |
|
config: default |
|
split: test |
|
revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
|
metrics: |
|
- type: map |
|
value: 55.40099299306496 |
|
- type: mrr |
|
value: 56.411316420507596 |
|
- task: |
|
type: Summarization |
|
dataset: |
|
type: mteb/summeval |
|
name: MTEB SummEval |
|
config: default |
|
split: test |
|
revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 30.940008734510055 |
|
- type: cos_sim_spearman |
|
value: 31.606997026865212 |
|
- type: dot_pearson |
|
value: 30.940010256206353 |
|
- type: dot_spearman |
|
value: 31.62194110302714 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: trec-covid |
|
name: MTEB TRECCOVID |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.197 |
|
- type: map_at_10 |
|
value: 1.6549999999999998 |
|
- type: map_at_100 |
|
value: 8.939 |
|
- type: map_at_1000 |
|
value: 22.402 |
|
- type: map_at_3 |
|
value: 0.587 |
|
- type: map_at_5 |
|
value: 0.931 |
|
- type: mrr_at_1 |
|
value: 74.0 |
|
- type: mrr_at_10 |
|
value: 84.667 |
|
- type: mrr_at_100 |
|
value: 84.667 |
|
- type: mrr_at_1000 |
|
value: 84.667 |
|
- type: mrr_at_3 |
|
value: 83.667 |
|
- type: mrr_at_5 |
|
value: 84.667 |
|
- type: ndcg_at_1 |
|
value: 69.0 |
|
- type: ndcg_at_10 |
|
value: 66.574 |
|
- type: ndcg_at_100 |
|
value: 51.074 |
|
- type: ndcg_at_1000 |
|
value: 47.263 |
|
- type: ndcg_at_3 |
|
value: 71.95 |
|
- type: ndcg_at_5 |
|
value: 70.52000000000001 |
|
- type: precision_at_1 |
|
value: 74.0 |
|
- type: precision_at_10 |
|
value: 70.39999999999999 |
|
- type: precision_at_100 |
|
value: 52.580000000000005 |
|
- type: precision_at_1000 |
|
value: 20.93 |
|
- type: precision_at_3 |
|
value: 76.667 |
|
- type: precision_at_5 |
|
value: 75.6 |
|
- type: recall_at_1 |
|
value: 0.197 |
|
- type: recall_at_10 |
|
value: 1.92 |
|
- type: recall_at_100 |
|
value: 12.655 |
|
- type: recall_at_1000 |
|
value: 44.522 |
|
- type: recall_at_3 |
|
value: 0.639 |
|
- type: recall_at_5 |
|
value: 1.03 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: webis-touche2020 |
|
name: MTEB Touche2020 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 1.735 |
|
- type: map_at_10 |
|
value: 9.064 |
|
- type: map_at_100 |
|
value: 15.021999999999998 |
|
- type: map_at_1000 |
|
value: 16.596 |
|
- type: map_at_3 |
|
value: 4.188 |
|
- type: map_at_5 |
|
value: 6.194999999999999 |
|
- type: mrr_at_1 |
|
value: 26.531 |
|
- type: mrr_at_10 |
|
value: 44.413000000000004 |
|
- type: mrr_at_100 |
|
value: 45.433 |
|
- type: mrr_at_1000 |
|
value: 45.452999999999996 |
|
- type: mrr_at_3 |
|
value: 41.497 |
|
- type: mrr_at_5 |
|
value: 42.925000000000004 |
|
- type: ndcg_at_1 |
|
value: 22.448999999999998 |
|
- type: ndcg_at_10 |
|
value: 22.597 |
|
- type: ndcg_at_100 |
|
value: 34.893 |
|
- type: ndcg_at_1000 |
|
value: 46.763 |
|
- type: ndcg_at_3 |
|
value: 24.366 |
|
- type: ndcg_at_5 |
|
value: 23.959 |
|
- type: precision_at_1 |
|
value: 26.531 |
|
- type: precision_at_10 |
|
value: 21.02 |
|
- type: precision_at_100 |
|
value: 7.51 |
|
- type: precision_at_1000 |
|
value: 1.541 |
|
- type: precision_at_3 |
|
value: 27.211000000000002 |
|
- type: precision_at_5 |
|
value: 25.306 |
|
- type: recall_at_1 |
|
value: 1.735 |
|
- type: recall_at_10 |
|
value: 15.870999999999999 |
|
- type: recall_at_100 |
|
value: 47.385 |
|
- type: recall_at_1000 |
|
value: 83.55 |
|
- type: recall_at_3 |
|
value: 5.813 |
|
- type: recall_at_5 |
|
value: 9.707 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/toxic_conversations_50k |
|
name: MTEB ToxicConversationsClassification |
|
config: default |
|
split: test |
|
revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
|
metrics: |
|
- type: accuracy |
|
value: 71.19 |
|
- type: ap |
|
value: 15.106812062408629 |
|
- type: f1 |
|
value: 55.254852511954255 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/tweet_sentiment_extraction |
|
name: MTEB TweetSentimentExtractionClassification |
|
config: default |
|
split: test |
|
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
|
metrics: |
|
- type: accuracy |
|
value: 61.553480475382 |
|
- type: f1 |
|
value: 61.697424438626435 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/twentynewsgroups-clustering |
|
name: MTEB TwentyNewsgroupsClustering |
|
config: default |
|
split: test |
|
revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
|
metrics: |
|
- type: v_measure |
|
value: 53.12092298453447 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twittersemeval2015-pairclassification |
|
name: MTEB TwitterSemEval2015 |
|
config: default |
|
split: test |
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 87.35173153722357 |
|
- type: cos_sim_ap |
|
value: 78.22985044080261 |
|
- type: cos_sim_f1 |
|
value: 71.23356926188069 |
|
- type: cos_sim_precision |
|
value: 68.36487142163999 |
|
- type: cos_sim_recall |
|
value: 74.35356200527704 |
|
- type: dot_accuracy |
|
value: 87.35173153722357 |
|
- type: dot_ap |
|
value: 78.22985958574529 |
|
- type: dot_f1 |
|
value: 71.23356926188069 |
|
- type: dot_precision |
|
value: 68.36487142163999 |
|
- type: dot_recall |
|
value: 74.35356200527704 |
|
- type: euclidean_accuracy |
|
value: 87.35173153722357 |
|
- type: euclidean_ap |
|
value: 78.22985909816191 |
|
- type: euclidean_f1 |
|
value: 71.23356926188069 |
|
- type: euclidean_precision |
|
value: 68.36487142163999 |
|
- type: euclidean_recall |
|
value: 74.35356200527704 |
|
- type: manhattan_accuracy |
|
value: 87.36365261965786 |
|
- type: manhattan_ap |
|
value: 78.18108280854142 |
|
- type: manhattan_f1 |
|
value: 71.19958634953466 |
|
- type: manhattan_precision |
|
value: 69.79219462747086 |
|
- type: manhattan_recall |
|
value: 72.66490765171504 |
|
- type: max_accuracy |
|
value: 87.36365261965786 |
|
- type: max_ap |
|
value: 78.22985958574529 |
|
- type: max_f1 |
|
value: 71.23356926188069 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twitterurlcorpus-pairclassification |
|
name: MTEB TwitterURLCorpus |
|
config: default |
|
split: test |
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 88.71424690495596 |
|
- type: cos_sim_ap |
|
value: 85.53000600450122 |
|
- type: cos_sim_f1 |
|
value: 77.95508274231679 |
|
- type: cos_sim_precision |
|
value: 74.92189718829879 |
|
- type: cos_sim_recall |
|
value: 81.24422543886665 |
|
- type: dot_accuracy |
|
value: 88.71424690495596 |
|
- type: dot_ap |
|
value: 85.53000387261983 |
|
- type: dot_f1 |
|
value: 77.95508274231679 |
|
- type: dot_precision |
|
value: 74.92189718829879 |
|
- type: dot_recall |
|
value: 81.24422543886665 |
|
- type: euclidean_accuracy |
|
value: 88.71424690495596 |
|
- type: euclidean_ap |
|
value: 85.53000527321076 |
|
- type: euclidean_f1 |
|
value: 77.95508274231679 |
|
- type: euclidean_precision |
|
value: 74.92189718829879 |
|
- type: euclidean_recall |
|
value: 81.24422543886665 |
|
- type: manhattan_accuracy |
|
value: 88.7297706368611 |
|
- type: manhattan_ap |
|
value: 85.49670114967172 |
|
- type: manhattan_f1 |
|
value: 77.91265729089562 |
|
- type: manhattan_precision |
|
value: 75.01425313568986 |
|
- type: manhattan_recall |
|
value: 81.04404065291038 |
|
- type: max_accuracy |
|
value: 88.7297706368611 |
|
- type: max_ap |
|
value: 85.53000600450122 |
|
- type: max_f1 |
|
value: 77.95508274231679 |
|
--- |
|
|
|
# {MODEL_NAME} |
|
|
|
This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search. |
|
|
|
<!--- Describe your model here --> |
|
|
|
## Usage (Sentence-Transformers) |
|
|
|
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: |
|
|
|
``` |
|
pip install -U sentence-transformers |
|
``` |
|
|
|
Then you can use the model like this: |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
sentences = ["This is an example sentence", "Each sentence is converted"] |
|
|
|
model = SentenceTransformer('{MODEL_NAME}') |
|
embeddings = model.encode(sentences) |
|
print(embeddings) |
|
``` |
|
|
|
|
|
|
|
## Evaluation Results |
|
|
|
<!--- Describe how your model was evaluated --> |
|
|
|
For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME}) |
|
|
|
|
|
## Training |
|
The model was trained with the parameters: |
|
|
|
**DataLoader**: |
|
|
|
`torch.utils.data.dataloader.DataLoader` of length 1196 with parameters: |
|
``` |
|
{'batch_size': 10, 'sampler': 'torch.utils.data.sampler.SequentialSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'} |
|
``` |
|
|
|
**Loss**: |
|
|
|
`sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss` with parameters: |
|
``` |
|
{'scale': 20.0, 'similarity_fct': 'cos_sim'} |
|
``` |
|
|
|
Parameters of the fit()-Method: |
|
``` |
|
{ |
|
"epochs": 5, |
|
"evaluation_steps": 50, |
|
"evaluator": "sentence_transformers.evaluation.InformationRetrievalEvaluator.InformationRetrievalEvaluator", |
|
"max_grad_norm": 1, |
|
"optimizer_class": "<class 'torch.optim.adamw.AdamW'>", |
|
"optimizer_params": { |
|
"lr": 2e-05 |
|
}, |
|
"scheduler": "WarmupLinear", |
|
"steps_per_epoch": null, |
|
"warmup_steps": 598, |
|
"weight_decay": 0.01 |
|
} |
|
``` |
|
|
|
|
|
## Full Model Architecture |
|
``` |
|
SentenceTransformer( |
|
(0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel |
|
(1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False}) |
|
(2): Normalize() |
|
) |
|
``` |
|
|
|
## Citing & Authors |
|
|
|
<!--- Describe where people can find more information --> |
