--- tags: - mteb model-index: - name: sf_model_e5 results: - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en) config: en split: test revision: e8379541af4e31359cca9fbcf4b00f2671dba205 metrics: - type: accuracy value: 70.85074626865672 - type: ap value: 33.779217850079206 - type: f1 value: 64.96977487239377 - task: type: Classification dataset: type: mteb/amazon_polarity name: MTEB AmazonPolarityClassification config: default split: test revision: e2d317d38cd51312af73b3d32a06d1a08b442046 metrics: - type: accuracy value: 91.80945 - type: ap value: 88.22978189506895 - type: f1 value: 91.7858219911604 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (en) config: en split: test revision: 1399c76144fd37290681b995c656ef9b2e06e26d metrics: - type: accuracy value: 48.94200000000001 - type: f1 value: 47.911934405973895 - task: type: Retrieval dataset: type: arguana name: MTEB ArguAna config: default split: test revision: None metrics: - type: map_at_1 value: 39.616 - type: map_at_10 value: 55.938 - type: map_at_100 value: 56.552 - type: map_at_1000 value: 56.556 - type: map_at_3 value: 51.754 - type: map_at_5 value: 54.623999999999995 - type: mrr_at_1 value: 40.967 - type: mrr_at_10 value: 56.452999999999996 - type: mrr_at_100 value: 57.053 - type: mrr_at_1000 value: 57.057 - type: mrr_at_3 value: 52.312000000000005 - type: mrr_at_5 value: 55.1 - type: ndcg_at_1 value: 39.616 - type: ndcg_at_10 value: 64.067 - type: ndcg_at_100 value: 66.384 - type: ndcg_at_1000 value: 66.468 - type: ndcg_at_3 value: 55.74 - type: ndcg_at_5 value: 60.889 - type: precision_at_1 value: 39.616 - type: precision_at_10 value: 8.953999999999999 - type: precision_at_100 value: 0.9900000000000001 - type: precision_at_1000 value: 0.1 - type: precision_at_3 value: 22.428 - type: precision_at_5 value: 15.946 - type: recall_at_1 value: 39.616 - type: recall_at_10 value: 89.545 - type: recall_at_100 value: 99.004 - type: recall_at_1000 value: 99.644 - type: recall_at_3 value: 67.283 - type: recall_at_5 value: 79.73 - task: type: Clustering dataset: type: mteb/arxiv-clustering-p2p name: MTEB ArxivClusteringP2P config: default split: test revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d metrics: - type: v_measure value: 48.72923923743124 - task: type: Clustering dataset: type: mteb/arxiv-clustering-s2s name: MTEB ArxivClusteringS2S config: default split: test revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 metrics: - type: v_measure value: 42.87449955203238 - task: type: Reranking dataset: type: mteb/askubuntudupquestions-reranking name: MTEB AskUbuntuDupQuestions config: default split: test revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 metrics: - type: map value: 64.3214434754065 - type: mrr value: 77.87879787187265 - task: type: STS dataset: type: mteb/biosses-sts name: MTEB BIOSSES config: default split: test revision: d3fb88f8f02e40887cd149695127462bbcf29b4a metrics: - type: cos_sim_pearson value: 88.82418607751953 - type: cos_sim_spearman value: 86.74535004562274 - type: euclidean_pearson value: 86.58792166831103 - type: euclidean_spearman value: 86.74535004562274 - type: manhattan_pearson value: 86.23957813056677 - type: manhattan_spearman value: 86.41522204150452 - task: type: Classification dataset: type: mteb/banking77 name: MTEB Banking77Classification config: default split: test revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 metrics: - type: accuracy value: 84.61363636363636 - type: f1 value: 83.98373241136187 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-p2p name: MTEB BiorxivClusteringP2P config: default split: test revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 metrics: - type: v_measure value: 39.73148995791471 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-s2s name: MTEB BiorxivClusteringS2S config: default split: test revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 metrics: - type: v_measure value: 37.23723038699733 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackAndroidRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 32.217 - type: map_at_10 value: 43.453 - type: map_at_100 value: 45.038 - type: map_at_1000 value: 45.162 - type: map_at_3 value: 39.589 - type: map_at_5 value: 41.697 - type: mrr_at_1 value: 39.628 - type: mrr_at_10 value: 49.698 - type: mrr_at_100 value: 50.44 - type: mrr_at_1000 value: 50.482000000000006 - type: mrr_at_3 value: 46.781 - type: mrr_at_5 value: 48.548 - type: ndcg_at_1 value: 39.628 - type: ndcg_at_10 value: 50.158 - type: ndcg_at_100 value: 55.687 - type: ndcg_at_1000 value: 57.499 - type: ndcg_at_3 value: 44.594 - type: ndcg_at_5 value: 47.198 - type: precision_at_1 value: 39.628 - type: precision_at_10 value: 9.828000000000001 - type: precision_at_100 value: 1.591 - type: precision_at_1000 value: 0.20600000000000002 - type: precision_at_3 value: 21.507 - type: precision_at_5 value: 15.765 - type: recall_at_1 value: 32.217 - type: recall_at_10 value: 62.717999999999996 - type: recall_at_100 value: 85.992 - type: recall_at_1000 value: 97.271 - type: recall_at_3 value: 46.694 - type: recall_at_5 value: 53.952 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackEnglishRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 30.862000000000002 - type: map_at_10 value: 41.287 - type: map_at_100 value: 42.526 - type: map_at_1000 value: 42.653999999999996 - type: map_at_3 value: 38.055 - type: map_at_5 value: 40.022000000000006 - type: mrr_at_1 value: 38.408 - type: mrr_at_10 value: 46.943 - type: mrr_at_100 value: 47.597 - type: mrr_at_1000 value: 47.64 - type: mrr_at_3 value: 44.607 - type: mrr_at_5 value: 46.079 - type: ndcg_at_1 value: 38.408 - type: ndcg_at_10 value: 46.936 - type: ndcg_at_100 value: 51.307 - type: ndcg_at_1000 value: 53.312000000000005 - type: ndcg_at_3 value: 42.579 - type: ndcg_at_5 value: 44.877 - type: precision_at_1 value: 38.408 - type: precision_at_10 value: 8.885 - type: precision_at_100 value: 1.4449999999999998 - type: precision_at_1000 value: 0.192 - type: precision_at_3 value: 20.616 - type: precision_at_5 value: 14.841 - type: recall_at_1 value: 30.862000000000002 - type: recall_at_10 value: 56.994 - type: recall_at_100 value: 75.347 - type: recall_at_1000 value: 87.911 - type: recall_at_3 value: 44.230000000000004 - type: recall_at_5 value: 50.625 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGamingRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 39.076 - type: map_at_10 value: 52.535 - type: map_at_100 value: 53.537 - type: map_at_1000 value: 53.591 - type: map_at_3 value: 48.961 - type: map_at_5 value: 50.96000000000001 - type: mrr_at_1 value: 44.765 - type: mrr_at_10 value: 55.615 - type: mrr_at_100 value: 56.24 - type: mrr_at_1000 value: 56.264 - type: mrr_at_3 value: 52.925999999999995 - type: mrr_at_5 value: 54.493 - type: ndcg_at_1 value: 44.765 - type: ndcg_at_10 value: 58.777 - type: ndcg_at_100 value: 62.574 - type: ndcg_at_1000 value: 63.624 - type: ndcg_at_3 value: 52.81 - type: ndcg_at_5 value: 55.657999999999994 - type: precision_at_1 value: 44.765 - type: precision_at_10 value: 9.693 - type: precision_at_100 value: 1.248 - type: precision_at_1000 value: 0.13799999999999998 - type: precision_at_3 value: 23.866 - type: precision_at_5 value: 16.489 - type: recall_at_1 value: 39.076 - type: recall_at_10 value: 74.01299999999999 - type: recall_at_100 value: 90.363 - type: recall_at_1000 value: 97.782 - type: recall_at_3 value: 58.056 - type: recall_at_5 value: 65.029 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGisRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 26.357000000000003 - type: map_at_10 value: 35.492000000000004 - type: map_at_100 value: 36.504999999999995 - type: map_at_1000 value: 36.578 - type: map_at_3 value: 32.696999999999996 - type: map_at_5 value: 34.388999999999996 - type: mrr_at_1 value: 28.136 - type: mrr_at_10 value: 37.383 - type: mrr_at_100 value: 38.271 - type: mrr_at_1000 value: 38.324999999999996 - type: mrr_at_3 value: 34.782999999999994 - type: mrr_at_5 value: 36.416 - type: ndcg_at_1 value: 28.136 - type: ndcg_at_10 value: 40.741 - type: ndcg_at_100 value: 45.803 - type: ndcg_at_1000 value: 47.637 - type: ndcg_at_3 value: 35.412 - type: ndcg_at_5 value: 38.251000000000005 - type: precision_at_1 value: 28.136 - type: precision_at_10 value: 6.315999999999999 - type: precision_at_100 value: 0.931 - type: precision_at_1000 value: 0.11299999999999999 - type: precision_at_3 value: 15.254000000000001 - type: precision_at_5 value: 10.757 - type: recall_at_1 value: 26.357000000000003 - type: recall_at_10 value: 55.021 - type: recall_at_100 value: 78.501 - type: recall_at_1000 value: 92.133 - type: recall_at_3 value: 40.798 - type: recall_at_5 value: 47.591 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackMathematicaRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 17.302 - type: map_at_10 value: 26.365 - type: map_at_100 value: 27.581 - type: map_at_1000 value: 27.705999999999996 - type: map_at_3 value: 23.682 - type: map_at_5 value: 25.304 - type: mrr_at_1 value: 21.891 - type: mrr_at_10 value: 31.227 - type: mrr_at_100 value: 32.22 - type: mrr_at_1000 value: 32.282 - type: mrr_at_3 value: 28.711 - type: mrr_at_5 value: 30.314999999999998 - type: ndcg_at_1 value: 21.891 - type: ndcg_at_10 value: 31.965 - type: ndcg_at_100 value: 37.869 - type: ndcg_at_1000 value: 40.642 - type: ndcg_at_3 value: 27.184 - type: ndcg_at_5 value: 29.686 - type: precision_at_1 value: 21.891 - type: precision_at_10 value: 5.9830000000000005 - type: precision_at_100 value: 1.0250000000000001 - type: precision_at_1000 value: 0.14100000000000001 - type: precision_at_3 value: 13.391 - type: precision_at_5 value: 9.801 - type: recall_at_1 value: 17.302 - type: recall_at_10 value: 44.312000000000005 - type: recall_at_100 value: 70.274 - type: recall_at_1000 value: 89.709 - type: recall_at_3 value: 31.117 - type: recall_at_5 value: 37.511 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackPhysicsRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 29.404000000000003 - type: map_at_10 value: 40.571 - type: map_at_100 value: 42.049 - type: map_at_1000 value: 42.156 - type: map_at_3 value: 37.413000000000004 - type: map_at_5 value: 39.206 - type: mrr_at_1 value: 36.285000000000004 - type: mrr_at_10 value: 46.213 - type: mrr_at_100 value: 47.129 - type: mrr_at_1000 value: 47.168 - type: mrr_at_3 value: 43.84 - type: mrr_at_5 value: 45.226 - type: ndcg_at_1 value: 36.285000000000004 - type: ndcg_at_10 value: 46.809 - type: ndcg_at_100 value: 52.615 - type: ndcg_at_1000 value: 54.538 - type: ndcg_at_3 value: 41.91 - type: ndcg_at_5 value: 44.224999999999994 - type: precision_at_1 value: 36.285000000000004 - type: precision_at_10 value: 8.527 - type: precision_at_100 value: 1.3259999999999998 - type: precision_at_1000 value: 0.167 - type: precision_at_3 value: 20.083000000000002 - type: precision_at_5 value: 14.071 - type: recall_at_1 value: 29.404000000000003 - type: recall_at_10 value: 59.611999999999995 - type: recall_at_100 value: 83.383 - type: recall_at_1000 value: 95.703 - type: recall_at_3 value: 45.663 - type: recall_at_5 value: 51.971999999999994 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackProgrammersRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 25.317 - type: map_at_10 value: 35.217999999999996 - type: map_at_100 value: 36.665 - type: map_at_1000 value: 36.768 - type: map_at_3 value: 31.924000000000003 - type: map_at_5 value: 33.591 - type: mrr_at_1 value: 31.507 - type: mrr_at_10 value: 40.671 - type: mrr_at_100 value: 41.609 - type: mrr_at_1000 value: 41.657 - type: mrr_at_3 value: 38.261 - type: mrr_at_5 value: 39.431 - type: ndcg_at_1 value: 31.507 - type: ndcg_at_10 value: 41.375 - type: ndcg_at_100 value: 47.426 - type: ndcg_at_1000 value: 49.504 - type: ndcg_at_3 value: 35.989 - type: ndcg_at_5 value: 38.068000000000005 - type: precision_at_1 value: 31.507 - type: precision_at_10 value: 7.8420000000000005 - type: precision_at_100 value: 1.257 - type: precision_at_1000 value: 0.16199999999999998 - type: precision_at_3 value: 17.352 - type: precision_at_5 value: 12.328999999999999 - type: recall_at_1 value: 25.317 - type: recall_at_10 value: 54.254999999999995 - type: recall_at_100 value: 80.184 - type: recall_at_1000 value: 94.07 - type: recall_at_3 value: 39.117000000000004 - type: recall_at_5 value: 44.711 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 25.813000000000002 - type: map_at_10 value: 35.47183333333334 - type: map_at_100 value: 36.71775 - type: map_at_1000 value: 36.833000000000006 - type: map_at_3 value: 32.449916666666674 - type: map_at_5 value: 34.1235 - type: mrr_at_1 value: 30.766750000000005 - type: mrr_at_10 value: 39.77508333333334 - type: mrr_at_100 value: 40.64233333333333 - type: mrr_at_1000 value: 40.69658333333333 - type: mrr_at_3 value: 37.27349999999999 - type: mrr_at_5 value: 38.723416666666665 - type: ndcg_at_1 value: 30.766750000000005 - type: ndcg_at_10 value: 41.141416666666665 - type: ndcg_at_100 value: 46.42016666666666 - type: ndcg_at_1000 value: 48.61916666666667 - type: ndcg_at_3 value: 36.06883333333333 - type: ndcg_at_5 value: 38.43966666666666 - type: precision_at_1 value: 30.766750000000005 - type: precision_at_10 value: 7.340000000000001 - type: precision_at_100 value: 1.1796666666666666 - type: precision_at_1000 value: 0.15625 - type: precision_at_3 value: 16.763833333333334 - type: precision_at_5 value: 11.972166666666666 - type: recall_at_1 value: 25.813000000000002 - type: recall_at_10 value: 53.62741666666667 - type: recall_at_100 value: 76.70125000000002 - type: recall_at_1000 value: 91.85566666666666 - type: recall_at_3 value: 39.55075 - type: recall_at_5 value: 45.645250000000004 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackStatsRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 23.249 - type: map_at_10 value: 31.095 - type: map_at_100 value: 32.056000000000004 - type: map_at_1000 value: 32.163000000000004 - type: map_at_3 value: 29.275000000000002 - type: map_at_5 value: 30.333 - type: mrr_at_1 value: 26.687 - type: mrr_at_10 value: 34.122 - type: mrr_at_100 value: 34.958 - type: mrr_at_1000 value: 35.039 - type: mrr_at_3 value: 32.541 - type: mrr_at_5 value: 33.43 - type: ndcg_at_1 value: 26.687 - type: ndcg_at_10 value: 35.248000000000005 - type: ndcg_at_100 value: 39.933 - type: ndcg_at_1000 value: 42.616 - type: ndcg_at_3 value: 31.980999999999998 - type: ndcg_at_5 value: 33.583 - type: precision_at_1 value: 26.687 - type: precision_at_10 value: 5.445 - type: precision_at_100 value: 0.848 - type: precision_at_1000 value: 0.11499999999999999 - type: precision_at_3 value: 13.957 - type: precision_at_5 value: 9.479 - type: recall_at_1 value: 23.249 - type: recall_at_10 value: 45.005 - type: recall_at_100 value: 66.175 - type: recall_at_1000 value: 86.116 - type: recall_at_3 value: 36.03 - type: recall_at_5 value: 40.037 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackTexRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 17.592 - type: map_at_10 value: 25.003999999999998 - type: map_at_100 value: 26.208 - type: map_at_1000 value: 26.333000000000002 - type: map_at_3 value: 22.479 - type: map_at_5 value: 23.712 - type: mrr_at_1 value: 21.37 - type: mrr_at_10 value: 28.951999999999998 - type: mrr_at_100 value: 29.915999999999997 - type: mrr_at_1000 value: 29.99 - type: mrr_at_3 value: 26.503 - type: mrr_at_5 value: 27.728 - type: ndcg_at_1 value: 21.37 - type: ndcg_at_10 value: 29.944 - type: ndcg_at_100 value: 35.632000000000005 - type: ndcg_at_1000 value: 38.393 - type: ndcg_at_3 value: 25.263999999999996 - type: ndcg_at_5 value: 27.115000000000002 - type: precision_at_1 value: 21.37 - type: precision_at_10 value: 5.568 - type: precision_at_100 value: 0.992 - type: precision_at_1000 value: 0.13999999999999999 - type: precision_at_3 value: 11.895 - type: precision_at_5 value: 8.61 - type: recall_at_1 value: 17.592 - type: recall_at_10 value: 40.976 - type: recall_at_100 value: 66.487 - type: recall_at_1000 value: 85.954 - type: recall_at_3 value: 27.797 - type: recall_at_5 value: 32.553 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackUnixRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 25.173000000000002 - type: map_at_10 value: 34.611999999999995 - type: map_at_100 value: 35.735 - type: map_at_1000 value: 35.842 - type: map_at_3 value: 31.345 - type: map_at_5 value: 33.123000000000005 - type: mrr_at_1 value: 29.570999999999998 - type: mrr_at_10 value: 38.775999999999996 - type: mrr_at_100 value: 39.621 - type: mrr_at_1000 value: 39.684000000000005 - type: mrr_at_3 value: 35.992000000000004 - type: mrr_at_5 value: 37.586999999999996 - type: ndcg_at_1 value: 29.570999999999998 - type: ndcg_at_10 value: 40.388000000000005 - type: ndcg_at_100 value: 45.59 - type: ndcg_at_1000 value: 47.948 - type: ndcg_at_3 value: 34.497 - type: ndcg_at_5 value: 37.201 - type: precision_at_1 value: 29.570999999999998 - type: precision_at_10 value: 6.931 - type: precision_at_100 value: 1.082 - type: precision_at_1000 value: 0.13999999999999999 - type: precision_at_3 value: 15.609 - type: precision_at_5 value: 11.286999999999999 - type: recall_at_1 value: 25.173000000000002 - type: recall_at_10 value: 53.949000000000005 - type: recall_at_100 value: 76.536 - type: recall_at_1000 value: 92.979 - type: recall_at_3 value: 37.987 - type: recall_at_5 value: 44.689 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWebmastersRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 24.224 - type: map_at_10 value: 32.903 - type: map_at_100 value: 34.65 - type: map_at_1000 value: 34.873 - type: map_at_3 value: 29.673 - type: map_at_5 value: 31.361 - type: mrr_at_1 value: 30.435000000000002 - type: mrr_at_10 value: 38.677 - type: mrr_at_100 value: 39.805 - type: mrr_at_1000 value: 39.851 - type: mrr_at_3 value: 35.935 - type: mrr_at_5 value: 37.566 - type: ndcg_at_1 value: 30.435000000000002 - type: ndcg_at_10 value: 39.012 - type: ndcg_at_100 value: 45.553 - type: ndcg_at_1000 value: 47.919 - type: ndcg_at_3 value: 33.809 - type: ndcg_at_5 value: 36.120999999999995 - type: precision_at_1 value: 30.435000000000002 - type: precision_at_10 value: 7.628 - type: precision_at_100 value: 1.5810000000000002 - type: precision_at_1000 value: 0.243 - type: precision_at_3 value: 15.744 - type: precision_at_5 value: 11.66 - type: recall_at_1 value: 24.224 - type: recall_at_10 value: 50.009 - type: recall_at_100 value: 78.839 - type: recall_at_1000 value: 93.71300000000001 - type: recall_at_3 value: 35.512 - type: recall_at_5 value: 41.541 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWordpressRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 18.983 - type: map_at_10 value: 27.127000000000002 - type: map_at_100 value: 28.063 - type: map_at_1000 value: 28.17 - type: map_at_3 value: 24.306 - type: map_at_5 value: 25.784000000000002 - type: mrr_at_1 value: 20.518 - type: mrr_at_10 value: 29.024 - type: mrr_at_100 value: 29.902 - type: mrr_at_1000 value: 29.976999999999997 - type: mrr_at_3 value: 26.401999999999997 - type: mrr_at_5 value: 27.862 - type: ndcg_at_1 value: 20.518 - type: ndcg_at_10 value: 32.344 - type: ndcg_at_100 value: 37.053000000000004 - type: ndcg_at_1000 value: 39.798 - type: ndcg_at_3 value: 26.796999999999997 - type: ndcg_at_5 value: 29.293000000000003 - type: precision_at_1 value: 20.518 - type: precision_at_10 value: 5.434 - type: precision_at_100 value: 0.83 - type: precision_at_1000 value: 0.11800000000000001 - type: precision_at_3 value: 11.892 - type: precision_at_5 value: 8.577 - type: recall_at_1 value: 18.983 - type: recall_at_10 value: 46.665 - type: recall_at_100 value: 68.33399999999999 - type: recall_at_1000 value: 88.927 - type: recall_at_3 value: 31.608000000000004 - type: recall_at_5 value: 37.532 - task: type: Retrieval dataset: type: climate-fever name: MTEB ClimateFEVER config: default split: test revision: None metrics: - type: map_at_1 value: 11.200000000000001 - type: map_at_10 value: 20.241999999999997 - type: map_at_100 value: 22.357 - type: map_at_1000 value: 22.556 - type: map_at_3 value: 16.564999999999998 - type: map_at_5 value: 18.443 - type: mrr_at_1 value: 25.277 - type: mrr_at_10 value: 37.582 - type: mrr_at_100 value: 38.525999999999996 - type: mrr_at_1000 value: 38.564 - type: mrr_at_3 value: 33.898 - type: mrr_at_5 value: 36.191 - type: ndcg_at_1 value: 25.277 - type: ndcg_at_10 value: 28.74 - type: ndcg_at_100 value: 36.665 - type: ndcg_at_1000 value: 40.08 - type: ndcg_at_3 value: 22.888 - type: ndcg_at_5 value: 25.081999999999997 - type: precision_at_1 value: 25.277 - type: precision_at_10 value: 9.251 - type: precision_at_100 value: 1.773 - type: precision_at_1000 value: 0.241 - type: precision_at_3 value: 17.329 - type: precision_at_5 value: 13.746 - type: recall_at_1 value: 11.200000000000001 - type: recall_at_10 value: 35.419 - type: recall_at_100 value: 62.41 - type: recall_at_1000 value: 81.467 - type: recall_at_3 value: 21.275 - type: recall_at_5 value: 27.201999999999998 - task: type: Retrieval dataset: type: dbpedia-entity name: MTEB DBPedia config: default split: test revision: None metrics: - type: map_at_1 value: 9.396 - type: map_at_10 value: 20.735 - type: map_at_100 value: 30.098000000000003 - type: map_at_1000 value: 31.866 - type: map_at_3 value: 14.71 - type: map_at_5 value: 17.259 - type: mrr_at_1 value: 70.25 - type: mrr_at_10 value: 77.09700000000001 - type: mrr_at_100 value: 77.398 - type: mrr_at_1000 value: 77.40899999999999 - type: mrr_at_3 value: 75.542 - type: mrr_at_5 value: 76.354 - type: ndcg_at_1 value: 57.75 - type: ndcg_at_10 value: 42.509 - type: ndcg_at_100 value: 48.94 - type: ndcg_at_1000 value: 56.501000000000005 - type: ndcg_at_3 value: 46.827000000000005 - type: ndcg_at_5 value: 44.033 - type: precision_at_1 value: 70.25 - type: precision_at_10 value: 33.85 - type: precision_at_100 value: 11.373 - type: precision_at_1000 value: 2.136 - type: precision_at_3 value: 50.917 - type: precision_at_5 value: 42.8 - type: recall_at_1 value: 9.396 - type: recall_at_10 value: 26.472 - type: recall_at_100 value: 57.30800000000001 - type: recall_at_1000 value: 80.983 - type: recall_at_3 value: 15.859000000000002 - type: recall_at_5 value: 19.758 - task: type: Classification dataset: type: mteb/emotion name: MTEB EmotionClassification config: default split: test revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 metrics: - type: accuracy value: 54.900000000000006 - type: f1 value: 48.14707395235448 - task: type: Retrieval dataset: type: fever name: MTEB FEVER config: default split: test revision: None metrics: - type: map_at_1 value: 66.369 - type: map_at_10 value: 76.708 - type: map_at_100 value: 76.981 - type: map_at_1000 value: 76.995 - type: map_at_3 value: 75.114 - type: map_at_5 value: 76.116 - type: mrr_at_1 value: 71.557 - type: mrr_at_10 value: 80.95 - type: mrr_at_100 value: 81.075 - type: mrr_at_1000 value: 81.07900000000001 - type: mrr_at_3 value: 79.728 - type: mrr_at_5 value: 80.522 - type: ndcg_at_1 value: 71.557 - type: ndcg_at_10 value: 81.381 - type: ndcg_at_100 value: 82.421 - type: ndcg_at_1000 value: 82.709 - type: ndcg_at_3 value: 78.671 - type: ndcg_at_5 value: 80.17 - type: precision_at_1 value: 71.557 - type: precision_at_10 value: 10.159 - type: precision_at_100 value: 1.089 - type: precision_at_1000 value: 0.11299999999999999 - type: precision_at_3 value: 30.668 - type: precision_at_5 value: 19.337 - type: recall_at_1 value: 66.369 - type: recall_at_10 value: 91.482 - type: recall_at_100 value: 95.848 - type: recall_at_1000 value: 97.749 - type: recall_at_3 value: 84.185 - type: recall_at_5 value: 87.908 - task: type: Retrieval dataset: type: fiqa name: MTEB FiQA2018 config: default split: test revision: None metrics: - type: map_at_1 value: 20.902 - type: map_at_10 value: 34.554 - type: map_at_100 value: 36.632 - type: map_at_1000 value: 36.811 - type: map_at_3 value: 30.264000000000003 - type: map_at_5 value: 32.714999999999996 - type: mrr_at_1 value: 42.13 - type: mrr_at_10 value: 51.224000000000004 - type: mrr_at_100 value: 52.044999999999995 - type: mrr_at_1000 value: 52.075 - type: mrr_at_3 value: 48.842999999999996 - type: mrr_at_5 value: 50.108 - type: ndcg_at_1 value: 42.13 - type: ndcg_at_10 value: 42.643 - type: ndcg_at_100 value: 49.806 - type: ndcg_at_1000 value: 52.583 - type: ndcg_at_3 value: 38.927 - type: ndcg_at_5 value: 40.071 - type: precision_at_1 value: 42.13 - type: precision_at_10 value: 11.928999999999998 - type: precision_at_100 value: 1.931 - type: precision_at_1000 value: 0.243 - type: precision_at_3 value: 26.337 - type: precision_at_5 value: 19.29 - type: recall_at_1 value: 20.902 - type: recall_at_10 value: 49.527 - type: recall_at_100 value: 75.754 - type: recall_at_1000 value: 92.171 - type: recall_at_3 value: 35.024 - type: recall_at_5 value: 41.207 - task: type: Retrieval dataset: type: hotpotqa name: MTEB HotpotQA config: default split: test revision: None metrics: - type: map_at_1 value: 39.831 - type: map_at_10 value: 63.958999999999996 - type: map_at_100 value: 64.869 - type: map_at_1000 value: 64.924 - type: map_at_3 value: 60.25 - type: map_at_5 value: 62.572 - type: mrr_at_1 value: 79.662 - type: mrr_at_10 value: 85.57900000000001 - type: mrr_at_100 value: 85.744 - type: mrr_at_1000 value: 85.748 - type: mrr_at_3 value: 84.718 - type: mrr_at_5 value: 85.312 - type: ndcg_at_1 value: 79.662 - type: ndcg_at_10 value: 72.366 - type: ndcg_at_100 value: 75.42999999999999 - type: ndcg_at_1000 value: 76.469 - type: ndcg_at_3 value: 67.258 - type: ndcg_at_5 value: 70.14099999999999 - type: precision_at_1 value: 79.662 - type: precision_at_10 value: 15.254999999999999 - type: precision_at_100 value: 1.763 - type: precision_at_1000 value: 0.19 - type: precision_at_3 value: 43.358000000000004 - type: precision_at_5 value: 28.288999999999998 - type: recall_at_1 value: 39.831 - type: recall_at_10 value: 76.273 - type: recall_at_100 value: 88.163 - type: recall_at_1000 value: 95.017 - type: recall_at_3 value: 65.037 - type: recall_at_5 value: 70.722 - task: type: Classification dataset: type: mteb/imdb name: MTEB ImdbClassification config: default split: test revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 metrics: - type: accuracy value: 93.13879999999999 - type: ap value: 89.94638859649079 - type: f1 value: 93.13371537570421 - task: type: Retrieval dataset: type: msmarco name: MTEB MSMARCO config: default split: dev revision: None metrics: - type: map_at_1 value: 21.482 - type: map_at_10 value: 33.635999999999996 - type: map_at_100 value: 34.792 - type: map_at_1000 value: 34.839999999999996 - type: map_at_3 value: 29.553 - type: map_at_5 value: 31.892 - type: mrr_at_1 value: 22.076999999999998 - type: mrr_at_10 value: 34.247 - type: mrr_at_100 value: 35.337 - type: mrr_at_1000 value: 35.38 - type: mrr_at_3 value: 30.208000000000002 - type: mrr_at_5 value: 32.554 - type: ndcg_at_1 value: 22.092 - type: ndcg_at_10 value: 40.657 - type: ndcg_at_100 value: 46.251999999999995 - type: ndcg_at_1000 value: 47.466 - type: ndcg_at_3 value: 32.353 - type: ndcg_at_5 value: 36.532 - type: precision_at_1 value: 22.092 - type: precision_at_10 value: 6.5040000000000004 - type: precision_at_100 value: 0.9329999999999999 - type: precision_at_1000 value: 0.104 - type: precision_at_3 value: 13.719999999999999 - type: precision_at_5 value: 10.344000000000001 - type: recall_at_1 value: 21.482 - type: recall_at_10 value: 62.316 - type: recall_at_100 value: 88.283 - type: recall_at_1000 value: 97.554 - type: recall_at_3 value: 39.822 - type: recall_at_5 value: 49.805 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (en) config: en split: test revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf metrics: - type: accuracy value: 93.63657090743274 - type: f1 value: 93.49355466580484 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (en) config: en split: test revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba metrics: - type: accuracy value: 66.01459188326493 - type: f1 value: 48.48386472180784 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (en) config: en split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 73.49024882313383 - type: f1 value: 71.8750196914349 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (en) config: en split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 77.38063214525891 - type: f1 value: 76.87364042122763 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-p2p name: MTEB MedrxivClusteringP2P config: default split: test revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 metrics: - type: v_measure value: 34.30572302322684 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-s2s name: MTEB MedrxivClusteringS2S config: default split: test revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 metrics: - type: v_measure value: 32.18418556367587 - task: type: Reranking dataset: type: mteb/mind_small name: MTEB MindSmallReranking config: default split: test revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 metrics: - type: map value: 32.268707296386154 - type: mrr value: 33.481925531215055 - task: type: Retrieval dataset: type: nfcorpus name: MTEB NFCorpus config: default split: test revision: None metrics: - type: map_at_1 value: 6.586 - type: map_at_10 value: 14.954999999999998 - type: map_at_100 value: 19.03 - type: map_at_1000 value: 20.653 - type: map_at_3 value: 10.859 - type: map_at_5 value: 12.577 - type: mrr_at_1 value: 47.988 - type: mrr_at_10 value: 57.57 - type: mrr_at_100 value: 58.050000000000004 - type: mrr_at_1000 value: 58.083 - type: mrr_at_3 value: 55.212 - type: mrr_at_5 value: 56.713 - type: ndcg_at_1 value: 45.975 - type: ndcg_at_10 value: 38.432 - type: ndcg_at_100 value: 35.287 - type: ndcg_at_1000 value: 44.35 - type: ndcg_at_3 value: 43.077 - type: ndcg_at_5 value: 40.952 - type: precision_at_1 value: 47.368 - type: precision_at_10 value: 28.483000000000004 - type: precision_at_100 value: 8.882 - type: precision_at_1000 value: 2.217 - type: precision_at_3 value: 40.144000000000005 - type: precision_at_5 value: 35.17 - type: recall_at_1 value: 6.586 - type: recall_at_10 value: 19.688 - type: recall_at_100 value: 35.426 - type: recall_at_1000 value: 68.09100000000001 - type: recall_at_3 value: 12.234 - type: recall_at_5 value: 14.937000000000001 - task: type: Retrieval dataset: type: nq name: MTEB NQ config: default split: test revision: None metrics: - type: map_at_1 value: 27.322000000000003 - type: map_at_10 value: 43.224000000000004 - type: map_at_100 value: 44.275999999999996 - type: map_at_1000 value: 44.308 - type: map_at_3 value: 38.239000000000004 - type: map_at_5 value: 41.244 - type: mrr_at_1 value: 31.025000000000002 - type: mrr_at_10 value: 45.635 - type: mrr_at_100 value: 46.425 - type: mrr_at_1000 value: 46.445 - type: mrr_at_3 value: 41.42 - type: mrr_at_5 value: 44.038 - type: ndcg_at_1 value: 30.997000000000003 - type: ndcg_at_10 value: 51.55499999999999 - type: ndcg_at_100 value: 55.964999999999996 - type: ndcg_at_1000 value: 56.657000000000004 - type: ndcg_at_3 value: 42.185 - type: ndcg_at_5 value: 47.229 - type: precision_at_1 value: 30.997000000000003 - type: precision_at_10 value: 8.885 - type: precision_at_100 value: 1.1360000000000001 - type: precision_at_1000 value: 0.12 - type: precision_at_3 value: 19.457 - type: precision_at_5 value: 14.554 - type: recall_at_1 value: 27.322000000000003 - type: recall_at_10 value: 74.59400000000001 - type: recall_at_100 value: 93.699 - type: recall_at_1000 value: 98.76599999999999 - type: recall_at_3 value: 50.43 - type: recall_at_5 value: 62.073 - task: type: Retrieval dataset: type: quora name: MTEB QuoraRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 71.109 - type: map_at_10 value: 85.137 - type: map_at_100 value: 85.759 - type: map_at_1000 value: 85.774 - type: map_at_3 value: 82.25200000000001 - type: map_at_5 value: 84.031 - type: mrr_at_1 value: 82.01 - type: mrr_at_10 value: 87.97 - type: mrr_at_100 value: 88.076 - type: mrr_at_1000 value: 88.076 - type: mrr_at_3 value: 87.06 - type: mrr_at_5 value: 87.694 - type: ndcg_at_1 value: 81.99 - type: ndcg_at_10 value: 88.738 - type: ndcg_at_100 value: 89.928 - type: ndcg_at_1000 value: 90.01400000000001 - type: ndcg_at_3 value: 86.042 - type: ndcg_at_5 value: 87.505 - type: precision_at_1 value: 81.99 - type: precision_at_10 value: 13.468 - type: precision_at_100 value: 1.534 - type: precision_at_1000 value: 0.157 - type: precision_at_3 value: 37.702999999999996 - type: precision_at_5 value: 24.706 - type: recall_at_1 value: 71.109 - type: recall_at_10 value: 95.58 - type: recall_at_100 value: 99.62299999999999 - type: recall_at_1000 value: 99.98899999999999 - type: recall_at_3 value: 87.69 - type: recall_at_5 value: 91.982 - task: type: Clustering dataset: type: mteb/reddit-clustering name: MTEB RedditClustering config: default split: test revision: 24640382cdbf8abc73003fb0fa6d111a705499eb metrics: - type: v_measure value: 59.43361510023748 - task: type: Clustering dataset: type: mteb/reddit-clustering-p2p name: MTEB RedditClusteringP2P config: default split: test revision: 282350215ef01743dc01b456c7f5241fa8937f16 metrics: - type: v_measure value: 64.53582642500159 - task: type: Retrieval dataset: type: scidocs name: MTEB SCIDOCS config: default split: test revision: None metrics: - type: map_at_1 value: 4.2299999999999995 - type: map_at_10 value: 11.802 - type: map_at_100 value: 14.454 - type: map_at_1000 value: 14.865 - type: map_at_3 value: 7.911 - type: map_at_5 value: 9.912 - type: mrr_at_1 value: 21.0 - type: mrr_at_10 value: 32.722 - type: mrr_at_100 value: 33.989000000000004 - type: mrr_at_1000 value: 34.026 - type: mrr_at_3 value: 28.65 - type: mrr_at_5 value: 31.075000000000003 - type: ndcg_at_1 value: 21.0 - type: ndcg_at_10 value: 20.161 - type: ndcg_at_100 value: 30.122 - type: ndcg_at_1000 value: 36.399 - type: ndcg_at_3 value: 17.881 - type: ndcg_at_5 value: 16.439999999999998 - type: precision_at_1 value: 21.0 - type: precision_at_10 value: 10.94 - type: precision_at_100 value: 2.5340000000000003 - type: precision_at_1000 value: 0.402 - type: precision_at_3 value: 17.067 - type: precision_at_5 value: 15.120000000000001 - type: recall_at_1 value: 4.2299999999999995 - type: recall_at_10 value: 22.163 - type: recall_at_100 value: 51.42 - type: recall_at_1000 value: 81.652 - type: recall_at_3 value: 10.353 - type: recall_at_5 value: 15.323 - task: type: STS dataset: type: mteb/sickr-sts name: MTEB SICK-R config: default split: test revision: a6ea5a8cab320b040a23452cc28066d9beae2cee metrics: - type: cos_sim_pearson value: 86.44056731476951 - type: cos_sim_spearman value: 82.32974396072802 - type: euclidean_pearson value: 83.63616080755894 - type: euclidean_spearman value: 82.32974071069209 - type: manhattan_pearson value: 83.64149958303744 - type: manhattan_spearman value: 82.32161014878858 - task: type: STS dataset: type: mteb/sts12-sts name: MTEB STS12 config: default split: test revision: a0d554a64d88156834ff5ae9920b964011b16384 metrics: - type: cos_sim_pearson value: 85.65083720426293 - type: cos_sim_spearman value: 77.60786500521749 - type: euclidean_pearson value: 81.8149634918642 - type: euclidean_spearman value: 77.60637450428892 - type: manhattan_pearson value: 81.83507575657566 - type: manhattan_spearman value: 77.613220311151 - task: type: STS dataset: type: mteb/sts13-sts name: MTEB STS13 config: default split: test revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca metrics: - type: cos_sim_pearson value: 87.35683624595698 - type: cos_sim_spearman value: 87.94550696434106 - type: euclidean_pearson value: 87.50272679030367 - type: euclidean_spearman value: 87.94550696434106 - type: manhattan_pearson value: 87.4759786099497 - type: manhattan_spearman value: 87.90226811166427 - task: type: STS dataset: type: mteb/sts14-sts name: MTEB STS14 config: default split: test revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 metrics: - type: cos_sim_pearson value: 86.27438743391316 - type: cos_sim_spearman value: 83.85378984594779 - type: euclidean_pearson value: 85.25840635223642 - type: euclidean_spearman value: 83.85378983163673 - type: manhattan_pearson value: 85.24936075631025 - type: manhattan_spearman value: 83.85052479958138 - task: type: STS dataset: type: mteb/sts15-sts name: MTEB STS15 config: default split: test revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 metrics: - type: cos_sim_pearson value: 87.4783814521557 - type: cos_sim_spearman value: 88.473284566453 - type: euclidean_pearson value: 87.94757741870404 - type: euclidean_spearman value: 88.47327698999878 - type: manhattan_pearson value: 87.93617414057984 - type: manhattan_spearman value: 88.45889274229359 - task: type: STS dataset: type: mteb/sts16-sts name: MTEB STS16 config: default split: test revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 metrics: - type: cos_sim_pearson value: 84.68359147631057 - type: cos_sim_spearman value: 86.46426572535646 - type: euclidean_pearson value: 85.98303971468599 - type: euclidean_spearman value: 86.46426572535646 - type: manhattan_pearson value: 85.95109710640726 - type: manhattan_spearman value: 86.43282632541583 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-en) config: en-en split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 88.88758959688604 - type: cos_sim_spearman value: 88.70384784133324 - type: euclidean_pearson value: 89.27293800474978 - type: euclidean_spearman value: 88.70384784133324 - type: manhattan_pearson value: 89.41494348093664 - type: manhattan_spearman value: 88.8330050824941 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (en) config: en split: test revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 metrics: - type: cos_sim_pearson value: 67.66759812551814 - type: cos_sim_spearman value: 68.02368115471576 - type: euclidean_pearson value: 69.52859542757353 - type: euclidean_spearman value: 68.02368115471576 - type: manhattan_pearson value: 69.50332399468952 - type: manhattan_spearman value: 67.91228681203849 - task: type: STS dataset: type: mteb/stsbenchmark-sts name: MTEB STSBenchmark config: default split: test revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 metrics: - type: cos_sim_pearson value: 87.75891320010409 - type: cos_sim_spearman value: 88.33063922402347 - type: euclidean_pearson value: 88.02964654543274 - type: euclidean_spearman value: 88.33063922402347 - type: manhattan_pearson value: 88.03029440701458 - type: manhattan_spearman value: 88.3158691488696 - task: type: Reranking dataset: type: mteb/scidocs-reranking name: MTEB SciDocsRR config: default split: test revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab metrics: - type: map value: 87.46897310470844 - type: mrr value: 96.29042072669523 - task: type: Retrieval dataset: type: scifact name: MTEB SciFact config: default split: test revision: None metrics: - type: map_at_1 value: 62.261 - type: map_at_10 value: 71.023 - type: map_at_100 value: 71.5 - type: map_at_1000 value: 71.518 - type: map_at_3 value: 67.857 - type: map_at_5 value: 69.44500000000001 - type: mrr_at_1 value: 65.0 - type: mrr_at_10 value: 72.11 - type: mrr_at_100 value: 72.479 - type: mrr_at_1000 value: 72.49600000000001 - type: mrr_at_3 value: 69.722 - type: mrr_at_5 value: 71.02199999999999 - type: ndcg_at_1 value: 65.0 - type: ndcg_at_10 value: 75.40599999999999 - type: ndcg_at_100 value: 77.41 - type: ndcg_at_1000 value: 77.83200000000001 - type: ndcg_at_3 value: 69.95599999999999 - type: ndcg_at_5 value: 72.296 - type: precision_at_1 value: 65.0 - type: precision_at_10 value: 9.966999999999999 - type: precision_at_100 value: 1.097 - type: precision_at_1000 value: 0.11299999999999999 - type: precision_at_3 value: 26.667 - type: precision_at_5 value: 17.666999999999998 - type: recall_at_1 value: 62.261 - type: recall_at_10 value: 87.822 - type: recall_at_100 value: 96.833 - type: recall_at_1000 value: 100.0 - type: recall_at_3 value: 73.06099999999999 - type: recall_at_5 value: 78.88300000000001 - task: type: PairClassification dataset: type: mteb/sprintduplicatequestions-pairclassification name: MTEB SprintDuplicateQuestions config: default split: test revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 metrics: - type: cos_sim_accuracy value: 99.86138613861387 - type: cos_sim_ap value: 96.7851799601876 - type: cos_sim_f1 value: 92.94354838709677 - type: cos_sim_precision value: 93.69918699186992 - type: cos_sim_recall value: 92.2 - type: dot_accuracy value: 99.86138613861387 - type: dot_ap value: 96.78517996018759 - type: dot_f1 value: 92.94354838709677 - type: dot_precision value: 93.69918699186992 - type: dot_recall value: 92.2 - type: euclidean_accuracy value: 99.86138613861387 - type: euclidean_ap value: 96.78517996018759 - type: euclidean_f1 value: 92.94354838709677 - type: euclidean_precision value: 93.69918699186992 - type: euclidean_recall value: 92.2 - type: manhattan_accuracy value: 99.86336633663366 - type: manhattan_ap value: 96.79790073128503 - type: manhattan_f1 value: 93.0930930930931 - type: manhattan_precision value: 93.18637274549098 - type: manhattan_recall value: 93.0 - type: max_accuracy value: 99.86336633663366 - type: max_ap value: 96.79790073128503 - type: max_f1 value: 93.0930930930931 - task: type: Clustering dataset: type: mteb/stackexchange-clustering name: MTEB StackExchangeClustering config: default split: test revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 metrics: - type: v_measure value: 65.07696952556874 - task: type: Clustering dataset: type: mteb/stackexchange-clustering-p2p name: MTEB StackExchangeClusteringP2P config: default split: test revision: 815ca46b2622cec33ccafc3735d572c266efdb44 metrics: - type: v_measure value: 35.51701116515262 - task: type: Reranking dataset: type: mteb/stackoverflowdupquestions-reranking name: MTEB StackOverflowDupQuestions config: default split: test revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 metrics: - type: map value: 55.40099299306496 - type: mrr value: 56.411316420507596 - task: type: Summarization dataset: type: mteb/summeval name: MTEB SummEval config: default split: test revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c metrics: - type: cos_sim_pearson value: 30.940008734510055 - type: cos_sim_spearman value: 31.606997026865212 - type: dot_pearson value: 30.940010256206353 - type: dot_spearman value: 31.62194110302714 - task: type: Retrieval dataset: type: trec-covid name: MTEB TRECCOVID config: default split: test revision: None metrics: - type: map_at_1 value: 0.197 - type: map_at_10 value: 1.6549999999999998 - type: map_at_100 value: 8.939 - type: map_at_1000 value: 22.402 - type: map_at_3 value: 0.587 - type: map_at_5 value: 0.931 - type: mrr_at_1 value: 74.0 - type: mrr_at_10 value: 84.667 - type: mrr_at_100 value: 84.667 - type: mrr_at_1000 value: 84.667 - type: mrr_at_3 value: 83.667 - type: mrr_at_5 value: 84.667 - type: ndcg_at_1 value: 69.0 - type: ndcg_at_10 value: 66.574 - type: ndcg_at_100 value: 51.074 - type: ndcg_at_1000 value: 47.263 - type: ndcg_at_3 value: 71.95 - type: ndcg_at_5 value: 70.52000000000001 - type: precision_at_1 value: 74.0 - type: precision_at_10 value: 70.39999999999999 - type: precision_at_100 value: 52.580000000000005 - type: precision_at_1000 value: 20.93 - type: precision_at_3 value: 76.667 - type: precision_at_5 value: 75.6 - type: recall_at_1 value: 0.197 - type: recall_at_10 value: 1.92 - type: recall_at_100 value: 12.655 - type: recall_at_1000 value: 44.522 - type: recall_at_3 value: 0.639 - type: recall_at_5 value: 1.03 - task: type: Retrieval dataset: type: webis-touche2020 name: MTEB Touche2020 config: default split: test revision: None metrics: - type: map_at_1 value: 1.735 - type: map_at_10 value: 9.064 - type: map_at_100 value: 15.021999999999998 - type: map_at_1000 value: 16.596 - type: map_at_3 value: 4.188 - type: map_at_5 value: 6.194999999999999 - type: mrr_at_1 value: 26.531 - type: mrr_at_10 value: 44.413000000000004 - type: mrr_at_100 value: 45.433 - type: mrr_at_1000 value: 45.452999999999996 - type: mrr_at_3 value: 41.497 - type: mrr_at_5 value: 42.925000000000004 - type: ndcg_at_1 value: 22.448999999999998 - type: ndcg_at_10 value: 22.597 - type: ndcg_at_100 value: 34.893 - type: ndcg_at_1000 value: 46.763 - type: ndcg_at_3 value: 24.366 - type: ndcg_at_5 value: 23.959 - type: precision_at_1 value: 26.531 - type: precision_at_10 value: 21.02 - type: precision_at_100 value: 7.51 - type: precision_at_1000 value: 1.541 - type: precision_at_3 value: 27.211000000000002 - type: precision_at_5 value: 25.306 - type: recall_at_1 value: 1.735 - type: recall_at_10 value: 15.870999999999999 - type: recall_at_100 value: 47.385 - type: recall_at_1000 value: 83.55 - type: recall_at_3 value: 5.813 - type: recall_at_5 value: 9.707 - task: type: Classification dataset: type: mteb/toxic_conversations_50k name: MTEB ToxicConversationsClassification config: default split: test revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c metrics: - type: accuracy value: 71.19 - type: ap value: 15.106812062408629 - type: f1 value: 55.254852511954255 - task: type: Classification dataset: type: mteb/tweet_sentiment_extraction name: MTEB TweetSentimentExtractionClassification config: default split: test revision: d604517c81ca91fe16a244d1248fc021f9ecee7a metrics: - type: accuracy value: 61.553480475382 - type: f1 value: 61.697424438626435 - task: type: Clustering dataset: type: mteb/twentynewsgroups-clustering name: MTEB TwentyNewsgroupsClustering config: default split: test revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 metrics: - type: v_measure value: 53.12092298453447 - task: type: PairClassification dataset: type: mteb/twittersemeval2015-pairclassification name: MTEB TwitterSemEval2015 config: default split: test revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 metrics: - type: cos_sim_accuracy value: 87.35173153722357 - type: cos_sim_ap value: 78.22985044080261 - type: cos_sim_f1 value: 71.23356926188069 - type: cos_sim_precision value: 68.36487142163999 - type: cos_sim_recall value: 74.35356200527704 - type: dot_accuracy value: 87.35173153722357 - type: dot_ap value: 78.22985958574529 - type: dot_f1 value: 71.23356926188069 - type: dot_precision value: 68.36487142163999 - type: dot_recall value: 74.35356200527704 - type: euclidean_accuracy value: 87.35173153722357 - type: euclidean_ap value: 78.22985909816191 - type: euclidean_f1 value: 71.23356926188069 - type: euclidean_precision value: 68.36487142163999 - type: euclidean_recall value: 74.35356200527704 - type: manhattan_accuracy value: 87.36365261965786 - type: manhattan_ap value: 78.18108280854142 - type: manhattan_f1 value: 71.19958634953466 - type: manhattan_precision value: 69.79219462747086 - type: manhattan_recall value: 72.66490765171504 - type: max_accuracy value: 87.36365261965786 - type: max_ap value: 78.22985958574529 - type: max_f1 value: 71.23356926188069 - task: type: PairClassification dataset: type: mteb/twitterurlcorpus-pairclassification name: MTEB TwitterURLCorpus config: default split: test revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf metrics: - type: cos_sim_accuracy value: 88.71424690495596 - type: cos_sim_ap value: 85.53000600450122 - type: cos_sim_f1 value: 77.95508274231679 - type: cos_sim_precision value: 74.92189718829879 - type: cos_sim_recall value: 81.24422543886665 - type: dot_accuracy value: 88.71424690495596 - type: dot_ap value: 85.53000387261983 - type: dot_f1 value: 77.95508274231679 - type: dot_precision value: 74.92189718829879 - type: dot_recall value: 81.24422543886665 - type: euclidean_accuracy value: 88.71424690495596 - type: euclidean_ap value: 85.53000527321076 - type: euclidean_f1 value: 77.95508274231679 - type: euclidean_precision value: 74.92189718829879 - type: euclidean_recall value: 81.24422543886665 - type: manhattan_accuracy value: 88.7297706368611 - type: manhattan_ap value: 85.49670114967172 - type: manhattan_f1 value: 77.91265729089562 - type: manhattan_precision value: 75.01425313568986 - type: manhattan_recall value: 81.04404065291038 - type: max_accuracy value: 88.7297706368611 - type: max_ap value: 85.53000600450122 - type: max_f1 value: 77.95508274231679 --- # {MODEL_NAME} This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search. ## Usage (Sentence-Transformers) Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: ``` pip install -U sentence-transformers ``` Then you can use the model like this: ```python from sentence_transformers import SentenceTransformer sentences = ["This is an example sentence", "Each sentence is converted"] model = SentenceTransformer('{MODEL_NAME}') embeddings = model.encode(sentences) print(embeddings) ``` ## Evaluation Results For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME}) ## Training The model was trained with the parameters: **DataLoader**: `torch.utils.data.dataloader.DataLoader` of length 1196 with parameters: ``` {'batch_size': 10, 'sampler': 'torch.utils.data.sampler.SequentialSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'} ``` **Loss**: `sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss` with parameters: ``` {'scale': 20.0, 'similarity_fct': 'cos_sim'} ``` Parameters of the fit()-Method: ``` { "epochs": 5, "evaluation_steps": 50, "evaluator": "sentence_transformers.evaluation.InformationRetrievalEvaluator.InformationRetrievalEvaluator", "max_grad_norm": 1, "optimizer_class": "", "optimizer_params": { "lr": 2e-05 }, "scheduler": "WarmupLinear", "steps_per_epoch": null, "warmup_steps": 598, "weight_decay": 0.01 } ``` ## Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False}) (2): Normalize() ) ``` ## Citing & Authors