--- pipeline_tag: sentence-similarity language: en license: apache-2.0 tags: - sentence-transformers - feature-extraction - sentence-similarity - transformers - mteb model-index: - name: sentence-t5-base results: - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en) config: en split: test metrics: - type: accuracy value: 75.82089552238807 - type: ap value: 40.58809426967639 - type: f1 value: 70.5050115572668 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (de) config: de split: test metrics: - type: accuracy value: 69.97858672376874 - type: ap value: 80.89622545806847 - type: f1 value: 68.09770164363411 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en-ext) config: en-ext split: test metrics: - type: accuracy value: 76.80659670164917 - type: ap value: 26.663544686227127 - type: f1 value: 64.52406535274052 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (ja) config: ja split: test metrics: - type: accuracy value: 46.04925053533191 - type: ap value: 10.574096802771448 - type: f1 value: 36.74441737116304 - task: type: Classification dataset: type: mteb/amazon_polarity name: MTEB AmazonPolarityClassification config: default split: test metrics: - type: accuracy value: 85.11737500000001 - type: ap value: 81.28435308927632 - type: f1 value: 85.01612484917347 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (en) config: en split: test metrics: - type: accuracy value: 44.943999999999996 - type: f1 value: 42.681783855948844 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (de) config: de split: test metrics: - type: accuracy value: 37.895999999999994 - type: f1 value: 35.428429230946115 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (es) config: es split: test metrics: - type: accuracy value: 37.328 - type: f1 value: 34.26335456752553 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (fr) config: fr split: test metrics: - type: accuracy value: 37.35 - type: f1 value: 34.644931974230495 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (ja) config: ja split: test metrics: - type: accuracy value: 22.290000000000003 - type: f1 value: 20.438677904046305 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (zh) config: zh split: test metrics: - type: accuracy value: 21.529999999999998 - type: f1 value: 18.273004097867844 - task: type: Retrieval dataset: type: arguana name: MTEB ArguAna config: default split: test metrics: - type: map_at_1 value: 21.906 - type: map_at_10 value: 35.993 - type: map_at_100 value: 37.14 - type: map_at_1000 value: 37.153999999999996 - type: map_at_3 value: 30.642000000000003 - type: map_at_5 value: 33.534000000000006 - type: ndcg_at_1 value: 21.906 - type: ndcg_at_10 value: 44.846000000000004 - type: ndcg_at_100 value: 49.95 - type: ndcg_at_1000 value: 50.29 - type: ndcg_at_3 value: 33.579 - type: ndcg_at_5 value: 38.807 - type: precision_at_1 value: 21.906 - type: precision_at_10 value: 7.367999999999999 - type: precision_at_100 value: 0.966 - type: precision_at_1000 value: 0.099 - type: precision_at_3 value: 14.035 - type: precision_at_5 value: 10.967 - type: recall_at_1 value: 21.906 - type: recall_at_10 value: 73.68400000000001 - type: recall_at_100 value: 96.586 - type: recall_at_1000 value: 99.14699999999999 - type: recall_at_3 value: 42.105 - type: recall_at_5 value: 54.836 - task: type: Clustering dataset: type: mteb/arxiv-clustering-p2p name: MTEB ArxivClusteringP2P config: default split: test metrics: - type: v_measure value: 39.27529166223639 - task: type: Clustering dataset: type: mteb/arxiv-clustering-s2s name: MTEB ArxivClusteringS2S config: default split: test metrics: - type: v_measure value: 27.261128959373327 - task: type: Reranking dataset: type: mteb/askubuntudupquestions-reranking name: MTEB AskUbuntuDupQuestions config: default split: test metrics: - type: map value: 59.72875661091822 - type: mrr value: 72.76997317856043 - task: type: STS dataset: type: mteb/biosses-sts name: MTEB BIOSSES config: default split: test metrics: - type: cos_sim_pearson value: 75.50587493517146 - type: cos_sim_spearman value: 75.89088585182279 - type: euclidean_pearson value: 75.74627833999679 - type: euclidean_spearman value: 75.89088585182279 - type: manhattan_pearson value: 76.10746255262428 - type: manhattan_spearman value: 75.93968214440233 - task: type: Classification dataset: type: mteb/banking77 name: MTEB Banking77Classification config: default split: test metrics: - type: accuracy value: 76.47727272727273 - type: f1 value: 75.41900393828456 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-p2p name: MTEB BiorxivClusteringP2P config: default split: test metrics: - type: v_measure value: 33.98533095653499 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-s2s name: MTEB BiorxivClusteringS2S config: default split: test metrics: - type: v_measure value: 22.921149832439514 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackAndroidRetrieval config: default split: test metrics: - type: map_at_1 value: 27.97 - type: map_at_10 value: 39.523 - type: map_at_100 value: 41.101 - type: map_at_1000 value: 41.221000000000004 - type: map_at_3 value: 36.193999999999996 - type: map_at_5 value: 37.952000000000005 - type: ndcg_at_1 value: 34.621 - type: ndcg_at_10 value: 46.18 - type: ndcg_at_100 value: 51.93600000000001 - type: ndcg_at_1000 value: 53.833 - type: ndcg_at_3 value: 41.091 - type: ndcg_at_5 value: 43.230000000000004 - type: precision_at_1 value: 34.621 - type: precision_at_10 value: 9.041 - type: precision_at_100 value: 1.525 - type: precision_at_1000 value: 0.19499999999999998 - type: precision_at_3 value: 20.029 - type: precision_at_5 value: 14.335 - type: recall_at_1 value: 27.97 - type: recall_at_10 value: 59.325 - type: recall_at_100 value: 82.917 - type: recall_at_1000 value: 95.175 - type: recall_at_3 value: 44.251000000000005 - type: recall_at_5 value: 50.383 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackEnglishRetrieval config: default split: test metrics: - type: map_at_1 value: 23.701 - type: map_at_10 value: 32.094 - type: map_at_100 value: 33.293 - type: map_at_1000 value: 33.434999999999995 - type: map_at_3 value: 29.609999999999996 - type: map_at_5 value: 31.16 - type: ndcg_at_1 value: 30.573 - type: ndcg_at_10 value: 37.031 - type: ndcg_at_100 value: 42.001 - type: ndcg_at_1000 value: 44.714 - type: ndcg_at_3 value: 33.434999999999995 - type: ndcg_at_5 value: 35.356 - type: precision_at_1 value: 30.573 - type: precision_at_10 value: 6.854 - type: precision_at_100 value: 1.192 - type: precision_at_1000 value: 0.174 - type: precision_at_3 value: 16.178 - type: precision_at_5 value: 11.567 - type: recall_at_1 value: 23.701 - type: recall_at_10 value: 45.755 - type: recall_at_100 value: 67.035 - type: recall_at_1000 value: 84.893 - type: recall_at_3 value: 34.977999999999994 - type: recall_at_5 value: 40.357 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGamingRetrieval config: default split: test metrics: - type: map_at_1 value: 35.617 - type: map_at_10 value: 47.774 - type: map_at_100 value: 48.943999999999996 - type: map_at_1000 value: 49.007 - type: map_at_3 value: 44.214999999999996 - type: map_at_5 value: 46.291 - type: ndcg_at_1 value: 40.627 - type: ndcg_at_10 value: 53.952 - type: ndcg_at_100 value: 58.55200000000001 - type: ndcg_at_1000 value: 59.824 - type: ndcg_at_3 value: 47.911 - type: ndcg_at_5 value: 50.966 - type: precision_at_1 value: 40.627 - type: precision_at_10 value: 8.884 - type: precision_at_100 value: 1.213 - type: precision_at_1000 value: 0.13699999999999998 - type: precision_at_3 value: 21.337999999999997 - type: precision_at_5 value: 15.034 - type: recall_at_1 value: 35.617 - type: recall_at_10 value: 68.73599999999999 - type: recall_at_100 value: 88.42999999999999 - type: recall_at_1000 value: 97.455 - type: recall_at_3 value: 52.915 - type: recall_at_5 value: 60.182 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGisRetrieval config: default split: test metrics: - type: map_at_1 value: 21.029999999999998 - type: map_at_10 value: 27.915 - type: map_at_100 value: 28.924 - type: map_at_1000 value: 29.023 - type: map_at_3 value: 25.634 - type: map_at_5 value: 26.934 - type: ndcg_at_1 value: 22.599 - type: ndcg_at_10 value: 32.340999999999994 - type: ndcg_at_100 value: 37.422 - type: ndcg_at_1000 value: 40.014 - type: ndcg_at_3 value: 27.604 - type: ndcg_at_5 value: 29.872 - type: precision_at_1 value: 22.599 - type: precision_at_10 value: 5.051 - type: precision_at_100 value: 0.799 - type: precision_at_1000 value: 0.106 - type: precision_at_3 value: 11.562999999999999 - type: precision_at_5 value: 8.225999999999999 - type: recall_at_1 value: 21.029999999999998 - type: recall_at_10 value: 44.226 - type: recall_at_100 value: 67.902 - type: recall_at_1000 value: 87.497 - type: recall_at_3 value: 31.389 - type: recall_at_5 value: 36.888 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackMathematicaRetrieval config: default split: test metrics: - type: map_at_1 value: 12.592 - type: map_at_10 value: 20.054 - type: map_at_100 value: 21.384 - type: map_at_1000 value: 21.52 - type: map_at_3 value: 17.718999999999998 - type: map_at_5 value: 19.189999999999998 - type: ndcg_at_1 value: 15.299 - type: ndcg_at_10 value: 24.698 - type: ndcg_at_100 value: 31.080000000000002 - type: ndcg_at_1000 value: 34.266000000000005 - type: ndcg_at_3 value: 20.331 - type: ndcg_at_5 value: 22.735 - type: precision_at_1 value: 15.299 - type: precision_at_10 value: 4.776 - type: precision_at_100 value: 0.928 - type: precision_at_1000 value: 0.133 - type: precision_at_3 value: 10.033 - type: precision_at_5 value: 7.761 - type: recall_at_1 value: 12.592 - type: recall_at_10 value: 35.386 - type: recall_at_100 value: 63.412 - type: recall_at_1000 value: 86.20400000000001 - type: recall_at_3 value: 23.768 - type: recall_at_5 value: 29.557 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackPhysicsRetrieval config: default split: test metrics: - type: map_at_1 value: 23.549 - type: map_at_10 value: 32.875 - type: map_at_100 value: 34.247 - type: map_at_1000 value: 34.374 - type: map_at_3 value: 29.774 - type: map_at_5 value: 31.535000000000004 - type: ndcg_at_1 value: 28.874 - type: ndcg_at_10 value: 38.801 - type: ndcg_at_100 value: 44.727 - type: ndcg_at_1000 value: 47.154 - type: ndcg_at_3 value: 33.643 - type: ndcg_at_5 value: 36.046 - type: precision_at_1 value: 28.874 - type: precision_at_10 value: 7.305000000000001 - type: precision_at_100 value: 1.21 - type: precision_at_1000 value: 0.16199999999999998 - type: precision_at_3 value: 16.009 - type: precision_at_5 value: 11.741999999999999 - type: recall_at_1 value: 23.549 - type: recall_at_10 value: 51.15 - type: recall_at_100 value: 76.32900000000001 - type: recall_at_1000 value: 92.167 - type: recall_at_3 value: 36.544 - type: recall_at_5 value: 42.75 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackProgrammersRetrieval config: default split: test metrics: - type: map_at_1 value: 24.524 - type: map_at_10 value: 34.288999999999994 - type: map_at_100 value: 35.67 - type: map_at_1000 value: 35.788 - type: map_at_3 value: 31.029 - type: map_at_5 value: 32.767 - type: ndcg_at_1 value: 29.794999999999998 - type: ndcg_at_10 value: 40.164 - type: ndcg_at_100 value: 46.278999999999996 - type: ndcg_at_1000 value: 48.698 - type: ndcg_at_3 value: 34.648 - type: ndcg_at_5 value: 36.982 - type: precision_at_1 value: 29.794999999999998 - type: precision_at_10 value: 7.580000000000001 - type: precision_at_100 value: 1.248 - type: precision_at_1000 value: 0.165 - type: precision_at_3 value: 16.628999999999998 - type: precision_at_5 value: 12.055 - type: recall_at_1 value: 24.524 - type: recall_at_10 value: 52.782 - type: recall_at_100 value: 79.108 - type: recall_at_1000 value: 95.62899999999999 - type: recall_at_3 value: 37.330999999999996 - type: recall_at_5 value: 43.502 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackRetrieval config: default split: test metrics: - type: map_at_1 value: 21.669083333333333 - type: map_at_10 value: 30.095166666666668 - type: map_at_100 value: 31.35275 - type: map_at_1000 value: 31.476166666666668 - type: map_at_3 value: 27.41675 - type: map_at_5 value: 28.91216666666667 - type: ndcg_at_1 value: 25.666833333333333 - type: ndcg_at_10 value: 35.23175 - type: ndcg_at_100 value: 40.822833333333335 - type: ndcg_at_1000 value: 43.33783333333334 - type: ndcg_at_3 value: 30.516333333333336 - type: ndcg_at_5 value: 32.723 - type: precision_at_1 value: 25.666833333333333 - type: precision_at_10 value: 6.345583333333332 - type: precision_at_100 value: 1.0886666666666667 - type: precision_at_1000 value: 0.14974999999999997 - type: precision_at_3 value: 14.185583333333335 - type: precision_at_5 value: 10.265333333333334 - type: recall_at_1 value: 21.669083333333333 - type: recall_at_10 value: 46.69591666666667 - type: recall_at_100 value: 71.36999999999999 - type: recall_at_1000 value: 88.98216666666666 - type: recall_at_3 value: 33.59675 - type: recall_at_5 value: 39.2065 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackStatsRetrieval config: default split: test metrics: - type: map_at_1 value: 18.587999999999997 - type: map_at_10 value: 25.452 - type: map_at_100 value: 26.296999999999997 - type: map_at_1000 value: 26.394000000000002 - type: map_at_3 value: 23.474 - type: map_at_5 value: 24.629 - type: ndcg_at_1 value: 21.012 - type: ndcg_at_10 value: 29.369 - type: ndcg_at_100 value: 33.782000000000004 - type: ndcg_at_1000 value: 36.406 - type: ndcg_at_3 value: 25.45 - type: ndcg_at_5 value: 27.384999999999998 - type: precision_at_1 value: 21.012 - type: precision_at_10 value: 4.723999999999999 - type: precision_at_100 value: 0.753 - type: precision_at_1000 value: 0.105 - type: precision_at_3 value: 11.094 - type: precision_at_5 value: 7.914000000000001 - type: recall_at_1 value: 18.587999999999997 - type: recall_at_10 value: 39.413 - type: recall_at_100 value: 59.78 - type: recall_at_1000 value: 79.49199999999999 - type: recall_at_3 value: 28.485 - type: recall_at_5 value: 33.367999999999995 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackTexRetrieval config: default split: test metrics: - type: map_at_1 value: 12.76 - type: map_at_10 value: 18.859 - type: map_at_100 value: 19.865 - type: map_at_1000 value: 19.994 - type: map_at_3 value: 16.817 - type: map_at_5 value: 17.837 - type: ndcg_at_1 value: 15.415999999999999 - type: ndcg_at_10 value: 23.037 - type: ndcg_at_100 value: 28.164 - type: ndcg_at_1000 value: 31.404 - type: ndcg_at_3 value: 19.134999999999998 - type: ndcg_at_5 value: 20.711 - type: precision_at_1 value: 15.415999999999999 - type: precision_at_10 value: 4.387 - type: precision_at_100 value: 0.826 - type: precision_at_1000 value: 0.127 - type: precision_at_3 value: 9.257 - type: precision_at_5 value: 6.696000000000001 - type: recall_at_1 value: 12.76 - type: recall_at_10 value: 32.657000000000004 - type: recall_at_100 value: 56.023 - type: recall_at_1000 value: 79.572 - type: recall_at_3 value: 21.608 - type: recall_at_5 value: 25.726 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackUnixRetrieval config: default split: test metrics: - type: map_at_1 value: 22.415 - type: map_at_10 value: 29.957 - type: map_at_100 value: 31.234 - type: map_at_1000 value: 31.351000000000003 - type: map_at_3 value: 27.261999999999997 - type: map_at_5 value: 28.708 - type: ndcg_at_1 value: 26.118999999999996 - type: ndcg_at_10 value: 34.961999999999996 - type: ndcg_at_100 value: 40.876000000000005 - type: ndcg_at_1000 value: 43.586000000000006 - type: ndcg_at_3 value: 29.958000000000002 - type: ndcg_at_5 value: 32.228 - type: precision_at_1 value: 26.118999999999996 - type: precision_at_10 value: 6.053999999999999 - type: precision_at_100 value: 1.012 - type: precision_at_1000 value: 0.13699999999999998 - type: precision_at_3 value: 13.65 - type: precision_at_5 value: 9.795 - type: recall_at_1 value: 22.415 - type: recall_at_10 value: 46.339000000000006 - type: recall_at_100 value: 72.30799999999999 - type: recall_at_1000 value: 91.448 - type: recall_at_3 value: 32.673 - type: recall_at_5 value: 38.467 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWebmastersRetrieval config: default split: test metrics: - type: map_at_1 value: 21.624 - type: map_at_10 value: 30.0 - type: map_at_100 value: 31.776 - type: map_at_1000 value: 32.005 - type: map_at_3 value: 27.314 - type: map_at_5 value: 28.741 - type: ndcg_at_1 value: 25.691999999999997 - type: ndcg_at_10 value: 35.64 - type: ndcg_at_100 value: 42.488 - type: ndcg_at_1000 value: 44.978 - type: ndcg_at_3 value: 31.147000000000002 - type: ndcg_at_5 value: 33.241 - type: precision_at_1 value: 25.691999999999997 - type: precision_at_10 value: 7.0360000000000005 - type: precision_at_100 value: 1.547 - type: precision_at_1000 value: 0.244 - type: precision_at_3 value: 15.02 - type: precision_at_5 value: 11.146 - type: recall_at_1 value: 21.624 - type: recall_at_10 value: 46.415 - type: recall_at_100 value: 77.086 - type: recall_at_1000 value: 92.72500000000001 - type: recall_at_3 value: 33.911 - type: recall_at_5 value: 39.116 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWordpressRetrieval config: default split: test metrics: - type: map_at_1 value: 15.659 - type: map_at_10 value: 22.35 - type: map_at_100 value: 23.498 - type: map_at_1000 value: 23.602 - type: map_at_3 value: 19.959 - type: map_at_5 value: 21.201999999999998 - type: ndcg_at_1 value: 17.375 - type: ndcg_at_10 value: 26.606 - type: ndcg_at_100 value: 32.567 - type: ndcg_at_1000 value: 35.177 - type: ndcg_at_3 value: 21.843 - type: ndcg_at_5 value: 23.924 - type: precision_at_1 value: 17.375 - type: precision_at_10 value: 4.455 - type: precision_at_100 value: 0.8109999999999999 - type: precision_at_1000 value: 0.11199999999999999 - type: precision_at_3 value: 9.427000000000001 - type: precision_at_5 value: 6.912999999999999 - type: recall_at_1 value: 15.659 - type: recall_at_10 value: 38.167 - type: recall_at_100 value: 66.11 - type: recall_at_1000 value: 85.529 - type: recall_at_3 value: 25.308000000000003 - type: recall_at_5 value: 30.182 - task: type: Retrieval dataset: type: climate-fever name: MTEB ClimateFEVER config: default split: test metrics: - type: map_at_1 value: 3.9469999999999996 - type: map_at_10 value: 6.816999999999999 - type: map_at_100 value: 7.7170000000000005 - type: map_at_1000 value: 7.887 - type: map_at_3 value: 5.6739999999999995 - type: map_at_5 value: 6.243 - type: ndcg_at_1 value: 8.73 - type: ndcg_at_10 value: 10.366999999999999 - type: ndcg_at_100 value: 15.343000000000002 - type: ndcg_at_1000 value: 19.535 - type: ndcg_at_3 value: 7.976 - type: ndcg_at_5 value: 8.786 - type: precision_at_1 value: 8.73 - type: precision_at_10 value: 3.3160000000000003 - type: precision_at_100 value: 0.857 - type: precision_at_1000 value: 0.16199999999999998 - type: precision_at_3 value: 5.776 - type: precision_at_5 value: 4.534 - type: recall_at_1 value: 3.9469999999999996 - type: recall_at_10 value: 13.385 - type: recall_at_100 value: 31.612000000000002 - type: recall_at_1000 value: 56.252 - type: recall_at_3 value: 7.686 - type: recall_at_5 value: 9.879 - task: type: Retrieval dataset: type: dbpedia-entity name: MTEB DBPedia config: default split: test metrics: - type: map_at_1 value: 5.75 - type: map_at_10 value: 11.632000000000001 - type: map_at_100 value: 16.400000000000002 - type: map_at_1000 value: 17.580000000000002 - type: map_at_3 value: 8.49 - type: map_at_5 value: 9.626999999999999 - type: ndcg_at_1 value: 35.75 - type: ndcg_at_10 value: 27.766000000000002 - type: ndcg_at_100 value: 31.424000000000003 - type: ndcg_at_1000 value: 38.998 - type: ndcg_at_3 value: 30.807000000000002 - type: ndcg_at_5 value: 28.62 - type: precision_at_1 value: 44.25 - type: precision_at_10 value: 22.625 - type: precision_at_100 value: 7.163 - type: precision_at_1000 value: 1.619 - type: precision_at_3 value: 33.75 - type: precision_at_5 value: 28.199999999999996 - type: recall_at_1 value: 5.75 - type: recall_at_10 value: 16.918 - type: recall_at_100 value: 37.645 - type: recall_at_1000 value: 62.197 - type: recall_at_3 value: 9.721 - type: recall_at_5 value: 11.974 - task: type: Classification dataset: type: mteb/emotion name: MTEB EmotionClassification config: default split: test metrics: - type: accuracy value: 51.355000000000004 - type: f1 value: 44.27505726378252 - task: type: Retrieval dataset: type: fever name: MTEB FEVER config: default split: test metrics: - type: map_at_1 value: 13.81 - type: map_at_10 value: 21.567 - type: map_at_100 value: 22.461000000000002 - type: map_at_1000 value: 22.545 - type: map_at_3 value: 19.282 - type: map_at_5 value: 20.535999999999998 - type: ndcg_at_1 value: 15.032 - type: ndcg_at_10 value: 26.165 - type: ndcg_at_100 value: 30.819999999999997 - type: ndcg_at_1000 value: 33.209 - type: ndcg_at_3 value: 21.488 - type: ndcg_at_5 value: 23.721999999999998 - type: precision_at_1 value: 15.032 - type: precision_at_10 value: 4.292 - type: precision_at_100 value: 0.6779999999999999 - type: precision_at_1000 value: 0.09 - type: precision_at_3 value: 9.551 - type: precision_at_5 value: 6.927999999999999 - type: recall_at_1 value: 13.81 - type: recall_at_10 value: 39.009 - type: recall_at_100 value: 60.99400000000001 - type: recall_at_1000 value: 79.703 - type: recall_at_3 value: 26.221 - type: recall_at_5 value: 31.604 - task: type: Retrieval dataset: type: fiqa name: MTEB FiQA2018 config: default split: test metrics: - type: map_at_1 value: 16.55 - type: map_at_10 value: 27.101 - type: map_at_100 value: 28.941 - type: map_at_1000 value: 29.137 - type: map_at_3 value: 22.926 - type: map_at_5 value: 25.217 - type: ndcg_at_1 value: 33.951 - type: ndcg_at_10 value: 34.832 - type: ndcg_at_100 value: 41.989 - type: ndcg_at_1000 value: 45.262 - type: ndcg_at_3 value: 30.427 - type: ndcg_at_5 value: 31.985999999999997 - type: precision_at_1 value: 33.951 - type: precision_at_10 value: 10.139 - type: precision_at_100 value: 1.735 - type: precision_at_1000 value: 0.233 - type: precision_at_3 value: 20.576 - type: precision_at_5 value: 15.556000000000001 - type: recall_at_1 value: 16.55 - type: recall_at_10 value: 42.153 - type: recall_at_100 value: 69.19999999999999 - type: recall_at_1000 value: 88.631 - type: recall_at_3 value: 27.071 - type: recall_at_5 value: 33.432 - task: type: Retrieval dataset: type: hotpotqa name: MTEB HotpotQA config: default split: test metrics: - type: map_at_1 value: 18.102999999999998 - type: map_at_10 value: 26.006 - type: map_at_100 value: 27.060000000000002 - type: map_at_1000 value: 27.173000000000002 - type: map_at_3 value: 23.815 - type: map_at_5 value: 24.978 - type: ndcg_at_1 value: 36.205 - type: ndcg_at_10 value: 33.198 - type: ndcg_at_100 value: 37.836999999999996 - type: ndcg_at_1000 value: 40.499 - type: ndcg_at_3 value: 29.108 - type: ndcg_at_5 value: 30.993 - type: precision_at_1 value: 36.205 - type: precision_at_10 value: 7.404 - type: precision_at_100 value: 1.109 - type: precision_at_1000 value: 0.146 - type: precision_at_3 value: 18.479 - type: precision_at_5 value: 12.581000000000001 - type: recall_at_1 value: 18.102999999999998 - type: recall_at_10 value: 37.022 - type: recall_at_100 value: 55.449000000000005 - type: recall_at_1000 value: 73.214 - type: recall_at_3 value: 27.717999999999996 - type: recall_at_5 value: 31.452 - task: type: Classification dataset: type: mteb/imdb name: MTEB ImdbClassification config: default split: test metrics: - type: accuracy value: 77.3372 - type: ap value: 71.64946791935137 - type: f1 value: 77.13428403424751 - task: type: Retrieval dataset: type: msmarco name: MTEB MSMARCO config: default split: dev metrics: - type: map_at_1 value: 9.093 - type: map_at_10 value: 16.227 - type: map_at_100 value: 17.477999999999998 - type: map_at_1000 value: 17.579 - type: map_at_3 value: 13.541 - type: map_at_5 value: 14.921000000000001 - type: ndcg_at_1 value: 9.370000000000001 - type: ndcg_at_10 value: 20.705000000000002 - type: ndcg_at_100 value: 27.331 - type: ndcg_at_1000 value: 30.104 - type: ndcg_at_3 value: 15.081 - type: ndcg_at_5 value: 17.551 - type: precision_at_1 value: 9.370000000000001 - type: precision_at_10 value: 3.633 - type: precision_at_100 value: 0.7040000000000001 - type: precision_at_1000 value: 0.094 - type: precision_at_3 value: 6.648 - type: precision_at_5 value: 5.241 - type: recall_at_1 value: 9.093 - type: recall_at_10 value: 34.777 - type: recall_at_100 value: 66.673 - type: recall_at_1000 value: 88.44999999999999 - type: recall_at_3 value: 19.194 - type: recall_at_5 value: 25.124999999999996 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (en) config: en split: test metrics: - type: accuracy value: 90.3374373005016 - type: f1 value: 90.25497662319412 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (de) config: de split: test metrics: - type: accuracy value: 76.98224852071004 - type: f1 value: 75.10443724253962 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (es) config: es split: test metrics: - type: accuracy value: 73.60907271514343 - type: f1 value: 73.15530983235772 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (fr) config: fr split: test metrics: - type: accuracy value: 75.02975258377701 - type: f1 value: 75.53083321964739 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (hi) config: hi split: test metrics: - type: accuracy value: 21.40193617784152 - type: f1 value: 15.465217146460256 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (th) config: th split: test metrics: - type: accuracy value: 16.206148282097647 - type: f1 value: 11.580229602870345 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (en) config: en split: test metrics: - type: accuracy value: 63.32421340629275 - type: f1 value: 45.42341063027956 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (de) config: de split: test metrics: - type: accuracy value: 44.426599041983664 - type: f1 value: 27.205947872504428 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (es) config: es split: test metrics: - type: accuracy value: 42.02801867911942 - type: f1 value: 26.314909946795733 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (fr) config: fr split: test metrics: - type: accuracy value: 43.845912934544316 - type: f1 value: 29.519701972859792 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (hi) config: hi split: test metrics: - type: accuracy value: 3.80064539261384 - type: f1 value: 1.2078686392462628 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (th) config: th split: test metrics: - type: accuracy value: 5.207956600361665 - type: f1 value: 1.5365513001536746 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (af) config: af split: test metrics: - type: accuracy value: 34.32078009414929 - type: f1 value: 31.969428974847435 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (am) config: am split: test metrics: - type: accuracy value: 2.3772696704774714 - type: f1 value: 1.027013290806954 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ar) config: ar split: test metrics: - type: accuracy value: 4.53261600537996 - type: f1 value: 2.793131265571347 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (az) config: az split: test metrics: - type: accuracy value: 31.758574310692673 - type: f1 value: 30.162299253522708 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (bn) config: bn split: test metrics: - type: accuracy value: 2.5823806321452594 - type: f1 value: 1.0918434877949255 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (cy) config: cy split: test metrics: - type: accuracy value: 28.94418291862811 - type: f1 value: 27.498874158049468 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (da) config: da split: test metrics: - type: accuracy value: 38.81977135171486 - type: f1 value: 36.44688565156101 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (de) config: de split: test metrics: - type: accuracy value: 45.22864828513786 - type: f1 value: 41.61460113481098 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (el) config: el split: test metrics: - type: accuracy value: 10.053799596503026 - type: f1 value: 5.1615743271775285 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (en) config: en split: test metrics: - type: accuracy value: 69.74445191661063 - type: f1 value: 67.00099408297854 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (es) config: es split: test metrics: - type: accuracy value: 45.31943510423672 - type: f1 value: 43.92469151179908 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fa) config: fa split: test metrics: - type: accuracy value: 3.5810356422326834 - type: f1 value: 0.8057464198110936 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fi) config: fi split: test metrics: - type: accuracy value: 33.52387357094821 - type: f1 value: 30.686159550520415 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fr) config: fr split: test metrics: - type: accuracy value: 51.13315400134499 - type: f1 value: 48.84533274433444 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (he) config: he split: test metrics: - type: accuracy value: 2.632817753866846 - type: f1 value: 0.7565304035292157 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hi) config: hi split: test metrics: - type: accuracy value: 2.6798924008069935 - type: f1 value: 1.5577100383199163 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hu) config: hu split: test metrics: - type: accuracy value: 32.306657700067255 - type: f1 value: 29.508334412971788 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hy) config: hy split: test metrics: - type: accuracy value: 3.3254875588433084 - type: f1 value: 0.9498561670625558 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (id) config: id split: test metrics: - type: accuracy value: 35.497646267652996 - type: f1 value: 32.919473578262014 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (is) config: is split: test metrics: - type: accuracy value: 29.818426361802285 - type: f1 value: 27.968522255792134 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (it) config: it split: test metrics: - type: accuracy value: 45.585070611970416 - type: f1 value: 43.85609178763681 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ja) config: ja split: test metrics: - type: accuracy value: 3.6718224613315398 - type: f1 value: 1.5834733153849028 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (jv) config: jv split: test metrics: - type: accuracy value: 31.149966375252188 - type: f1 value: 28.77156087445068 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ka) config: ka split: test metrics: - type: accuracy value: 2.767316745124411 - type: f1 value: 1.0163373847923576 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (km) config: km split: test metrics: - type: accuracy value: 5.655682582380632 - type: f1 value: 1.6046205246119 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (kn) config: kn split: test metrics: - type: accuracy value: 2.5924680564895763 - type: f1 value: 1.3338404330308657 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ko) config: ko split: test metrics: - type: accuracy value: 2.3436449226630804 - type: f1 value: 0.5935093070394912 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (lv) config: lv split: test metrics: - type: accuracy value: 33.97108271687962 - type: f1 value: 33.35695453571926 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ml) config: ml split: test metrics: - type: accuracy value: 2.5453934095494284 - type: f1 value: 0.5515796181696971 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (mn) config: mn split: test metrics: - type: accuracy value: 14.704102219233356 - type: f1 value: 12.444230806799856 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ms) config: ms split: test metrics: - type: accuracy value: 33.12037659717552 - type: f1 value: 29.867258908899636 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (my) config: my split: test metrics: - type: accuracy value: 4.421654337592468 - type: f1 value: 1.3125497683444283 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (nb) config: nb split: test metrics: - type: accuracy value: 38.53059852051109 - type: f1 value: 35.473185172465996 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (nl) config: nl split: test metrics: - type: accuracy value: 37.96234028244788 - type: f1 value: 34.24786837723274 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (pl) config: pl split: test metrics: - type: accuracy value: 34.40820443846672 - type: f1 value: 32.06121218840769 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (pt) config: pt split: test metrics: - type: accuracy value: 43.35238735709483 - type: f1 value: 41.66578945324342 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ro) config: ro split: test metrics: - type: accuracy value: 42.68997982515131 - type: f1 value: 40.12799296163562 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ru) config: ru split: test metrics: - type: accuracy value: 14.815063887020846 - type: f1 value: 14.113147395663178 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sl) config: sl split: test metrics: - type: accuracy value: 34.54270342972428 - type: f1 value: 32.556771565845736 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sq) config: sq split: test metrics: - type: accuracy value: 38.54068594485541 - type: f1 value: 34.63112962341627 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sv) config: sv split: test metrics: - type: accuracy value: 35.978480161398785 - type: f1 value: 32.896757123025 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sw) config: sw split: test metrics: - type: accuracy value: 32.135171486213856 - type: f1 value: 30.338211408536065 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ta) config: ta split: test metrics: - type: accuracy value: 1.4055144586415602 - type: f1 value: 0.4241059307791662 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (te) config: te split: test metrics: - type: accuracy value: 2.501681237390719 - type: f1 value: 0.9247691466655827 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (th) config: th split: test metrics: - type: accuracy value: 3.7088096839273708 - type: f1 value: 1.7117159563540418 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (tl) config: tl split: test metrics: - type: accuracy value: 36.04236718224613 - type: f1 value: 31.848270154788448 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (tr) config: tr split: test metrics: - type: accuracy value: 33.765971755211844 - type: f1 value: 31.25189279882076 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ur) config: ur split: test metrics: - type: accuracy value: 2.992602555480834 - type: f1 value: 1.6243709826215067 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (vi) config: vi split: test metrics: - type: accuracy value: 22.61936785474109 - type: f1 value: 20.616493213131037 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (zh-CN) config: zh-CN split: test metrics: - type: accuracy value: 1.1163416274377942 - type: f1 value: 0.17708079483587733 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (zh-TW) config: zh-TW split: test metrics: - type: accuracy value: 4.6267652992602555 - type: f1 value: 1.8933559386335084 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (af) config: af split: test metrics: - type: accuracy value: 44.45191661062542 - type: f1 value: 40.78791900409084 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (am) config: am split: test metrics: - type: accuracy value: 7.508406186953598 - type: f1 value: 2.821623266241915 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ar) config: ar split: test metrics: - type: accuracy value: 12.323470073974445 - type: f1 value: 7.033060282091267 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (az) config: az split: test metrics: - type: accuracy value: 38.40954942837929 - type: f1 value: 35.118197115254915 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (bn) config: bn split: test metrics: - type: accuracy value: 8.453261600537996 - type: f1 value: 3.7954675930409887 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (cy) config: cy split: test metrics: - type: accuracy value: 35.036987222595826 - type: f1 value: 32.66245000430742 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (da) config: da split: test metrics: - type: accuracy value: 48.35911230665769 - type: f1 value: 45.007102253202646 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (de) config: de split: test metrics: - type: accuracy value: 59.11903160726294 - type: f1 value: 56.29359157547811 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (el) config: el split: test metrics: - type: accuracy value: 17.679892400806995 - type: f1 value: 12.298039805650598 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (en) config: en split: test metrics: - type: accuracy value: 72.32347007397445 - type: f1 value: 71.44743919882103 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (es) config: es split: test metrics: - type: accuracy value: 55.61197041022192 - type: f1 value: 53.34238608745496 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fa) config: fa split: test metrics: - type: accuracy value: 6.859448554135844 - type: f1 value: 2.6611096235427603 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fi) config: fi split: test metrics: - type: accuracy value: 41.34162743779422 - type: f1 value: 37.16587245902462 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fr) config: fr split: test metrics: - type: accuracy value: 59.91930060524545 - type: f1 value: 58.82978451275059 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (he) config: he split: test metrics: - type: accuracy value: 7.864828513786147 - type: f1 value: 2.1478868713724184 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hi) config: hi split: test metrics: - type: accuracy value: 7.6328177538668465 - type: f1 value: 4.359158983520571 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hu) config: hu split: test metrics: - type: accuracy value: 41.31136516476127 - type: f1 value: 37.56187834379295 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hy) config: hy split: test metrics: - type: accuracy value: 9.226630800268998 - type: f1 value: 4.272151036150232 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (id) config: id split: test metrics: - type: accuracy value: 44.64357767316745 - type: f1 value: 42.836643630228295 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (is) config: is split: test metrics: - type: accuracy value: 39.633490248823136 - type: f1 value: 35.97498169100485 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (it) config: it split: test metrics: - type: accuracy value: 54.57969065232011 - type: f1 value: 51.96887024053593 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ja) config: ja split: test metrics: - type: accuracy value: 4.956287827841291 - type: f1 value: 2.885322950712636 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (jv) config: jv split: test metrics: - type: accuracy value: 40.73301950235374 - type: f1 value: 37.77421393601645 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ka) config: ka split: test metrics: - type: accuracy value: 7.505043712172157 - type: f1 value: 2.885798319215117 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (km) config: km split: test metrics: - type: accuracy value: 8.732347007397442 - type: f1 value: 3.3227736931128002 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (kn) config: kn split: test metrics: - type: accuracy value: 7.985877605917954 - type: f1 value: 4.454873367911008 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ko) config: ko split: test metrics: - type: accuracy value: 6.028917283120376 - type: f1 value: 2.1947301613241064 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (lv) config: lv split: test metrics: - type: accuracy value: 36.415601882985875 - type: f1 value: 34.328528362740194 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ml) config: ml split: test metrics: - type: accuracy value: 6.960322797579018 - type: f1 value: 2.5948923502086636 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (mn) config: mn split: test metrics: - type: accuracy value: 19.84868863483524 - type: f1 value: 16.571872313821913 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ms) config: ms split: test metrics: - type: accuracy value: 43.18090114324143 - type: f1 value: 39.21386548691514 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (my) config: my split: test metrics: - type: accuracy value: 9.462004034969738 - type: f1 value: 4.439617481980993 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (nb) config: nb split: test metrics: - type: accuracy value: 46.60390047074647 - type: f1 value: 42.97761990856311 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (nl) config: nl split: test metrics: - type: accuracy value: 50.00336247478143 - type: f1 value: 45.623347729802326 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (pl) config: pl split: test metrics: - type: accuracy value: 42.29993275050437 - type: f1 value: 39.93988660787354 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (pt) config: pt split: test metrics: - type: accuracy value: 52.23940820443846 - type: f1 value: 50.63740128715899 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ro) config: ro split: test metrics: - type: accuracy value: 53.69535978480161 - type: f1 value: 51.48099264354228 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ru) config: ru split: test metrics: - type: accuracy value: 20.68594485541358 - type: f1 value: 19.732532357173614 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sl) config: sl split: test metrics: - type: accuracy value: 39.788164088769335 - type: f1 value: 37.63884763800779 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sq) config: sq split: test metrics: - type: accuracy value: 50.164761264290526 - type: f1 value: 46.40832635120166 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sv) config: sv split: test metrics: - type: accuracy value: 46.68796234028245 - type: f1 value: 42.65714705554807 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sw) config: sw split: test metrics: - type: accuracy value: 40.480833893745796 - type: f1 value: 37.28728896543833 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ta) config: ta split: test metrics: - type: accuracy value: 7.474781439139207 - type: f1 value: 2.114620869965788 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (te) config: te split: test metrics: - type: accuracy value: 6.866173503698722 - type: f1 value: 3.0078405064872755 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (th) config: th split: test metrics: - type: accuracy value: 8.258238063214526 - type: f1 value: 4.082391072869187 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (tl) config: tl split: test metrics: - type: accuracy value: 48.93745796906523 - type: f1 value: 46.427786382184266 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (tr) config: tr split: test metrics: - type: accuracy value: 41.82918628110289 - type: f1 value: 40.642360044818325 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ur) config: ur split: test metrics: - type: accuracy value: 9.767989240080698 - type: f1 value: 4.6812848634278925 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (vi) config: vi split: test metrics: - type: accuracy value: 30.013449899125753 - type: f1 value: 28.091947569862718 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (zh-CN) config: zh-CN split: test metrics: - type: accuracy value: 4.169468728984533 - type: f1 value: 0.8211582847545612 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (zh-TW) config: zh-TW split: test metrics: - type: accuracy value: 7.908540685944855 - type: f1 value: 3.5358754800289534 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-p2p name: MTEB MedrxivClusteringP2P config: default split: test metrics: - type: v_measure value: 33.20164128507875 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-s2s name: MTEB MedrxivClusteringS2S config: default split: test metrics: - type: v_measure value: 26.13067187536383 - task: type: Reranking dataset: type: mteb/mind_small name: MTEB MindSmallReranking config: default split: test metrics: - type: map value: 30.196798710053223 - type: mrr value: 31.098179519159814 - task: type: Retrieval dataset: type: nfcorpus name: MTEB NFCorpus config: default split: test metrics: - type: map_at_1 value: 4.7010000000000005 - type: map_at_10 value: 9.756 - type: map_at_100 value: 12.631999999999998 - type: map_at_1000 value: 14.046 - type: map_at_3 value: 7.053 - type: map_at_5 value: 8.244 - type: ndcg_at_1 value: 35.604 - type: ndcg_at_10 value: 28.645 - type: ndcg_at_100 value: 27.431 - type: ndcg_at_1000 value: 36.378 - type: ndcg_at_3 value: 32.533 - type: ndcg_at_5 value: 30.737 - type: precision_at_1 value: 37.771 - type: precision_at_10 value: 21.517 - type: precision_at_100 value: 7.567 - type: precision_at_1000 value: 2.026 - type: precision_at_3 value: 30.753000000000004 - type: precision_at_5 value: 26.811 - type: recall_at_1 value: 4.7010000000000005 - type: recall_at_10 value: 14.302999999999999 - type: recall_at_100 value: 29.304000000000002 - type: recall_at_1000 value: 62.202999999999996 - type: recall_at_3 value: 8.419 - type: recall_at_5 value: 10.656 - task: type: Retrieval dataset: type: nq name: MTEB NQ config: default split: test metrics: - type: map_at_1 value: 17.444000000000003 - type: map_at_10 value: 29.198 - type: map_at_100 value: 30.553 - type: map_at_1000 value: 30.614 - type: map_at_3 value: 24.992 - type: map_at_5 value: 27.322999999999997 - type: ndcg_at_1 value: 19.641000000000002 - type: ndcg_at_10 value: 36.324 - type: ndcg_at_100 value: 42.641 - type: ndcg_at_1000 value: 44.089 - type: ndcg_at_3 value: 28.000000000000004 - type: ndcg_at_5 value: 32.025 - type: precision_at_1 value: 19.641000000000002 - type: precision_at_10 value: 6.6339999999999995 - type: precision_at_100 value: 1.024 - type: precision_at_1000 value: 0.116 - type: precision_at_3 value: 13.209999999999999 - type: precision_at_5 value: 10.162 - type: recall_at_1 value: 17.444000000000003 - type: recall_at_10 value: 56.230999999999995 - type: recall_at_100 value: 84.61800000000001 - type: recall_at_1000 value: 95.416 - type: recall_at_3 value: 34.245999999999995 - type: recall_at_5 value: 43.617 - task: type: Retrieval dataset: type: quora name: MTEB QuoraRetrieval config: default split: test metrics: - type: map_at_1 value: 67.604 - type: map_at_10 value: 81.364 - type: map_at_100 value: 82.092 - type: map_at_1000 value: 82.112 - type: map_at_3 value: 78.321 - type: map_at_5 value: 80.203 - type: ndcg_at_1 value: 77.92 - type: ndcg_at_10 value: 85.491 - type: ndcg_at_100 value: 87.102 - type: ndcg_at_1000 value: 87.246 - type: ndcg_at_3 value: 82.219 - type: ndcg_at_5 value: 83.991 - type: precision_at_1 value: 77.92 - type: precision_at_10 value: 13.065 - type: precision_at_100 value: 1.525 - type: precision_at_1000 value: 0.157 - type: precision_at_3 value: 35.96 - type: precision_at_5 value: 23.785999999999998 - type: recall_at_1 value: 67.604 - type: recall_at_10 value: 93.57 - type: recall_at_100 value: 99.20400000000001 - type: recall_at_1000 value: 99.958 - type: recall_at_3 value: 84.38900000000001 - type: recall_at_5 value: 89.223 - task: type: Clustering dataset: type: mteb/reddit-clustering name: MTEB RedditClustering config: default split: test metrics: - type: v_measure value: 52.930534839708464 - task: type: Clustering dataset: type: mteb/reddit-clustering-p2p name: MTEB RedditClusteringP2P config: default split: test metrics: - type: v_measure value: 59.6686566444821 - task: type: Retrieval dataset: type: scidocs name: MTEB SCIDOCS config: default split: test metrics: - type: map_at_1 value: 3.267 - type: map_at_10 value: 8.061 - type: map_at_100 value: 9.66 - type: map_at_1000 value: 9.926 - type: map_at_3 value: 5.733 - type: map_at_5 value: 6.894 - type: ndcg_at_1 value: 16.0 - type: ndcg_at_10 value: 14.155000000000001 - type: ndcg_at_100 value: 20.973 - type: ndcg_at_1000 value: 26.163999999999998 - type: ndcg_at_3 value: 12.994 - type: ndcg_at_5 value: 11.58 - type: precision_at_1 value: 16.0 - type: precision_at_10 value: 7.470000000000001 - type: precision_at_100 value: 1.7389999999999999 - type: precision_at_1000 value: 0.299 - type: precision_at_3 value: 12.167 - type: precision_at_5 value: 10.280000000000001 - type: recall_at_1 value: 3.267 - type: recall_at_10 value: 15.152 - type: recall_at_100 value: 35.248000000000005 - type: recall_at_1000 value: 60.742 - type: recall_at_3 value: 7.4319999999999995 - type: recall_at_5 value: 10.452 - task: type: STS dataset: type: mteb/sickr-sts name: MTEB SICK-R config: default split: test metrics: - type: cos_sim_pearson value: 84.12684378047692 - type: cos_sim_spearman value: 80.18231249099851 - type: euclidean_pearson value: 81.10311004134292 - type: euclidean_spearman value: 80.18231162371262 - type: manhattan_pearson value: 81.06660654194627 - type: manhattan_spearman value: 80.15421301055235 - task: type: STS dataset: type: mteb/sts12-sts name: MTEB STS12 config: default split: test metrics: - type: cos_sim_pearson value: 86.39022108792102 - type: cos_sim_spearman value: 78.0511871449349 - type: euclidean_pearson value: 83.55414895785707 - type: euclidean_spearman value: 78.04999900363751 - type: manhattan_pearson value: 83.58122709700247 - type: manhattan_spearman value: 78.09617051485085 - task: type: STS dataset: type: mteb/sts13-sts name: MTEB STS13 config: default split: test metrics: - type: cos_sim_pearson value: 85.20665643089602 - type: cos_sim_spearman value: 85.84897342040492 - type: euclidean_pearson value: 85.07344348481206 - type: euclidean_spearman value: 85.84897334409469 - type: manhattan_pearson value: 85.05095172720918 - type: manhattan_spearman value: 85.82539599484174 - task: type: STS dataset: type: mteb/sts14-sts name: MTEB STS14 config: default split: test metrics: - type: cos_sim_pearson value: 83.89550144541676 - type: cos_sim_spearman value: 82.18926664662587 - type: euclidean_pearson value: 83.2979886572065 - type: euclidean_spearman value: 82.18927470901535 - type: manhattan_pearson value: 83.26470031355984 - type: manhattan_spearman value: 82.18712042624048 - task: type: STS dataset: type: mteb/sts15-sts name: MTEB STS15 config: default split: test metrics: - type: cos_sim_pearson value: 86.24309164032012 - type: cos_sim_spearman value: 87.45860981918769 - type: euclidean_pearson value: 87.04473506428359 - type: euclidean_spearman value: 87.45861561864089 - type: manhattan_pearson value: 87.02002615328881 - type: manhattan_spearman value: 87.43661746711435 - task: type: STS dataset: type: mteb/sts16-sts name: MTEB STS16 config: default split: test metrics: - type: cos_sim_pearson value: 82.24202172291855 - type: cos_sim_spearman value: 84.03233567112525 - type: euclidean_pearson value: 83.5361433714169 - type: euclidean_spearman value: 84.03233506665642 - type: manhattan_pearson value: 83.51738829906122 - type: manhattan_spearman value: 84.02036537979589 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (ko-ko) config: ko-ko split: test metrics: - type: cos_sim_pearson value: 9.160685666083912 - type: cos_sim_spearman value: 10.0553422118037 - type: euclidean_pearson value: 9.589155152132493 - type: euclidean_spearman value: 10.215143153291868 - type: manhattan_pearson value: 9.570908402796292 - type: manhattan_spearman value: 10.214075999964175 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (ar-ar) config: ar-ar split: test metrics: - type: cos_sim_pearson value: 10.60353259635145 - type: cos_sim_spearman value: 13.355557088500165 - type: euclidean_pearson value: 14.636463268109537 - type: euclidean_spearman value: 14.35296057730866 - type: manhattan_pearson value: 14.553161459629774 - type: manhattan_spearman value: 14.267005982719008 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-ar) config: en-ar split: test metrics: - type: cos_sim_pearson value: -4.869359628676264 - type: cos_sim_spearman value: -5.6460908267056835 - type: euclidean_pearson value: -4.9763689688023245 - type: euclidean_spearman value: -5.642707032163295 - type: manhattan_pearson value: -2.1980242988428276 - type: manhattan_spearman value: -1.854801657544592 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-de) config: en-de split: test metrics: - type: cos_sim_pearson value: 66.79239834758799 - type: cos_sim_spearman value: 67.11298548130333 - type: euclidean_pearson value: 66.77948456698994 - type: euclidean_spearman value: 67.11298548130333 - type: manhattan_pearson value: 66.5459479074496 - type: manhattan_spearman value: 66.85517071449804 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-en) config: en-en split: test metrics: - type: cos_sim_pearson value: 89.5692743691406 - type: cos_sim_spearman value: 89.56885540021487 - type: euclidean_pearson value: 89.78111903652413 - type: euclidean_spearman value: 89.56885540021487 - type: manhattan_pearson value: 89.68974590722112 - type: manhattan_spearman value: 89.40694757290255 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-tr) config: en-tr split: test metrics: - type: cos_sim_pearson value: 2.5434531383947427 - type: cos_sim_spearman value: -0.015686409614414636 - type: euclidean_pearson value: 3.3562612023763454 - type: euclidean_spearman value: -0.015686409614414636 - type: manhattan_pearson value: 3.06029066490911 - type: manhattan_spearman value: 0.9087736864115655 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (es-en) config: es-en split: test metrics: - type: cos_sim_pearson value: 48.19554059380143 - type: cos_sim_spearman value: 47.72387836409936 - type: euclidean_pearson value: 48.566966490440386 - type: euclidean_spearman value: 47.72387836409936 - type: manhattan_pearson value: 48.47970171544757 - type: manhattan_spearman value: 48.06448477123342 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (es-es) config: es-es split: test metrics: - type: cos_sim_pearson value: 80.72325991736295 - type: cos_sim_spearman value: 79.94411571627043 - type: euclidean_pearson value: 81.66909260117279 - type: euclidean_spearman value: 79.94284742229813 - type: manhattan_pearson value: 81.78261278000369 - type: manhattan_spearman value: 80.18524960358721 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (fr-en) config: fr-en split: test metrics: - type: cos_sim_pearson value: 57.385906804319916 - type: cos_sim_spearman value: 56.60927284389835 - type: euclidean_pearson value: 58.220472472555414 - type: euclidean_spearman value: 56.60927284389835 - type: manhattan_pearson value: 57.974972842168704 - type: manhattan_spearman value: 56.38609220634484 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (it-en) config: it-en split: test metrics: - type: cos_sim_pearson value: 33.56148532200449 - type: cos_sim_spearman value: 30.46169688812801 - type: euclidean_pearson value: 34.03749511332228 - type: euclidean_spearman value: 30.46169688812801 - type: manhattan_pearson value: 33.51842606041771 - type: manhattan_spearman value: 30.87826743052681 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (nl-en) config: nl-en split: test metrics: - type: cos_sim_pearson value: 37.01008110523631 - type: cos_sim_spearman value: 36.46124293832437 - type: euclidean_pearson value: 37.860431566730725 - type: euclidean_spearman value: 36.46124293832437 - type: manhattan_pearson value: 37.84974555851177 - type: manhattan_spearman value: 37.026498066678556 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (en) config: en split: test metrics: - type: cos_sim_pearson value: 56.563590445291226 - type: cos_sim_spearman value: 62.65994888539158 - type: euclidean_pearson value: 61.43083003163841 - type: euclidean_spearman value: 62.65994888539158 - type: manhattan_pearson value: 61.530512036243564 - type: manhattan_spearman value: 62.65300646176863 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de) config: de split: test metrics: - type: cos_sim_pearson value: 28.29024604941182 - type: cos_sim_spearman value: 42.084625834786046 - type: euclidean_pearson value: 30.271611311423545 - type: euclidean_spearman value: 42.084625834786046 - type: manhattan_pearson value: 30.19034939394144 - type: manhattan_spearman value: 42.02260224541176 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es) config: es split: test metrics: - type: cos_sim_pearson value: 42.79846662914459 - type: cos_sim_spearman value: 53.8129210907069 - type: euclidean_pearson value: 48.21779716691527 - type: euclidean_spearman value: 53.8129210907069 - type: manhattan_pearson value: 48.35900342355713 - type: manhattan_spearman value: 53.94896150957018 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (pl) config: pl split: test metrics: - type: cos_sim_pearson value: 9.818882867657061 - type: cos_sim_spearman value: 24.41994279795319 - type: euclidean_pearson value: 4.813919367736767 - type: euclidean_spearman value: 24.41994279795319 - type: manhattan_pearson value: 4.602063702670144 - type: manhattan_spearman value: 24.218951967147824 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (tr) config: tr split: test metrics: - type: cos_sim_pearson value: 27.410972788257048 - type: cos_sim_spearman value: 40.44872572093327 - type: euclidean_pearson value: 33.742359285090565 - type: euclidean_spearman value: 40.44872572093327 - type: manhattan_pearson value: 33.90231904900396 - type: manhattan_spearman value: 40.19149257794821 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (ar) config: ar split: test metrics: - type: cos_sim_pearson value: 31.3322380268429 - type: cos_sim_spearman value: 31.200490449337714 - type: euclidean_pearson value: 32.130848968646525 - type: euclidean_spearman value: 31.200490449337714 - type: manhattan_pearson value: 32.14834980954443 - type: manhattan_spearman value: 31.427049121627025 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (ru) config: ru split: test metrics: - type: cos_sim_pearson value: 3.4365537430337683 - type: cos_sim_spearman value: 12.125486695771288 - type: euclidean_pearson value: 8.134889656987513 - type: euclidean_spearman value: 12.125486695771288 - type: manhattan_pearson value: 8.163310600014055 - type: manhattan_spearman value: 12.129258700591722 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (zh) config: zh split: test metrics: - type: cos_sim_pearson value: 27.320332340418773 - type: cos_sim_spearman value: 32.900042162025 - type: euclidean_pearson value: 30.195166197236723 - type: euclidean_spearman value: 32.900041812396196 - type: manhattan_pearson value: 30.146557575933087 - type: manhattan_spearman value: 32.96907086076731 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (fr) config: fr split: test metrics: - type: cos_sim_pearson value: 69.2830779511937 - type: cos_sim_spearman value: 77.68846630995027 - type: euclidean_pearson value: 73.034747757096 - type: euclidean_spearman value: 77.68846630995027 - type: manhattan_pearson value: 73.03548141166142 - type: manhattan_spearman value: 77.65745427658017 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-en) config: de-en split: test metrics: - type: cos_sim_pearson value: 39.71606949409573 - type: cos_sim_spearman value: 46.8990508231622 - type: euclidean_pearson value: 46.606091669710025 - type: euclidean_spearman value: 46.8990508231622 - type: manhattan_pearson value: 46.39554347396642 - type: manhattan_spearman value: 46.59771734872816 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es-en) config: es-en split: test metrics: - type: cos_sim_pearson value: 54.53158773665186 - type: cos_sim_spearman value: 65.18822674266846 - type: euclidean_pearson value: 58.19324925326185 - type: euclidean_spearman value: 65.18822674266846 - type: manhattan_pearson value: 57.83750769005698 - type: manhattan_spearman value: 65.02074812497972 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (it) config: it split: test metrics: - type: cos_sim_pearson value: 56.77648080772914 - type: cos_sim_spearman value: 60.64694762935356 - type: euclidean_pearson value: 58.1456140359783 - type: euclidean_spearman value: 60.64694762935356 - type: manhattan_pearson value: 58.03342495626636 - type: manhattan_spearman value: 60.384166246014914 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (pl-en) config: pl-en split: test metrics: - type: cos_sim_pearson value: 47.2314368564716 - type: cos_sim_spearman value: 42.96651621279448 - type: euclidean_pearson value: 47.136522518411184 - type: euclidean_spearman value: 42.96651621279448 - type: manhattan_pearson value: 48.71469489220069 - type: manhattan_spearman value: 44.518895193324646 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (zh-en) config: zh-en split: test metrics: - type: cos_sim_pearson value: 25.589949160802995 - type: cos_sim_spearman value: 20.153084379882284 - type: euclidean_pearson value: 26.82363451623337 - type: euclidean_spearman value: 20.153084379882284 - type: manhattan_pearson value: 25.843715884495634 - type: manhattan_spearman value: 18.901328744286676 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es-it) config: es-it split: test metrics: - type: cos_sim_pearson value: 48.45790617159233 - type: cos_sim_spearman value: 55.28609467652911 - type: euclidean_pearson value: 51.88464425822175 - type: euclidean_spearman value: 55.28609467652911 - type: manhattan_pearson value: 51.815736921803136 - type: manhattan_spearman value: 55.33932627352348 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-fr) config: de-fr split: test metrics: - type: cos_sim_pearson value: 44.7093430670243 - type: cos_sim_spearman value: 55.04493953270152 - type: euclidean_pearson value: 47.90591946944973 - type: euclidean_spearman value: 55.04493953270152 - type: manhattan_pearson value: 47.964230618301606 - type: manhattan_spearman value: 56.09186738739794 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-pl) config: de-pl split: test metrics: - type: cos_sim_pearson value: 25.093485946833393 - type: cos_sim_spearman value: 33.93510205658959 - type: euclidean_pearson value: 27.454896639869027 - type: euclidean_spearman value: 33.93510205658959 - type: manhattan_pearson value: 24.299109196300538 - type: manhattan_spearman value: 32.51857329560673 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (fr-pl) config: fr-pl split: test metrics: - type: cos_sim_pearson value: 40.9753045484768 - type: cos_sim_spearman value: 28.17180849095055 - type: euclidean_pearson value: 40.382800203298906 - type: euclidean_spearman value: 28.17180849095055 - type: manhattan_pearson value: 34.084425723423486 - type: manhattan_spearman value: 28.17180849095055 - task: type: STS dataset: type: mteb/stsbenchmark-sts name: MTEB STSBenchmark config: default split: test metrics: - type: cos_sim_pearson value: 84.76003618726351 - type: cos_sim_spearman value: 85.52030817522575 - type: euclidean_pearson value: 85.5039926987335 - type: euclidean_spearman value: 85.52030817522575 - type: manhattan_pearson value: 85.51493965359182 - type: manhattan_spearman value: 85.52189380846832 - task: type: Reranking dataset: type: mteb/scidocs-reranking name: MTEB SciDocsRR config: default split: test metrics: - type: map value: 73.96228332723271 - type: mrr value: 91.34847813769382 - task: type: Retrieval dataset: type: scifact name: MTEB SciFact config: default split: test metrics: - type: map_at_1 value: 32.372 - type: map_at_10 value: 41.02 - type: map_at_100 value: 41.907 - type: map_at_1000 value: 41.967 - type: map_at_3 value: 38.244 - type: map_at_5 value: 39.786 - type: ndcg_at_1 value: 34.666999999999994 - type: ndcg_at_10 value: 45.76 - type: ndcg_at_100 value: 50.163999999999994 - type: ndcg_at_1000 value: 51.956 - type: ndcg_at_3 value: 40.687 - type: ndcg_at_5 value: 43.143 - type: precision_at_1 value: 34.666999999999994 - type: precision_at_10 value: 6.7 - type: precision_at_100 value: 0.907 - type: precision_at_1000 value: 0.107 - type: precision_at_3 value: 16.667 - type: precision_at_5 value: 11.466999999999999 - type: recall_at_1 value: 32.372 - type: recall_at_10 value: 59.061 - type: recall_at_100 value: 79.733 - type: recall_at_1000 value: 94.167 - type: recall_at_3 value: 45.161 - type: recall_at_5 value: 51.439 - task: type: PairClassification dataset: type: mteb/sprintduplicatequestions-pairclassification name: MTEB SprintDuplicateQuestions config: default split: test metrics: - type: cos_sim_accuracy value: 99.6960396039604 - type: cos_sim_ap value: 91.22814257221482 - type: cos_sim_f1 value: 84.43775100401606 - type: cos_sim_precision value: 84.77822580645162 - type: cos_sim_recall value: 84.1 - type: dot_accuracy value: 99.6960396039604 - type: dot_ap value: 91.22814257221482 - type: dot_f1 value: 84.43775100401606 - type: dot_precision value: 84.77822580645162 - type: dot_recall value: 84.1 - type: euclidean_accuracy value: 99.6960396039604 - type: euclidean_ap value: 91.22814257221482 - type: euclidean_f1 value: 84.43775100401606 - type: euclidean_precision value: 84.77822580645162 - type: euclidean_recall value: 84.1 - type: manhattan_accuracy value: 99.6960396039604 - type: manhattan_ap value: 91.18887077921163 - type: manhattan_f1 value: 84.27991886409735 - type: manhattan_precision value: 85.49382716049382 - type: manhattan_recall value: 83.1 - type: max_accuracy value: 99.6960396039604 - type: max_ap value: 91.22814257221482 - type: max_f1 value: 84.43775100401606 - task: type: Clustering dataset: type: mteb/stackexchange-clustering name: MTEB StackExchangeClustering config: default split: test metrics: - type: v_measure value: 63.13072579524015 - task: type: Clustering dataset: type: mteb/stackexchange-clustering-p2p name: MTEB StackExchangeClusteringP2P config: default split: test metrics: - type: v_measure value: 35.681141375580225 - task: type: Reranking dataset: type: mteb/stackoverflowdupquestions-reranking name: MTEB StackOverflowDupQuestions config: default split: test metrics: - type: map value: 48.46269194141537 - type: mrr value: 49.11958343943638 - task: type: Summarization dataset: type: mteb/summeval name: MTEB SummEval config: default split: test metrics: - type: cos_sim_pearson value: 30.709572612837498 - type: cos_sim_spearman value: 31.3940211538976 - type: dot_pearson value: 30.709578240668765 - type: dot_spearman value: 31.3940211538976 - task: type: Retrieval dataset: type: trec-covid name: MTEB TRECCOVID config: default split: test metrics: - type: map_at_1 value: 0.151 - type: map_at_10 value: 0.822 - type: map_at_100 value: 4.846 - type: map_at_1000 value: 13.117 - type: map_at_3 value: 0.349 - type: map_at_5 value: 0.49500000000000005 - type: ndcg_at_1 value: 48.0 - type: ndcg_at_10 value: 40.699000000000005 - type: ndcg_at_100 value: 35.455 - type: ndcg_at_1000 value: 35.067 - type: ndcg_at_3 value: 44.519999999999996 - type: ndcg_at_5 value: 42.697 - type: precision_at_1 value: 54.0 - type: precision_at_10 value: 44.0 - type: precision_at_100 value: 37.72 - type: precision_at_1000 value: 16.302 - type: precision_at_3 value: 50.0 - type: precision_at_5 value: 47.199999999999996 - type: recall_at_1 value: 0.151 - type: recall_at_10 value: 1.109 - type: recall_at_100 value: 8.644 - type: recall_at_1000 value: 34.566 - type: recall_at_3 value: 0.394 - type: recall_at_5 value: 0.601 - task: type: Retrieval dataset: type: webis-touche2020 name: MTEB Touche2020 config: default split: test metrics: - type: map_at_1 value: 1.786 - type: map_at_10 value: 8.379 - type: map_at_100 value: 13.618 - type: map_at_1000 value: 15.15 - type: map_at_3 value: 3.7900000000000005 - type: map_at_5 value: 6.1530000000000005 - type: ndcg_at_1 value: 19.387999999999998 - type: ndcg_at_10 value: 20.296 - type: ndcg_at_100 value: 31.828 - type: ndcg_at_1000 value: 43.968 - type: ndcg_at_3 value: 19.583000000000002 - type: ndcg_at_5 value: 21.066 - type: precision_at_1 value: 22.448999999999998 - type: precision_at_10 value: 19.592000000000002 - type: precision_at_100 value: 7.041 - type: precision_at_1000 value: 1.49 - type: precision_at_3 value: 22.448999999999998 - type: precision_at_5 value: 24.490000000000002 - type: recall_at_1 value: 1.786 - type: recall_at_10 value: 14.571000000000002 - type: recall_at_100 value: 44.247 - type: recall_at_1000 value: 80.36 - type: recall_at_3 value: 5.117 - type: recall_at_5 value: 9.449 - task: type: Classification dataset: type: mteb/toxic_conversations_50k name: MTEB ToxicConversationsClassification config: default split: test metrics: - type: accuracy value: 68.19919999999999 - type: ap value: 14.328836562980976 - type: f1 value: 53.33893474325896 - task: type: Classification dataset: type: mteb/tweet_sentiment_extraction name: MTEB TweetSentimentExtractionClassification config: default split: test metrics: - type: accuracy value: 62.71080928126768 - type: f1 value: 62.35221892617029 - task: type: Clustering dataset: type: mteb/twentynewsgroups-clustering name: MTEB TwentyNewsgroupsClustering config: default split: test metrics: - type: v_measure value: 48.099101871064484 - task: type: PairClassification dataset: type: mteb/twittersemeval2015-pairclassification name: MTEB TwitterSemEval2015 config: default split: test metrics: - type: cos_sim_accuracy value: 87.60207426834357 - type: cos_sim_ap value: 78.25096573546108 - type: cos_sim_f1 value: 71.740233384069 - type: cos_sim_precision value: 69.07669760625306 - type: cos_sim_recall value: 74.6174142480211 - type: dot_accuracy value: 87.60207426834357 - type: dot_ap value: 78.25097910093768 - type: dot_f1 value: 71.740233384069 - type: dot_precision value: 69.07669760625306 - type: dot_recall value: 74.6174142480211 - type: euclidean_accuracy value: 87.60207426834357 - type: euclidean_ap value: 78.25097099603116 - type: euclidean_f1 value: 71.740233384069 - type: euclidean_precision value: 69.07669760625306 - type: euclidean_recall value: 74.6174142480211 - type: manhattan_accuracy value: 87.61399535077786 - type: manhattan_ap value: 78.238484943708 - type: manhattan_f1 value: 71.77797490812317 - type: manhattan_precision value: 69.05632772494513 - type: manhattan_recall value: 74.72295514511873 - type: max_accuracy value: 87.61399535077786 - type: max_ap value: 78.25097910093768 - type: max_f1 value: 71.77797490812317 - task: type: PairClassification dataset: type: mteb/twitterurlcorpus-pairclassification name: MTEB TwitterURLCorpus config: default split: test metrics: - type: cos_sim_accuracy value: 89.17413746264602 - type: cos_sim_ap value: 86.04575990028458 - type: cos_sim_f1 value: 78.52034894604814 - type: cos_sim_precision value: 76.42300123897675 - type: cos_sim_recall value: 80.73606405913151 - type: dot_accuracy value: 89.17413746264602 - type: dot_ap value: 86.04575880500646 - type: dot_f1 value: 78.52034894604814 - type: dot_precision value: 76.42300123897675 - type: dot_recall value: 80.73606405913151 - type: euclidean_accuracy value: 89.17413746264602 - type: euclidean_ap value: 86.04575106124874 - type: euclidean_f1 value: 78.52034894604814 - type: euclidean_precision value: 76.42300123897675 - type: euclidean_recall value: 80.73606405913151 - type: manhattan_accuracy value: 89.14891139830016 - type: manhattan_ap value: 86.01748033351211 - type: manhattan_f1 value: 78.48817724818471 - type: manhattan_precision value: 76.00057690920892 - type: manhattan_recall value: 81.14413304588851 - type: max_accuracy value: 89.17413746264602 - type: max_ap value: 86.04575990028458 - type: max_f1 value: 78.52034894604814 --- # sentence-transformers/sentence-t5-base This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space. The model works well for sentence similarity tasks, but doesn't perform that well for semantic search tasks. This model was converted from the Tensorflow model [st5-base-1](https://tfhub.dev/google/sentence-t5/st5-base/1) to PyTorch. When using this model, have a look at the publication: [Sentence-T5: Scalable sentence encoders from pre-trained text-to-text models](https://arxiv.org/abs/2108.08877). The tfhub model and this PyTorch model can produce slightly different embeddings, however, when run on the same benchmarks, they produce identical results. The model uses only the encoder from a T5-base model. The weights are stored in FP16. ## Usage (Sentence-Transformers) Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: ``` pip install -U sentence-transformers ``` Then you can use the model like this: ```python from sentence_transformers import SentenceTransformer sentences = ["This is an example sentence", "Each sentence is converted"] model = SentenceTransformer('sentence-transformers/sentence-t5-base') embeddings = model.encode(sentences) print(embeddings) ``` The model requires sentence-transformers version 2.2.0 or newer. ## Evaluation Results For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=sentence-transformers/sentence-t5-base) ## Citing & Authors If you find this model helpful, please cite the respective publication: [Sentence-T5: Scalable sentence encoders from pre-trained text-to-text models](https://arxiv.org/abs/2108.08877)