|
--- |
|
license: cc-by-nc-4.0 |
|
tags: |
|
- mteb |
|
model-index: |
|
- name: text_sonar_basic_encoder_normalized |
|
results: |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: PL-MTEB/8tags-clustering |
|
name: MTEB 8TagsClustering |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 18.787544117314575 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/AFQMC |
|
name: MTEB AFQMC |
|
config: default |
|
split: validation |
|
revision: b44c3b011063adb25877c13823db83bb193913c4 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 17.97026675319667 |
|
- type: cos_sim_spearman |
|
value: 17.63407829948615 |
|
- type: euclidean_pearson |
|
value: 17.704571608660725 |
|
- type: euclidean_spearman |
|
value: 17.634078298828143 |
|
- type: manhattan_pearson |
|
value: 17.606959101509464 |
|
- type: manhattan_spearman |
|
value: 17.549620164990085 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/ATEC |
|
name: MTEB ATEC |
|
config: default |
|
split: test |
|
revision: 0f319b1142f28d00e055a6770f3f726ae9b7d865 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 27.670887504789675 |
|
- type: cos_sim_spearman |
|
value: 26.176629407301782 |
|
- type: euclidean_pearson |
|
value: 28.878485717935586 |
|
- type: euclidean_spearman |
|
value: 26.176635036613355 |
|
- type: manhattan_pearson |
|
value: 28.782373978690103 |
|
- type: manhattan_spearman |
|
value: 26.055266444113794 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/allegro-reviews |
|
name: MTEB AllegroReviews |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 29.62226640159046 |
|
- type: f1 |
|
value: 27.632722290701047 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (en) |
|
config: en |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 81.49253731343285 |
|
- type: ap |
|
value: 46.61440947240349 |
|
- type: f1 |
|
value: 75.68925212232107 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (de) |
|
config: de |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 72.02355460385438 |
|
- type: ap |
|
value: 83.13664983282676 |
|
- type: f1 |
|
value: 70.48997817871013 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (en-ext) |
|
config: en-ext |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 82.09145427286357 |
|
- type: ap |
|
value: 31.45181004731995 |
|
- type: f1 |
|
value: 69.41750580313406 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (ja) |
|
config: ja |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 71.78800856531049 |
|
- type: ap |
|
value: 19.65443896353892 |
|
- type: f1 |
|
value: 58.436688187826334 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_polarity |
|
name: MTEB AmazonPolarityClassification |
|
config: default |
|
split: test |
|
revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
|
metrics: |
|
- type: accuracy |
|
value: 62.73074999999999 |
|
- type: ap |
|
value: 58.2839375458089 |
|
- type: f1 |
|
value: 62.16204082406629 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (en) |
|
config: en |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 31.552000000000003 |
|
- type: f1 |
|
value: 31.125328770568277 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (de) |
|
config: de |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 34.611999999999995 |
|
- type: f1 |
|
value: 33.93738697105999 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (es) |
|
config: es |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 35.172 |
|
- type: f1 |
|
value: 34.14112656493798 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (fr) |
|
config: fr |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 34.910000000000004 |
|
- type: f1 |
|
value: 34.276631172288965 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (ja) |
|
config: ja |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 31.844 |
|
- type: f1 |
|
value: 31.478780923476368 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (zh) |
|
config: zh |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 31.912000000000003 |
|
- type: f1 |
|
value: 31.384992191831312 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: DDSC/angry-tweets |
|
name: MTEB AngryTweetsClassification |
|
config: default |
|
split: test |
|
revision: 20b0e6081892e78179356fada741b7afa381443d |
|
metrics: |
|
- type: accuracy |
|
value: 49.61795606494747 |
|
- type: f1 |
|
value: 48.63625944670304 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana |
|
name: MTEB ArguAna |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 8.677 |
|
- type: map_at_10 |
|
value: 14.732000000000001 |
|
- type: map_at_100 |
|
value: 15.501999999999999 |
|
- type: map_at_1000 |
|
value: 15.583 |
|
- type: map_at_3 |
|
value: 12.553 |
|
- type: map_at_5 |
|
value: 13.822999999999999 |
|
- type: mrr_at_1 |
|
value: 8.819 |
|
- type: mrr_at_10 |
|
value: 14.787 |
|
- type: mrr_at_100 |
|
value: 15.557000000000002 |
|
- type: mrr_at_1000 |
|
value: 15.638 |
|
- type: mrr_at_3 |
|
value: 12.648000000000001 |
|
- type: mrr_at_5 |
|
value: 13.879 |
|
- type: ndcg_at_1 |
|
value: 8.677 |
|
- type: ndcg_at_10 |
|
value: 18.295 |
|
- type: ndcg_at_100 |
|
value: 22.353 |
|
- type: ndcg_at_1000 |
|
value: 24.948999999999998 |
|
- type: ndcg_at_3 |
|
value: 13.789000000000001 |
|
- type: ndcg_at_5 |
|
value: 16.075 |
|
- type: precision_at_1 |
|
value: 8.677 |
|
- type: precision_at_10 |
|
value: 2.98 |
|
- type: precision_at_100 |
|
value: 0.49500000000000005 |
|
- type: precision_at_1000 |
|
value: 0.07100000000000001 |
|
- type: precision_at_3 |
|
value: 5.785 |
|
- type: precision_at_5 |
|
value: 4.58 |
|
- type: recall_at_1 |
|
value: 8.677 |
|
- type: recall_at_10 |
|
value: 29.801 |
|
- type: recall_at_100 |
|
value: 49.502 |
|
- type: recall_at_1000 |
|
value: 70.91 |
|
- type: recall_at_3 |
|
value: 17.354 |
|
- type: recall_at_5 |
|
value: 22.902 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana-pl |
|
name: MTEB ArguAna-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 7.752000000000001 |
|
- type: map_at_10 |
|
value: 12.248000000000001 |
|
- type: map_at_100 |
|
value: 12.882 |
|
- type: map_at_1000 |
|
value: 12.963 |
|
- type: map_at_3 |
|
value: 10.574 |
|
- type: map_at_5 |
|
value: 11.566 |
|
- type: mrr_at_1 |
|
value: 7.824000000000001 |
|
- type: mrr_at_10 |
|
value: 12.293 |
|
- type: mrr_at_100 |
|
value: 12.928 |
|
- type: mrr_at_1000 |
|
value: 13.008000000000001 |
|
- type: mrr_at_3 |
|
value: 10.586 |
|
- type: mrr_at_5 |
|
value: 11.599 |
|
- type: ndcg_at_1 |
|
value: 7.752000000000001 |
|
- type: ndcg_at_10 |
|
value: 15.035000000000002 |
|
- type: ndcg_at_100 |
|
value: 18.497 |
|
- type: ndcg_at_1000 |
|
value: 20.896 |
|
- type: ndcg_at_3 |
|
value: 11.578 |
|
- type: ndcg_at_5 |
|
value: 13.38 |
|
- type: precision_at_1 |
|
value: 7.752000000000001 |
|
- type: precision_at_10 |
|
value: 2.404 |
|
- type: precision_at_100 |
|
value: 0.411 |
|
- type: precision_at_1000 |
|
value: 0.061 |
|
- type: precision_at_3 |
|
value: 4.836 |
|
- type: precision_at_5 |
|
value: 3.784 |
|
- type: recall_at_1 |
|
value: 7.752000000000001 |
|
- type: recall_at_10 |
|
value: 24.04 |
|
- type: recall_at_100 |
|
value: 41.11 |
|
- type: recall_at_1000 |
|
value: 60.597 |
|
- type: recall_at_3 |
|
value: 14.509 |
|
- type: recall_at_5 |
|
value: 18.919 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-p2p |
|
name: MTEB ArxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
|
metrics: |
|
- type: v_measure |
|
value: 26.81177290816682 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-s2s |
|
name: MTEB ArxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 |
|
metrics: |
|
- type: v_measure |
|
value: 24.346811178757022 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/askubuntudupquestions-reranking |
|
name: MTEB AskUbuntuDupQuestions |
|
config: default |
|
split: test |
|
revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
|
metrics: |
|
- type: map |
|
value: 50.88606427049027 |
|
- type: mrr |
|
value: 65.13004001231148 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/biosses-sts |
|
name: MTEB BIOSSES |
|
config: default |
|
split: test |
|
revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 77.15058512395619 |
|
- type: cos_sim_spearman |
|
value: 79.10541692841936 |
|
- type: euclidean_pearson |
|
value: 75.30525535929353 |
|
- type: euclidean_spearman |
|
value: 79.10541692841936 |
|
- type: manhattan_pearson |
|
value: 75.33508042552984 |
|
- type: manhattan_spearman |
|
value: 78.84577245802708 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/BQ |
|
name: MTEB BQ |
|
config: default |
|
split: test |
|
revision: e3dda5e115e487b39ec7e618c0c6a29137052a55 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 37.84739189558895 |
|
- type: cos_sim_spearman |
|
value: 37.662710610486265 |
|
- type: euclidean_pearson |
|
value: 37.5407537185213 |
|
- type: euclidean_spearman |
|
value: 37.66272446700578 |
|
- type: manhattan_pearson |
|
value: 37.863820146709706 |
|
- type: manhattan_spearman |
|
value: 38.09120266204032 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/bucc-bitext-mining |
|
name: MTEB BUCC (de-en) |
|
config: de-en |
|
split: test |
|
revision: d51519689f32196a32af33b075a01d0e7c51e252 |
|
metrics: |
|
- type: accuracy |
|
value: 98.97703549060543 |
|
- type: f1 |
|
value: 98.82393876130828 |
|
- type: precision |
|
value: 98.74913013221992 |
|
- type: recall |
|
value: 98.97703549060543 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/bucc-bitext-mining |
|
name: MTEB BUCC (fr-en) |
|
config: fr-en |
|
split: test |
|
revision: d51519689f32196a32af33b075a01d0e7c51e252 |
|
metrics: |
|
- type: accuracy |
|
value: 98.34910851860005 |
|
- type: f1 |
|
value: 98.09487123046446 |
|
- type: precision |
|
value: 97.97032063981217 |
|
- type: recall |
|
value: 98.34910851860005 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/bucc-bitext-mining |
|
name: MTEB BUCC (ru-en) |
|
config: ru-en |
|
split: test |
|
revision: d51519689f32196a32af33b075a01d0e7c51e252 |
|
metrics: |
|
- type: accuracy |
|
value: 97.60304814686526 |
|
- type: f1 |
|
value: 97.36520032328832 |
|
- type: precision |
|
value: 97.24743101258517 |
|
- type: recall |
|
value: 97.60304814686526 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/bucc-bitext-mining |
|
name: MTEB BUCC (zh-en) |
|
config: zh-en |
|
split: test |
|
revision: d51519689f32196a32af33b075a01d0e7c51e252 |
|
metrics: |
|
- type: accuracy |
|
value: 98.78883622959452 |
|
- type: f1 |
|
value: 98.71862383710724 |
|
- type: precision |
|
value: 98.68351764086361 |
|
- type: recall |
|
value: 98.78883622959452 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/banking77 |
|
name: MTEB Banking77Classification |
|
config: default |
|
split: test |
|
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
|
metrics: |
|
- type: accuracy |
|
value: 73.49675324675324 |
|
- type: f1 |
|
value: 72.88538992490979 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: jinaai/big-patent-clustering |
|
name: MTEB BigPatentClustering |
|
config: default |
|
split: test |
|
revision: 62d5330920bca426ce9d3c76ea914f15fc83e891 |
|
metrics: |
|
- type: v_measure |
|
value: 6.801245618724224 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-p2p |
|
name: MTEB BiorxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 |
|
metrics: |
|
- type: v_measure |
|
value: 20.6156033971932 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-s2s |
|
name: MTEB BiorxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 |
|
metrics: |
|
- type: v_measure |
|
value: 19.077587707743156 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: slvnwhrl/blurbs-clustering-p2p |
|
name: MTEB BlurbsClusteringP2P |
|
config: default |
|
split: test |
|
revision: a2dd5b02a77de3466a3eaa98ae586b5610314496 |
|
metrics: |
|
- type: v_measure |
|
value: 27.00349462858046 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: slvnwhrl/blurbs-clustering-s2s |
|
name: MTEB BlurbsClusteringS2S |
|
config: default |
|
split: test |
|
revision: 9bfff9a7f8f6dc6ffc9da71c48dd48b68696471d |
|
metrics: |
|
- type: v_measure |
|
value: 14.845348131791589 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: strombergnlp/bornholmsk_parallel |
|
name: MTEB BornholmBitextMining |
|
config: default |
|
split: test |
|
revision: 3bc5cfb4ec514264fe2db5615fac9016f7251552 |
|
metrics: |
|
- type: accuracy |
|
value: 54.0 |
|
- type: f1 |
|
value: 47.37026862026861 |
|
- type: precision |
|
value: 45.0734126984127 |
|
- type: recall |
|
value: 54.0 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/cbd |
|
name: MTEB CBD |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 63.83000000000001 |
|
- type: ap |
|
value: 18.511972946438764 |
|
- type: f1 |
|
value: 53.16787370496645 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/cdsce-pairclassification |
|
name: MTEB CDSC-E |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 84.39999999999999 |
|
- type: cos_sim_ap |
|
value: 59.968589741258036 |
|
- type: cos_sim_f1 |
|
value: 54.90909090909091 |
|
- type: cos_sim_precision |
|
value: 41.94444444444444 |
|
- type: cos_sim_recall |
|
value: 79.47368421052632 |
|
- type: dot_accuracy |
|
value: 84.39999999999999 |
|
- type: dot_ap |
|
value: 59.968589741258036 |
|
- type: dot_f1 |
|
value: 54.90909090909091 |
|
- type: dot_precision |
|
value: 41.94444444444444 |
|
- type: dot_recall |
|
value: 79.47368421052632 |
|
- type: euclidean_accuracy |
|
value: 84.39999999999999 |
|
- type: euclidean_ap |
|
value: 59.968589741258036 |
|
- type: euclidean_f1 |
|
value: 54.90909090909091 |
|
- type: euclidean_precision |
|
value: 41.94444444444444 |
|
- type: euclidean_recall |
|
value: 79.47368421052632 |
|
- type: manhattan_accuracy |
|
value: 84.39999999999999 |
|
- type: manhattan_ap |
|
value: 60.094893481041154 |
|
- type: manhattan_f1 |
|
value: 55.452865064695004 |
|
- type: manhattan_precision |
|
value: 42.73504273504273 |
|
- type: manhattan_recall |
|
value: 78.94736842105263 |
|
- type: max_accuracy |
|
value: 84.39999999999999 |
|
- type: max_ap |
|
value: 60.094893481041154 |
|
- type: max_f1 |
|
value: 55.452865064695004 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: PL-MTEB/cdscr-sts |
|
name: MTEB CDSC-R |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 83.8427417206754 |
|
- type: cos_sim_spearman |
|
value: 85.76946319798301 |
|
- type: euclidean_pearson |
|
value: 79.43901249477852 |
|
- type: euclidean_spearman |
|
value: 85.76946319798301 |
|
- type: manhattan_pearson |
|
value: 79.81046681362531 |
|
- type: manhattan_spearman |
|
value: 86.24115514951988 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/CLSClusteringP2P |
|
name: MTEB CLSClusteringP2P |
|
config: default |
|
split: test |
|
revision: 4b6227591c6c1a73bc76b1055f3b7f3588e72476 |
|
metrics: |
|
- type: v_measure |
|
value: 27.432031859995952 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/CLSClusteringS2S |
|
name: MTEB CLSClusteringS2S |
|
config: default |
|
split: test |
|
revision: e458b3f5414b62b7f9f83499ac1f5497ae2e869f |
|
metrics: |
|
- type: v_measure |
|
value: 28.32367305628197 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/CMedQAv1-reranking |
|
name: MTEB CMedQAv1 |
|
config: default |
|
split: test |
|
revision: 8d7f1e942507dac42dc58017c1a001c3717da7df |
|
metrics: |
|
- type: map |
|
value: 34.30720667137015 |
|
- type: mrr |
|
value: 40.24416666666666 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/CMedQAv2-reranking |
|
name: MTEB CMedQAv2 |
|
config: default |
|
split: test |
|
revision: 23d186750531a14a0357ca22cd92d712fd512ea0 |
|
metrics: |
|
- type: map |
|
value: 35.87700379259406 |
|
- type: mrr |
|
value: 40.80206349206349 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackAndroidRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 7.655000000000001 |
|
- type: map_at_10 |
|
value: 11.681999999999999 |
|
- type: map_at_100 |
|
value: 12.464 |
|
- type: map_at_1000 |
|
value: 12.603 |
|
- type: map_at_3 |
|
value: 10.514 |
|
- type: map_at_5 |
|
value: 11.083 |
|
- type: mrr_at_1 |
|
value: 10.157 |
|
- type: mrr_at_10 |
|
value: 14.773 |
|
- type: mrr_at_100 |
|
value: 15.581999999999999 |
|
- type: mrr_at_1000 |
|
value: 15.68 |
|
- type: mrr_at_3 |
|
value: 13.519 |
|
- type: mrr_at_5 |
|
value: 14.049 |
|
- type: ndcg_at_1 |
|
value: 10.157 |
|
- type: ndcg_at_10 |
|
value: 14.527999999999999 |
|
- type: ndcg_at_100 |
|
value: 18.695999999999998 |
|
- type: ndcg_at_1000 |
|
value: 22.709 |
|
- type: ndcg_at_3 |
|
value: 12.458 |
|
- type: ndcg_at_5 |
|
value: 13.152 |
|
- type: precision_at_1 |
|
value: 10.157 |
|
- type: precision_at_10 |
|
value: 2.976 |
|
- type: precision_at_100 |
|
value: 0.634 |
|
- type: precision_at_1000 |
|
value: 0.131 |
|
- type: precision_at_3 |
|
value: 6.152 |
|
- type: precision_at_5 |
|
value: 4.378 |
|
- type: recall_at_1 |
|
value: 7.655000000000001 |
|
- type: recall_at_10 |
|
value: 20.105 |
|
- type: recall_at_100 |
|
value: 39.181 |
|
- type: recall_at_1000 |
|
value: 68.06400000000001 |
|
- type: recall_at_3 |
|
value: 14.033000000000001 |
|
- type: recall_at_5 |
|
value: 16.209 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackEnglishRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.2329999999999997 |
|
- type: map_at_10 |
|
value: 5.378 |
|
- type: map_at_100 |
|
value: 5.774 |
|
- type: map_at_1000 |
|
value: 5.863 |
|
- type: map_at_3 |
|
value: 4.598 |
|
- type: map_at_5 |
|
value: 4.9750000000000005 |
|
- type: mrr_at_1 |
|
value: 4.076 |
|
- type: mrr_at_10 |
|
value: 6.679 |
|
- type: mrr_at_100 |
|
value: 7.151000000000001 |
|
- type: mrr_at_1000 |
|
value: 7.24 |
|
- type: mrr_at_3 |
|
value: 5.722 |
|
- type: mrr_at_5 |
|
value: 6.2059999999999995 |
|
- type: ndcg_at_1 |
|
value: 4.076 |
|
- type: ndcg_at_10 |
|
value: 6.994 |
|
- type: ndcg_at_100 |
|
value: 9.366 |
|
- type: ndcg_at_1000 |
|
value: 12.181000000000001 |
|
- type: ndcg_at_3 |
|
value: 5.356000000000001 |
|
- type: ndcg_at_5 |
|
value: 6.008 |
|
- type: precision_at_1 |
|
value: 4.076 |
|
- type: precision_at_10 |
|
value: 1.459 |
|
- type: precision_at_100 |
|
value: 0.334 |
|
- type: precision_at_1000 |
|
value: 0.075 |
|
- type: precision_at_3 |
|
value: 2.718 |
|
- type: precision_at_5 |
|
value: 2.089 |
|
- type: recall_at_1 |
|
value: 3.2329999999999997 |
|
- type: recall_at_10 |
|
value: 10.749 |
|
- type: recall_at_100 |
|
value: 21.776 |
|
- type: recall_at_1000 |
|
value: 42.278999999999996 |
|
- type: recall_at_3 |
|
value: 6.146999999999999 |
|
- type: recall_at_5 |
|
value: 7.779999999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGamingRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 8.036 |
|
- type: map_at_10 |
|
value: 12.727 |
|
- type: map_at_100 |
|
value: 13.532 |
|
- type: map_at_1000 |
|
value: 13.653 |
|
- type: map_at_3 |
|
value: 11.15 |
|
- type: map_at_5 |
|
value: 11.965 |
|
- type: mrr_at_1 |
|
value: 9.404 |
|
- type: mrr_at_10 |
|
value: 14.493 |
|
- type: mrr_at_100 |
|
value: 15.274 |
|
- type: mrr_at_1000 |
|
value: 15.370000000000001 |
|
- type: mrr_at_3 |
|
value: 12.853 |
|
- type: mrr_at_5 |
|
value: 13.696 |
|
- type: ndcg_at_1 |
|
value: 9.404 |
|
- type: ndcg_at_10 |
|
value: 15.784 |
|
- type: ndcg_at_100 |
|
value: 20.104 |
|
- type: ndcg_at_1000 |
|
value: 23.357 |
|
- type: ndcg_at_3 |
|
value: 12.61 |
|
- type: ndcg_at_5 |
|
value: 13.988 |
|
- type: precision_at_1 |
|
value: 9.404 |
|
- type: precision_at_10 |
|
value: 2.947 |
|
- type: precision_at_100 |
|
value: 0.562 |
|
- type: precision_at_1000 |
|
value: 0.093 |
|
- type: precision_at_3 |
|
value: 6.04 |
|
- type: precision_at_5 |
|
value: 4.4639999999999995 |
|
- type: recall_at_1 |
|
value: 8.036 |
|
- type: recall_at_10 |
|
value: 23.429 |
|
- type: recall_at_100 |
|
value: 43.728 |
|
- type: recall_at_1000 |
|
value: 68.10000000000001 |
|
- type: recall_at_3 |
|
value: 14.99 |
|
- type: recall_at_5 |
|
value: 18.274 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGisRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.653 |
|
- type: map_at_10 |
|
value: 5.941 |
|
- type: map_at_100 |
|
value: 6.512 |
|
- type: map_at_1000 |
|
value: 6.6129999999999995 |
|
- type: map_at_3 |
|
value: 5.2540000000000004 |
|
- type: map_at_5 |
|
value: 5.645 |
|
- type: mrr_at_1 |
|
value: 3.955 |
|
- type: mrr_at_10 |
|
value: 6.4079999999999995 |
|
- type: mrr_at_100 |
|
value: 7.005999999999999 |
|
- type: mrr_at_1000 |
|
value: 7.105 |
|
- type: mrr_at_3 |
|
value: 5.593 |
|
- type: mrr_at_5 |
|
value: 6.051 |
|
- type: ndcg_at_1 |
|
value: 3.955 |
|
- type: ndcg_at_10 |
|
value: 7.342 |
|
- type: ndcg_at_100 |
|
value: 10.543 |
|
- type: ndcg_at_1000 |
|
value: 14.011000000000001 |
|
- type: ndcg_at_3 |
|
value: 5.853 |
|
- type: ndcg_at_5 |
|
value: 6.586 |
|
- type: precision_at_1 |
|
value: 3.955 |
|
- type: precision_at_10 |
|
value: 1.266 |
|
- type: precision_at_100 |
|
value: 0.315 |
|
- type: precision_at_1000 |
|
value: 0.066 |
|
- type: precision_at_3 |
|
value: 2.5989999999999998 |
|
- type: precision_at_5 |
|
value: 1.966 |
|
- type: recall_at_1 |
|
value: 3.653 |
|
- type: recall_at_10 |
|
value: 11.232000000000001 |
|
- type: recall_at_100 |
|
value: 26.625 |
|
- type: recall_at_1000 |
|
value: 54.476 |
|
- type: recall_at_3 |
|
value: 7.269 |
|
- type: recall_at_5 |
|
value: 8.982999999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackMathematicaRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 2.257 |
|
- type: map_at_10 |
|
value: 3.881 |
|
- type: map_at_100 |
|
value: 4.279 |
|
- type: map_at_1000 |
|
value: 4.417 |
|
- type: map_at_3 |
|
value: 3.4070000000000005 |
|
- type: map_at_5 |
|
value: 3.744 |
|
- type: mrr_at_1 |
|
value: 2.9850000000000003 |
|
- type: mrr_at_10 |
|
value: 4.756 |
|
- type: mrr_at_100 |
|
value: 5.228 |
|
- type: mrr_at_1000 |
|
value: 5.354 |
|
- type: mrr_at_3 |
|
value: 4.125 |
|
- type: mrr_at_5 |
|
value: 4.567 |
|
- type: ndcg_at_1 |
|
value: 2.9850000000000003 |
|
- type: ndcg_at_10 |
|
value: 4.936999999999999 |
|
- type: ndcg_at_100 |
|
value: 7.664 |
|
- type: ndcg_at_1000 |
|
value: 12.045 |
|
- type: ndcg_at_3 |
|
value: 3.956 |
|
- type: ndcg_at_5 |
|
value: 4.584 |
|
- type: precision_at_1 |
|
value: 2.9850000000000003 |
|
- type: precision_at_10 |
|
value: 0.9329999999999999 |
|
- type: precision_at_100 |
|
value: 0.29 |
|
- type: precision_at_1000 |
|
value: 0.083 |
|
- type: precision_at_3 |
|
value: 1.949 |
|
- type: precision_at_5 |
|
value: 1.567 |
|
- type: recall_at_1 |
|
value: 2.257 |
|
- type: recall_at_10 |
|
value: 7.382 |
|
- type: recall_at_100 |
|
value: 20.689 |
|
- type: recall_at_1000 |
|
value: 53.586 |
|
- type: recall_at_3 |
|
value: 4.786 |
|
- type: recall_at_5 |
|
value: 6.2829999999999995 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackPhysicsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 6.691 |
|
- type: map_at_10 |
|
value: 9.447 |
|
- type: map_at_100 |
|
value: 10.174 |
|
- type: map_at_1000 |
|
value: 10.308 |
|
- type: map_at_3 |
|
value: 8.187999999999999 |
|
- type: map_at_5 |
|
value: 8.852 |
|
- type: mrr_at_1 |
|
value: 8.566 |
|
- type: mrr_at_10 |
|
value: 12.036 |
|
- type: mrr_at_100 |
|
value: 12.817 |
|
- type: mrr_at_1000 |
|
value: 12.918 |
|
- type: mrr_at_3 |
|
value: 10.539 |
|
- type: mrr_at_5 |
|
value: 11.381 |
|
- type: ndcg_at_1 |
|
value: 8.566 |
|
- type: ndcg_at_10 |
|
value: 11.95 |
|
- type: ndcg_at_100 |
|
value: 15.831000000000001 |
|
- type: ndcg_at_1000 |
|
value: 19.561 |
|
- type: ndcg_at_3 |
|
value: 9.467 |
|
- type: ndcg_at_5 |
|
value: 10.544 |
|
- type: precision_at_1 |
|
value: 8.566 |
|
- type: precision_at_10 |
|
value: 2.387 |
|
- type: precision_at_100 |
|
value: 0.538 |
|
- type: precision_at_1000 |
|
value: 0.104 |
|
- type: precision_at_3 |
|
value: 4.556 |
|
- type: precision_at_5 |
|
value: 3.5029999999999997 |
|
- type: recall_at_1 |
|
value: 6.691 |
|
- type: recall_at_10 |
|
value: 17.375 |
|
- type: recall_at_100 |
|
value: 34.503 |
|
- type: recall_at_1000 |
|
value: 61.492000000000004 |
|
- type: recall_at_3 |
|
value: 10.134 |
|
- type: recall_at_5 |
|
value: 13.056999999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackProgrammersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 4.68 |
|
- type: map_at_10 |
|
value: 6.776999999999999 |
|
- type: map_at_100 |
|
value: 7.207 |
|
- type: map_at_1000 |
|
value: 7.321999999999999 |
|
- type: map_at_3 |
|
value: 6.007 |
|
- type: map_at_5 |
|
value: 6.356000000000001 |
|
- type: mrr_at_1 |
|
value: 5.479 |
|
- type: mrr_at_10 |
|
value: 8.094999999999999 |
|
- type: mrr_at_100 |
|
value: 8.622 |
|
- type: mrr_at_1000 |
|
value: 8.729000000000001 |
|
- type: mrr_at_3 |
|
value: 7.249 |
|
- type: mrr_at_5 |
|
value: 7.6770000000000005 |
|
- type: ndcg_at_1 |
|
value: 5.479 |
|
- type: ndcg_at_10 |
|
value: 8.474 |
|
- type: ndcg_at_100 |
|
value: 11.134 |
|
- type: ndcg_at_1000 |
|
value: 14.759 |
|
- type: ndcg_at_3 |
|
value: 6.888 |
|
- type: ndcg_at_5 |
|
value: 7.504 |
|
- type: precision_at_1 |
|
value: 5.479 |
|
- type: precision_at_10 |
|
value: 1.575 |
|
- type: precision_at_100 |
|
value: 0.35000000000000003 |
|
- type: precision_at_1000 |
|
value: 0.08099999999999999 |
|
- type: precision_at_3 |
|
value: 3.272 |
|
- type: precision_at_5 |
|
value: 2.374 |
|
- type: recall_at_1 |
|
value: 4.68 |
|
- type: recall_at_10 |
|
value: 12.552 |
|
- type: recall_at_100 |
|
value: 24.91 |
|
- type: recall_at_1000 |
|
value: 52.019999999999996 |
|
- type: recall_at_3 |
|
value: 8.057 |
|
- type: recall_at_5 |
|
value: 9.629999999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 4.741750000000001 |
|
- type: map_at_10 |
|
value: 7.103916666666667 |
|
- type: map_at_100 |
|
value: 7.656499999999998 |
|
- type: map_at_1000 |
|
value: 7.767583333333332 |
|
- type: map_at_3 |
|
value: 6.262416666666668 |
|
- type: map_at_5 |
|
value: 6.693916666666667 |
|
- type: mrr_at_1 |
|
value: 5.780583333333332 |
|
- type: mrr_at_10 |
|
value: 8.576333333333332 |
|
- type: mrr_at_100 |
|
value: 9.17975 |
|
- type: mrr_at_1000 |
|
value: 9.279083333333334 |
|
- type: mrr_at_3 |
|
value: 7.608833333333333 |
|
- type: mrr_at_5 |
|
value: 8.111333333333333 |
|
- type: ndcg_at_1 |
|
value: 5.780583333333332 |
|
- type: ndcg_at_10 |
|
value: 8.866166666666668 |
|
- type: ndcg_at_100 |
|
value: 12.037083333333333 |
|
- type: ndcg_at_1000 |
|
value: 15.4555 |
|
- type: ndcg_at_3 |
|
value: 7.179083333333335 |
|
- type: ndcg_at_5 |
|
value: 7.897166666666666 |
|
- type: precision_at_1 |
|
value: 5.780583333333332 |
|
- type: precision_at_10 |
|
value: 1.6935833333333334 |
|
- type: precision_at_100 |
|
value: 0.3921666666666667 |
|
- type: precision_at_1000 |
|
value: 0.08391666666666667 |
|
- type: precision_at_3 |
|
value: 3.425416666666666 |
|
- type: precision_at_5 |
|
value: 2.5570833333333334 |
|
- type: recall_at_1 |
|
value: 4.741750000000001 |
|
- type: recall_at_10 |
|
value: 12.889083333333334 |
|
- type: recall_at_100 |
|
value: 27.81866666666667 |
|
- type: recall_at_1000 |
|
value: 53.52316666666667 |
|
- type: recall_at_3 |
|
value: 8.179333333333332 |
|
- type: recall_at_5 |
|
value: 10.004083333333334 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackStatsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.7130000000000005 |
|
- type: map_at_10 |
|
value: 5.734 |
|
- type: map_at_100 |
|
value: 6.297999999999999 |
|
- type: map_at_1000 |
|
value: 6.388000000000001 |
|
- type: map_at_3 |
|
value: 5.119 |
|
- type: map_at_5 |
|
value: 5.432 |
|
- type: mrr_at_1 |
|
value: 4.9079999999999995 |
|
- type: mrr_at_10 |
|
value: 7.2940000000000005 |
|
- type: mrr_at_100 |
|
value: 7.8549999999999995 |
|
- type: mrr_at_1000 |
|
value: 7.95 |
|
- type: mrr_at_3 |
|
value: 6.621 |
|
- type: mrr_at_5 |
|
value: 6.950000000000001 |
|
- type: ndcg_at_1 |
|
value: 4.9079999999999995 |
|
- type: ndcg_at_10 |
|
value: 7.167999999999999 |
|
- type: ndcg_at_100 |
|
value: 10.436 |
|
- type: ndcg_at_1000 |
|
value: 13.370999999999999 |
|
- type: ndcg_at_3 |
|
value: 5.959 |
|
- type: ndcg_at_5 |
|
value: 6.481000000000001 |
|
- type: precision_at_1 |
|
value: 4.9079999999999995 |
|
- type: precision_at_10 |
|
value: 1.3339999999999999 |
|
- type: precision_at_100 |
|
value: 0.33899999999999997 |
|
- type: precision_at_1000 |
|
value: 0.065 |
|
- type: precision_at_3 |
|
value: 2.965 |
|
- type: precision_at_5 |
|
value: 2.117 |
|
- type: recall_at_1 |
|
value: 3.7130000000000005 |
|
- type: recall_at_10 |
|
value: 10.156 |
|
- type: recall_at_100 |
|
value: 25.955000000000002 |
|
- type: recall_at_1000 |
|
value: 48.891 |
|
- type: recall_at_3 |
|
value: 6.795 |
|
- type: recall_at_5 |
|
value: 8.187999999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackTexRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 2.114 |
|
- type: map_at_10 |
|
value: 3.4290000000000003 |
|
- type: map_at_100 |
|
value: 3.789 |
|
- type: map_at_1000 |
|
value: 3.878 |
|
- type: map_at_3 |
|
value: 2.9139999999999997 |
|
- type: map_at_5 |
|
value: 3.148 |
|
- type: mrr_at_1 |
|
value: 2.65 |
|
- type: mrr_at_10 |
|
value: 4.252000000000001 |
|
- type: mrr_at_100 |
|
value: 4.689 |
|
- type: mrr_at_1000 |
|
value: 4.782 |
|
- type: mrr_at_3 |
|
value: 3.671 |
|
- type: mrr_at_5 |
|
value: 3.9370000000000003 |
|
- type: ndcg_at_1 |
|
value: 2.65 |
|
- type: ndcg_at_10 |
|
value: 4.47 |
|
- type: ndcg_at_100 |
|
value: 6.654 |
|
- type: ndcg_at_1000 |
|
value: 9.713 |
|
- type: ndcg_at_3 |
|
value: 3.424 |
|
- type: ndcg_at_5 |
|
value: 3.794 |
|
- type: precision_at_1 |
|
value: 2.65 |
|
- type: precision_at_10 |
|
value: 0.9119999999999999 |
|
- type: precision_at_100 |
|
value: 0.248 |
|
- type: precision_at_1000 |
|
value: 0.063 |
|
- type: precision_at_3 |
|
value: 1.7209999999999999 |
|
- type: precision_at_5 |
|
value: 1.287 |
|
- type: recall_at_1 |
|
value: 2.114 |
|
- type: recall_at_10 |
|
value: 6.927 |
|
- type: recall_at_100 |
|
value: 17.26 |
|
- type: recall_at_1000 |
|
value: 40.672999999999995 |
|
- type: recall_at_3 |
|
value: 3.8859999999999997 |
|
- type: recall_at_5 |
|
value: 4.861 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackUnixRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 6.055 |
|
- type: map_at_10 |
|
value: 7.704999999999999 |
|
- type: map_at_100 |
|
value: 8.169 |
|
- type: map_at_1000 |
|
value: 8.257 |
|
- type: map_at_3 |
|
value: 7.033 |
|
- type: map_at_5 |
|
value: 7.4079999999999995 |
|
- type: mrr_at_1 |
|
value: 6.81 |
|
- type: mrr_at_10 |
|
value: 8.955 |
|
- type: mrr_at_100 |
|
value: 9.497 |
|
- type: mrr_at_1000 |
|
value: 9.583 |
|
- type: mrr_at_3 |
|
value: 8.116 |
|
- type: mrr_at_5 |
|
value: 8.526 |
|
- type: ndcg_at_1 |
|
value: 6.81 |
|
- type: ndcg_at_10 |
|
value: 9.113 |
|
- type: ndcg_at_100 |
|
value: 11.884 |
|
- type: ndcg_at_1000 |
|
value: 14.762 |
|
- type: ndcg_at_3 |
|
value: 7.675999999999999 |
|
- type: ndcg_at_5 |
|
value: 8.325000000000001 |
|
- type: precision_at_1 |
|
value: 6.81 |
|
- type: precision_at_10 |
|
value: 1.558 |
|
- type: precision_at_100 |
|
value: 0.34299999999999997 |
|
- type: precision_at_1000 |
|
value: 0.068 |
|
- type: precision_at_3 |
|
value: 3.2960000000000003 |
|
- type: precision_at_5 |
|
value: 2.388 |
|
- type: recall_at_1 |
|
value: 6.055 |
|
- type: recall_at_10 |
|
value: 12.219 |
|
- type: recall_at_100 |
|
value: 25.304 |
|
- type: recall_at_1000 |
|
value: 47.204 |
|
- type: recall_at_3 |
|
value: 8.387 |
|
- type: recall_at_5 |
|
value: 9.991 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWebmastersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 5.043 |
|
- type: map_at_10 |
|
value: 7.394 |
|
- type: map_at_100 |
|
value: 8.096 |
|
- type: map_at_1000 |
|
value: 8.243 |
|
- type: map_at_3 |
|
value: 6.300999999999999 |
|
- type: map_at_5 |
|
value: 6.7780000000000005 |
|
- type: mrr_at_1 |
|
value: 6.126 |
|
- type: mrr_at_10 |
|
value: 9.308 |
|
- type: mrr_at_100 |
|
value: 10.091 |
|
- type: mrr_at_1000 |
|
value: 10.206 |
|
- type: mrr_at_3 |
|
value: 7.938000000000001 |
|
- type: mrr_at_5 |
|
value: 8.64 |
|
- type: ndcg_at_1 |
|
value: 6.126 |
|
- type: ndcg_at_10 |
|
value: 9.474 |
|
- type: ndcg_at_100 |
|
value: 13.238 |
|
- type: ndcg_at_1000 |
|
value: 17.366 |
|
- type: ndcg_at_3 |
|
value: 7.3260000000000005 |
|
- type: ndcg_at_5 |
|
value: 8.167 |
|
- type: precision_at_1 |
|
value: 6.126 |
|
- type: precision_at_10 |
|
value: 1.9959999999999998 |
|
- type: precision_at_100 |
|
value: 0.494 |
|
- type: precision_at_1000 |
|
value: 0.125 |
|
- type: precision_at_3 |
|
value: 3.557 |
|
- type: precision_at_5 |
|
value: 2.9250000000000003 |
|
- type: recall_at_1 |
|
value: 5.043 |
|
- type: recall_at_10 |
|
value: 13.812 |
|
- type: recall_at_100 |
|
value: 31.375999999999998 |
|
- type: recall_at_1000 |
|
value: 61.309999999999995 |
|
- type: recall_at_3 |
|
value: 7.8020000000000005 |
|
- type: recall_at_5 |
|
value: 9.725999999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWordpressRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.771 |
|
- type: map_at_10 |
|
value: 5.152 |
|
- type: map_at_100 |
|
value: 5.584 |
|
- type: map_at_1000 |
|
value: 5.666 |
|
- type: map_at_3 |
|
value: 4.664 |
|
- type: map_at_5 |
|
value: 4.941 |
|
- type: mrr_at_1 |
|
value: 4.251 |
|
- type: mrr_at_10 |
|
value: 5.867 |
|
- type: mrr_at_100 |
|
value: 6.345000000000001 |
|
- type: mrr_at_1000 |
|
value: 6.432 |
|
- type: mrr_at_3 |
|
value: 5.36 |
|
- type: mrr_at_5 |
|
value: 5.656 |
|
- type: ndcg_at_1 |
|
value: 4.251 |
|
- type: ndcg_at_10 |
|
value: 6.16 |
|
- type: ndcg_at_100 |
|
value: 8.895 |
|
- type: ndcg_at_1000 |
|
value: 11.631 |
|
- type: ndcg_at_3 |
|
value: 5.176 |
|
- type: ndcg_at_5 |
|
value: 5.633 |
|
- type: precision_at_1 |
|
value: 4.251 |
|
- type: precision_at_10 |
|
value: 0.98 |
|
- type: precision_at_100 |
|
value: 0.259 |
|
- type: precision_at_1000 |
|
value: 0.053 |
|
- type: precision_at_3 |
|
value: 2.2800000000000002 |
|
- type: precision_at_5 |
|
value: 1.627 |
|
- type: recall_at_1 |
|
value: 3.771 |
|
- type: recall_at_10 |
|
value: 8.731 |
|
- type: recall_at_100 |
|
value: 22.517 |
|
- type: recall_at_1000 |
|
value: 44.183 |
|
- type: recall_at_3 |
|
value: 5.866 |
|
- type: recall_at_5 |
|
value: 7.066999999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: climate-fever |
|
name: MTEB ClimateFEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.543 |
|
- type: map_at_10 |
|
value: 1.027 |
|
- type: map_at_100 |
|
value: 1.228 |
|
- type: map_at_1000 |
|
value: 1.266 |
|
- type: map_at_3 |
|
value: 0.756 |
|
- type: map_at_5 |
|
value: 0.877 |
|
- type: mrr_at_1 |
|
value: 1.3679999999999999 |
|
- type: mrr_at_10 |
|
value: 2.474 |
|
- type: mrr_at_100 |
|
value: 2.8369999999999997 |
|
- type: mrr_at_1000 |
|
value: 2.894 |
|
- type: mrr_at_3 |
|
value: 1.8780000000000001 |
|
- type: mrr_at_5 |
|
value: 2.1319999999999997 |
|
- type: ndcg_at_1 |
|
value: 1.3679999999999999 |
|
- type: ndcg_at_10 |
|
value: 1.791 |
|
- type: ndcg_at_100 |
|
value: 3.06 |
|
- type: ndcg_at_1000 |
|
value: 4.501 |
|
- type: ndcg_at_3 |
|
value: 1.16 |
|
- type: ndcg_at_5 |
|
value: 1.3419999999999999 |
|
- type: precision_at_1 |
|
value: 1.3679999999999999 |
|
- type: precision_at_10 |
|
value: 0.697 |
|
- type: precision_at_100 |
|
value: 0.193 |
|
- type: precision_at_1000 |
|
value: 0.045 |
|
- type: precision_at_3 |
|
value: 0.9339999999999999 |
|
- type: precision_at_5 |
|
value: 0.808 |
|
- type: recall_at_1 |
|
value: 0.543 |
|
- type: recall_at_10 |
|
value: 2.5149999999999997 |
|
- type: recall_at_100 |
|
value: 7.356999999999999 |
|
- type: recall_at_1000 |
|
value: 16.233 |
|
- type: recall_at_3 |
|
value: 1.018 |
|
- type: recall_at_5 |
|
value: 1.5150000000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/CmedqaRetrieval |
|
name: MTEB CmedqaRetrieval |
|
config: default |
|
split: dev |
|
revision: cd540c506dae1cf9e9a59c3e06f42030d54e7301 |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.7289999999999996 |
|
- type: map_at_10 |
|
value: 5.524 |
|
- type: map_at_100 |
|
value: 5.984 |
|
- type: map_at_1000 |
|
value: 6.087 |
|
- type: map_at_3 |
|
value: 4.854 |
|
- type: map_at_5 |
|
value: 5.2299999999999995 |
|
- type: mrr_at_1 |
|
value: 6.177 |
|
- type: mrr_at_10 |
|
value: 8.541 |
|
- type: mrr_at_100 |
|
value: 9.073 |
|
- type: mrr_at_1000 |
|
value: 9.161 |
|
- type: mrr_at_3 |
|
value: 7.71 |
|
- type: mrr_at_5 |
|
value: 8.148 |
|
- type: ndcg_at_1 |
|
value: 6.177 |
|
- type: ndcg_at_10 |
|
value: 7.217999999999999 |
|
- type: ndcg_at_100 |
|
value: 9.927 |
|
- type: ndcg_at_1000 |
|
value: 13.062000000000001 |
|
- type: ndcg_at_3 |
|
value: 6.0569999999999995 |
|
- type: ndcg_at_5 |
|
value: 6.544999999999999 |
|
- type: precision_at_1 |
|
value: 6.177 |
|
- type: precision_at_10 |
|
value: 1.6729999999999998 |
|
- type: precision_at_100 |
|
value: 0.38999999999999996 |
|
- type: precision_at_1000 |
|
value: 0.082 |
|
- type: precision_at_3 |
|
value: 3.5090000000000003 |
|
- type: precision_at_5 |
|
value: 2.596 |
|
- type: recall_at_1 |
|
value: 3.7289999999999996 |
|
- type: recall_at_10 |
|
value: 9.501 |
|
- type: recall_at_100 |
|
value: 21.444 |
|
- type: recall_at_1000 |
|
value: 43.891999999999996 |
|
- type: recall_at_3 |
|
value: 6.053 |
|
- type: recall_at_5 |
|
value: 7.531000000000001 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: C-MTEB/CMNLI |
|
name: MTEB Cmnli |
|
config: default |
|
split: validation |
|
revision: 41bc36f332156f7adc9e38f53777c959b2ae9766 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 58.123872519543 |
|
- type: cos_sim_ap |
|
value: 61.86046509726734 |
|
- type: cos_sim_f1 |
|
value: 68.18181818181817 |
|
- type: cos_sim_precision |
|
value: 52.4198617221873 |
|
- type: cos_sim_recall |
|
value: 97.49824643441664 |
|
- type: dot_accuracy |
|
value: 58.123872519543 |
|
- type: dot_ap |
|
value: 61.860555259802986 |
|
- type: dot_f1 |
|
value: 68.18181818181817 |
|
- type: dot_precision |
|
value: 52.4198617221873 |
|
- type: dot_recall |
|
value: 97.49824643441664 |
|
- type: euclidean_accuracy |
|
value: 58.123872519543 |
|
- type: euclidean_ap |
|
value: 61.87698627731538 |
|
- type: euclidean_f1 |
|
value: 68.18181818181817 |
|
- type: euclidean_precision |
|
value: 52.4198617221873 |
|
- type: euclidean_recall |
|
value: 97.49824643441664 |
|
- type: manhattan_accuracy |
|
value: 58.123872519543 |
|
- type: manhattan_ap |
|
value: 61.99468883207791 |
|
- type: manhattan_f1 |
|
value: 68.33675564681727 |
|
- type: manhattan_precision |
|
value: 52.671562420866046 |
|
- type: manhattan_recall |
|
value: 97.26443768996961 |
|
- type: max_accuracy |
|
value: 58.123872519543 |
|
- type: max_ap |
|
value: 61.99468883207791 |
|
- type: max_f1 |
|
value: 68.33675564681727 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/CovidRetrieval |
|
name: MTEB CovidRetrieval |
|
config: default |
|
split: dev |
|
revision: 1271c7809071a13532e05f25fb53511ffce77117 |
|
metrics: |
|
- type: map_at_1 |
|
value: 6.428000000000001 |
|
- type: map_at_10 |
|
value: 8.883000000000001 |
|
- type: map_at_100 |
|
value: 9.549000000000001 |
|
- type: map_at_1000 |
|
value: 9.665 |
|
- type: map_at_3 |
|
value: 8.061 |
|
- type: map_at_5 |
|
value: 8.475000000000001 |
|
- type: mrr_at_1 |
|
value: 6.428000000000001 |
|
- type: mrr_at_10 |
|
value: 8.896999999999998 |
|
- type: mrr_at_100 |
|
value: 9.557 |
|
- type: mrr_at_1000 |
|
value: 9.674000000000001 |
|
- type: mrr_at_3 |
|
value: 8.061 |
|
- type: mrr_at_5 |
|
value: 8.488 |
|
- type: ndcg_at_1 |
|
value: 6.428000000000001 |
|
- type: ndcg_at_10 |
|
value: 10.382 |
|
- type: ndcg_at_100 |
|
value: 14.235999999999999 |
|
- type: ndcg_at_1000 |
|
value: 18.04 |
|
- type: ndcg_at_3 |
|
value: 8.613999999999999 |
|
- type: ndcg_at_5 |
|
value: 9.372 |
|
- type: precision_at_1 |
|
value: 6.428000000000001 |
|
- type: precision_at_10 |
|
value: 1.528 |
|
- type: precision_at_100 |
|
value: 0.349 |
|
- type: precision_at_1000 |
|
value: 0.067 |
|
- type: precision_at_3 |
|
value: 3.4070000000000005 |
|
- type: precision_at_5 |
|
value: 2.424 |
|
- type: recall_at_1 |
|
value: 6.428000000000001 |
|
- type: recall_at_10 |
|
value: 15.226999999999999 |
|
- type: recall_at_100 |
|
value: 34.694 |
|
- type: recall_at_1000 |
|
value: 66.07 |
|
- type: recall_at_3 |
|
value: 10.221 |
|
- type: recall_at_5 |
|
value: 12.065 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: dbpedia-entity |
|
name: MTEB DBPedia |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.541 |
|
- type: map_at_10 |
|
value: 1.1560000000000001 |
|
- type: map_at_100 |
|
value: 1.508 |
|
- type: map_at_1000 |
|
value: 1.598 |
|
- type: map_at_3 |
|
value: 0.918 |
|
- type: map_at_5 |
|
value: 0.992 |
|
- type: mrr_at_1 |
|
value: 9.5 |
|
- type: mrr_at_10 |
|
value: 13.446 |
|
- type: mrr_at_100 |
|
value: 13.935 |
|
- type: mrr_at_1000 |
|
value: 14.008999999999999 |
|
- type: mrr_at_3 |
|
value: 12.083 |
|
- type: mrr_at_5 |
|
value: 12.733 |
|
- type: ndcg_at_1 |
|
value: 5.75 |
|
- type: ndcg_at_10 |
|
value: 3.9210000000000003 |
|
- type: ndcg_at_100 |
|
value: 3.975 |
|
- type: ndcg_at_1000 |
|
value: 5.634 |
|
- type: ndcg_at_3 |
|
value: 4.87 |
|
- type: ndcg_at_5 |
|
value: 4.259 |
|
- type: precision_at_1 |
|
value: 9.5 |
|
- type: precision_at_10 |
|
value: 3.9 |
|
- type: precision_at_100 |
|
value: 1.015 |
|
- type: precision_at_1000 |
|
value: 0.297 |
|
- type: precision_at_3 |
|
value: 6.75 |
|
- type: precision_at_5 |
|
value: 5.25 |
|
- type: recall_at_1 |
|
value: 0.541 |
|
- type: recall_at_10 |
|
value: 2.228 |
|
- type: recall_at_100 |
|
value: 4.9430000000000005 |
|
- type: recall_at_1000 |
|
value: 11.661000000000001 |
|
- type: recall_at_3 |
|
value: 1.264 |
|
- type: recall_at_5 |
|
value: 1.4869999999999999 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: DDSC/dkhate |
|
name: MTEB DKHateClassification |
|
config: default |
|
split: test |
|
revision: 59d12749a3c91a186063c7d729ec392fda94681c |
|
metrics: |
|
- type: accuracy |
|
value: 69.96960486322187 |
|
- type: ap |
|
value: 91.23131906690253 |
|
- type: f1 |
|
value: 57.11872970138122 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: AI-Sweden/SuperLim |
|
name: MTEB DalajClassification |
|
config: default |
|
split: test |
|
revision: 7ebf0b4caa7b2ae39698a889de782c09e6f5ee56 |
|
metrics: |
|
- type: accuracy |
|
value: 49.75225225225225 |
|
- type: ap |
|
value: 49.88223192425368 |
|
- type: f1 |
|
value: 49.55059044107012 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: danish_political_comments |
|
name: MTEB DanishPoliticalCommentsClassification |
|
config: default |
|
split: train |
|
revision: edbb03726c04a0efab14fc8c3b8b79e4d420e5a1 |
|
metrics: |
|
- type: accuracy |
|
value: 37.58534554537886 |
|
- type: f1 |
|
value: 33.99440115952713 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/DuRetrieval |
|
name: MTEB DuRetrieval |
|
config: default |
|
split: dev |
|
revision: a1a333e290fe30b10f3f56498e3a0d911a693ced |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.608 |
|
- type: map_at_10 |
|
value: 0.882 |
|
- type: map_at_100 |
|
value: 0.962 |
|
- type: map_at_1000 |
|
value: 1.028 |
|
- type: map_at_3 |
|
value: 0.749 |
|
- type: map_at_5 |
|
value: 0.8240000000000001 |
|
- type: mrr_at_1 |
|
value: 2.0500000000000003 |
|
- type: mrr_at_10 |
|
value: 2.796 |
|
- type: mrr_at_100 |
|
value: 2.983 |
|
- type: mrr_at_1000 |
|
value: 3.09 |
|
- type: mrr_at_3 |
|
value: 2.483 |
|
- type: mrr_at_5 |
|
value: 2.661 |
|
- type: ndcg_at_1 |
|
value: 2.0500000000000003 |
|
- type: ndcg_at_10 |
|
value: 1.435 |
|
- type: ndcg_at_100 |
|
value: 1.991 |
|
- type: ndcg_at_1000 |
|
value: 4.961 |
|
- type: ndcg_at_3 |
|
value: 1.428 |
|
- type: ndcg_at_5 |
|
value: 1.369 |
|
- type: precision_at_1 |
|
value: 2.0500000000000003 |
|
- type: precision_at_10 |
|
value: 0.5349999999999999 |
|
- type: precision_at_100 |
|
value: 0.127 |
|
- type: precision_at_1000 |
|
value: 0.086 |
|
- type: precision_at_3 |
|
value: 1.05 |
|
- type: precision_at_5 |
|
value: 0.84 |
|
- type: recall_at_1 |
|
value: 0.608 |
|
- type: recall_at_10 |
|
value: 1.54 |
|
- type: recall_at_100 |
|
value: 3.5069999999999997 |
|
- type: recall_at_1000 |
|
value: 20.531 |
|
- type: recall_at_3 |
|
value: 0.901 |
|
- type: recall_at_5 |
|
value: 1.168 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/EcomRetrieval |
|
name: MTEB EcomRetrieval |
|
config: default |
|
split: dev |
|
revision: 687de13dc7294d6fd9be10c6945f9e8fec8166b9 |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.1 |
|
- type: map_at_10 |
|
value: 4.016 |
|
- type: map_at_100 |
|
value: 4.455 |
|
- type: map_at_1000 |
|
value: 4.579 |
|
- type: map_at_3 |
|
value: 3.567 |
|
- type: map_at_5 |
|
value: 3.8019999999999996 |
|
- type: mrr_at_1 |
|
value: 3.1 |
|
- type: mrr_at_10 |
|
value: 4.016 |
|
- type: mrr_at_100 |
|
value: 4.455 |
|
- type: mrr_at_1000 |
|
value: 4.579 |
|
- type: mrr_at_3 |
|
value: 3.567 |
|
- type: mrr_at_5 |
|
value: 3.8019999999999996 |
|
- type: ndcg_at_1 |
|
value: 3.1 |
|
- type: ndcg_at_10 |
|
value: 4.684 |
|
- type: ndcg_at_100 |
|
value: 7.284 |
|
- type: ndcg_at_1000 |
|
value: 11.689 |
|
- type: ndcg_at_3 |
|
value: 3.7289999999999996 |
|
- type: ndcg_at_5 |
|
value: 4.146 |
|
- type: precision_at_1 |
|
value: 3.1 |
|
- type: precision_at_10 |
|
value: 0.69 |
|
- type: precision_at_100 |
|
value: 0.202 |
|
- type: precision_at_1000 |
|
value: 0.056999999999999995 |
|
- type: precision_at_3 |
|
value: 1.4000000000000001 |
|
- type: precision_at_5 |
|
value: 1.04 |
|
- type: recall_at_1 |
|
value: 3.1 |
|
- type: recall_at_10 |
|
value: 6.9 |
|
- type: recall_at_100 |
|
value: 20.200000000000003 |
|
- type: recall_at_1000 |
|
value: 57.3 |
|
- type: recall_at_3 |
|
value: 4.2 |
|
- type: recall_at_5 |
|
value: 5.2 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/emotion |
|
name: MTEB EmotionClassification |
|
config: default |
|
split: test |
|
revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
|
metrics: |
|
- type: accuracy |
|
value: 38.285000000000004 |
|
- type: f1 |
|
value: 35.35979931355028 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fever |
|
name: MTEB FEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.9249999999999999 |
|
- type: map_at_10 |
|
value: 1.311 |
|
- type: map_at_100 |
|
value: 1.363 |
|
- type: map_at_1000 |
|
value: 1.376 |
|
- type: map_at_3 |
|
value: 1.145 |
|
- type: map_at_5 |
|
value: 1.233 |
|
- type: mrr_at_1 |
|
value: 0.975 |
|
- type: mrr_at_10 |
|
value: 1.371 |
|
- type: mrr_at_100 |
|
value: 1.426 |
|
- type: mrr_at_1000 |
|
value: 1.439 |
|
- type: mrr_at_3 |
|
value: 1.195 |
|
- type: mrr_at_5 |
|
value: 1.286 |
|
- type: ndcg_at_1 |
|
value: 0.975 |
|
- type: ndcg_at_10 |
|
value: 1.5859999999999999 |
|
- type: ndcg_at_100 |
|
value: 1.8800000000000001 |
|
- type: ndcg_at_1000 |
|
value: 2.313 |
|
- type: ndcg_at_3 |
|
value: 1.229 |
|
- type: ndcg_at_5 |
|
value: 1.388 |
|
- type: precision_at_1 |
|
value: 0.975 |
|
- type: precision_at_10 |
|
value: 0.254 |
|
- type: precision_at_100 |
|
value: 0.041 |
|
- type: precision_at_1000 |
|
value: 0.008 |
|
- type: precision_at_3 |
|
value: 0.49 |
|
- type: precision_at_5 |
|
value: 0.375 |
|
- type: recall_at_1 |
|
value: 0.9249999999999999 |
|
- type: recall_at_10 |
|
value: 2.4250000000000003 |
|
- type: recall_at_100 |
|
value: 3.866 |
|
- type: recall_at_1000 |
|
value: 7.401000000000001 |
|
- type: recall_at_3 |
|
value: 1.4200000000000002 |
|
- type: recall_at_5 |
|
value: 1.81 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fiqa-pl |
|
name: MTEB FiQA-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.959 |
|
- type: map_at_10 |
|
value: 1.952 |
|
- type: map_at_100 |
|
value: 2.281 |
|
- type: map_at_1000 |
|
value: 2.393 |
|
- type: map_at_3 |
|
value: 1.703 |
|
- type: map_at_5 |
|
value: 1.8319999999999999 |
|
- type: mrr_at_1 |
|
value: 2.469 |
|
- type: mrr_at_10 |
|
value: 4.547 |
|
- type: mrr_at_100 |
|
value: 5.021 |
|
- type: mrr_at_1000 |
|
value: 5.1339999999999995 |
|
- type: mrr_at_3 |
|
value: 3.884 |
|
- type: mrr_at_5 |
|
value: 4.223 |
|
- type: ndcg_at_1 |
|
value: 2.469 |
|
- type: ndcg_at_10 |
|
value: 3.098 |
|
- type: ndcg_at_100 |
|
value: 5.177 |
|
- type: ndcg_at_1000 |
|
value: 8.889 |
|
- type: ndcg_at_3 |
|
value: 2.7119999999999997 |
|
- type: ndcg_at_5 |
|
value: 2.8000000000000003 |
|
- type: precision_at_1 |
|
value: 2.469 |
|
- type: precision_at_10 |
|
value: 1.065 |
|
- type: precision_at_100 |
|
value: 0.321 |
|
- type: precision_at_1000 |
|
value: 0.095 |
|
- type: precision_at_3 |
|
value: 2.109 |
|
- type: precision_at_5 |
|
value: 1.574 |
|
- type: recall_at_1 |
|
value: 0.959 |
|
- type: recall_at_10 |
|
value: 4.075 |
|
- type: recall_at_100 |
|
value: 12.487 |
|
- type: recall_at_1000 |
|
value: 36.854 |
|
- type: recall_at_3 |
|
value: 2.632 |
|
- type: recall_at_5 |
|
value: 3.231 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fiqa |
|
name: MTEB FiQA2018 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 1.032 |
|
- type: map_at_10 |
|
value: 1.8849999999999998 |
|
- type: map_at_100 |
|
value: 2.167 |
|
- type: map_at_1000 |
|
value: 2.266 |
|
- type: map_at_3 |
|
value: 1.609 |
|
- type: map_at_5 |
|
value: 1.7680000000000002 |
|
- type: mrr_at_1 |
|
value: 2.6229999999999998 |
|
- type: mrr_at_10 |
|
value: 4.479 |
|
- type: mrr_at_100 |
|
value: 4.92 |
|
- type: mrr_at_1000 |
|
value: 5.029999999999999 |
|
- type: mrr_at_3 |
|
value: 3.7289999999999996 |
|
- type: mrr_at_5 |
|
value: 4.138 |
|
- type: ndcg_at_1 |
|
value: 2.6229999999999998 |
|
- type: ndcg_at_10 |
|
value: 3.005 |
|
- type: ndcg_at_100 |
|
value: 5.01 |
|
- type: ndcg_at_1000 |
|
value: 8.312 |
|
- type: ndcg_at_3 |
|
value: 2.548 |
|
- type: ndcg_at_5 |
|
value: 2.735 |
|
- type: precision_at_1 |
|
value: 2.6229999999999998 |
|
- type: precision_at_10 |
|
value: 1.049 |
|
- type: precision_at_100 |
|
value: 0.31 |
|
- type: precision_at_1000 |
|
value: 0.089 |
|
- type: precision_at_3 |
|
value: 1.955 |
|
- type: precision_at_5 |
|
value: 1.574 |
|
- type: recall_at_1 |
|
value: 1.032 |
|
- type: recall_at_10 |
|
value: 3.888 |
|
- type: recall_at_100 |
|
value: 12.414 |
|
- type: recall_at_1000 |
|
value: 33.823 |
|
- type: recall_at_3 |
|
value: 2.37 |
|
- type: recall_at_5 |
|
value: 3.077 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: jinaai/ger_da_lir |
|
name: MTEB GerDaLIR |
|
config: default |
|
split: test |
|
revision: 0bb47f1d73827e96964edb84dfe552f62f4fd5eb |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.542 |
|
- type: map_at_10 |
|
value: 0.8130000000000001 |
|
- type: map_at_100 |
|
value: 0.898 |
|
- type: map_at_1000 |
|
value: 0.9209999999999999 |
|
- type: map_at_3 |
|
value: 0.709 |
|
- type: map_at_5 |
|
value: 0.764 |
|
- type: mrr_at_1 |
|
value: 0.594 |
|
- type: mrr_at_10 |
|
value: 0.8880000000000001 |
|
- type: mrr_at_100 |
|
value: 0.9820000000000001 |
|
- type: mrr_at_1000 |
|
value: 1.008 |
|
- type: mrr_at_3 |
|
value: 0.774 |
|
- type: mrr_at_5 |
|
value: 0.832 |
|
- type: ndcg_at_1 |
|
value: 0.594 |
|
- type: ndcg_at_10 |
|
value: 1.0030000000000001 |
|
- type: ndcg_at_100 |
|
value: 1.537 |
|
- type: ndcg_at_1000 |
|
value: 2.4330000000000003 |
|
- type: ndcg_at_3 |
|
value: 0.782 |
|
- type: ndcg_at_5 |
|
value: 0.882 |
|
- type: precision_at_1 |
|
value: 0.594 |
|
- type: precision_at_10 |
|
value: 0.16999999999999998 |
|
- type: precision_at_100 |
|
value: 0.048 |
|
- type: precision_at_1000 |
|
value: 0.013 |
|
- type: precision_at_3 |
|
value: 0.33899999999999997 |
|
- type: precision_at_5 |
|
value: 0.255 |
|
- type: recall_at_1 |
|
value: 0.542 |
|
- type: recall_at_10 |
|
value: 1.533 |
|
- type: recall_at_100 |
|
value: 4.204 |
|
- type: recall_at_1000 |
|
value: 11.574 |
|
- type: recall_at_3 |
|
value: 0.932 |
|
- type: recall_at_5 |
|
value: 1.172 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: deepset/germandpr |
|
name: MTEB GermanDPR |
|
config: default |
|
split: test |
|
revision: 5129d02422a66be600ac89cd3e8531b4f97d347d |
|
metrics: |
|
- type: map_at_1 |
|
value: 25.561 |
|
- type: map_at_10 |
|
value: 38.873000000000005 |
|
- type: map_at_100 |
|
value: 40.004 |
|
- type: map_at_1000 |
|
value: 40.03 |
|
- type: map_at_3 |
|
value: 34.585 |
|
- type: map_at_5 |
|
value: 36.980000000000004 |
|
- type: mrr_at_1 |
|
value: 25.463 |
|
- type: mrr_at_10 |
|
value: 38.792 |
|
- type: mrr_at_100 |
|
value: 39.922000000000004 |
|
- type: mrr_at_1000 |
|
value: 39.949 |
|
- type: mrr_at_3 |
|
value: 34.504000000000005 |
|
- type: mrr_at_5 |
|
value: 36.899 |
|
- type: ndcg_at_1 |
|
value: 25.561 |
|
- type: ndcg_at_10 |
|
value: 46.477000000000004 |
|
- type: ndcg_at_100 |
|
value: 51.751999999999995 |
|
- type: ndcg_at_1000 |
|
value: 52.366 |
|
- type: ndcg_at_3 |
|
value: 37.645 |
|
- type: ndcg_at_5 |
|
value: 41.953 |
|
- type: precision_at_1 |
|
value: 25.561 |
|
- type: precision_at_10 |
|
value: 7.083 |
|
- type: precision_at_100 |
|
value: 0.9490000000000001 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 15.512 |
|
- type: precision_at_5 |
|
value: 11.395 |
|
- type: recall_at_1 |
|
value: 25.561 |
|
- type: recall_at_10 |
|
value: 70.829 |
|
- type: recall_at_100 |
|
value: 94.92699999999999 |
|
- type: recall_at_1000 |
|
value: 99.61 |
|
- type: recall_at_3 |
|
value: 46.537 |
|
- type: recall_at_5 |
|
value: 56.976000000000006 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: mteb/germanquad-retrieval |
|
name: MTEB GermanQuAD-Retrieval |
|
config: default |
|
split: test |
|
revision: f5c87ae5a2e7a5106606314eef45255f03151bb3 |
|
metrics: |
|
- type: map_at_1 |
|
value: 53.539 |
|
- type: map_at_10 |
|
value: 65.144 |
|
- type: map_at_100 |
|
value: 65.627 |
|
- type: map_at_1000 |
|
value: 65.63900000000001 |
|
- type: map_at_3 |
|
value: 62.598 |
|
- type: map_at_5 |
|
value: 64.302 |
|
- type: mrr_at_1 |
|
value: 53.539 |
|
- type: mrr_at_10 |
|
value: 65.144 |
|
- type: mrr_at_100 |
|
value: 65.627 |
|
- type: mrr_at_1000 |
|
value: 65.63900000000001 |
|
- type: mrr_at_3 |
|
value: 62.598 |
|
- type: mrr_at_5 |
|
value: 64.302 |
|
- type: ndcg_at_1 |
|
value: 53.539 |
|
- type: ndcg_at_10 |
|
value: 70.602 |
|
- type: ndcg_at_100 |
|
value: 72.886 |
|
- type: ndcg_at_1000 |
|
value: 73.14500000000001 |
|
- type: ndcg_at_3 |
|
value: 65.52900000000001 |
|
- type: ndcg_at_5 |
|
value: 68.596 |
|
- type: precision_at_1 |
|
value: 53.539 |
|
- type: precision_at_10 |
|
value: 8.757 |
|
- type: precision_at_100 |
|
value: 0.9809999999999999 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 24.667 |
|
- type: precision_at_5 |
|
value: 16.289 |
|
- type: recall_at_1 |
|
value: 53.539 |
|
- type: recall_at_10 |
|
value: 87.568 |
|
- type: recall_at_100 |
|
value: 98.09400000000001 |
|
- type: recall_at_1000 |
|
value: 100.0 |
|
- type: recall_at_3 |
|
value: 74.002 |
|
- type: recall_at_5 |
|
value: 81.443 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: jinaai/german-STSbenchmark |
|
name: MTEB GermanSTSBenchmark |
|
config: default |
|
split: test |
|
revision: e36907544d44c3a247898ed81540310442329e20 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 68.82052535790737 |
|
- type: cos_sim_spearman |
|
value: 67.9356892072251 |
|
- type: euclidean_pearson |
|
value: 67.2308663006278 |
|
- type: euclidean_spearman |
|
value: 67.93572522920142 |
|
- type: manhattan_pearson |
|
value: 67.23568952733595 |
|
- type: manhattan_spearman |
|
value: 67.91660489262797 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: hotpotqa |
|
name: MTEB HotpotQA |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 6.813 |
|
- type: map_at_10 |
|
value: 9.49 |
|
- type: map_at_100 |
|
value: 9.959 |
|
- type: map_at_1000 |
|
value: 10.024 |
|
- type: map_at_3 |
|
value: 8.618 |
|
- type: map_at_5 |
|
value: 9.084 |
|
- type: mrr_at_1 |
|
value: 13.626 |
|
- type: mrr_at_10 |
|
value: 17.818 |
|
- type: mrr_at_100 |
|
value: 18.412 |
|
- type: mrr_at_1000 |
|
value: 18.482000000000003 |
|
- type: mrr_at_3 |
|
value: 16.506999999999998 |
|
- type: mrr_at_5 |
|
value: 17.219 |
|
- type: ndcg_at_1 |
|
value: 13.626 |
|
- type: ndcg_at_10 |
|
value: 12.959999999999999 |
|
- type: ndcg_at_100 |
|
value: 15.562999999999999 |
|
- type: ndcg_at_1000 |
|
value: 17.571 |
|
- type: ndcg_at_3 |
|
value: 10.995000000000001 |
|
- type: ndcg_at_5 |
|
value: 11.908000000000001 |
|
- type: precision_at_1 |
|
value: 13.626 |
|
- type: precision_at_10 |
|
value: 2.995 |
|
- type: precision_at_100 |
|
value: 0.51 |
|
- type: precision_at_1000 |
|
value: 0.078 |
|
- type: precision_at_3 |
|
value: 7.000000000000001 |
|
- type: precision_at_5 |
|
value: 4.926 |
|
- type: recall_at_1 |
|
value: 6.813 |
|
- type: recall_at_10 |
|
value: 14.976 |
|
- type: recall_at_100 |
|
value: 25.517 |
|
- type: recall_at_1000 |
|
value: 39.095 |
|
- type: recall_at_3 |
|
value: 10.5 |
|
- type: recall_at_5 |
|
value: 12.316 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/IFlyTek-classification |
|
name: MTEB IFlyTek |
|
config: default |
|
split: validation |
|
revision: 421605374b29664c5fc098418fe20ada9bd55f8a |
|
metrics: |
|
- type: accuracy |
|
value: 38.01462100808003 |
|
- type: f1 |
|
value: 26.680357453754215 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/imdb |
|
name: MTEB ImdbClassification |
|
config: default |
|
split: test |
|
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
|
metrics: |
|
- type: accuracy |
|
value: 55.7508 |
|
- type: ap |
|
value: 53.28158993124153 |
|
- type: f1 |
|
value: 55.34571379744637 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/JDReview-classification |
|
name: MTEB JDReview |
|
config: default |
|
split: test |
|
revision: b7c64bd89eb87f8ded463478346f76731f07bf8b |
|
metrics: |
|
- type: accuracy |
|
value: 69.58724202626641 |
|
- type: ap |
|
value: 30.04577466931377 |
|
- type: f1 |
|
value: 62.46921898313143 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/LCQMC |
|
name: MTEB LCQMC |
|
config: default |
|
split: test |
|
revision: 17f9b096f80380fce5ed12a9be8be7784b337daf |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 48.80585861169271 |
|
- type: cos_sim_spearman |
|
value: 50.11025991147549 |
|
- type: euclidean_pearson |
|
value: 50.055425341198934 |
|
- type: euclidean_spearman |
|
value: 50.11024862622995 |
|
- type: manhattan_pearson |
|
value: 50.029980024931064 |
|
- type: manhattan_spearman |
|
value: 50.074388245963384 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: DDSC/lcc |
|
name: MTEB LccSentimentClassification |
|
config: default |
|
split: test |
|
revision: de7ba3406ee55ea2cc52a0a41408fa6aede6d3c6 |
|
metrics: |
|
- type: accuracy |
|
value: 54.266666666666666 |
|
- type: f1 |
|
value: 52.181931818742875 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: jinaai/miracl |
|
name: MTEB MIRACL |
|
config: default |
|
split: test |
|
revision: d28a029f35c4ff7f616df47b0edf54e6882395e6 |
|
metrics: |
|
- type: map |
|
value: 51.40745004398599 |
|
- type: mrr |
|
value: 56.71940267335004 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/Mmarco-reranking |
|
name: MTEB MMarcoReranking |
|
config: default |
|
split: dev |
|
revision: 8e0c766dbe9e16e1d221116a3f36795fbade07f6 |
|
metrics: |
|
- type: map |
|
value: 5.831060174627054 |
|
- type: mrr |
|
value: 4.019047619047618 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/MMarcoRetrieval |
|
name: MTEB MMarcoRetrieval |
|
config: default |
|
split: dev |
|
revision: 539bbde593d947e2a124ba72651aafc09eb33fc2 |
|
metrics: |
|
- type: map_at_1 |
|
value: 5.826 |
|
- type: map_at_10 |
|
value: 8.956999999999999 |
|
- type: map_at_100 |
|
value: 9.746 |
|
- type: map_at_1000 |
|
value: 9.873999999999999 |
|
- type: map_at_3 |
|
value: 7.757 |
|
- type: map_at_5 |
|
value: 8.373 |
|
- type: mrr_at_1 |
|
value: 6.046 |
|
- type: mrr_at_10 |
|
value: 9.251 |
|
- type: mrr_at_100 |
|
value: 10.044 |
|
- type: mrr_at_1000 |
|
value: 10.167 |
|
- type: mrr_at_3 |
|
value: 8.028 |
|
- type: mrr_at_5 |
|
value: 8.66 |
|
- type: ndcg_at_1 |
|
value: 6.046 |
|
- type: ndcg_at_10 |
|
value: 10.998 |
|
- type: ndcg_at_100 |
|
value: 15.568999999999999 |
|
- type: ndcg_at_1000 |
|
value: 19.453 |
|
- type: ndcg_at_3 |
|
value: 8.468 |
|
- type: ndcg_at_5 |
|
value: 9.582 |
|
- type: precision_at_1 |
|
value: 6.046 |
|
- type: precision_at_10 |
|
value: 1.807 |
|
- type: precision_at_100 |
|
value: 0.42500000000000004 |
|
- type: precision_at_1000 |
|
value: 0.076 |
|
- type: precision_at_3 |
|
value: 3.572 |
|
- type: precision_at_5 |
|
value: 2.702 |
|
- type: recall_at_1 |
|
value: 5.826 |
|
- type: recall_at_10 |
|
value: 17.291 |
|
- type: recall_at_100 |
|
value: 40.037 |
|
- type: recall_at_1000 |
|
value: 71.351 |
|
- type: recall_at_3 |
|
value: 10.269 |
|
- type: recall_at_5 |
|
value: 12.950000000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: msmarco |
|
name: MTEB MSMARCO |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 1.203 |
|
- type: map_at_10 |
|
value: 2.27 |
|
- type: map_at_100 |
|
value: 2.5860000000000003 |
|
- type: map_at_1000 |
|
value: 2.661 |
|
- type: map_at_3 |
|
value: 1.8159999999999998 |
|
- type: map_at_5 |
|
value: 2.037 |
|
- type: mrr_at_1 |
|
value: 1.232 |
|
- type: mrr_at_10 |
|
value: 2.338 |
|
- type: mrr_at_100 |
|
value: 2.665 |
|
- type: mrr_at_1000 |
|
value: 2.7390000000000003 |
|
- type: mrr_at_3 |
|
value: 1.87 |
|
- type: mrr_at_5 |
|
value: 2.1 |
|
- type: ndcg_at_1 |
|
value: 1.232 |
|
- type: ndcg_at_10 |
|
value: 3.005 |
|
- type: ndcg_at_100 |
|
value: 4.936 |
|
- type: ndcg_at_1000 |
|
value: 7.441000000000001 |
|
- type: ndcg_at_3 |
|
value: 2.036 |
|
- type: ndcg_at_5 |
|
value: 2.435 |
|
- type: precision_at_1 |
|
value: 1.232 |
|
- type: precision_at_10 |
|
value: 0.549 |
|
- type: precision_at_100 |
|
value: 0.158 |
|
- type: precision_at_1000 |
|
value: 0.038 |
|
- type: precision_at_3 |
|
value: 0.903 |
|
- type: precision_at_5 |
|
value: 0.739 |
|
- type: recall_at_1 |
|
value: 1.203 |
|
- type: recall_at_10 |
|
value: 5.332 |
|
- type: recall_at_100 |
|
value: 15.164 |
|
- type: recall_at_1000 |
|
value: 35.831 |
|
- type: recall_at_3 |
|
value: 2.622 |
|
- type: recall_at_5 |
|
value: 3.572 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (en) |
|
config: en |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 89.92476060191518 |
|
- type: f1 |
|
value: 89.30222882069823 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (de) |
|
config: de |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 89.54353338968724 |
|
- type: f1 |
|
value: 88.23043644828002 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (es) |
|
config: es |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 90.62374916611076 |
|
- type: f1 |
|
value: 89.68544977510335 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (fr) |
|
config: fr |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 86.18540557469466 |
|
- type: f1 |
|
value: 85.7362674669331 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (hi) |
|
config: hi |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 89.41556113302258 |
|
- type: f1 |
|
value: 89.04934651990581 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (th) |
|
config: th |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 85.89511754068715 |
|
- type: f1 |
|
value: 85.57630467968119 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 70.85043319653442 |
|
- type: f1 |
|
value: 46.0794069318026 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (de) |
|
config: de |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 73.43195266272188 |
|
- type: f1 |
|
value: 48.08015719781981 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (es) |
|
config: es |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 73.8425617078052 |
|
- type: f1 |
|
value: 49.37915156189611 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (fr) |
|
config: fr |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 66.75227059191982 |
|
- type: f1 |
|
value: 43.4642946741452 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (hi) |
|
config: hi |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 69.13589100035855 |
|
- type: f1 |
|
value: 46.25935961966482 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (th) |
|
config: th |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 68.47016274864377 |
|
- type: f1 |
|
value: 46.197113305277796 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (af) |
|
config: af |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 58.14727639542704 |
|
- type: f1 |
|
value: 55.58745169431752 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (am) |
|
config: am |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 57.91190316072628 |
|
- type: f1 |
|
value: 55.46589962622107 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ar) |
|
config: ar |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 57.22932078009414 |
|
- type: f1 |
|
value: 53.661218041561334 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (az) |
|
config: az |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 57.16543375924681 |
|
- type: f1 |
|
value: 55.16504653263189 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (bn) |
|
config: bn |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 62.239408204438476 |
|
- type: f1 |
|
value: 58.941991707183874 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (cy) |
|
config: cy |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 51.186953597848 |
|
- type: f1 |
|
value: 49.59432722397084 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (da) |
|
config: da |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 62.030934767989244 |
|
- type: f1 |
|
value: 58.836302050830966 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (de) |
|
config: de |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 61.314727639542696 |
|
- type: f1 |
|
value: 57.80700293522655 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (el) |
|
config: el |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 64.20645595158037 |
|
- type: f1 |
|
value: 61.36755812840151 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 64.36785474108943 |
|
- type: f1 |
|
value: 61.15645935863754 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (es) |
|
config: es |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 63.97108271687962 |
|
- type: f1 |
|
value: 62.07352472659557 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (fa) |
|
config: fa |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 66.67114996637525 |
|
- type: f1 |
|
value: 63.420170447126324 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (fi) |
|
config: fi |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 62.864828513786144 |
|
- type: f1 |
|
value: 59.655860488861926 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (fr) |
|
config: fr |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 58.55077336919974 |
|
- type: f1 |
|
value: 55.28215385204243 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (he) |
|
config: he |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 63.453261600538 |
|
- type: f1 |
|
value: 59.991998820039186 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (hi) |
|
config: hi |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 61.32145258910558 |
|
- type: f1 |
|
value: 58.9676667104426 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (hu) |
|
config: hu |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 62.905178211163424 |
|
- type: f1 |
|
value: 59.645126480791674 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (hy) |
|
config: hy |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 60.03026227303295 |
|
- type: f1 |
|
value: 56.68905593909442 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (id) |
|
config: id |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 63.28850033624749 |
|
- type: f1 |
|
value: 60.21862015326403 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (is) |
|
config: is |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 56.0221923335575 |
|
- type: f1 |
|
value: 53.388473451598315 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (it) |
|
config: it |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 64.44182918628111 |
|
- type: f1 |
|
value: 62.14806714489123 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ja) |
|
config: ja |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 63.69535978480162 |
|
- type: f1 |
|
value: 62.40231098840202 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (jv) |
|
config: jv |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 52.00067249495628 |
|
- type: f1 |
|
value: 48.871263427511984 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ka) |
|
config: ka |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 54.088769334229994 |
|
- type: f1 |
|
value: 52.68998451556 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (km) |
|
config: km |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 43.34229993275051 |
|
- type: f1 |
|
value: 40.578510490463024 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (kn) |
|
config: kn |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 57.87491593813046 |
|
- type: f1 |
|
value: 55.19579071673386 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ko) |
|
config: ko |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 62.69334229993275 |
|
- type: f1 |
|
value: 60.90210922623679 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (lv) |
|
config: lv |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 56.240753194351036 |
|
- type: f1 |
|
value: 54.137519761157485 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ml) |
|
config: ml |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 62.81439139206457 |
|
- type: f1 |
|
value: 60.46554841337619 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (mn) |
|
config: mn |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 58.49361129791527 |
|
- type: f1 |
|
value: 55.12919894175168 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ms) |
|
config: ms |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 61.55682582380633 |
|
- type: f1 |
|
value: 58.81763499302702 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (my) |
|
config: my |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 59.3981170141224 |
|
- type: f1 |
|
value: 56.31810441546048 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (nb) |
|
config: nb |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 59.89576328177538 |
|
- type: f1 |
|
value: 57.35130066022407 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (nl) |
|
config: nl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 64.55951580363148 |
|
- type: f1 |
|
value: 61.50868742463585 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 65.86079354404842 |
|
- type: f1 |
|
value: 61.94702597578807 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (pt) |
|
config: pt |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 63.49024882313383 |
|
- type: f1 |
|
value: 60.796412851533454 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ro) |
|
config: ro |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 62.53194351042366 |
|
- type: f1 |
|
value: 59.9167382336848 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ru) |
|
config: ru |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 62.62945527908541 |
|
- type: f1 |
|
value: 59.195444230665096 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (sl) |
|
config: sl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 63.43308675184935 |
|
- type: f1 |
|
value: 60.605749901316145 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (sq) |
|
config: sq |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 61.44586415601883 |
|
- type: f1 |
|
value: 58.635066561729396 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (sv) |
|
config: sv |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 62.86482851378615 |
|
- type: f1 |
|
value: 59.75440194153033 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (sw) |
|
config: sw |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 56.250840618695364 |
|
- type: f1 |
|
value: 54.84944007944625 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ta) |
|
config: ta |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 59.747814391392076 |
|
- type: f1 |
|
value: 56.83761137925043 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (te) |
|
config: te |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 59.60995292535306 |
|
- type: f1 |
|
value: 57.106776457430705 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (th) |
|
config: th |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 59.421654337592464 |
|
- type: f1 |
|
value: 57.81013790437749 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (tl) |
|
config: tl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 58.120376597175515 |
|
- type: f1 |
|
value: 55.27690756097837 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (tr) |
|
config: tr |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 60.907868190988566 |
|
- type: f1 |
|
value: 57.43015543162361 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ur) |
|
config: ur |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 59.492266308002705 |
|
- type: f1 |
|
value: 56.885590563156455 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (vi) |
|
config: vi |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 60.477471418964356 |
|
- type: f1 |
|
value: 57.87047944039945 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (zh-CN) |
|
config: zh-CN |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 62.07800941492938 |
|
- type: f1 |
|
value: 59.340232908410265 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (zh-TW) |
|
config: zh-TW |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 56.73167451244117 |
|
- type: f1 |
|
value: 55.29236319279749 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (af) |
|
config: af |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 64.05850706119705 |
|
- type: f1 |
|
value: 62.20100273658395 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (am) |
|
config: am |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 63.24142568930733 |
|
- type: f1 |
|
value: 62.045023522098205 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ar) |
|
config: ar |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 63.685272360457304 |
|
- type: f1 |
|
value: 63.315744557403285 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (az) |
|
config: az |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 60.85743106926698 |
|
- type: f1 |
|
value: 59.106917986505636 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (bn) |
|
config: bn |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 67.1654337592468 |
|
- type: f1 |
|
value: 65.66986920813582 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (cy) |
|
config: cy |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 56.519838601210495 |
|
- type: f1 |
|
value: 54.73278620356587 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (da) |
|
config: da |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 67.76395427034298 |
|
- type: f1 |
|
value: 66.3447645997219 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (de) |
|
config: de |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 67.47814391392065 |
|
- type: f1 |
|
value: 66.32841368787447 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (el) |
|
config: el |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 70.22864828513787 |
|
- type: f1 |
|
value: 69.02774052818218 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (en) |
|
config: en |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 69.04841963685273 |
|
- type: f1 |
|
value: 67.70789401248665 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (es) |
|
config: es |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 69.08204438466711 |
|
- type: f1 |
|
value: 68.39277940460933 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (fa) |
|
config: fa |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 72.10154673839946 |
|
- type: f1 |
|
value: 70.7737194288215 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (fi) |
|
config: fi |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 67.16207128446537 |
|
- type: f1 |
|
value: 66.2311820377212 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (fr) |
|
config: fr |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 63.019502353732335 |
|
- type: f1 |
|
value: 62.105500895318656 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (he) |
|
config: he |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 68.82985877605918 |
|
- type: f1 |
|
value: 67.4894449433449 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (hi) |
|
config: hi |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 66.89643577673168 |
|
- type: f1 |
|
value: 65.45745898521055 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (hu) |
|
config: hu |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 69.32750504371216 |
|
- type: f1 |
|
value: 68.19665323990438 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (hy) |
|
config: hy |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 65.8238063214526 |
|
- type: f1 |
|
value: 64.60872984606974 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (id) |
|
config: id |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 68.98117014122394 |
|
- type: f1 |
|
value: 67.66697147027641 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (is) |
|
config: is |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 63.137188971082715 |
|
- type: f1 |
|
value: 61.58358081191463 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (it) |
|
config: it |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 70.0437121721587 |
|
- type: f1 |
|
value: 69.06747206775307 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ja) |
|
config: ja |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 70.67585743106926 |
|
- type: f1 |
|
value: 70.08618915891508 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (jv) |
|
config: jv |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 59.788164088769335 |
|
- type: f1 |
|
value: 57.91398932676417 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ka) |
|
config: ka |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 61.03227975790182 |
|
- type: f1 |
|
value: 60.044432258486715 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (km) |
|
config: km |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 49.051782111634154 |
|
- type: f1 |
|
value: 45.434581931581555 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (kn) |
|
config: kn |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 63.78278412911902 |
|
- type: f1 |
|
value: 62.106197625881535 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ko) |
|
config: ko |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 69.59986550100874 |
|
- type: f1 |
|
value: 68.94355682848476 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (lv) |
|
config: lv |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 59.97310020174847 |
|
- type: f1 |
|
value: 59.09912773329623 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ml) |
|
config: ml |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 69.20309347679893 |
|
- type: f1 |
|
value: 67.90665916607239 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (mn) |
|
config: mn |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 62.72024209818427 |
|
- type: f1 |
|
value: 60.77165334831407 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ms) |
|
config: ms |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 67.87155346334902 |
|
- type: f1 |
|
value: 65.7906032446679 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (my) |
|
config: my |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 64.97646267652992 |
|
- type: f1 |
|
value: 63.89390215791396 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (nb) |
|
config: nb |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 65.81371889710827 |
|
- type: f1 |
|
value: 64.39323436519936 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (nl) |
|
config: nl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 69.79825151311366 |
|
- type: f1 |
|
value: 68.53789900442244 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 69.98991257565568 |
|
- type: f1 |
|
value: 68.93867074879778 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (pt) |
|
config: pt |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 67.50168123739071 |
|
- type: f1 |
|
value: 66.7457644903972 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ro) |
|
config: ro |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 67.52521856086078 |
|
- type: f1 |
|
value: 66.83370797374445 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ru) |
|
config: ru |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 67.96234028244787 |
|
- type: f1 |
|
value: 67.58983110064196 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (sl) |
|
config: sl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 69.56624075319435 |
|
- type: f1 |
|
value: 68.35270162147211 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (sq) |
|
config: sq |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 68.48352387357095 |
|
- type: f1 |
|
value: 66.66973143886908 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (sv) |
|
config: sv |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 67.92535305985206 |
|
- type: f1 |
|
value: 66.52058462942483 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (sw) |
|
config: sw |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 63.184263618022875 |
|
- type: f1 |
|
value: 61.71153164960602 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ta) |
|
config: ta |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 64.8453261600538 |
|
- type: f1 |
|
value: 63.863209439112346 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (te) |
|
config: te |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 65.39340954942838 |
|
- type: f1 |
|
value: 63.85484524633183 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (th) |
|
config: th |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 67.9892400806994 |
|
- type: f1 |
|
value: 66.57022479007357 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (tl) |
|
config: tl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 63.399462004034966 |
|
- type: f1 |
|
value: 61.62381473991175 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (tr) |
|
config: tr |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 65.773369199731 |
|
- type: f1 |
|
value: 65.58317907780943 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ur) |
|
config: ur |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 65.8069939475454 |
|
- type: f1 |
|
value: 64.47027323557235 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (vi) |
|
config: vi |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 66.51647612642904 |
|
- type: f1 |
|
value: 65.66061210324213 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (zh-CN) |
|
config: zh-CN |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 68.88365837256221 |
|
- type: f1 |
|
value: 67.56956454874091 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (zh-TW) |
|
config: zh-TW |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 63.29858776059179 |
|
- type: f1 |
|
value: 62.76318771484755 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/MedicalRetrieval |
|
name: MTEB MedicalRetrieval |
|
config: default |
|
split: dev |
|
revision: 2039188fb5800a9803ba5048df7b76e6fb151fc6 |
|
metrics: |
|
- type: map_at_1 |
|
value: 2.9000000000000004 |
|
- type: map_at_10 |
|
value: 3.5360000000000005 |
|
- type: map_at_100 |
|
value: 3.703 |
|
- type: map_at_1000 |
|
value: 3.734 |
|
- type: map_at_3 |
|
value: 3.167 |
|
- type: map_at_5 |
|
value: 3.322 |
|
- type: mrr_at_1 |
|
value: 2.9000000000000004 |
|
- type: mrr_at_10 |
|
value: 3.5360000000000005 |
|
- type: mrr_at_100 |
|
value: 3.703 |
|
- type: mrr_at_1000 |
|
value: 3.734 |
|
- type: mrr_at_3 |
|
value: 3.167 |
|
- type: mrr_at_5 |
|
value: 3.322 |
|
- type: ndcg_at_1 |
|
value: 2.9000000000000004 |
|
- type: ndcg_at_10 |
|
value: 4.079 |
|
- type: ndcg_at_100 |
|
value: 5.101 |
|
- type: ndcg_at_1000 |
|
value: 6.295000000000001 |
|
- type: ndcg_at_3 |
|
value: 3.276 |
|
- type: ndcg_at_5 |
|
value: 3.56 |
|
- type: precision_at_1 |
|
value: 2.9000000000000004 |
|
- type: precision_at_10 |
|
value: 0.59 |
|
- type: precision_at_100 |
|
value: 0.11199999999999999 |
|
- type: precision_at_1000 |
|
value: 0.022000000000000002 |
|
- type: precision_at_3 |
|
value: 1.2 |
|
- type: precision_at_5 |
|
value: 0.86 |
|
- type: recall_at_1 |
|
value: 2.9000000000000004 |
|
- type: recall_at_10 |
|
value: 5.8999999999999995 |
|
- type: recall_at_100 |
|
value: 11.200000000000001 |
|
- type: recall_at_1000 |
|
value: 21.5 |
|
- type: recall_at_3 |
|
value: 3.5999999999999996 |
|
- type: recall_at_5 |
|
value: 4.3 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-p2p |
|
name: MTEB MedrxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
|
metrics: |
|
- type: v_measure |
|
value: 19.061819627060558 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-s2s |
|
name: MTEB MedrxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
|
metrics: |
|
- type: v_measure |
|
value: 19.79520446745267 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/mind_small |
|
name: MTEB MindSmallReranking |
|
config: default |
|
split: test |
|
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
|
metrics: |
|
- type: map |
|
value: 26.881162218991285 |
|
- type: mrr |
|
value: 27.23201335662217 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/MultilingualSentiment-classification |
|
name: MTEB MultilingualSentiment |
|
config: default |
|
split: validation |
|
revision: 46958b007a63fdbf239b7672c25d0bea67b5ea1a |
|
metrics: |
|
- type: accuracy |
|
value: 57.69 |
|
- type: f1 |
|
value: 57.370451927892695 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nfcorpus |
|
name: MTEB NFCorpus |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.443 |
|
- type: map_at_10 |
|
value: 1.189 |
|
- type: map_at_100 |
|
value: 2.221 |
|
- type: map_at_1000 |
|
value: 3.034 |
|
- type: map_at_3 |
|
value: 0.683 |
|
- type: map_at_5 |
|
value: 0.882 |
|
- type: mrr_at_1 |
|
value: 4.334 |
|
- type: mrr_at_10 |
|
value: 10.908 |
|
- type: mrr_at_100 |
|
value: 12.536 |
|
- type: mrr_at_1000 |
|
value: 12.642000000000001 |
|
- type: mrr_at_3 |
|
value: 7.481999999999999 |
|
- type: mrr_at_5 |
|
value: 9.324 |
|
- type: ndcg_at_1 |
|
value: 3.7150000000000003 |
|
- type: ndcg_at_10 |
|
value: 5.591 |
|
- type: ndcg_at_100 |
|
value: 9.522 |
|
- type: ndcg_at_1000 |
|
value: 19.705000000000002 |
|
- type: ndcg_at_3 |
|
value: 4.292 |
|
- type: ndcg_at_5 |
|
value: 5.038 |
|
- type: precision_at_1 |
|
value: 4.334 |
|
- type: precision_at_10 |
|
value: 5.077 |
|
- type: precision_at_100 |
|
value: 3.2910000000000004 |
|
- type: precision_at_1000 |
|
value: 1.568 |
|
- type: precision_at_3 |
|
value: 4.644 |
|
- type: precision_at_5 |
|
value: 5.139 |
|
- type: recall_at_1 |
|
value: 0.443 |
|
- type: recall_at_10 |
|
value: 3.3520000000000003 |
|
- type: recall_at_100 |
|
value: 15.515 |
|
- type: recall_at_1000 |
|
value: 50.505 |
|
- type: recall_at_3 |
|
value: 0.931 |
|
- type: recall_at_5 |
|
value: 1.698 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nfcorpus-pl |
|
name: MTEB NFCorpus-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.307 |
|
- type: map_at_10 |
|
value: 0.835 |
|
- type: map_at_100 |
|
value: 1.503 |
|
- type: map_at_1000 |
|
value: 2.263 |
|
- type: map_at_3 |
|
value: 0.503 |
|
- type: map_at_5 |
|
value: 0.567 |
|
- type: mrr_at_1 |
|
value: 4.025 |
|
- type: mrr_at_10 |
|
value: 9.731 |
|
- type: mrr_at_100 |
|
value: 11.229 |
|
- type: mrr_at_1000 |
|
value: 11.34 |
|
- type: mrr_at_3 |
|
value: 6.811 |
|
- type: mrr_at_5 |
|
value: 8.126999999999999 |
|
- type: ndcg_at_1 |
|
value: 3.56 |
|
- type: ndcg_at_10 |
|
value: 4.596 |
|
- type: ndcg_at_100 |
|
value: 7.567 |
|
- type: ndcg_at_1000 |
|
value: 17.76 |
|
- type: ndcg_at_3 |
|
value: 3.52 |
|
- type: ndcg_at_5 |
|
value: 3.823 |
|
- type: precision_at_1 |
|
value: 4.025 |
|
- type: precision_at_10 |
|
value: 4.334 |
|
- type: precision_at_100 |
|
value: 2.842 |
|
- type: precision_at_1000 |
|
value: 1.506 |
|
- type: precision_at_3 |
|
value: 3.818 |
|
- type: precision_at_5 |
|
value: 4.149 |
|
- type: recall_at_1 |
|
value: 0.307 |
|
- type: recall_at_10 |
|
value: 2.543 |
|
- type: recall_at_100 |
|
value: 12.152000000000001 |
|
- type: recall_at_1000 |
|
value: 46.878 |
|
- type: recall_at_3 |
|
value: 0.755 |
|
- type: recall_at_5 |
|
value: 0.975 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nq |
|
name: MTEB NQ |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.439 |
|
- type: map_at_10 |
|
value: 0.6839999999999999 |
|
- type: map_at_100 |
|
value: 0.769 |
|
- type: map_at_1000 |
|
value: 0.79 |
|
- type: map_at_3 |
|
value: 0.584 |
|
- type: map_at_5 |
|
value: 0.621 |
|
- type: mrr_at_1 |
|
value: 0.5499999999999999 |
|
- type: mrr_at_10 |
|
value: 0.819 |
|
- type: mrr_at_100 |
|
value: 0.9169999999999999 |
|
- type: mrr_at_1000 |
|
value: 0.9400000000000001 |
|
- type: mrr_at_3 |
|
value: 0.705 |
|
- type: mrr_at_5 |
|
value: 0.75 |
|
- type: ndcg_at_1 |
|
value: 0.5499999999999999 |
|
- type: ndcg_at_10 |
|
value: 0.886 |
|
- type: ndcg_at_100 |
|
value: 1.422 |
|
- type: ndcg_at_1000 |
|
value: 2.2079999999999997 |
|
- type: ndcg_at_3 |
|
value: 0.6629999999999999 |
|
- type: ndcg_at_5 |
|
value: 0.735 |
|
- type: precision_at_1 |
|
value: 0.5499999999999999 |
|
- type: precision_at_10 |
|
value: 0.16199999999999998 |
|
- type: precision_at_100 |
|
value: 0.048 |
|
- type: precision_at_1000 |
|
value: 0.012 |
|
- type: precision_at_3 |
|
value: 0.309 |
|
- type: precision_at_5 |
|
value: 0.22599999999999998 |
|
- type: recall_at_1 |
|
value: 0.439 |
|
- type: recall_at_10 |
|
value: 1.405 |
|
- type: recall_at_100 |
|
value: 4.051 |
|
- type: recall_at_1000 |
|
value: 10.487 |
|
- type: recall_at_3 |
|
value: 0.787 |
|
- type: recall_at_5 |
|
value: 0.9560000000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: narrativeqa |
|
name: MTEB NarrativeQARetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 5.93 |
|
- type: map_at_10 |
|
value: 7.349 |
|
- type: map_at_100 |
|
value: 8.011 |
|
- type: map_at_1000 |
|
value: 8.351 |
|
- type: map_at_3 |
|
value: 6.787 |
|
- type: map_at_5 |
|
value: 7.02 |
|
- type: mrr_at_1 |
|
value: 5.93 |
|
- type: mrr_at_10 |
|
value: 7.349 |
|
- type: mrr_at_100 |
|
value: 8.011 |
|
- type: mrr_at_1000 |
|
value: 8.351 |
|
- type: mrr_at_3 |
|
value: 6.787 |
|
- type: mrr_at_5 |
|
value: 7.02 |
|
- type: ndcg_at_1 |
|
value: 5.93 |
|
- type: ndcg_at_10 |
|
value: 8.291 |
|
- type: ndcg_at_100 |
|
value: 12.833 |
|
- type: ndcg_at_1000 |
|
value: 21.253 |
|
- type: ndcg_at_3 |
|
value: 7.072000000000001 |
|
- type: ndcg_at_5 |
|
value: 7.495 |
|
- type: precision_at_1 |
|
value: 5.93 |
|
- type: precision_at_10 |
|
value: 1.1400000000000001 |
|
- type: precision_at_100 |
|
value: 0.359 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 2.633 |
|
- type: precision_at_5 |
|
value: 1.786 |
|
- type: recall_at_1 |
|
value: 5.93 |
|
- type: recall_at_10 |
|
value: 11.395 |
|
- type: recall_at_100 |
|
value: 35.929 |
|
- type: recall_at_1000 |
|
value: 100.0 |
|
- type: recall_at_3 |
|
value: 7.9 |
|
- type: recall_at_5 |
|
value: 8.932 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: ScandEval/norec-mini |
|
name: MTEB NoRecClassification |
|
config: default |
|
split: test |
|
revision: 07b99ab3363c2e7f8f87015b01c21f4d9b917ce3 |
|
metrics: |
|
- type: accuracy |
|
value: 48.251953125 |
|
- type: f1 |
|
value: 45.42526611578402 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: strombergnlp/nordic_langid |
|
name: MTEB NordicLangClassification |
|
config: default |
|
split: test |
|
revision: e254179d18ab0165fdb6dbef91178266222bee2a |
|
metrics: |
|
- type: accuracy |
|
value: 48.403333333333336 |
|
- type: f1 |
|
value: 47.9287124185198 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: kardosdrur/norwegian-courts |
|
name: MTEB NorwegianCourtsBitextMining |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 93.85964912280701 |
|
- type: f1 |
|
value: 92.98245614035088 |
|
- type: precision |
|
value: 92.54385964912281 |
|
- type: recall |
|
value: 93.85964912280701 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: NbAiLab/norwegian_parliament |
|
name: MTEB NorwegianParliament |
|
config: default |
|
split: test |
|
revision: f7393532774c66312378d30b197610b43d751972 |
|
metrics: |
|
- type: accuracy |
|
value: 55.991666666666674 |
|
- type: ap |
|
value: 53.417849849746226 |
|
- type: f1 |
|
value: 55.757916182475384 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: C-MTEB/OCNLI |
|
name: MTEB Ocnli |
|
config: default |
|
split: validation |
|
revision: 66e76a618a34d6d565d5538088562851e6daa7ec |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 54.68327016783974 |
|
- type: cos_sim_ap |
|
value: 55.175059616546406 |
|
- type: cos_sim_f1 |
|
value: 67.81733189500179 |
|
- type: cos_sim_precision |
|
value: 51.41766630316249 |
|
- type: cos_sim_recall |
|
value: 99.57761351636748 |
|
- type: dot_accuracy |
|
value: 54.68327016783974 |
|
- type: dot_ap |
|
value: 55.175059616546406 |
|
- type: dot_f1 |
|
value: 67.81733189500179 |
|
- type: dot_precision |
|
value: 51.41766630316249 |
|
- type: dot_recall |
|
value: 99.57761351636748 |
|
- type: euclidean_accuracy |
|
value: 54.68327016783974 |
|
- type: euclidean_ap |
|
value: 55.17510180566365 |
|
- type: euclidean_f1 |
|
value: 67.81733189500179 |
|
- type: euclidean_precision |
|
value: 51.41766630316249 |
|
- type: euclidean_recall |
|
value: 99.57761351636748 |
|
- type: manhattan_accuracy |
|
value: 55.44125609095831 |
|
- type: manhattan_ap |
|
value: 55.76283671826867 |
|
- type: manhattan_f1 |
|
value: 68.05905653583004 |
|
- type: manhattan_precision |
|
value: 51.63934426229508 |
|
- type: manhattan_recall |
|
value: 99.78880675818374 |
|
- type: max_accuracy |
|
value: 55.44125609095831 |
|
- type: max_ap |
|
value: 55.76283671826867 |
|
- type: max_f1 |
|
value: 68.05905653583004 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/OnlineShopping-classification |
|
name: MTEB OnlineShopping |
|
config: default |
|
split: test |
|
revision: e610f2ebd179a8fda30ae534c3878750a96db120 |
|
metrics: |
|
- type: accuracy |
|
value: 75.64 |
|
- type: ap |
|
value: 71.45085103287833 |
|
- type: f1 |
|
value: 75.52254495697326 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: laugustyniak/abusive-clauses-pl |
|
name: MTEB PAC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 73.86620330147699 |
|
- type: ap |
|
value: 80.58015815306322 |
|
- type: f1 |
|
value: 71.49082510883872 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/PAWSX |
|
name: MTEB PAWSX |
|
config: default |
|
split: test |
|
revision: 9c6a90e430ac22b5779fb019a23e820b11a8b5e1 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 29.52361689421863 |
|
- type: cos_sim_spearman |
|
value: 32.750058577257875 |
|
- type: euclidean_pearson |
|
value: 34.583472972871796 |
|
- type: euclidean_spearman |
|
value: 32.75328764421994 |
|
- type: manhattan_pearson |
|
value: 34.727366510326995 |
|
- type: manhattan_spearman |
|
value: 32.787167142114214 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/ppc-pairclassification |
|
name: MTEB PPC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 71.1 |
|
- type: cos_sim_ap |
|
value: 85.36544548691205 |
|
- type: cos_sim_f1 |
|
value: 75.23393636930756 |
|
- type: cos_sim_precision |
|
value: 60.36036036036037 |
|
- type: cos_sim_recall |
|
value: 99.83443708609272 |
|
- type: dot_accuracy |
|
value: 71.1 |
|
- type: dot_ap |
|
value: 85.36544548691204 |
|
- type: dot_f1 |
|
value: 75.23393636930756 |
|
- type: dot_precision |
|
value: 60.36036036036037 |
|
- type: dot_recall |
|
value: 99.83443708609272 |
|
- type: euclidean_accuracy |
|
value: 71.1 |
|
- type: euclidean_ap |
|
value: 85.36544548691205 |
|
- type: euclidean_f1 |
|
value: 75.23393636930756 |
|
- type: euclidean_precision |
|
value: 60.36036036036037 |
|
- type: euclidean_recall |
|
value: 99.83443708609272 |
|
- type: manhattan_accuracy |
|
value: 71.1 |
|
- type: manhattan_ap |
|
value: 85.33853868545614 |
|
- type: manhattan_f1 |
|
value: 75.23393636930756 |
|
- type: manhattan_precision |
|
value: 60.36036036036037 |
|
- type: manhattan_recall |
|
value: 99.83443708609272 |
|
- type: max_accuracy |
|
value: 71.1 |
|
- type: max_ap |
|
value: 85.36544548691205 |
|
- type: max_f1 |
|
value: 75.23393636930756 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/psc-pairclassification |
|
name: MTEB PSC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 90.81632653061224 |
|
- type: cos_sim_ap |
|
value: 91.97693749083473 |
|
- type: cos_sim_f1 |
|
value: 85.55078683834049 |
|
- type: cos_sim_precision |
|
value: 80.59299191374663 |
|
- type: cos_sim_recall |
|
value: 91.15853658536585 |
|
- type: dot_accuracy |
|
value: 90.81632653061224 |
|
- type: dot_ap |
|
value: 91.97693749083473 |
|
- type: dot_f1 |
|
value: 85.55078683834049 |
|
- type: dot_precision |
|
value: 80.59299191374663 |
|
- type: dot_recall |
|
value: 91.15853658536585 |
|
- type: euclidean_accuracy |
|
value: 90.81632653061224 |
|
- type: euclidean_ap |
|
value: 91.97693749083473 |
|
- type: euclidean_f1 |
|
value: 85.55078683834049 |
|
- type: euclidean_precision |
|
value: 80.59299191374663 |
|
- type: euclidean_recall |
|
value: 91.15853658536585 |
|
- type: manhattan_accuracy |
|
value: 90.9090909090909 |
|
- type: manhattan_ap |
|
value: 92.043441286281 |
|
- type: manhattan_f1 |
|
value: 85.34482758620689 |
|
- type: manhattan_precision |
|
value: 80.70652173913044 |
|
- type: manhattan_recall |
|
value: 90.54878048780488 |
|
- type: max_accuracy |
|
value: 90.9090909090909 |
|
- type: max_ap |
|
value: 92.043441286281 |
|
- type: max_f1 |
|
value: 85.55078683834049 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: paws-x |
|
name: MTEB PawsX (de) |
|
config: de |
|
split: test |
|
revision: 8a04d940a42cd40658986fdd8e3da561533a3646 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 70.35 |
|
- type: cos_sim_ap |
|
value: 72.01641717127626 |
|
- type: cos_sim_f1 |
|
value: 64.49511400651467 |
|
- type: cos_sim_precision |
|
value: 55.26315789473685 |
|
- type: cos_sim_recall |
|
value: 77.43016759776536 |
|
- type: dot_accuracy |
|
value: 70.35 |
|
- type: dot_ap |
|
value: 72.06599137974572 |
|
- type: dot_f1 |
|
value: 64.49511400651467 |
|
- type: dot_precision |
|
value: 55.26315789473685 |
|
- type: dot_recall |
|
value: 77.43016759776536 |
|
- type: euclidean_accuracy |
|
value: 70.35 |
|
- type: euclidean_ap |
|
value: 71.92019289154159 |
|
- type: euclidean_f1 |
|
value: 64.49511400651467 |
|
- type: euclidean_precision |
|
value: 55.26315789473685 |
|
- type: euclidean_recall |
|
value: 77.43016759776536 |
|
- type: manhattan_accuracy |
|
value: 70.35 |
|
- type: manhattan_ap |
|
value: 71.92979188519502 |
|
- type: manhattan_f1 |
|
value: 64.60409019402202 |
|
- type: manhattan_precision |
|
value: 60.86956521739131 |
|
- type: manhattan_recall |
|
value: 68.8268156424581 |
|
- type: max_accuracy |
|
value: 70.35 |
|
- type: max_ap |
|
value: 72.06599137974572 |
|
- type: max_f1 |
|
value: 64.60409019402202 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: paws-x |
|
name: MTEB PawsX (en) |
|
config: en |
|
split: test |
|
revision: 8a04d940a42cd40658986fdd8e3da561533a3646 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 71.0 |
|
- type: cos_sim_ap |
|
value: 74.73017292645147 |
|
- type: cos_sim_f1 |
|
value: 66.73427991886409 |
|
- type: cos_sim_precision |
|
value: 61.78403755868545 |
|
- type: cos_sim_recall |
|
value: 72.54685777287762 |
|
- type: dot_accuracy |
|
value: 71.0 |
|
- type: dot_ap |
|
value: 74.73017292645147 |
|
- type: dot_f1 |
|
value: 66.73427991886409 |
|
- type: dot_precision |
|
value: 61.78403755868545 |
|
- type: dot_recall |
|
value: 72.54685777287762 |
|
- type: euclidean_accuracy |
|
value: 71.0 |
|
- type: euclidean_ap |
|
value: 74.73013082197343 |
|
- type: euclidean_f1 |
|
value: 66.73427991886409 |
|
- type: euclidean_precision |
|
value: 61.78403755868545 |
|
- type: euclidean_recall |
|
value: 72.54685777287762 |
|
- type: manhattan_accuracy |
|
value: 70.95 |
|
- type: manhattan_ap |
|
value: 74.71203917486744 |
|
- type: manhattan_f1 |
|
value: 66.86868686868686 |
|
- type: manhattan_precision |
|
value: 61.696178937558244 |
|
- type: manhattan_recall |
|
value: 72.98787210584344 |
|
- type: max_accuracy |
|
value: 71.0 |
|
- type: max_ap |
|
value: 74.73017292645147 |
|
- type: max_f1 |
|
value: 66.86868686868686 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: paws-x |
|
name: MTEB PawsX (es) |
|
config: es |
|
split: test |
|
revision: 8a04d940a42cd40658986fdd8e3da561533a3646 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 67.7 |
|
- type: cos_sim_ap |
|
value: 69.70320170421651 |
|
- type: cos_sim_f1 |
|
value: 62.55625562556255 |
|
- type: cos_sim_precision |
|
value: 52.851711026615966 |
|
- type: cos_sim_recall |
|
value: 76.62624035281146 |
|
- type: dot_accuracy |
|
value: 67.7 |
|
- type: dot_ap |
|
value: 69.70320170421651 |
|
- type: dot_f1 |
|
value: 62.55625562556255 |
|
- type: dot_precision |
|
value: 52.851711026615966 |
|
- type: dot_recall |
|
value: 76.62624035281146 |
|
- type: euclidean_accuracy |
|
value: 67.7 |
|
- type: euclidean_ap |
|
value: 69.70320170421651 |
|
- type: euclidean_f1 |
|
value: 62.55625562556255 |
|
- type: euclidean_precision |
|
value: 52.851711026615966 |
|
- type: euclidean_recall |
|
value: 76.62624035281146 |
|
- type: manhattan_accuracy |
|
value: 67.75 |
|
- type: manhattan_ap |
|
value: 69.67833816050764 |
|
- type: manhattan_f1 |
|
value: 62.734082397003746 |
|
- type: manhattan_precision |
|
value: 54.515866558177386 |
|
- type: manhattan_recall |
|
value: 73.8699007717751 |
|
- type: max_accuracy |
|
value: 67.75 |
|
- type: max_ap |
|
value: 69.70320170421651 |
|
- type: max_f1 |
|
value: 62.734082397003746 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: paws-x |
|
name: MTEB PawsX (fr) |
|
config: fr |
|
split: test |
|
revision: 8a04d940a42cd40658986fdd8e3da561533a3646 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 69.0 |
|
- type: cos_sim_ap |
|
value: 71.36406639969131 |
|
- type: cos_sim_f1 |
|
value: 64.45993031358886 |
|
- type: cos_sim_precision |
|
value: 53.12275664034458 |
|
- type: cos_sim_recall |
|
value: 81.94905869324474 |
|
- type: dot_accuracy |
|
value: 69.0 |
|
- type: dot_ap |
|
value: 71.2599779415656 |
|
- type: dot_f1 |
|
value: 64.45993031358886 |
|
- type: dot_precision |
|
value: 53.12275664034458 |
|
- type: dot_recall |
|
value: 81.94905869324474 |
|
- type: euclidean_accuracy |
|
value: 69.0 |
|
- type: euclidean_ap |
|
value: 71.3126257271965 |
|
- type: euclidean_f1 |
|
value: 64.45993031358886 |
|
- type: euclidean_precision |
|
value: 53.12275664034458 |
|
- type: euclidean_recall |
|
value: 81.94905869324474 |
|
- type: manhattan_accuracy |
|
value: 69.0 |
|
- type: manhattan_ap |
|
value: 71.29361764028188 |
|
- type: manhattan_f1 |
|
value: 64.54789615040288 |
|
- type: manhattan_precision |
|
value: 54.16979714500376 |
|
- type: manhattan_recall |
|
value: 79.84496124031007 |
|
- type: max_accuracy |
|
value: 69.0 |
|
- type: max_ap |
|
value: 71.36406639969131 |
|
- type: max_f1 |
|
value: 64.54789615040288 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: paws-x |
|
name: MTEB PawsX (ja) |
|
config: ja |
|
split: test |
|
revision: 8a04d940a42cd40658986fdd8e3da561533a3646 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 63.849999999999994 |
|
- type: cos_sim_ap |
|
value: 60.914955950361026 |
|
- type: cos_sim_f1 |
|
value: 62.4556422995032 |
|
- type: cos_sim_precision |
|
value: 45.47803617571059 |
|
- type: cos_sim_recall |
|
value: 99.66024915062289 |
|
- type: dot_accuracy |
|
value: 63.849999999999994 |
|
- type: dot_ap |
|
value: 60.808056565465506 |
|
- type: dot_f1 |
|
value: 62.4556422995032 |
|
- type: dot_precision |
|
value: 45.47803617571059 |
|
- type: dot_recall |
|
value: 99.66024915062289 |
|
- type: euclidean_accuracy |
|
value: 63.849999999999994 |
|
- type: euclidean_ap |
|
value: 60.8231492677072 |
|
- type: euclidean_f1 |
|
value: 62.4556422995032 |
|
- type: euclidean_precision |
|
value: 45.47803617571059 |
|
- type: euclidean_recall |
|
value: 99.66024915062289 |
|
- type: manhattan_accuracy |
|
value: 63.800000000000004 |
|
- type: manhattan_ap |
|
value: 60.86392751846975 |
|
- type: manhattan_f1 |
|
value: 62.43348705214614 |
|
- type: manhattan_precision |
|
value: 45.45454545454545 |
|
- type: manhattan_recall |
|
value: 99.66024915062289 |
|
- type: max_accuracy |
|
value: 63.849999999999994 |
|
- type: max_ap |
|
value: 60.914955950361026 |
|
- type: max_f1 |
|
value: 62.4556422995032 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: paws-x |
|
name: MTEB PawsX (ko) |
|
config: ko |
|
split: test |
|
revision: 8a04d940a42cd40658986fdd8e3da561533a3646 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 61.1 |
|
- type: cos_sim_ap |
|
value: 58.40339411735916 |
|
- type: cos_sim_f1 |
|
value: 62.7906976744186 |
|
- type: cos_sim_precision |
|
value: 46.55172413793103 |
|
- type: cos_sim_recall |
|
value: 96.42857142857143 |
|
- type: dot_accuracy |
|
value: 61.1 |
|
- type: dot_ap |
|
value: 58.439189685586456 |
|
- type: dot_f1 |
|
value: 62.7906976744186 |
|
- type: dot_precision |
|
value: 46.55172413793103 |
|
- type: dot_recall |
|
value: 96.42857142857143 |
|
- type: euclidean_accuracy |
|
value: 61.1 |
|
- type: euclidean_ap |
|
value: 58.34968788203145 |
|
- type: euclidean_f1 |
|
value: 62.7906976744186 |
|
- type: euclidean_precision |
|
value: 46.55172413793103 |
|
- type: euclidean_recall |
|
value: 96.42857142857143 |
|
- type: manhattan_accuracy |
|
value: 61.1 |
|
- type: manhattan_ap |
|
value: 58.31504446861402 |
|
- type: manhattan_f1 |
|
value: 62.636562272396226 |
|
- type: manhattan_precision |
|
value: 46.48648648648649 |
|
- type: manhattan_recall |
|
value: 95.98214285714286 |
|
- type: max_accuracy |
|
value: 61.1 |
|
- type: max_ap |
|
value: 58.439189685586456 |
|
- type: max_f1 |
|
value: 62.7906976744186 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: paws-x |
|
name: MTEB PawsX (zh) |
|
config: zh |
|
split: test |
|
revision: 8a04d940a42cd40658986fdd8e3da561533a3646 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 64.2 |
|
- type: cos_sim_ap |
|
value: 63.73722153283802 |
|
- type: cos_sim_f1 |
|
value: 62.52707581227437 |
|
- type: cos_sim_precision |
|
value: 46.16204690831556 |
|
- type: cos_sim_recall |
|
value: 96.86800894854586 |
|
- type: dot_accuracy |
|
value: 64.2 |
|
- type: dot_ap |
|
value: 63.67335241021108 |
|
- type: dot_f1 |
|
value: 62.52707581227437 |
|
- type: dot_precision |
|
value: 46.16204690831556 |
|
- type: dot_recall |
|
value: 96.86800894854586 |
|
- type: euclidean_accuracy |
|
value: 64.2 |
|
- type: euclidean_ap |
|
value: 63.77399571117368 |
|
- type: euclidean_f1 |
|
value: 62.52707581227437 |
|
- type: euclidean_precision |
|
value: 46.16204690831556 |
|
- type: euclidean_recall |
|
value: 96.86800894854586 |
|
- type: manhattan_accuracy |
|
value: 64.5 |
|
- type: manhattan_ap |
|
value: 63.747406783360816 |
|
- type: manhattan_f1 |
|
value: 62.58601955813112 |
|
- type: manhattan_precision |
|
value: 46.27745045527584 |
|
- type: manhattan_recall |
|
value: 96.64429530201343 |
|
- type: max_accuracy |
|
value: 64.5 |
|
- type: max_ap |
|
value: 63.77399571117368 |
|
- type: max_f1 |
|
value: 62.58601955813112 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/polemo2_in |
|
name: MTEB PolEmo2.0-IN |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 52.797783933518005 |
|
- type: f1 |
|
value: 53.84971294048786 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/polemo2_out |
|
name: MTEB PolEmo2.0-OUT |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 38.40080971659919 |
|
- type: f1 |
|
value: 30.38990873840624 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/QBQTC |
|
name: MTEB QBQTC |
|
config: default |
|
split: test |
|
revision: 790b0510dc52b1553e8c49f3d2afb48c0e5c48b7 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 23.34232568997104 |
|
- type: cos_sim_spearman |
|
value: 24.47961936211083 |
|
- type: euclidean_pearson |
|
value: 22.03140944610336 |
|
- type: euclidean_spearman |
|
value: 24.47949166265398 |
|
- type: manhattan_pearson |
|
value: 25.542406448726908 |
|
- type: manhattan_spearman |
|
value: 28.655724283839533 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: quora-pl |
|
name: MTEB Quora-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 59.938 |
|
- type: map_at_10 |
|
value: 72.734 |
|
- type: map_at_100 |
|
value: 73.564 |
|
- type: map_at_1000 |
|
value: 73.602 |
|
- type: map_at_3 |
|
value: 69.707 |
|
- type: map_at_5 |
|
value: 71.515 |
|
- type: mrr_at_1 |
|
value: 69.28 |
|
- type: mrr_at_10 |
|
value: 76.97500000000001 |
|
- type: mrr_at_100 |
|
value: 77.27199999999999 |
|
- type: mrr_at_1000 |
|
value: 77.28 |
|
- type: mrr_at_3 |
|
value: 75.355 |
|
- type: mrr_at_5 |
|
value: 76.389 |
|
- type: ndcg_at_1 |
|
value: 69.33 |
|
- type: ndcg_at_10 |
|
value: 77.61099999999999 |
|
- type: ndcg_at_100 |
|
value: 80.02 |
|
- type: ndcg_at_1000 |
|
value: 80.487 |
|
- type: ndcg_at_3 |
|
value: 73.764 |
|
- type: ndcg_at_5 |
|
value: 75.723 |
|
- type: precision_at_1 |
|
value: 69.33 |
|
- type: precision_at_10 |
|
value: 11.917 |
|
- type: precision_at_100 |
|
value: 1.447 |
|
- type: precision_at_1000 |
|
value: 0.154 |
|
- type: precision_at_3 |
|
value: 32.29 |
|
- type: precision_at_5 |
|
value: 21.432000000000002 |
|
- type: recall_at_1 |
|
value: 59.938 |
|
- type: recall_at_10 |
|
value: 87.252 |
|
- type: recall_at_100 |
|
value: 96.612 |
|
- type: recall_at_1000 |
|
value: 99.388 |
|
- type: recall_at_3 |
|
value: 76.264 |
|
- type: recall_at_5 |
|
value: 81.71000000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: quora |
|
name: MTEB QuoraRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 61.458999999999996 |
|
- type: map_at_10 |
|
value: 73.90299999999999 |
|
- type: map_at_100 |
|
value: 74.733 |
|
- type: map_at_1000 |
|
value: 74.771 |
|
- type: map_at_3 |
|
value: 70.999 |
|
- type: map_at_5 |
|
value: 72.745 |
|
- type: mrr_at_1 |
|
value: 70.93 |
|
- type: mrr_at_10 |
|
value: 78.353 |
|
- type: mrr_at_100 |
|
value: 78.636 |
|
- type: mrr_at_1000 |
|
value: 78.644 |
|
- type: mrr_at_3 |
|
value: 76.908 |
|
- type: mrr_at_5 |
|
value: 77.807 |
|
- type: ndcg_at_1 |
|
value: 70.93 |
|
- type: ndcg_at_10 |
|
value: 78.625 |
|
- type: ndcg_at_100 |
|
value: 81.01 |
|
- type: ndcg_at_1000 |
|
value: 81.45700000000001 |
|
- type: ndcg_at_3 |
|
value: 75.045 |
|
- type: ndcg_at_5 |
|
value: 76.84299999999999 |
|
- type: precision_at_1 |
|
value: 70.93 |
|
- type: precision_at_10 |
|
value: 11.953 |
|
- type: precision_at_100 |
|
value: 1.4489999999999998 |
|
- type: precision_at_1000 |
|
value: 0.154 |
|
- type: precision_at_3 |
|
value: 32.65 |
|
- type: precision_at_5 |
|
value: 21.598 |
|
- type: recall_at_1 |
|
value: 61.458999999999996 |
|
- type: recall_at_10 |
|
value: 87.608 |
|
- type: recall_at_100 |
|
value: 96.818 |
|
- type: recall_at_1000 |
|
value: 99.445 |
|
- type: recall_at_3 |
|
value: 77.354 |
|
- type: recall_at_5 |
|
value: 82.334 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering |
|
name: MTEB RedditClustering |
|
config: default |
|
split: test |
|
revision: 24640382cdbf8abc73003fb0fa6d111a705499eb |
|
metrics: |
|
- type: v_measure |
|
value: 28.519889100999958 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering-p2p |
|
name: MTEB RedditClusteringP2P |
|
config: default |
|
split: test |
|
revision: 282350215ef01743dc01b456c7f5241fa8937f16 |
|
metrics: |
|
- type: v_measure |
|
value: 38.62765374782771 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scidocs |
|
name: MTEB SCIDOCS |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.52 |
|
- type: map_at_10 |
|
value: 0.893 |
|
- type: map_at_100 |
|
value: 1.113 |
|
- type: map_at_1000 |
|
value: 1.304 |
|
- type: map_at_3 |
|
value: 0.7779999999999999 |
|
- type: map_at_5 |
|
value: 0.8200000000000001 |
|
- type: mrr_at_1 |
|
value: 2.6 |
|
- type: mrr_at_10 |
|
value: 4.0680000000000005 |
|
- type: mrr_at_100 |
|
value: 4.6080000000000005 |
|
- type: mrr_at_1000 |
|
value: 4.797 |
|
- type: mrr_at_3 |
|
value: 3.5999999999999996 |
|
- type: mrr_at_5 |
|
value: 3.8150000000000004 |
|
- type: ndcg_at_1 |
|
value: 2.6 |
|
- type: ndcg_at_10 |
|
value: 1.79 |
|
- type: ndcg_at_100 |
|
value: 3.5549999999999997 |
|
- type: ndcg_at_1000 |
|
value: 9.942 |
|
- type: ndcg_at_3 |
|
value: 1.94 |
|
- type: ndcg_at_5 |
|
value: 1.543 |
|
- type: precision_at_1 |
|
value: 2.6 |
|
- type: precision_at_10 |
|
value: 0.8500000000000001 |
|
- type: precision_at_100 |
|
value: 0.361 |
|
- type: precision_at_1000 |
|
value: 0.197 |
|
- type: precision_at_3 |
|
value: 1.7670000000000001 |
|
- type: precision_at_5 |
|
value: 1.26 |
|
- type: recall_at_1 |
|
value: 0.52 |
|
- type: recall_at_10 |
|
value: 1.7149999999999999 |
|
- type: recall_at_100 |
|
value: 7.318 |
|
- type: recall_at_1000 |
|
value: 39.915 |
|
- type: recall_at_3 |
|
value: 1.0699999999999998 |
|
- type: recall_at_5 |
|
value: 1.27 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scidocs-pl |
|
name: MTEB SCIDOCS-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.32 |
|
- type: map_at_10 |
|
value: 0.676 |
|
- type: map_at_100 |
|
value: 0.847 |
|
- type: map_at_1000 |
|
value: 1.032 |
|
- type: map_at_3 |
|
value: 0.5369999999999999 |
|
- type: map_at_5 |
|
value: 0.592 |
|
- type: mrr_at_1 |
|
value: 1.6 |
|
- type: mrr_at_10 |
|
value: 2.863 |
|
- type: mrr_at_100 |
|
value: 3.334 |
|
- type: mrr_at_1000 |
|
value: 3.5479999999999996 |
|
- type: mrr_at_3 |
|
value: 2.317 |
|
- type: mrr_at_5 |
|
value: 2.587 |
|
- type: ndcg_at_1 |
|
value: 1.6 |
|
- type: ndcg_at_10 |
|
value: 1.397 |
|
- type: ndcg_at_100 |
|
value: 2.819 |
|
- type: ndcg_at_1000 |
|
value: 9.349 |
|
- type: ndcg_at_3 |
|
value: 1.3 |
|
- type: ndcg_at_5 |
|
value: 1.1079999999999999 |
|
- type: precision_at_1 |
|
value: 1.6 |
|
- type: precision_at_10 |
|
value: 0.74 |
|
- type: precision_at_100 |
|
value: 0.295 |
|
- type: precision_at_1000 |
|
value: 0.194 |
|
- type: precision_at_3 |
|
value: 1.2 |
|
- type: precision_at_5 |
|
value: 0.96 |
|
- type: recall_at_1 |
|
value: 0.32 |
|
- type: recall_at_10 |
|
value: 1.505 |
|
- type: recall_at_100 |
|
value: 5.988 |
|
- type: recall_at_1000 |
|
value: 39.308 |
|
- type: recall_at_3 |
|
value: 0.72 |
|
- type: recall_at_5 |
|
value: 0.9650000000000001 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/sicke-pl-pairclassification |
|
name: MTEB SICK-E-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 73.84834896045659 |
|
- type: cos_sim_ap |
|
value: 55.484124732566606 |
|
- type: cos_sim_f1 |
|
value: 57.34228187919464 |
|
- type: cos_sim_precision |
|
value: 46.01464885825076 |
|
- type: cos_sim_recall |
|
value: 76.06837606837607 |
|
- type: dot_accuracy |
|
value: 73.84834896045659 |
|
- type: dot_ap |
|
value: 55.48400003295399 |
|
- type: dot_f1 |
|
value: 57.34228187919464 |
|
- type: dot_precision |
|
value: 46.01464885825076 |
|
- type: dot_recall |
|
value: 76.06837606837607 |
|
- type: euclidean_accuracy |
|
value: 73.84834896045659 |
|
- type: euclidean_ap |
|
value: 55.48407331902175 |
|
- type: euclidean_f1 |
|
value: 57.34228187919464 |
|
- type: euclidean_precision |
|
value: 46.01464885825076 |
|
- type: euclidean_recall |
|
value: 76.06837606837607 |
|
- type: manhattan_accuracy |
|
value: 73.80758255197716 |
|
- type: manhattan_ap |
|
value: 55.42477275597209 |
|
- type: manhattan_f1 |
|
value: 57.55860953920776 |
|
- type: manhattan_precision |
|
value: 46.29388816644994 |
|
- type: manhattan_recall |
|
value: 76.06837606837607 |
|
- type: max_accuracy |
|
value: 73.84834896045659 |
|
- type: max_ap |
|
value: 55.484124732566606 |
|
- type: max_f1 |
|
value: 57.55860953920776 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sickr-sts |
|
name: MTEB SICK-R |
|
config: default |
|
split: test |
|
revision: a6ea5a8cab320b040a23452cc28066d9beae2cee |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 67.03943120783973 |
|
- type: cos_sim_spearman |
|
value: 62.93971145260584 |
|
- type: euclidean_pearson |
|
value: 64.13947263916926 |
|
- type: euclidean_spearman |
|
value: 62.93972324235839 |
|
- type: manhattan_pearson |
|
value: 64.11295322654566 |
|
- type: manhattan_spearman |
|
value: 62.92816122293202 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: PL-MTEB/sickr-pl-sts |
|
name: MTEB SICK-R-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 67.75034167381077 |
|
- type: cos_sim_spearman |
|
value: 62.98158872758643 |
|
- type: euclidean_pearson |
|
value: 64.25794794439082 |
|
- type: euclidean_spearman |
|
value: 62.981566596223125 |
|
- type: manhattan_pearson |
|
value: 64.25439446502435 |
|
- type: manhattan_spearman |
|
value: 63.01301439900365 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts12-sts |
|
name: MTEB STS12 |
|
config: default |
|
split: test |
|
revision: a0d554a64d88156834ff5ae9920b964011b16384 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 61.622204530882755 |
|
- type: cos_sim_spearman |
|
value: 65.4632047656541 |
|
- type: euclidean_pearson |
|
value: 59.21529585527598 |
|
- type: euclidean_spearman |
|
value: 65.4638163967956 |
|
- type: manhattan_pearson |
|
value: 59.39341472707122 |
|
- type: manhattan_spearman |
|
value: 65.57635757250173 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts13-sts |
|
name: MTEB STS13 |
|
config: default |
|
split: test |
|
revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 60.329743331971486 |
|
- type: cos_sim_spearman |
|
value: 62.78607195958339 |
|
- type: euclidean_pearson |
|
value: 62.07415212138581 |
|
- type: euclidean_spearman |
|
value: 62.78618151904129 |
|
- type: manhattan_pearson |
|
value: 62.41250554765521 |
|
- type: manhattan_spearman |
|
value: 62.87580558029627 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts14-sts |
|
name: MTEB STS14 |
|
config: default |
|
split: test |
|
revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 59.16277512775291 |
|
- type: cos_sim_spearman |
|
value: 57.53693422381856 |
|
- type: euclidean_pearson |
|
value: 57.85017283427473 |
|
- type: euclidean_spearman |
|
value: 57.53697385589326 |
|
- type: manhattan_pearson |
|
value: 58.049796184955596 |
|
- type: manhattan_spearman |
|
value: 57.76174789162225 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts15-sts |
|
name: MTEB STS15 |
|
config: default |
|
split: test |
|
revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 74.42588553600197 |
|
- type: cos_sim_spearman |
|
value: 74.25087788257943 |
|
- type: euclidean_pearson |
|
value: 73.35436018935222 |
|
- type: euclidean_spearman |
|
value: 74.25087694991477 |
|
- type: manhattan_pearson |
|
value: 73.33747415771185 |
|
- type: manhattan_spearman |
|
value: 74.21504509447377 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts16-sts |
|
name: MTEB STS16 |
|
config: default |
|
split: test |
|
revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 75.77242432372144 |
|
- type: cos_sim_spearman |
|
value: 75.72930700521489 |
|
- type: euclidean_pearson |
|
value: 75.6995220623788 |
|
- type: euclidean_spearman |
|
value: 75.72930646047212 |
|
- type: manhattan_pearson |
|
value: 75.65841087952896 |
|
- type: manhattan_spearman |
|
value: 75.69567692328437 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (ko-ko) |
|
config: ko-ko |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 66.2495297342053 |
|
- type: cos_sim_spearman |
|
value: 66.14124319602982 |
|
- type: euclidean_pearson |
|
value: 66.49498096178358 |
|
- type: euclidean_spearman |
|
value: 66.14121792287747 |
|
- type: manhattan_pearson |
|
value: 66.51560623835172 |
|
- type: manhattan_spearman |
|
value: 66.05794413582558 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (ar-ar) |
|
config: ar-ar |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 75.0045186560239 |
|
- type: cos_sim_spearman |
|
value: 74.96504390762252 |
|
- type: euclidean_pearson |
|
value: 74.20988464347049 |
|
- type: euclidean_spearman |
|
value: 74.98114602301776 |
|
- type: manhattan_pearson |
|
value: 74.37929169860529 |
|
- type: manhattan_spearman |
|
value: 75.37049827509504 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-ar) |
|
config: en-ar |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 73.88478151514396 |
|
- type: cos_sim_spearman |
|
value: 74.05322141272103 |
|
- type: euclidean_pearson |
|
value: 73.52175483343693 |
|
- type: euclidean_spearman |
|
value: 74.05322141272103 |
|
- type: manhattan_pearson |
|
value: 73.35875118828287 |
|
- type: manhattan_spearman |
|
value: 73.83972625384673 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-de) |
|
config: en-de |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 75.57014781622605 |
|
- type: cos_sim_spearman |
|
value: 74.95329129562734 |
|
- type: euclidean_pearson |
|
value: 75.5667786729257 |
|
- type: euclidean_spearman |
|
value: 74.95329129562734 |
|
- type: manhattan_pearson |
|
value: 75.39548673816147 |
|
- type: manhattan_spearman |
|
value: 74.89428642503749 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-en) |
|
config: en-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 80.04007129652777 |
|
- type: cos_sim_spearman |
|
value: 79.94429611477106 |
|
- type: euclidean_pearson |
|
value: 79.91583070858822 |
|
- type: euclidean_spearman |
|
value: 79.94429611477106 |
|
- type: manhattan_pearson |
|
value: 80.14382273152769 |
|
- type: manhattan_spearman |
|
value: 80.23862855392836 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-tr) |
|
config: en-tr |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 77.28740870194635 |
|
- type: cos_sim_spearman |
|
value: 77.18286391819586 |
|
- type: euclidean_pearson |
|
value: 77.05644328687119 |
|
- type: euclidean_spearman |
|
value: 77.18286391819586 |
|
- type: manhattan_pearson |
|
value: 77.15625898067294 |
|
- type: manhattan_spearman |
|
value: 77.03165154316278 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (es-en) |
|
config: es-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 72.99293002371301 |
|
- type: cos_sim_spearman |
|
value: 72.24657859872468 |
|
- type: euclidean_pearson |
|
value: 73.38839879755461 |
|
- type: euclidean_spearman |
|
value: 72.24657859872468 |
|
- type: manhattan_pearson |
|
value: 73.6627728800822 |
|
- type: manhattan_spearman |
|
value: 72.70893449698669 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (es-es) |
|
config: es-es |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 81.37213723705916 |
|
- type: cos_sim_spearman |
|
value: 80.64548512701263 |
|
- type: euclidean_pearson |
|
value: 80.94992193351284 |
|
- type: euclidean_spearman |
|
value: 80.64484963200427 |
|
- type: manhattan_pearson |
|
value: 80.92246813841794 |
|
- type: manhattan_spearman |
|
value: 80.68860823161657 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (fr-en) |
|
config: fr-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 77.54059604962391 |
|
- type: cos_sim_spearman |
|
value: 77.19559169700682 |
|
- type: euclidean_pearson |
|
value: 77.32739821317861 |
|
- type: euclidean_spearman |
|
value: 77.19559169700682 |
|
- type: manhattan_pearson |
|
value: 77.29224328831437 |
|
- type: manhattan_spearman |
|
value: 77.24394878313191 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (it-en) |
|
config: it-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 79.06397062195414 |
|
- type: cos_sim_spearman |
|
value: 78.66694637555244 |
|
- type: euclidean_pearson |
|
value: 79.34923290885872 |
|
- type: euclidean_spearman |
|
value: 78.66694637555244 |
|
- type: manhattan_pearson |
|
value: 79.50802161625809 |
|
- type: manhattan_spearman |
|
value: 78.79195213396169 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (nl-en) |
|
config: nl-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 78.66045829245238 |
|
- type: cos_sim_spearman |
|
value: 78.14055373851183 |
|
- type: euclidean_pearson |
|
value: 78.94489279300518 |
|
- type: euclidean_spearman |
|
value: 78.14055373851183 |
|
- type: manhattan_pearson |
|
value: 79.33473165536323 |
|
- type: manhattan_spearman |
|
value: 78.5783429705299 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (en) |
|
config: en |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 36.63454535818336 |
|
- type: cos_sim_spearman |
|
value: 47.12016162570126 |
|
- type: euclidean_pearson |
|
value: 39.07268779927362 |
|
- type: euclidean_spearman |
|
value: 47.12016162570126 |
|
- type: manhattan_pearson |
|
value: 41.723119770725944 |
|
- type: manhattan_spearman |
|
value: 47.90334362422989 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (de) |
|
config: de |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 13.325547358617957 |
|
- type: cos_sim_spearman |
|
value: 24.094051740693416 |
|
- type: euclidean_pearson |
|
value: 10.39110006005262 |
|
- type: euclidean_spearman |
|
value: 24.094051740693416 |
|
- type: manhattan_pearson |
|
value: 12.4380555005162 |
|
- type: manhattan_spearman |
|
value: 25.176800279885715 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (es) |
|
config: es |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 41.21281570342249 |
|
- type: cos_sim_spearman |
|
value: 55.397885077207974 |
|
- type: euclidean_pearson |
|
value: 43.96150945976646 |
|
- type: euclidean_spearman |
|
value: 55.397885077207974 |
|
- type: manhattan_pearson |
|
value: 49.58812224529121 |
|
- type: manhattan_spearman |
|
value: 55.35874879475974 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (pl) |
|
config: pl |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 5.985012243744998 |
|
- type: cos_sim_spearman |
|
value: 25.307464943919012 |
|
- type: euclidean_pearson |
|
value: -4.080537702499046 |
|
- type: euclidean_spearman |
|
value: 25.307464943919012 |
|
- type: manhattan_pearson |
|
value: -2.5058642304196543 |
|
- type: manhattan_spearman |
|
value: 26.751588484373233 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (tr) |
|
config: tr |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 34.44666578772084 |
|
- type: cos_sim_spearman |
|
value: 46.45977141800899 |
|
- type: euclidean_pearson |
|
value: 38.78305544036559 |
|
- type: euclidean_spearman |
|
value: 46.45977141800899 |
|
- type: manhattan_pearson |
|
value: 46.45101297876112 |
|
- type: manhattan_spearman |
|
value: 50.642972694093814 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (ar) |
|
config: ar |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 28.095327083873055 |
|
- type: cos_sim_spearman |
|
value: 40.24741745875892 |
|
- type: euclidean_pearson |
|
value: 29.141496784653892 |
|
- type: euclidean_spearman |
|
value: 40.24741745875892 |
|
- type: manhattan_pearson |
|
value: 32.013290716034064 |
|
- type: manhattan_spearman |
|
value: 40.85454084311211 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (ru) |
|
config: ru |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 27.46788309503312 |
|
- type: cos_sim_spearman |
|
value: 43.57385391855994 |
|
- type: euclidean_pearson |
|
value: 24.558349674326177 |
|
- type: euclidean_spearman |
|
value: 43.57385391855994 |
|
- type: manhattan_pearson |
|
value: 28.974505207055866 |
|
- type: manhattan_spearman |
|
value: 44.111553205713 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (zh) |
|
config: zh |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 34.87841073990563 |
|
- type: cos_sim_spearman |
|
value: 52.8221686505807 |
|
- type: euclidean_pearson |
|
value: 38.36114580544504 |
|
- type: euclidean_spearman |
|
value: 52.8221686505807 |
|
- type: manhattan_pearson |
|
value: 46.69329448756753 |
|
- type: manhattan_spearman |
|
value: 53.9140781097337 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (fr) |
|
config: fr |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 49.999267528357 |
|
- type: cos_sim_spearman |
|
value: 61.71837669697145 |
|
- type: euclidean_pearson |
|
value: 53.578476744372274 |
|
- type: euclidean_spearman |
|
value: 61.71837669697145 |
|
- type: manhattan_pearson |
|
value: 56.410294227490795 |
|
- type: manhattan_spearman |
|
value: 60.684457655864875 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (de-en) |
|
config: de-en |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 22.43564137760586 |
|
- type: cos_sim_spearman |
|
value: 34.28346144104183 |
|
- type: euclidean_pearson |
|
value: 27.41326011184764 |
|
- type: euclidean_spearman |
|
value: 34.28346144104183 |
|
- type: manhattan_pearson |
|
value: 35.62923154232163 |
|
- type: manhattan_spearman |
|
value: 37.937151135297185 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (es-en) |
|
config: es-en |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 44.34071611983998 |
|
- type: cos_sim_spearman |
|
value: 57.823185616169646 |
|
- type: euclidean_pearson |
|
value: 49.29310650157244 |
|
- type: euclidean_spearman |
|
value: 57.823185616169646 |
|
- type: manhattan_pearson |
|
value: 55.93298736518848 |
|
- type: manhattan_spearman |
|
value: 58.57556581684834 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (it) |
|
config: it |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 56.07027840344927 |
|
- type: cos_sim_spearman |
|
value: 62.20158260763411 |
|
- type: euclidean_pearson |
|
value: 55.887969718543616 |
|
- type: euclidean_spearman |
|
value: 62.20158260763411 |
|
- type: manhattan_pearson |
|
value: 56.081533365738444 |
|
- type: manhattan_spearman |
|
value: 62.018651361750685 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (pl-en) |
|
config: pl-en |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 41.41816324477061 |
|
- type: cos_sim_spearman |
|
value: 44.71684955996943 |
|
- type: euclidean_pearson |
|
value: 42.74585025834968 |
|
- type: euclidean_spearman |
|
value: 44.71684955996943 |
|
- type: manhattan_pearson |
|
value: 47.992481632815256 |
|
- type: manhattan_spearman |
|
value: 46.18587933349126 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (zh-en) |
|
config: zh-en |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 38.89140730917917 |
|
- type: cos_sim_spearman |
|
value: 49.18633779347391 |
|
- type: euclidean_pearson |
|
value: 43.27605428753535 |
|
- type: euclidean_spearman |
|
value: 49.18633779347391 |
|
- type: manhattan_pearson |
|
value: 48.22046568809415 |
|
- type: manhattan_spearman |
|
value: 49.248416391249464 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (es-it) |
|
config: es-it |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 40.31620568726327 |
|
- type: cos_sim_spearman |
|
value: 49.13034440774138 |
|
- type: euclidean_pearson |
|
value: 43.95169508285692 |
|
- type: euclidean_spearman |
|
value: 49.13034440774138 |
|
- type: manhattan_pearson |
|
value: 48.84250981398146 |
|
- type: manhattan_spearman |
|
value: 49.54216339903405 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (de-fr) |
|
config: de-fr |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 27.074582378144058 |
|
- type: cos_sim_spearman |
|
value: 41.29498619968451 |
|
- type: euclidean_pearson |
|
value: 28.993986097276505 |
|
- type: euclidean_spearman |
|
value: 41.29498619968451 |
|
- type: manhattan_pearson |
|
value: 32.079813951133254 |
|
- type: manhattan_spearman |
|
value: 43.664111732941464 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (de-pl) |
|
config: de-pl |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 6.864334110072116 |
|
- type: cos_sim_spearman |
|
value: 25.805458732687914 |
|
- type: euclidean_pearson |
|
value: 11.435920047618103 |
|
- type: euclidean_spearman |
|
value: 25.805458732687914 |
|
- type: manhattan_pearson |
|
value: 15.036308569899552 |
|
- type: manhattan_spearman |
|
value: 25.405135387192757 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (fr-pl) |
|
config: fr-pl |
|
split: test |
|
revision: eea2b4fe26a775864c896887d910b76a8098ad3f |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 65.44029549925597 |
|
- type: cos_sim_spearman |
|
value: 61.97797868009122 |
|
- type: euclidean_pearson |
|
value: 65.92740669959876 |
|
- type: euclidean_spearman |
|
value: 61.97797868009122 |
|
- type: manhattan_pearson |
|
value: 70.29575044091207 |
|
- type: manhattan_spearman |
|
value: 73.24670207647144 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/STSB |
|
name: MTEB STSB |
|
config: default |
|
split: test |
|
revision: 0cde68302b3541bb8b3c340dc0644b0b745b3dc0 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 51.35413149349556 |
|
- type: cos_sim_spearman |
|
value: 50.175051356729924 |
|
- type: euclidean_pearson |
|
value: 53.12039152785364 |
|
- type: euclidean_spearman |
|
value: 50.174289111089685 |
|
- type: manhattan_pearson |
|
value: 53.0731746793555 |
|
- type: manhattan_spearman |
|
value: 50.15176393928403 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/stsbenchmark-sts |
|
name: MTEB STSBenchmark |
|
config: default |
|
split: test |
|
revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 67.84222983023291 |
|
- type: cos_sim_spearman |
|
value: 67.39086924655895 |
|
- type: euclidean_pearson |
|
value: 67.3393327127967 |
|
- type: euclidean_spearman |
|
value: 67.39088047106472 |
|
- type: manhattan_pearson |
|
value: 67.40316731822271 |
|
- type: manhattan_spearman |
|
value: 67.49067800994015 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: ScandEval/scala-da |
|
name: MTEB ScalaDaClassification |
|
config: default |
|
split: test |
|
revision: 1de08520a7b361e92ffa2a2201ebd41942c54675 |
|
metrics: |
|
- type: accuracy |
|
value: 50.62988281250001 |
|
- type: ap |
|
value: 50.32274824114816 |
|
- type: f1 |
|
value: 50.37741703766756 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: ScandEval/scala-nb |
|
name: MTEB ScalaNbClassification |
|
config: default |
|
split: test |
|
revision: 237111a078ad5a834a55c57803d40bbe410ed03b |
|
metrics: |
|
- type: accuracy |
|
value: 51.181640625 |
|
- type: ap |
|
value: 50.60884394099696 |
|
- type: f1 |
|
value: 50.866988720930415 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: ScandEval/scala-nn |
|
name: MTEB ScalaNnClassification |
|
config: default |
|
split: test |
|
revision: 9d9a2a4092ed3cacf0744592f6d2f32ab8ef4c0b |
|
metrics: |
|
- type: accuracy |
|
value: 50.9375 |
|
- type: ap |
|
value: 50.47969135089731 |
|
- type: f1 |
|
value: 50.62913552324756 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: ScandEval/scala-sv |
|
name: MTEB ScalaSvClassification |
|
config: default |
|
split: test |
|
revision: 1b48e3dcb02872335ff985ff938a054a4ed99008 |
|
metrics: |
|
- type: accuracy |
|
value: 51.1474609375 |
|
- type: ap |
|
value: 50.5894187272385 |
|
- type: f1 |
|
value: 50.901812392367916 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/scidocs-reranking |
|
name: MTEB SciDocsRR |
|
config: default |
|
split: test |
|
revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
|
metrics: |
|
- type: map |
|
value: 68.36051662289248 |
|
- type: mrr |
|
value: 89.39224265204656 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scifact |
|
name: MTEB SciFact |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.721999999999998 |
|
- type: map_at_10 |
|
value: 31.335 |
|
- type: map_at_100 |
|
value: 32.461 |
|
- type: map_at_1000 |
|
value: 32.557 |
|
- type: map_at_3 |
|
value: 29.282000000000004 |
|
- type: map_at_5 |
|
value: 30.602 |
|
- type: mrr_at_1 |
|
value: 24.667 |
|
- type: mrr_at_10 |
|
value: 32.363 |
|
- type: mrr_at_100 |
|
value: 33.421 |
|
- type: mrr_at_1000 |
|
value: 33.499 |
|
- type: mrr_at_3 |
|
value: 30.444 |
|
- type: mrr_at_5 |
|
value: 31.628 |
|
- type: ndcg_at_1 |
|
value: 24.667 |
|
- type: ndcg_at_10 |
|
value: 35.29 |
|
- type: ndcg_at_100 |
|
value: 40.665 |
|
- type: ndcg_at_1000 |
|
value: 43.241 |
|
- type: ndcg_at_3 |
|
value: 31.238 |
|
- type: ndcg_at_5 |
|
value: 33.486 |
|
- type: precision_at_1 |
|
value: 24.667 |
|
- type: precision_at_10 |
|
value: 5.1 |
|
- type: precision_at_100 |
|
value: 0.7969999999999999 |
|
- type: precision_at_1000 |
|
value: 0.10300000000000001 |
|
- type: precision_at_3 |
|
value: 12.667 |
|
- type: precision_at_5 |
|
value: 8.933 |
|
- type: recall_at_1 |
|
value: 23.721999999999998 |
|
- type: recall_at_10 |
|
value: 46.417 |
|
- type: recall_at_100 |
|
value: 70.944 |
|
- type: recall_at_1000 |
|
value: 91.033 |
|
- type: recall_at_3 |
|
value: 35.693999999999996 |
|
- type: recall_at_5 |
|
value: 40.944 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scifact-pl |
|
name: MTEB SciFact-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 21.706 |
|
- type: map_at_10 |
|
value: 28.333000000000002 |
|
- type: map_at_100 |
|
value: 29.364 |
|
- type: map_at_1000 |
|
value: 29.451 |
|
- type: map_at_3 |
|
value: 26.112999999999996 |
|
- type: map_at_5 |
|
value: 27.502 |
|
- type: mrr_at_1 |
|
value: 23.0 |
|
- type: mrr_at_10 |
|
value: 29.555999999999997 |
|
- type: mrr_at_100 |
|
value: 30.536 |
|
- type: mrr_at_1000 |
|
value: 30.606 |
|
- type: mrr_at_3 |
|
value: 27.333000000000002 |
|
- type: mrr_at_5 |
|
value: 28.717 |
|
- type: ndcg_at_1 |
|
value: 23.0 |
|
- type: ndcg_at_10 |
|
value: 32.238 |
|
- type: ndcg_at_100 |
|
value: 37.785999999999994 |
|
- type: ndcg_at_1000 |
|
value: 40.266999999999996 |
|
- type: ndcg_at_3 |
|
value: 27.961000000000002 |
|
- type: ndcg_at_5 |
|
value: 30.322 |
|
- type: precision_at_1 |
|
value: 23.0 |
|
- type: precision_at_10 |
|
value: 4.7669999999999995 |
|
- type: precision_at_100 |
|
value: 0.787 |
|
- type: precision_at_1000 |
|
value: 0.10200000000000001 |
|
- type: precision_at_3 |
|
value: 11.444 |
|
- type: precision_at_5 |
|
value: 8.200000000000001 |
|
- type: recall_at_1 |
|
value: 21.706 |
|
- type: recall_at_10 |
|
value: 43.206 |
|
- type: recall_at_100 |
|
value: 69.678 |
|
- type: recall_at_1000 |
|
value: 89.333 |
|
- type: recall_at_3 |
|
value: 31.900000000000002 |
|
- type: recall_at_5 |
|
value: 37.594 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/sprintduplicatequestions-pairclassification |
|
name: MTEB SprintDuplicateQuestions |
|
config: default |
|
split: test |
|
revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 99.5 |
|
- type: cos_sim_ap |
|
value: 77.07584309978081 |
|
- type: cos_sim_f1 |
|
value: 71.8864950078823 |
|
- type: cos_sim_precision |
|
value: 75.74750830564784 |
|
- type: cos_sim_recall |
|
value: 68.4 |
|
- type: dot_accuracy |
|
value: 99.5 |
|
- type: dot_ap |
|
value: 77.07584309978081 |
|
- type: dot_f1 |
|
value: 71.8864950078823 |
|
- type: dot_precision |
|
value: 75.74750830564784 |
|
- type: dot_recall |
|
value: 68.4 |
|
- type: euclidean_accuracy |
|
value: 99.5 |
|
- type: euclidean_ap |
|
value: 77.07584309978081 |
|
- type: euclidean_f1 |
|
value: 71.8864950078823 |
|
- type: euclidean_precision |
|
value: 75.74750830564784 |
|
- type: euclidean_recall |
|
value: 68.4 |
|
- type: manhattan_accuracy |
|
value: 99.50594059405941 |
|
- type: manhattan_ap |
|
value: 77.41658577240027 |
|
- type: manhattan_f1 |
|
value: 71.91374663072777 |
|
- type: manhattan_precision |
|
value: 78.01169590643275 |
|
- type: manhattan_recall |
|
value: 66.7 |
|
- type: max_accuracy |
|
value: 99.50594059405941 |
|
- type: max_ap |
|
value: 77.41658577240027 |
|
- type: max_f1 |
|
value: 71.91374663072777 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering |
|
name: MTEB StackExchangeClustering |
|
config: default |
|
split: test |
|
revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
|
metrics: |
|
- type: v_measure |
|
value: 46.32521494308228 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering-p2p |
|
name: MTEB StackExchangeClusteringP2P |
|
config: default |
|
split: test |
|
revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
|
metrics: |
|
- type: v_measure |
|
value: 20.573273825125266 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/stackoverflowdupquestions-reranking |
|
name: MTEB StackOverflowDupQuestions |
|
config: default |
|
split: test |
|
revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
|
metrics: |
|
- type: map |
|
value: 38.612724125942385 |
|
- type: mrr |
|
value: 38.891130315762666 |
|
- task: |
|
type: Summarization |
|
dataset: |
|
type: mteb/summeval |
|
name: MTEB SummEval |
|
config: default |
|
split: test |
|
revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 29.305330424238836 |
|
- type: cos_sim_spearman |
|
value: 30.556621737388685 |
|
- type: dot_pearson |
|
value: 29.30533056265583 |
|
- type: dot_spearman |
|
value: 30.556621737388685 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: ScandEval/swerec-mini |
|
name: MTEB SweRecClassification |
|
config: default |
|
split: test |
|
revision: 3c62f26bafdc4c4e1c16401ad4b32f0a94b46612 |
|
metrics: |
|
- type: accuracy |
|
value: 68.4716796875 |
|
- type: f1 |
|
value: 59.865730786092364 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/T2Reranking |
|
name: MTEB T2Reranking |
|
config: default |
|
split: dev |
|
revision: 76631901a18387f85eaa53e5450019b87ad58ef9 |
|
metrics: |
|
- type: map |
|
value: 55.34794621490011 |
|
- type: mrr |
|
value: 59.22764129348421 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/T2Retrieval |
|
name: MTEB T2Retrieval |
|
config: default |
|
split: dev |
|
revision: 8731a845f1bf500a4f111cf1070785c793d10e64 |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.586 |
|
- type: map_at_10 |
|
value: 0.819 |
|
- type: map_at_100 |
|
value: 0.8920000000000001 |
|
- type: map_at_1000 |
|
value: 0.928 |
|
- type: map_at_3 |
|
value: 0.729 |
|
- type: map_at_5 |
|
value: 0.771 |
|
- type: mrr_at_1 |
|
value: 1.9949999999999999 |
|
- type: mrr_at_10 |
|
value: 2.608 |
|
- type: mrr_at_100 |
|
value: 2.771 |
|
- type: mrr_at_1000 |
|
value: 2.8289999999999997 |
|
- type: mrr_at_3 |
|
value: 2.365 |
|
- type: mrr_at_5 |
|
value: 2.483 |
|
- type: ndcg_at_1 |
|
value: 1.9949999999999999 |
|
- type: ndcg_at_10 |
|
value: 1.314 |
|
- type: ndcg_at_100 |
|
value: 1.831 |
|
- type: ndcg_at_1000 |
|
value: 3.4139999999999997 |
|
- type: ndcg_at_3 |
|
value: 1.377 |
|
- type: ndcg_at_5 |
|
value: 1.2630000000000001 |
|
- type: precision_at_1 |
|
value: 1.9949999999999999 |
|
- type: precision_at_10 |
|
value: 0.488 |
|
- type: precision_at_100 |
|
value: 0.123 |
|
- type: precision_at_1000 |
|
value: 0.054 |
|
- type: precision_at_3 |
|
value: 1.027 |
|
- type: precision_at_5 |
|
value: 0.737 |
|
- type: recall_at_1 |
|
value: 0.586 |
|
- type: recall_at_10 |
|
value: 1.3390000000000002 |
|
- type: recall_at_100 |
|
value: 3.15 |
|
- type: recall_at_1000 |
|
value: 11.859 |
|
- type: recall_at_3 |
|
value: 0.8710000000000001 |
|
- type: recall_at_5 |
|
value: 1.0290000000000001 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/TNews-classification |
|
name: MTEB TNews |
|
config: default |
|
split: validation |
|
revision: 317f262bf1e6126357bbe89e875451e4b0938fe4 |
|
metrics: |
|
- type: accuracy |
|
value: 40.946 |
|
- type: f1 |
|
value: 39.56517169731474 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: trec-covid |
|
name: MTEB TRECCOVID |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.08499999999999999 |
|
- type: map_at_10 |
|
value: 0.462 |
|
- type: map_at_100 |
|
value: 0.893 |
|
- type: map_at_1000 |
|
value: 1.129 |
|
- type: map_at_3 |
|
value: 0.232 |
|
- type: map_at_5 |
|
value: 0.3 |
|
- type: mrr_at_1 |
|
value: 38.0 |
|
- type: mrr_at_10 |
|
value: 50.629999999999995 |
|
- type: mrr_at_100 |
|
value: 51.315999999999995 |
|
- type: mrr_at_1000 |
|
value: 51.365 |
|
- type: mrr_at_3 |
|
value: 47.0 |
|
- type: mrr_at_5 |
|
value: 48.9 |
|
- type: ndcg_at_1 |
|
value: 31.0 |
|
- type: ndcg_at_10 |
|
value: 24.823 |
|
- type: ndcg_at_100 |
|
value: 10.583 |
|
- type: ndcg_at_1000 |
|
value: 6.497999999999999 |
|
- type: ndcg_at_3 |
|
value: 30.95 |
|
- type: ndcg_at_5 |
|
value: 27.899 |
|
- type: precision_at_1 |
|
value: 38.0 |
|
- type: precision_at_10 |
|
value: 25.6 |
|
- type: precision_at_100 |
|
value: 8.98 |
|
- type: precision_at_1000 |
|
value: 2.248 |
|
- type: precision_at_3 |
|
value: 34.666999999999994 |
|
- type: precision_at_5 |
|
value: 29.599999999999998 |
|
- type: recall_at_1 |
|
value: 0.08499999999999999 |
|
- type: recall_at_10 |
|
value: 0.641 |
|
- type: recall_at_100 |
|
value: 2.002 |
|
- type: recall_at_1000 |
|
value: 4.902 |
|
- type: recall_at_3 |
|
value: 0.28200000000000003 |
|
- type: recall_at_5 |
|
value: 0.379 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: trec-covid-pl |
|
name: MTEB TRECCOVID-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.124 |
|
- type: map_at_10 |
|
value: 0.45199999999999996 |
|
- type: map_at_100 |
|
value: 0.874 |
|
- type: map_at_1000 |
|
value: 1.1039999999999999 |
|
- type: map_at_3 |
|
value: 0.253 |
|
- type: map_at_5 |
|
value: 0.32299999999999995 |
|
- type: mrr_at_1 |
|
value: 36.0 |
|
- type: mrr_at_10 |
|
value: 47.56 |
|
- type: mrr_at_100 |
|
value: 48.532 |
|
- type: mrr_at_1000 |
|
value: 48.579 |
|
- type: mrr_at_3 |
|
value: 45.0 |
|
- type: mrr_at_5 |
|
value: 45.5 |
|
- type: ndcg_at_1 |
|
value: 34.0 |
|
- type: ndcg_at_10 |
|
value: 24.529 |
|
- type: ndcg_at_100 |
|
value: 10.427 |
|
- type: ndcg_at_1000 |
|
value: 6.457 |
|
- type: ndcg_at_3 |
|
value: 31.173000000000002 |
|
- type: ndcg_at_5 |
|
value: 27.738000000000003 |
|
- type: precision_at_1 |
|
value: 38.0 |
|
- type: precision_at_10 |
|
value: 25.4 |
|
- type: precision_at_100 |
|
value: 8.88 |
|
- type: precision_at_1000 |
|
value: 2.2159999999999997 |
|
- type: precision_at_3 |
|
value: 34.666999999999994 |
|
- type: precision_at_5 |
|
value: 29.2 |
|
- type: recall_at_1 |
|
value: 0.124 |
|
- type: recall_at_10 |
|
value: 0.618 |
|
- type: recall_at_100 |
|
value: 1.9349999999999998 |
|
- type: recall_at_1000 |
|
value: 4.808 |
|
- type: recall_at_3 |
|
value: 0.28300000000000003 |
|
- type: recall_at_5 |
|
value: 0.382 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (sqi-eng) |
|
config: sqi-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.9 |
|
- type: f1 |
|
value: 98.55000000000001 |
|
- type: precision |
|
value: 98.38333333333334 |
|
- type: recall |
|
value: 98.9 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (fry-eng) |
|
config: fry-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 65.3179190751445 |
|
- type: f1 |
|
value: 59.44582071749702 |
|
- type: precision |
|
value: 57.49678869621066 |
|
- type: recall |
|
value: 65.3179190751445 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (kur-eng) |
|
config: kur-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 38.53658536585366 |
|
- type: f1 |
|
value: 34.217555952803785 |
|
- type: precision |
|
value: 32.96511296649355 |
|
- type: recall |
|
value: 38.53658536585366 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (tur-eng) |
|
config: tur-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.7 |
|
- type: f1 |
|
value: 98.26666666666665 |
|
- type: precision |
|
value: 98.05 |
|
- type: recall |
|
value: 98.7 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (deu-eng) |
|
config: deu-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 99.3 |
|
- type: f1 |
|
value: 99.13333333333333 |
|
- type: precision |
|
value: 99.05000000000001 |
|
- type: recall |
|
value: 99.3 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (nld-eng) |
|
config: nld-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.89999999999999 |
|
- type: f1 |
|
value: 97.2 |
|
- type: precision |
|
value: 96.85000000000001 |
|
- type: recall |
|
value: 97.89999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ron-eng) |
|
config: ron-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.2 |
|
- type: f1 |
|
value: 97.6 |
|
- type: precision |
|
value: 97.3 |
|
- type: recall |
|
value: 98.2 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ang-eng) |
|
config: ang-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 52.23880597014925 |
|
- type: f1 |
|
value: 46.340992406389105 |
|
- type: precision |
|
value: 44.556384742951906 |
|
- type: recall |
|
value: 52.23880597014925 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ido-eng) |
|
config: ido-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.0 |
|
- type: f1 |
|
value: 93.67000000000002 |
|
- type: precision |
|
value: 93.075 |
|
- type: recall |
|
value: 95.0 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (jav-eng) |
|
config: jav-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 88.29268292682927 |
|
- type: f1 |
|
value: 85.76422764227642 |
|
- type: precision |
|
value: 84.84204413472706 |
|
- type: recall |
|
value: 88.29268292682927 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (isl-eng) |
|
config: isl-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.2 |
|
- type: f1 |
|
value: 96.46666666666667 |
|
- type: precision |
|
value: 96.1 |
|
- type: recall |
|
value: 97.2 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (slv-eng) |
|
config: slv-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.8408262454435 |
|
- type: f1 |
|
value: 95.9902794653706 |
|
- type: precision |
|
value: 95.56500607533415 |
|
- type: recall |
|
value: 96.8408262454435 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (cym-eng) |
|
config: cym-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 93.3913043478261 |
|
- type: f1 |
|
value: 91.30434782608695 |
|
- type: precision |
|
value: 90.28985507246377 |
|
- type: recall |
|
value: 93.3913043478261 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (kaz-eng) |
|
config: kaz-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 90.6086956521739 |
|
- type: f1 |
|
value: 88.1159420289855 |
|
- type: precision |
|
value: 86.9623188405797 |
|
- type: recall |
|
value: 90.6086956521739 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (est-eng) |
|
config: est-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.8 |
|
- type: f1 |
|
value: 97.16666666666667 |
|
- type: precision |
|
value: 96.86666666666667 |
|
- type: recall |
|
value: 97.8 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (heb-eng) |
|
config: heb-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 94.0 |
|
- type: f1 |
|
value: 92.34 |
|
- type: precision |
|
value: 91.54166666666667 |
|
- type: recall |
|
value: 94.0 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (gla-eng) |
|
config: gla-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 84.92159227985525 |
|
- type: f1 |
|
value: 80.8868975817106 |
|
- type: precision |
|
value: 79.11540008041817 |
|
- type: recall |
|
value: 84.92159227985525 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (mar-eng) |
|
config: mar-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 94.89999999999999 |
|
- type: f1 |
|
value: 93.35 |
|
- type: precision |
|
value: 92.58333333333334 |
|
- type: recall |
|
value: 94.89999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (lat-eng) |
|
config: lat-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 43.3 |
|
- type: f1 |
|
value: 36.64473116255726 |
|
- type: precision |
|
value: 34.64017752457381 |
|
- type: recall |
|
value: 43.3 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (bel-eng) |
|
config: bel-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.7 |
|
- type: f1 |
|
value: 95.68333333333332 |
|
- type: precision |
|
value: 95.19999999999999 |
|
- type: recall |
|
value: 96.7 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (pms-eng) |
|
config: pms-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 70.47619047619048 |
|
- type: f1 |
|
value: 66.63032734461306 |
|
- type: precision |
|
value: 65.46459191863879 |
|
- type: recall |
|
value: 70.47619047619048 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (gle-eng) |
|
config: gle-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 93.5 |
|
- type: f1 |
|
value: 91.63 |
|
- type: precision |
|
value: 90.75 |
|
- type: recall |
|
value: 93.5 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (pes-eng) |
|
config: pes-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.5 |
|
- type: f1 |
|
value: 94.36666666666666 |
|
- type: precision |
|
value: 93.83333333333333 |
|
- type: recall |
|
value: 95.5 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (nob-eng) |
|
config: nob-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 99.3 |
|
- type: f1 |
|
value: 99.06666666666666 |
|
- type: precision |
|
value: 98.95 |
|
- type: recall |
|
value: 99.3 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (bul-eng) |
|
config: bul-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.8 |
|
- type: f1 |
|
value: 94.51666666666667 |
|
- type: precision |
|
value: 93.88333333333334 |
|
- type: recall |
|
value: 95.8 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (cbk-eng) |
|
config: cbk-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 84.0 |
|
- type: f1 |
|
value: 80.46675324675326 |
|
- type: precision |
|
value: 78.95999999999998 |
|
- type: recall |
|
value: 84.0 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (hun-eng) |
|
config: hun-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.7 |
|
- type: f1 |
|
value: 96.93333333333332 |
|
- type: precision |
|
value: 96.55 |
|
- type: recall |
|
value: 97.7 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (uig-eng) |
|
config: uig-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 92.10000000000001 |
|
- type: f1 |
|
value: 90.07333333333334 |
|
- type: precision |
|
value: 89.16166666666668 |
|
- type: recall |
|
value: 92.10000000000001 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (rus-eng) |
|
config: rus-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.6 |
|
- type: f1 |
|
value: 94.35 |
|
- type: precision |
|
value: 93.75 |
|
- type: recall |
|
value: 95.6 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (spa-eng) |
|
config: spa-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.9 |
|
- type: f1 |
|
value: 98.53333333333335 |
|
- type: precision |
|
value: 98.35000000000001 |
|
- type: recall |
|
value: 98.9 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (hye-eng) |
|
config: hye-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.22641509433963 |
|
- type: f1 |
|
value: 95.14824797843666 |
|
- type: precision |
|
value: 94.60916442048517 |
|
- type: recall |
|
value: 96.22641509433963 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (tel-eng) |
|
config: tel-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 93.58974358974359 |
|
- type: f1 |
|
value: 91.59544159544159 |
|
- type: precision |
|
value: 90.66951566951566 |
|
- type: recall |
|
value: 93.58974358974359 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (afr-eng) |
|
config: afr-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.1 |
|
- type: f1 |
|
value: 97.46666666666668 |
|
- type: precision |
|
value: 97.15 |
|
- type: recall |
|
value: 98.1 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (mon-eng) |
|
config: mon-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 93.4090909090909 |
|
- type: f1 |
|
value: 91.5909090909091 |
|
- type: precision |
|
value: 90.71969696969697 |
|
- type: recall |
|
value: 93.4090909090909 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (arz-eng) |
|
config: arz-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 89.51781970649894 |
|
- type: f1 |
|
value: 86.76150544075072 |
|
- type: precision |
|
value: 85.55206149545772 |
|
- type: recall |
|
value: 89.51781970649894 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (hrv-eng) |
|
config: hrv-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.2 |
|
- type: f1 |
|
value: 97.65 |
|
- type: precision |
|
value: 97.38333333333333 |
|
- type: recall |
|
value: 98.2 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (nov-eng) |
|
config: nov-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 75.87548638132296 |
|
- type: f1 |
|
value: 71.24698906800073 |
|
- type: precision |
|
value: 69.66572338167668 |
|
- type: recall |
|
value: 75.87548638132296 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (gsw-eng) |
|
config: gsw-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 61.53846153846154 |
|
- type: f1 |
|
value: 54.83234714003944 |
|
- type: precision |
|
value: 52.06552706552707 |
|
- type: recall |
|
value: 61.53846153846154 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (nds-eng) |
|
config: nds-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 59.199999999999996 |
|
- type: f1 |
|
value: 54.183211233211225 |
|
- type: precision |
|
value: 52.48751719986241 |
|
- type: recall |
|
value: 59.199999999999996 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ukr-eng) |
|
config: ukr-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.6 |
|
- type: f1 |
|
value: 94.3 |
|
- type: precision |
|
value: 93.65 |
|
- type: recall |
|
value: 95.6 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (uzb-eng) |
|
config: uzb-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 87.85046728971963 |
|
- type: f1 |
|
value: 85.25700934579439 |
|
- type: precision |
|
value: 84.09267912772586 |
|
- type: recall |
|
value: 87.85046728971963 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (lit-eng) |
|
config: lit-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.0 |
|
- type: f1 |
|
value: 97.43333333333332 |
|
- type: precision |
|
value: 97.15 |
|
- type: recall |
|
value: 98.0 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ina-eng) |
|
config: ina-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 90.8 |
|
- type: f1 |
|
value: 88.66055555555555 |
|
- type: precision |
|
value: 87.81845238095238 |
|
- type: recall |
|
value: 90.8 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (lfn-eng) |
|
config: lfn-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 70.6 |
|
- type: f1 |
|
value: 65.538895353013 |
|
- type: precision |
|
value: 63.69531394330308 |
|
- type: recall |
|
value: 70.6 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (zsm-eng) |
|
config: zsm-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.89999999999999 |
|
- type: f1 |
|
value: 96.06666666666668 |
|
- type: precision |
|
value: 95.68333333333334 |
|
- type: recall |
|
value: 96.89999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ita-eng) |
|
config: ita-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.8 |
|
- type: f1 |
|
value: 95.95 |
|
- type: precision |
|
value: 95.55 |
|
- type: recall |
|
value: 96.8 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (cmn-eng) |
|
config: cmn-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.19999999999999 |
|
- type: f1 |
|
value: 93.8 |
|
- type: precision |
|
value: 93.13333333333334 |
|
- type: recall |
|
value: 95.19999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (lvs-eng) |
|
config: lvs-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.5 |
|
- type: f1 |
|
value: 95.45 |
|
- type: precision |
|
value: 94.93333333333334 |
|
- type: recall |
|
value: 96.5 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (glg-eng) |
|
config: glg-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.89999999999999 |
|
- type: f1 |
|
value: 97.28333333333332 |
|
- type: precision |
|
value: 96.98333333333333 |
|
- type: recall |
|
value: 97.89999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ceb-eng) |
|
config: ceb-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 78.16666666666666 |
|
- type: f1 |
|
value: 74.67336721249764 |
|
- type: precision |
|
value: 73.26035353535354 |
|
- type: recall |
|
value: 78.16666666666666 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (bre-eng) |
|
config: bre-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 11.200000000000001 |
|
- type: f1 |
|
value: 8.48123815073815 |
|
- type: precision |
|
value: 7.843657708032708 |
|
- type: recall |
|
value: 11.200000000000001 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ben-eng) |
|
config: ben-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 91.3 |
|
- type: f1 |
|
value: 89.02333333333333 |
|
- type: precision |
|
value: 87.97500000000001 |
|
- type: recall |
|
value: 91.3 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (swg-eng) |
|
config: swg-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 72.32142857142857 |
|
- type: f1 |
|
value: 67.69209956709956 |
|
- type: precision |
|
value: 66.19047619047619 |
|
- type: recall |
|
value: 72.32142857142857 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (arq-eng) |
|
config: arq-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 79.69264544456641 |
|
- type: f1 |
|
value: 75.40693115885212 |
|
- type: precision |
|
value: 73.67544822539335 |
|
- type: recall |
|
value: 79.69264544456641 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (kab-eng) |
|
config: kab-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 86.8 |
|
- type: f1 |
|
value: 83.65666666666667 |
|
- type: precision |
|
value: 82.24833333333333 |
|
- type: recall |
|
value: 86.8 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (fra-eng) |
|
config: fra-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.39999999999999 |
|
- type: f1 |
|
value: 95.36666666666666 |
|
- type: precision |
|
value: 94.86666666666666 |
|
- type: recall |
|
value: 96.39999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (por-eng) |
|
config: por-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.3 |
|
- type: f1 |
|
value: 95.49 |
|
- type: precision |
|
value: 95.10833333333333 |
|
- type: recall |
|
value: 96.3 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (tat-eng) |
|
config: tat-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 89.60000000000001 |
|
- type: f1 |
|
value: 87.04746031746032 |
|
- type: precision |
|
value: 85.89583333333333 |
|
- type: recall |
|
value: 89.60000000000001 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (oci-eng) |
|
config: oci-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 86.9 |
|
- type: f1 |
|
value: 84.57088023088022 |
|
- type: precision |
|
value: 83.6475 |
|
- type: recall |
|
value: 86.9 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (pol-eng) |
|
config: pol-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.2 |
|
- type: f1 |
|
value: 97.7 |
|
- type: precision |
|
value: 97.46666666666668 |
|
- type: recall |
|
value: 98.2 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (war-eng) |
|
config: war-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 85.39999999999999 |
|
- type: f1 |
|
value: 82.83333333333333 |
|
- type: precision |
|
value: 81.80137426900586 |
|
- type: recall |
|
value: 85.39999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (aze-eng) |
|
config: aze-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 91.4 |
|
- type: f1 |
|
value: 89.11999999999999 |
|
- type: precision |
|
value: 88.12777777777778 |
|
- type: recall |
|
value: 91.4 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (vie-eng) |
|
config: vie-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.8 |
|
- type: f1 |
|
value: 97.16666666666669 |
|
- type: precision |
|
value: 96.85000000000001 |
|
- type: recall |
|
value: 97.8 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (nno-eng) |
|
config: nno-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.89999999999999 |
|
- type: f1 |
|
value: 97.30666666666666 |
|
- type: precision |
|
value: 97.02499999999999 |
|
- type: recall |
|
value: 97.89999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (cha-eng) |
|
config: cha-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 27.00729927007299 |
|
- type: f1 |
|
value: 25.114895917815623 |
|
- type: precision |
|
value: 24.602283361407448 |
|
- type: recall |
|
value: 27.00729927007299 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (mhr-eng) |
|
config: mhr-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 14.099999999999998 |
|
- type: f1 |
|
value: 11.869284007509814 |
|
- type: precision |
|
value: 11.199695454818405 |
|
- type: recall |
|
value: 14.099999999999998 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (dan-eng) |
|
config: dan-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.7 |
|
- type: f1 |
|
value: 97.09 |
|
- type: precision |
|
value: 96.80833333333332 |
|
- type: recall |
|
value: 97.7 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ell-eng) |
|
config: ell-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.5 |
|
- type: f1 |
|
value: 95.47333333333333 |
|
- type: precision |
|
value: 94.975 |
|
- type: recall |
|
value: 96.5 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (amh-eng) |
|
config: amh-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 93.45238095238095 |
|
- type: f1 |
|
value: 91.66666666666666 |
|
- type: precision |
|
value: 90.77380952380952 |
|
- type: recall |
|
value: 93.45238095238095 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (pam-eng) |
|
config: pam-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 11.899999999999999 |
|
- type: f1 |
|
value: 10.303261315113037 |
|
- type: precision |
|
value: 9.902986584515606 |
|
- type: recall |
|
value: 11.899999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (hsb-eng) |
|
config: hsb-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 81.57349896480332 |
|
- type: f1 |
|
value: 77.86519438693352 |
|
- type: precision |
|
value: 76.35595081247254 |
|
- type: recall |
|
value: 81.57349896480332 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (srp-eng) |
|
config: srp-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.1 |
|
- type: f1 |
|
value: 94.86666666666667 |
|
- type: precision |
|
value: 94.25 |
|
- type: recall |
|
value: 96.1 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (epo-eng) |
|
config: epo-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.8 |
|
- type: f1 |
|
value: 98.46666666666667 |
|
- type: precision |
|
value: 98.3 |
|
- type: recall |
|
value: 98.8 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (kzj-eng) |
|
config: kzj-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 10.7 |
|
- type: f1 |
|
value: 8.621683883854935 |
|
- type: precision |
|
value: 8.188292731521031 |
|
- type: recall |
|
value: 10.7 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (awa-eng) |
|
config: awa-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 90.47619047619048 |
|
- type: f1 |
|
value: 87.8581735724593 |
|
- type: precision |
|
value: 86.72438672438673 |
|
- type: recall |
|
value: 90.47619047619048 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (fao-eng) |
|
config: fao-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.0381679389313 |
|
- type: f1 |
|
value: 93.60050890585242 |
|
- type: precision |
|
value: 92.970737913486 |
|
- type: recall |
|
value: 95.0381679389313 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (mal-eng) |
|
config: mal-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.2532751091703 |
|
- type: f1 |
|
value: 97.67103347889375 |
|
- type: precision |
|
value: 97.37991266375546 |
|
- type: recall |
|
value: 98.2532751091703 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ile-eng) |
|
config: ile-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 84.6 |
|
- type: f1 |
|
value: 80.99904761904763 |
|
- type: precision |
|
value: 79.54634920634919 |
|
- type: recall |
|
value: 84.6 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (bos-eng) |
|
config: bos-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.89265536723164 |
|
- type: f1 |
|
value: 95.90395480225989 |
|
- type: precision |
|
value: 95.4331450094162 |
|
- type: recall |
|
value: 96.89265536723164 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (cor-eng) |
|
config: cor-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 12.6 |
|
- type: f1 |
|
value: 9.981918087824628 |
|
- type: precision |
|
value: 9.326319147606549 |
|
- type: recall |
|
value: 12.6 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (cat-eng) |
|
config: cat-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.39999999999999 |
|
- type: f1 |
|
value: 96.65 |
|
- type: precision |
|
value: 96.28333333333333 |
|
- type: recall |
|
value: 97.39999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (eus-eng) |
|
config: eus-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.5 |
|
- type: f1 |
|
value: 95.38333333333333 |
|
- type: precision |
|
value: 94.83333333333333 |
|
- type: recall |
|
value: 96.5 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (yue-eng) |
|
config: yue-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 90.8 |
|
- type: f1 |
|
value: 88.43666666666665 |
|
- type: precision |
|
value: 87.395 |
|
- type: recall |
|
value: 90.8 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (swe-eng) |
|
config: swe-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.7 |
|
- type: f1 |
|
value: 97.03333333333333 |
|
- type: precision |
|
value: 96.71666666666667 |
|
- type: recall |
|
value: 97.7 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (dtp-eng) |
|
config: dtp-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 9.4 |
|
- type: f1 |
|
value: 7.946889105220061 |
|
- type: precision |
|
value: 7.665059865752875 |
|
- type: recall |
|
value: 9.4 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (kat-eng) |
|
config: kat-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.04021447721179 |
|
- type: f1 |
|
value: 93.68632707774799 |
|
- type: precision |
|
value: 93.08534405719392 |
|
- type: recall |
|
value: 95.04021447721179 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (jpn-eng) |
|
config: jpn-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.89999999999999 |
|
- type: f1 |
|
value: 94.66666666666667 |
|
- type: precision |
|
value: 94.08333333333334 |
|
- type: recall |
|
value: 95.89999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (csb-eng) |
|
config: csb-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 82.6086956521739 |
|
- type: f1 |
|
value: 77.98418972332016 |
|
- type: precision |
|
value: 75.96837944664031 |
|
- type: recall |
|
value: 82.6086956521739 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (xho-eng) |
|
config: xho-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.77464788732394 |
|
- type: f1 |
|
value: 94.8356807511737 |
|
- type: precision |
|
value: 94.36619718309859 |
|
- type: recall |
|
value: 95.77464788732394 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (orv-eng) |
|
config: orv-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 53.17365269461077 |
|
- type: f1 |
|
value: 47.07043056743655 |
|
- type: precision |
|
value: 45.161363241830784 |
|
- type: recall |
|
value: 53.17365269461077 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ind-eng) |
|
config: ind-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.5 |
|
- type: f1 |
|
value: 94.5 |
|
- type: precision |
|
value: 94.03333333333333 |
|
- type: recall |
|
value: 95.5 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (tuk-eng) |
|
config: tuk-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 93.59605911330048 |
|
- type: f1 |
|
value: 91.82266009852216 |
|
- type: precision |
|
value: 91.09195402298852 |
|
- type: recall |
|
value: 93.59605911330048 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (max-eng) |
|
config: max-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 76.40845070422534 |
|
- type: f1 |
|
value: 72.73082942097027 |
|
- type: precision |
|
value: 71.46686939820742 |
|
- type: recall |
|
value: 76.40845070422534 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (swh-eng) |
|
config: swh-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 93.58974358974359 |
|
- type: f1 |
|
value: 91.98290598290598 |
|
- type: precision |
|
value: 91.3119658119658 |
|
- type: recall |
|
value: 93.58974358974359 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (hin-eng) |
|
config: hin-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.8 |
|
- type: f1 |
|
value: 97.06666666666668 |
|
- type: precision |
|
value: 96.7 |
|
- type: recall |
|
value: 97.8 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (dsb-eng) |
|
config: dsb-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 68.89352818371609 |
|
- type: f1 |
|
value: 64.47860652453555 |
|
- type: precision |
|
value: 62.878651918592574 |
|
- type: recall |
|
value: 68.89352818371609 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ber-eng) |
|
config: ber-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 33.800000000000004 |
|
- type: f1 |
|
value: 29.290774344112368 |
|
- type: precision |
|
value: 28.066016735704647 |
|
- type: recall |
|
value: 33.800000000000004 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (tam-eng) |
|
config: tam-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 90.22801302931596 |
|
- type: f1 |
|
value: 88.07817589576547 |
|
- type: precision |
|
value: 87.171552660152 |
|
- type: recall |
|
value: 90.22801302931596 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (slk-eng) |
|
config: slk-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.2 |
|
- type: f1 |
|
value: 97.63333333333334 |
|
- type: precision |
|
value: 97.36666666666667 |
|
- type: recall |
|
value: 98.2 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (tgl-eng) |
|
config: tgl-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.7 |
|
- type: f1 |
|
value: 96.95 |
|
- type: precision |
|
value: 96.58333333333331 |
|
- type: recall |
|
value: 97.7 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ast-eng) |
|
config: ast-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 92.91338582677166 |
|
- type: f1 |
|
value: 90.81364829396327 |
|
- type: precision |
|
value: 89.89501312335958 |
|
- type: recall |
|
value: 92.91338582677166 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (mkd-eng) |
|
config: mkd-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.89999999999999 |
|
- type: f1 |
|
value: 95.98333333333332 |
|
- type: precision |
|
value: 95.56666666666668 |
|
- type: recall |
|
value: 96.89999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (khm-eng) |
|
config: khm-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 74.51523545706371 |
|
- type: f1 |
|
value: 70.20346919931407 |
|
- type: precision |
|
value: 68.6389565788895 |
|
- type: recall |
|
value: 74.51523545706371 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ces-eng) |
|
config: ces-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 97.6 |
|
- type: f1 |
|
value: 96.88333333333333 |
|
- type: precision |
|
value: 96.53333333333333 |
|
- type: recall |
|
value: 97.6 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (tzl-eng) |
|
config: tzl-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 46.15384615384615 |
|
- type: f1 |
|
value: 39.47885447885448 |
|
- type: precision |
|
value: 37.301528599605525 |
|
- type: recall |
|
value: 46.15384615384615 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (urd-eng) |
|
config: urd-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 94.69999999999999 |
|
- type: f1 |
|
value: 93.16666666666667 |
|
- type: precision |
|
value: 92.41666666666667 |
|
- type: recall |
|
value: 94.69999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (ara-eng) |
|
config: ara-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.19999999999999 |
|
- type: f1 |
|
value: 93.83333333333333 |
|
- type: precision |
|
value: 93.16666666666667 |
|
- type: recall |
|
value: 95.19999999999999 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (kor-eng) |
|
config: kor-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 92.0 |
|
- type: f1 |
|
value: 89.98666666666666 |
|
- type: precision |
|
value: 89.09166666666667 |
|
- type: recall |
|
value: 92.0 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (yid-eng) |
|
config: yid-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 95.51886792452831 |
|
- type: f1 |
|
value: 94.3003144654088 |
|
- type: precision |
|
value: 93.75 |
|
- type: recall |
|
value: 95.51886792452831 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (fin-eng) |
|
config: fin-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 98.2 |
|
- type: f1 |
|
value: 97.83333333333333 |
|
- type: precision |
|
value: 97.65 |
|
- type: recall |
|
value: 98.2 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (tha-eng) |
|
config: tha-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 96.8978102189781 |
|
- type: f1 |
|
value: 96.04622871046227 |
|
- type: precision |
|
value: 95.62043795620438 |
|
- type: recall |
|
value: 96.8978102189781 |
|
- task: |
|
type: BitextMining |
|
dataset: |
|
type: mteb/tatoeba-bitext-mining |
|
name: MTEB Tatoeba (wuu-eng) |
|
config: wuu-eng |
|
split: test |
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
|
metrics: |
|
- type: accuracy |
|
value: 85.1 |
|
- type: f1 |
|
value: 81.78564213564214 |
|
- type: precision |
|
value: 80.46416666666667 |
|
- type: recall |
|
value: 85.1 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: slvnwhrl/tenkgnad-clustering-p2p |
|
name: MTEB TenKGnadClusteringP2P |
|
config: default |
|
split: test |
|
revision: 5c59e41555244b7e45c9a6be2d720ab4bafae558 |
|
metrics: |
|
- type: v_measure |
|
value: 21.827519839402644 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: slvnwhrl/tenkgnad-clustering-s2s |
|
name: MTEB TenKGnadClusteringS2S |
|
config: default |
|
split: test |
|
revision: 6cddbe003f12b9b140aec477b583ac4191f01786 |
|
metrics: |
|
- type: v_measure |
|
value: 27.160188241713684 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/ThuNewsClusteringP2P |
|
name: MTEB ThuNewsClusteringP2P |
|
config: default |
|
split: test |
|
revision: 5798586b105c0434e4f0fe5e767abe619442cf93 |
|
metrics: |
|
- type: v_measure |
|
value: 38.54459276932986 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/ThuNewsClusteringS2S |
|
name: MTEB ThuNewsClusteringS2S |
|
config: default |
|
split: test |
|
revision: 8a8b2caeda43f39e13c4bc5bea0f8a667896e10d |
|
metrics: |
|
- type: v_measure |
|
value: 43.4460576234314 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: webis-touche2020 |
|
name: MTEB Touche2020 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.20500000000000002 |
|
- type: map_at_10 |
|
value: 0.391 |
|
- type: map_at_100 |
|
value: 0.612 |
|
- type: map_at_1000 |
|
value: 0.645 |
|
- type: map_at_3 |
|
value: 0.302 |
|
- type: map_at_5 |
|
value: 0.383 |
|
- type: mrr_at_1 |
|
value: 4.082 |
|
- type: mrr_at_10 |
|
value: 5.612 |
|
- type: mrr_at_100 |
|
value: 6.822 |
|
- type: mrr_at_1000 |
|
value: 6.929 |
|
- type: mrr_at_3 |
|
value: 4.082 |
|
- type: mrr_at_5 |
|
value: 5.408 |
|
- type: ndcg_at_1 |
|
value: 4.082 |
|
- type: ndcg_at_10 |
|
value: 1.6840000000000002 |
|
- type: ndcg_at_100 |
|
value: 2.876 |
|
- type: ndcg_at_1000 |
|
value: 4.114 |
|
- type: ndcg_at_3 |
|
value: 2.52 |
|
- type: ndcg_at_5 |
|
value: 2.3720000000000003 |
|
- type: precision_at_1 |
|
value: 4.082 |
|
- type: precision_at_10 |
|
value: 1.429 |
|
- type: precision_at_100 |
|
value: 0.755 |
|
- type: precision_at_1000 |
|
value: 0.18 |
|
- type: precision_at_3 |
|
value: 2.041 |
|
- type: precision_at_5 |
|
value: 2.4490000000000003 |
|
- type: recall_at_1 |
|
value: 0.20500000000000002 |
|
- type: recall_at_10 |
|
value: 0.761 |
|
- type: recall_at_100 |
|
value: 4.423 |
|
- type: recall_at_1000 |
|
value: 9.044 |
|
- type: recall_at_3 |
|
value: 0.302 |
|
- type: recall_at_5 |
|
value: 0.683 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/toxic_conversations_50k |
|
name: MTEB ToxicConversationsClassification |
|
config: default |
|
split: test |
|
revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
|
metrics: |
|
- type: accuracy |
|
value: 67.28359999999999 |
|
- type: ap |
|
value: 12.424592214862038 |
|
- type: f1 |
|
value: 51.53630450055703 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/tweet_sentiment_extraction |
|
name: MTEB TweetSentimentExtractionClassification |
|
config: default |
|
split: test |
|
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
|
metrics: |
|
- type: accuracy |
|
value: 56.23372948500284 |
|
- type: f1 |
|
value: 56.440924587214234 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/twentynewsgroups-clustering |
|
name: MTEB TwentyNewsgroupsClustering |
|
config: default |
|
split: test |
|
revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
|
metrics: |
|
- type: v_measure |
|
value: 24.410059815620116 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twittersemeval2015-pairclassification |
|
name: MTEB TwitterSemEval2015 |
|
config: default |
|
split: test |
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 80.3302139834297 |
|
- type: cos_sim_ap |
|
value: 53.57723069745093 |
|
- type: cos_sim_f1 |
|
value: 51.58639580004565 |
|
- type: cos_sim_precision |
|
value: 45.45454545454545 |
|
- type: cos_sim_recall |
|
value: 59.63060686015831 |
|
- type: dot_accuracy |
|
value: 80.3302139834297 |
|
- type: dot_ap |
|
value: 53.57723006705641 |
|
- type: dot_f1 |
|
value: 51.58639580004565 |
|
- type: dot_precision |
|
value: 45.45454545454545 |
|
- type: dot_recall |
|
value: 59.63060686015831 |
|
- type: euclidean_accuracy |
|
value: 80.3302139834297 |
|
- type: euclidean_ap |
|
value: 53.57723050286929 |
|
- type: euclidean_f1 |
|
value: 51.58639580004565 |
|
- type: euclidean_precision |
|
value: 45.45454545454545 |
|
- type: euclidean_recall |
|
value: 59.63060686015831 |
|
- type: manhattan_accuracy |
|
value: 80.31233235977827 |
|
- type: manhattan_ap |
|
value: 53.44943961562638 |
|
- type: manhattan_f1 |
|
value: 51.24183006535947 |
|
- type: manhattan_precision |
|
value: 43.63636363636363 |
|
- type: manhattan_recall |
|
value: 62.05804749340369 |
|
- type: max_accuracy |
|
value: 80.3302139834297 |
|
- type: max_ap |
|
value: 53.57723069745093 |
|
- type: max_f1 |
|
value: 51.58639580004565 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twitterurlcorpus-pairclassification |
|
name: MTEB TwitterURLCorpus |
|
config: default |
|
split: test |
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 87.45876508712695 |
|
- type: cos_sim_ap |
|
value: 83.5320716566614 |
|
- type: cos_sim_f1 |
|
value: 75.54560716284276 |
|
- type: cos_sim_precision |
|
value: 73.27929362379678 |
|
- type: cos_sim_recall |
|
value: 77.95657530027718 |
|
- type: dot_accuracy |
|
value: 87.45876508712695 |
|
- type: dot_ap |
|
value: 83.53209944887666 |
|
- type: dot_f1 |
|
value: 75.54560716284276 |
|
- type: dot_precision |
|
value: 73.27929362379678 |
|
- type: dot_recall |
|
value: 77.95657530027718 |
|
- type: euclidean_accuracy |
|
value: 87.45876508712695 |
|
- type: euclidean_ap |
|
value: 83.53205938307582 |
|
- type: euclidean_f1 |
|
value: 75.54560716284276 |
|
- type: euclidean_precision |
|
value: 73.27929362379678 |
|
- type: euclidean_recall |
|
value: 77.95657530027718 |
|
- type: manhattan_accuracy |
|
value: 87.52280048123569 |
|
- type: manhattan_ap |
|
value: 83.4884324728773 |
|
- type: manhattan_f1 |
|
value: 75.43366677906411 |
|
- type: manhattan_precision |
|
value: 73.46566445303948 |
|
- type: manhattan_recall |
|
value: 77.51000923929782 |
|
- type: max_accuracy |
|
value: 87.52280048123569 |
|
- type: max_ap |
|
value: 83.53209944887666 |
|
- type: max_f1 |
|
value: 75.54560716284276 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/VideoRetrieval |
|
name: MTEB VideoRetrieval |
|
config: default |
|
split: dev |
|
revision: 58c2597a5943a2ba48f4668c3b90d796283c5639 |
|
metrics: |
|
- type: map_at_1 |
|
value: 13.100000000000001 |
|
- type: map_at_10 |
|
value: 15.620000000000001 |
|
- type: map_at_100 |
|
value: 15.928 |
|
- type: map_at_1000 |
|
value: 15.976 |
|
- type: map_at_3 |
|
value: 14.817 |
|
- type: map_at_5 |
|
value: 15.322 |
|
- type: mrr_at_1 |
|
value: 13.0 |
|
- type: mrr_at_10 |
|
value: 15.57 |
|
- type: mrr_at_100 |
|
value: 15.878 |
|
- type: mrr_at_1000 |
|
value: 15.926000000000002 |
|
- type: mrr_at_3 |
|
value: 14.767 |
|
- type: mrr_at_5 |
|
value: 15.272 |
|
- type: ndcg_at_1 |
|
value: 13.100000000000001 |
|
- type: ndcg_at_10 |
|
value: 17.05 |
|
- type: ndcg_at_100 |
|
value: 18.801000000000002 |
|
- type: ndcg_at_1000 |
|
value: 20.436 |
|
- type: ndcg_at_3 |
|
value: 15.425 |
|
- type: ndcg_at_5 |
|
value: 16.333000000000002 |
|
- type: precision_at_1 |
|
value: 13.100000000000001 |
|
- type: precision_at_10 |
|
value: 2.16 |
|
- type: precision_at_100 |
|
value: 0.304 |
|
- type: precision_at_1000 |
|
value: 0.044000000000000004 |
|
- type: precision_at_3 |
|
value: 5.733 |
|
- type: precision_at_5 |
|
value: 3.88 |
|
- type: recall_at_1 |
|
value: 13.100000000000001 |
|
- type: recall_at_10 |
|
value: 21.6 |
|
- type: recall_at_100 |
|
value: 30.4 |
|
- type: recall_at_1000 |
|
value: 44.1 |
|
- type: recall_at_3 |
|
value: 17.2 |
|
- type: recall_at_5 |
|
value: 19.400000000000002 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/waimai-classification |
|
name: MTEB Waimai |
|
config: default |
|
split: test |
|
revision: 339287def212450dcaa9df8c22bf93e9980c7023 |
|
metrics: |
|
- type: accuracy |
|
value: 76.12 |
|
- type: ap |
|
value: 54.1619589378045 |
|
- type: f1 |
|
value: 74.32372858884229 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: jinaai/cities_wiki_clustering |
|
name: MTEB WikiCitiesClustering |
|
config: default |
|
split: test |
|
revision: ddc9ee9242fa65332597f70e967ecc38b9d734fa |
|
metrics: |
|
- type: v_measure |
|
value: 50.71744674029636 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: jinaai/xmarket_de |
|
name: MTEB XMarketDE |
|
config: default |
|
split: test |
|
revision: 2336818db4c06570fcdf263e1bcb9993b786f67a |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.182 |
|
- type: map_at_10 |
|
value: 0.266 |
|
- type: map_at_100 |
|
value: 0.295 |
|
- type: map_at_1000 |
|
value: 0.313 |
|
- type: map_at_3 |
|
value: 0.232 |
|
- type: map_at_5 |
|
value: 0.23800000000000002 |
|
- type: mrr_at_1 |
|
value: 1.3379999999999999 |
|
- type: mrr_at_10 |
|
value: 1.918 |
|
- type: mrr_at_100 |
|
value: 2.051 |
|
- type: mrr_at_1000 |
|
value: 2.084 |
|
- type: mrr_at_3 |
|
value: 1.7049999999999998 |
|
- type: mrr_at_5 |
|
value: 1.791 |
|
- type: ndcg_at_1 |
|
value: 1.3379999999999999 |
|
- type: ndcg_at_10 |
|
value: 0.859 |
|
- type: ndcg_at_100 |
|
value: 0.8500000000000001 |
|
- type: ndcg_at_1000 |
|
value: 1.345 |
|
- type: ndcg_at_3 |
|
value: 1.032 |
|
- type: ndcg_at_5 |
|
value: 0.918 |
|
- type: precision_at_1 |
|
value: 1.3379999999999999 |
|
- type: precision_at_10 |
|
value: 0.528 |
|
- type: precision_at_100 |
|
value: 0.22699999999999998 |
|
- type: precision_at_1000 |
|
value: 0.132 |
|
- type: precision_at_3 |
|
value: 0.8829999999999999 |
|
- type: precision_at_5 |
|
value: 0.6890000000000001 |
|
- type: recall_at_1 |
|
value: 0.182 |
|
- type: recall_at_10 |
|
value: 0.51 |
|
- type: recall_at_100 |
|
value: 1.2229999999999999 |
|
- type: recall_at_1000 |
|
value: 4.183 |
|
- type: recall_at_3 |
|
value: 0.292 |
|
- type: recall_at_5 |
|
value: 0.315 |
|
--- |
|
|
|
# SONAR |
|
[[Paper]](https://ai.meta.com/research/publications/sonar-sentence-level-multimodal-and-language-agnostic-representations/) |
|
|
|
We introduce SONAR, a new multilingual and multimodal fixed-size sentence embedding space, with a full suite of speech and text encoders and decoders. It substantially outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim and xsim++ multilingual similarity search tasks. |
|
|
|
Speech segments can be embedded in the same SONAR embedding space using language-specific speech encoders trained in a teacher-student setting on speech transcription data. We also provide a single text decoder, which allows us to perform text-to-text and speech-to-text machine translation, including for zero-shot language and modality combinations. |
|
|
|
*SONAR* stands for **S**entence-level multim**O**dal and la**N**guage-**A**gnostic **R**epresentations |
|
|
|
The full list of supported languages (along with download links) can be found here [below](#supported-languages-and-download-links). |
|
|
|
|
|
## Installing |
|
SONAR depends mainly on [Fairseq2](https://github.com/fairinternal/fairseq2) and can be installed using (tested with `python=3.8`) |
|
```bash |
|
pip install --upgrade pip |
|
pip config set global.extra-index-url https://test.pypi.org/simple/ |
|
pip install -e . |
|
``` |
|
|
|
## Usage |
|
fairseq2 will automatically download models into your `$TORCH_HOME/hub` directory upon using the commands below. |
|
|
|
### Compute text sentence embeddings with SONAR: |
|
```python |
|
from sonar.inference_pipelines.text import TextToEmbeddingModelPipeline |
|
t2vec_model = TextToEmbeddingModelPipeline(encoder="text_sonar_basic_encoder", |
|
tokenizer="text_sonar_basic_encoder") |
|
sentences = ['My name is SONAR.', 'I can embed the sentences into vectorial space.'] |
|
t2vec_model.predict(sentences, source_lang="eng_Latn").shape |
|
# torch.Size([2, 1024]) |
|
``` |
|
|
|
### Translate text with SONAR |
|
```python |
|
from sonar.inference_pipelines.text import TextToTextModelPipeline |
|
t2t_model = TextToTextModelPipeline(encoder="text_sonar_basic_encoder", |
|
decoder="text_sonar_basic_decoder", |
|
tokenizer="text_sonar_basic_encoder") # tokenizer is attached to both encoder and decoder cards |
|
|
|
sentences = ['My name is SONAR.', 'I can embed the sentences into vectorial space.'] |
|
t2t_model.predict(sentences, source_lang="eng_Latn", target_lang="fra_Latn") |
|
# ['Mon nom est SONAR.', "Je peux intégrer les phrases dans l'espace vectoriel."] |
|
``` |
|
|
|
### Compute speech sentence embeddings with SONAR |
|
```python |
|
from sonar.inference_pipelines.speech import SpeechToEmbeddingModelPipeline |
|
s2vec_model = SpeechToEmbeddingModelPipeline(encoder="sonar_speech_encoder_eng") |
|
|
|
s2vec_model.predict(["./tests/integration_tests/data/audio_files/audio_1.wav", |
|
"./tests/integration_tests/data/audio_files/audio_2.wav"]).shape |
|
# torch.Size([2, 1024]) |
|
import torchaudio |
|
inp, sr = torchaudio.load("./tests/integration_tests/data/audio_files/audio_1.wav") |
|
assert sr == 16000, "Sample rate should be 16kHz" |
|
|
|
s2vec_model.predict([inp]).shape |
|
# torch.Size([1, 1024]) |
|
``` |
|
|
|
### Speech-to-text translation with SONAR |
|
```python |
|
from sonar.inference_pipelines.speech import SpeechToTextModelPipeline |
|
|
|
s2t_model = SpeechToTextModelPipeline(encoder="sonar_speech_encoder_eng", |
|
decoder="text_sonar_basic_decoder", |
|
tokenizer="text_sonar_basic_decoder") |
|
|
|
import torchaudio |
|
inp, sr = torchaudio.load("./tests/integration_tests/data/audio_files/audio_1.wav") |
|
assert sr == 16000, "Sample rate should be 16kHz" |
|
|
|
# passing loaded audio files |
|
s2t_model.predict([inp], target_lang="eng_Latn") |
|
# ['Television reports show white smoke coming from the plant.'] |
|
|
|
# passing multiple wav files |
|
s2t_model.predict(["./tests/integration_tests/data/audio_files/audio_1.wav", |
|
"./tests/integration_tests/data/audio_files/audio_2.wav"], target_lang="eng_Latn") |
|
# ['Television reports show white smoke coming from the plant.', |
|
# 'These couples may choose to make an adoption plan for their baby.'] |
|
``` |
|
|
|
|
|
### Predicting [cross-lingual semantic similarity](https://github.com/facebookresearch/fairseq/tree/nllb/examples/nllb/human_XSTS_eval) with BLASER 2 models |
|
```Python |
|
import torch |
|
from sonar.models.blaser.loader import load_blaser_model |
|
|
|
blaser_ref = load_blaser_model("blaser_st2st_ref_v2_0").eval() |
|
blaser_qe = load_blaser_model("blaser_st2st_qe_v2_0").eval() |
|
# BLASER-2 is supposed to work with SONAR speech and text embeddings, |
|
# but we didn't include their extraction in this snippet, to keep it simple. |
|
emb = torch.ones([1, 1024]) |
|
print(blaser_ref(src=emb, ref=emb, mt=emb).item()) # 5.2552 |
|
print(blaser_qe(src=emb, mt=emb).item()) # 4.9819 |
|
``` |
|
|
|
See more complete demo notebooks : |
|
|
|
* [sonar text2text similarity and translation](examples/sonar_text_demo.ipynb) |
|
* [sonar speech2text and other data pipeline examples](examples/inference_pipelines.ipynb) |
|
|
|
|
|
## Model details |
|
|
|
- **Developed by:** Paul-Ambroise Duquenne et al. |
|
- **License:** CC-BY-NC 4.0 license |
|
- **Cite as:** |
|
|
|
``` |
|
@article{Duquenne:2023:sonar_arxiv, |
|
author = {Paul-Ambroise Duquenne and Holger Schwenk and Benoit Sagot}, |
|
title = {{SONAR:} Sentence-Level Multimodal and Language-Agnostic Representations}, |
|
publisher = {arXiv}, |
|
year = {2023}, |
|
url = {https://arxiv.org/abs/unk}, |
|
} |
|
``` |
