metadata
pipeline_tag: sentence-similarity
language: multilingual
license: apache-2.0
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers
- mteb
model-index:
- name: distiluse-base-multilingual-cased-v2
results:
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (en)
config: en
split: test
revision: e8379541af4e31359cca9fbcf4b00f2671dba205
metrics:
- type: accuracy
value: 71.80597014925372
- type: ap
value: 33.70263085714158
- type: f1
value: 65.44989712268762
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (de)
config: de
split: test
revision: e8379541af4e31359cca9fbcf4b00f2671dba205
metrics:
- type: accuracy
value: 68.13704496788009
- type: ap
value: 80.6706553308835
- type: f1
value: 66.6468090116337
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (en-ext)
config: en-ext
split: test
revision: e8379541af4e31359cca9fbcf4b00f2671dba205
metrics:
- type: accuracy
value: 72.96101949025487
- type: ap
value: 22.209148737301962
- type: f1
value: 60.428775420466906
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (ja)
config: ja
split: test
revision: e8379541af4e31359cca9fbcf4b00f2671dba205
metrics:
- type: accuracy
value: 65.38543897216275
- type: ap
value: 16.13590032328447
- type: f1
value: 53.20720298606364
- task:
type: Classification
dataset:
type: mteb/amazon_polarity
name: MTEB AmazonPolarityClassification
config: default
split: test
revision: e2d317d38cd51312af73b3d32a06d1a08b442046
metrics:
- type: accuracy
value: 67.9988
- type: ap
value: 62.59891275364823
- type: f1
value: 67.73408963897285
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (en)
config: en
split: test
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
metrics:
- type: accuracy
value: 35.454
- type: f1
value: 35.01958914240701
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (de)
config: de
split: test
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
metrics:
- type: accuracy
value: 35.032000000000004
- type: f1
value: 33.93976447064354
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (es)
config: es
split: test
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
metrics:
- type: accuracy
value: 36.242000000000004
- type: f1
value: 34.98879083946539
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (fr)
config: fr
split: test
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
metrics:
- type: accuracy
value: 35.699999999999996
- type: f1
value: 34.74911268048424
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (ja)
config: ja
split: test
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
metrics:
- type: accuracy
value: 31.075999999999997
- type: f1
value: 30.525865114811996
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (zh)
config: zh
split: test
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
metrics:
- type: accuracy
value: 33.894000000000005
- type: f1
value: 32.63851365829613
- task:
type: Clustering
dataset:
type: mteb/arxiv-clustering-p2p
name: MTEB ArxivClusteringP2P
config: default
split: test
revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d
metrics:
- type: v_measure
value: 33.59372253035037
- task:
type: Reranking
dataset:
type: mteb/askubuntudupquestions-reranking
name: MTEB AskUbuntuDupQuestions
config: default
split: test
revision: 2000358ca161889fa9c082cb41daa8dcfb161a54
metrics:
- type: map
value: 53.752292029725815
- type: mrr
value: 68.26968737633557
- task:
type: STS
dataset:
type: mteb/biosses-sts
name: MTEB BIOSSES
config: default
split: test
revision: d3fb88f8f02e40887cd149695127462bbcf29b4a
metrics:
- type: cos_sim_pearson
value: 79.26094784825986
- type: cos_sim_spearman
value: 78.34033925464169
- type: euclidean_pearson
value: 77.43607353262966
- type: euclidean_spearman
value: 76.77765304536669
- type: manhattan_pearson
value: 77.43287991423313
- type: manhattan_spearman
value: 76.849341425823
- task:
type: Classification
dataset:
type: mteb/banking77
name: MTEB Banking77Classification
config: default
split: test
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
metrics:
- type: accuracy
value: 71.48051948051949
- type: f1
value: 70.45713884617551
- task:
type: Classification
dataset:
type: mteb/emotion
name: MTEB EmotionClassification
config: default
split: test
revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37
metrics:
- type: accuracy
value: 40.045
- type: f1
value: 36.59544493168501
- task:
type: Classification
dataset:
type: mteb/imdb
name: MTEB ImdbClassification
config: default
split: test
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
metrics:
- type: accuracy
value: 61.516799999999996
- type: ap
value: 57.302114956239514
- type: f1
value: 61.24392423075582
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (en)
config: en
split: test
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
metrics:
- type: accuracy
value: 91.59142726858185
- type: f1
value: 91.16731589297895
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (de)
config: de
split: test
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
metrics:
- type: accuracy
value: 86.19047619047619
- type: f1
value: 84.42185095665184
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (es)
config: es
split: test
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
metrics:
- type: accuracy
value: 87.74516344229485
- type: f1
value: 86.89629934160831
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (fr)
config: fr
split: test
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
metrics:
- type: accuracy
value: 84.61321641089883
- type: f1
value: 83.86194715158408
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (hi)
config: hi
split: test
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
metrics:
- type: accuracy
value: 76.4144854786662
- type: f1
value: 74.66143814759417
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (th)
config: th
split: test
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
metrics:
- type: accuracy
value: 73.61663652802893
- type: f1
value: 71.59773512640322
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (en)
config: en
split: test
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
metrics:
- type: accuracy
value: 66.40218878248974
- type: f1
value: 44.0157655128108
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (de)
config: de
split: test
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
metrics:
- type: accuracy
value: 59.208227669766124
- type: f1
value: 36.59415374962454
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (es)
config: es
split: test
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
metrics:
- type: accuracy
value: 57.21147431621081
- type: f1
value: 38.46167201793877
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (fr)
config: fr
split: test
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
metrics:
- type: accuracy
value: 53.40745380519887
- type: f1
value: 36.87813951228687
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (hi)
config: hi
split: test
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
metrics:
- type: accuracy
value: 45.54320544998208
- type: f1
value: 28.091086881484788
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (th)
config: th
split: test
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
metrics:
- type: accuracy
value: 47.732368896925855
- type: f1
value: 29.87429451601028
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (af)
config: af
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 40.02017484868864
- type: f1
value: 35.75859698769357
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (am)
config: am
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 2.347007397444519
- type: f1
value: 0.7465390699534603
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ar)
config: ar
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 43.143913920645595
- type: f1
value: 38.85558637592047
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (az)
config: az
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 25.601882985877605
- type: f1
value: 25.205774742990254
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (bn)
config: bn
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 4.84196368527236
- type: f1
value: 1.7486302624639154
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (cy)
config: cy
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 15.43375924680565
- type: f1
value: 14.212012285498213
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (da)
config: da
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 52.33355749831876
- type: f1
value: 48.18484932318873
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (de)
config: de
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 51.573638197713514
- type: f1
value: 45.55934579164648
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (el)
config: el
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 49.65366509751178
- type: f1
value: 45.64683808611846
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (en)
config: en
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 66.71149966375253
- type: f1
value: 63.78255507050109
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (es)
config: es
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 56.573638197713514
- type: f1
value: 54.98029542986489
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (fa)
config: fa
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 55.35642232683256
- type: f1
value: 50.20214626269123
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (fi)
config: fi
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 45.71620712844654
- type: f1
value: 42.200836560817535
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (fr)
config: fr
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 57.02084734364491
- type: f1
value: 53.910650671151814
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (he)
config: he
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 46.7350369872226
- type: f1
value: 42.509857120773866
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (hi)
config: hi
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 48.55077336919973
- type: f1
value: 43.993275443482936
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (hu)
config: hu
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 50.64559515803631
- type: f1
value: 45.28464736653043
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (hy)
config: hy
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 40.79354404841964
- type: f1
value: 36.90100598587695
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (id)
config: id
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 55.99529253530599
- type: f1
value: 52.44999289764702
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (is)
config: is
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 16.079354404841965
- type: f1
value: 14.926428149458182
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (it)
config: it
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 57.64626765299259
- type: f1
value: 53.7737970315679
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ja)
config: ja
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 55.329522528581045
- type: f1
value: 50.89055472943818
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (jv)
config: jv
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 28.164088769334228
- type: f1
value: 25.896264320477325
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ka)
config: ka
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 29.411566913248148
- type: f1
value: 26.845594782986996
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (km)
config: km
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 4.791526563550773
- type: f1
value: 1.4491239093711443
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (kn)
config: kn
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 3.365837256220579
- type: f1
value: 1.3064783225018712
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ko)
config: ko
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 49.9663752521856
- type: f1
value: 46.28463081207797
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (lv)
config: lv
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 44.31405514458642
- type: f1
value: 41.59880687298492
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ml)
config: ml
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 3.2447881640887695
- type: f1
value: 1.1130430676330432
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (mn)
config: mn
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 40.36650975117687
- type: f1
value: 36.405182949755556
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ms)
config: ms
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 47.969065232010756
- type: f1
value: 43.564724873023735
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (my)
config: my
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 38.483523873570945
- type: f1
value: 33.325537301233815
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (nb)
config: nb
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 46.008742434431745
- type: f1
value: 43.1074675107609
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (nl)
config: nl
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 58.29186281102891
- type: f1
value: 53.383269502572276
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (pl)
config: pl
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 53.10020174848689
- type: f1
value: 48.491009241597
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (pt)
config: pt
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 58.62811028917283
- type: f1
value: 56.39037901287144
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ro)
config: ro
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 50.632145258910555
- type: f1
value: 47.52272047301657
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ru)
config: ru
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 57.95897780766644
- type: f1
value: 53.79707075942384
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (sl)
config: sl
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 50.65904505716207
- type: f1
value: 48.69839976207718
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (sq)
config: sq
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 50.25218560860794
- type: f1
value: 46.925456055473525
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (sv)
config: sv
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 52.410894418291875
- type: f1
value: 47.64228703598475
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (sw)
config: sw
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 19.293880295897782
- type: f1
value: 17.66502971829105
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ta)
config: ta
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 3.7861466039004705
- type: f1
value: 1.2869466371674323
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (te)
config: te
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 3.3591123066577
- type: f1
value: 1.3191646312270082
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (th)
config: th
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 45.279085406859444
- type: f1
value: 42.5424265903176
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (tl)
config: tl
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 28.43981170141224
- type: f1
value: 25.283226291015392
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (tr)
config: tr
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 50.474108944182916
- type: f1
value: 47.186574797430794
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ur)
config: ur
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 46.02891728312038
- type: f1
value: 41.42008348263186
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (vi)
config: vi
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 45.252185608607945
- type: f1
value: 41.69045540062304
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (zh-CN)
config: zh-CN
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 59.21990585070611
- type: f1
value: 56.214011316092495
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (zh-TW)
config: zh-TW
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 54.96301277740416
- type: f1
value: 53.020268356293045
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (af)
config: af
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 53.665097511768664
- type: f1
value: 48.81662825721646
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (am)
config: am
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 7.720242098184263
- type: f1
value: 3.0172360162047553
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ar)
config: ar
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 52.188971082716876
- type: f1
value: 52.360668734058116
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (az)
config: az
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 34.74781439139206
- type: f1
value: 32.55953852645334
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (bn)
config: bn
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 10.652320107599191
- type: f1
value: 6.439785272600618
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (cy)
config: cy
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 21.237390719569603
- type: f1
value: 18.428497244325158
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (da)
config: da
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 62.54875588433087
- type: f1
value: 60.69001958508912
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (de)
config: de
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 61.40215198386013
- type: f1
value: 58.07492599013545
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (el)
config: el
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 60.67585743106927
- type: f1
value: 58.055827627792056
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (en)
config: en
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 74.00470746469401
- type: f1
value: 72.22931856264793
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (es)
config: es
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 64.6133154001345
- type: f1
value: 63.345907502958184
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (fa)
config: fa
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 59.2434431741762
- type: f1
value: 57.40580117369346
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (fi)
config: fi
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 54.660390047074635
- type: f1
value: 51.45432689446743
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (fr)
config: fr
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 65.19502353732346
- type: f1
value: 63.50200684075783
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (he)
config: he
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 54.744451916610636
- type: f1
value: 52.621508448089294
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (hi)
config: hi
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 55.985205110961665
- type: f1
value: 53.70079438430524
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (hu)
config: hu
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 61.20040349697378
- type: f1
value: 58.5060672562612
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (hy)
config: hy
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 49.63349024882314
- type: f1
value: 47.39478501763526
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (id)
config: id
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 65.25218560860793
- type: f1
value: 63.45266636240826
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (is)
config: is
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 22.599193006052456
- type: f1
value: 21.93829297740852
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (it)
config: it
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 64.63349024882314
- type: f1
value: 63.15345402734339
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ja)
config: ja
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 62.32010759919301
- type: f1
value: 60.02914271738089
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (jv)
config: jv
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 35.76664425016812
- type: f1
value: 33.52830525064859
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ka)
config: ka
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 39.08204438466712
- type: f1
value: 37.312566552928736
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (km)
config: km
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 9.236718224613314
- type: f1
value: 3.41684484979606
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (kn)
config: kn
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 8.278412911903162
- type: f1
value: 3.9418094806677426
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ko)
config: ko
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 57.595830531271005
- type: f1
value: 56.42188880877947
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (lv)
config: lv
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 51.72158708809683
- type: f1
value: 49.903136843275256
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ml)
config: ml
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 8.254875588433089
- type: f1
value: 4.06813409809564
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (mn)
config: mn
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 47.20914593140552
- type: f1
value: 44.780121017940225
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ms)
config: ms
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 55.64559515803632
- type: f1
value: 53.10457083056076
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (my)
config: my
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 43.308675184936114
- type: f1
value: 40.40654924373442
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (nb)
config: nb
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 54.983187626092814
- type: f1
value: 54.22408282419106
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (nl)
config: nl
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 67.4915938130464
- type: f1
value: 64.66608521628295
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (pl)
config: pl
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 61.28782784129119
- type: f1
value: 59.364955179296544
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (pt)
config: pt
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 64.26361802286483
- type: f1
value: 63.01306314842478
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ro)
config: ro
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 58.02622730329523
- type: f1
value: 55.8928740774695
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ru)
config: ru
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 65.41358439811701
- type: f1
value: 64.15512608670188
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (sl)
config: sl
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 59.357767316745125
- type: f1
value: 58.284479078165106
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (sq)
config: sq
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 62.686617350369865
- type: f1
value: 59.49767603465277
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (sv)
config: sv
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 64.35104236718225
- type: f1
value: 61.62298238070601
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (sw)
config: sw
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 25.12104909213181
- type: f1
value: 22.063961287382483
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ta)
config: ta
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 8.671822461331539
- type: f1
value: 4.160922973001201
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (te)
config: te
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 7.821116341627439
- type: f1
value: 3.59600077788794
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (th)
config: th
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 54.64694014794888
- type: f1
value: 51.591586777977504
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (tl)
config: tl
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 36.08607935440485
- type: f1
value: 32.46731674317254
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (tr)
config: tr
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 60.89441829186282
- type: f1
value: 60.11999627480401
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ur)
config: ur
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 54.707464694014796
- type: f1
value: 52.46709289947395
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (vi)
config: vi
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 55.1546738399462
- type: f1
value: 54.110902262235584
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (zh-CN)
config: zh-CN
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 66.4357767316745
- type: f1
value: 64.94684758602547
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (zh-TW)
config: zh-TW
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 62.88836583725623
- type: f1
value: 61.7106895387137
- task:
type: Reranking
dataset:
type: mteb/mind_small
name: MTEB MindSmallReranking
config: default
split: test
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69
metrics:
- type: map
value: 30.389522323606887
- type: mrr
value: 31.507198662637208
- task:
type: STS
dataset:
type: mteb/sickr-sts
name: MTEB SICK-R
config: default
split: test
revision: a6ea5a8cab320b040a23452cc28066d9beae2cee
metrics:
- type: cos_sim_pearson
value: 81.18466748223793
- type: cos_sim_spearman
value: 75.24738784985722
- type: euclidean_pearson
value: 78.51159752223624
- type: euclidean_spearman
value: 75.46087065937311
- type: manhattan_pearson
value: 77.16743820738003
- type: manhattan_spearman
value: 73.49433694282183
- task:
type: STS
dataset:
type: mteb/sts12-sts
name: MTEB STS12
config: default
split: test
revision: a0d554a64d88156834ff5ae9920b964011b16384
metrics:
- type: cos_sim_pearson
value: 79.35237266605724
- type: cos_sim_spearman
value: 72.95904349793416
- type: euclidean_pearson
value: 73.07895490202789
- type: euclidean_spearman
value: 71.66451640969629
- type: manhattan_pearson
value: 73.08359981539324
- type: manhattan_spearman
value: 71.91126963073746
- task:
type: STS
dataset:
type: mteb/sts13-sts
name: MTEB STS13
config: default
split: test
revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca
metrics:
- type: cos_sim_pearson
value: 68.26126180159085
- type: cos_sim_spearman
value: 70.5821267642011
- type: euclidean_pearson
value: 69.32005598610408
- type: euclidean_spearman
value: 69.91767420734864
- type: manhattan_pearson
value: 69.65574245013867
- type: manhattan_spearman
value: 70.22188522513176
- task:
type: STS
dataset:
type: mteb/sts14-sts
name: MTEB STS14
config: default
split: test
revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375
metrics:
- type: cos_sim_pearson
value: 73.8304467062826
- type: cos_sim_spearman
value: 70.28565248557119
- type: euclidean_pearson
value: 72.80361711138981
- type: euclidean_spearman
value: 70.63777081958187
- type: manhattan_pearson
value: 72.88892597106383
- type: manhattan_spearman
value: 70.86449280993048
- task:
type: STS
dataset:
type: mteb/sts15-sts
name: MTEB STS15
config: default
split: test
revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3
metrics:
- type: cos_sim_pearson
value: 81.41478503988436
- type: cos_sim_spearman
value: 81.94087130039843
- type: euclidean_pearson
value: 81.23351470401855
- type: euclidean_spearman
value: 81.43266713211875
- type: manhattan_pearson
value: 81.16667353510842
- type: manhattan_spearman
value: 81.24163241523068
- task:
type: STS
dataset:
type: mteb/sts16-sts
name: MTEB STS16
config: default
split: test
revision: 4d8694f8f0e0100860b497b999b3dbed754a0513
metrics:
- type: cos_sim_pearson
value: 75.08475719822
- type: cos_sim_spearman
value: 76.80438358515593
- type: euclidean_pearson
value: 75.90649123881406
- type: euclidean_spearman
value: 75.9482319164023
- type: manhattan_pearson
value: 75.64396465387331
- type: manhattan_spearman
value: 75.56185817375638
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (ko-ko)
config: ko-ko
split: test
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
metrics:
- type: cos_sim_pearson
value: 76.57756740555968
- type: cos_sim_spearman
value: 76.39843364267264
- type: euclidean_pearson
value: 75.40424583472578
- type: euclidean_spearman
value: 75.31307938562327
- type: manhattan_pearson
value: 74.73109587053861
- type: manhattan_spearman
value: 74.54667368714956
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (ar-ar)
config: ar-ar
split: test
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
metrics:
- type: cos_sim_pearson
value: 76.54105158056127
- type: cos_sim_spearman
value: 77.34104635434048
- type: euclidean_pearson
value: 75.28125389103582
- type: euclidean_spearman
value: 75.42418151345
- type: manhattan_pearson
value: 74.2691880967768
- type: manhattan_spearman
value: 74.14253657856801
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (en-ar)
config: en-ar
split: test
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
metrics:
- type: cos_sim_pearson
value: 77.02928931510961
- type: cos_sim_spearman
value: 77.45907270306685
- type: euclidean_pearson
value: 77.47937379735676
- type: euclidean_spearman
value: 77.21301895586583
- type: manhattan_pearson
value: 76.6676288138473
- type: manhattan_spearman
value: 76.7187203876331
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (en-de)
config: en-de
split: test
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
metrics:
- type: cos_sim_pearson
value: 79.85147526701459
- type: cos_sim_spearman
value: 80.24439450219447
- type: euclidean_pearson
value: 80.16905693851314
- type: euclidean_spearman
value: 79.30869641757035
- type: manhattan_pearson
value: 79.4830024429918
- type: manhattan_spearman
value: 78.64845690144578
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (en-en)
config: en-en
split: test
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
metrics:
- type: cos_sim_pearson
value: 85.23328074603815
- type: cos_sim_spearman
value: 86.18847213007086
- type: euclidean_pearson
value: 85.91331577309407
- type: euclidean_spearman
value: 85.89967500124904
- type: manhattan_pearson
value: 85.13857617716477
- type: manhattan_spearman
value: 84.82259586513993
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (en-tr)
config: en-tr
split: test
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
metrics:
- type: cos_sim_pearson
value: 75.38182956463326
- type: cos_sim_spearman
value: 74.34143229429068
- type: euclidean_pearson
value: 76.66151217728661
- type: euclidean_spearman
value: 75.68846427284615
- type: manhattan_pearson
value: 75.55942040372382
- type: manhattan_spearman
value: 74.67284614447757
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (es-en)
config: es-en
split: test
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
metrics:
- type: cos_sim_pearson
value: 76.94108940753875
- type: cos_sim_spearman
value: 77.39619379750977
- type: euclidean_pearson
value: 76.7736720732895
- type: euclidean_spearman
value: 76.29160645031078
- type: manhattan_pearson
value: 74.69337188827635
- type: manhattan_spearman
value: 74.47874230344613
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (es-es)
config: es-es
split: test
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
metrics:
- type: cos_sim_pearson
value: 83.99450399002905
- type: cos_sim_spearman
value: 83.71182297187157
- type: euclidean_pearson
value: 85.14304799861979
- type: euclidean_spearman
value: 83.69127569618827
- type: manhattan_pearson
value: 84.90116866712872
- type: manhattan_spearman
value: 83.31690582990805
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (fr-en)
config: fr-en
split: test
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
metrics:
- type: cos_sim_pearson
value: 79.12525161262887
- type: cos_sim_spearman
value: 79.27905944348255
- type: euclidean_pearson
value: 80.37847361563627
- type: euclidean_spearman
value: 79.45430583111714
- type: manhattan_pearson
value: 79.39311209355259
- type: manhattan_spearman
value: 78.35224091918822
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (it-en)
config: it-en
split: test
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
metrics:
- type: cos_sim_pearson
value: 80.35229136945712
- type: cos_sim_spearman
value: 80.82110464777067
- type: euclidean_pearson
value: 80.8820546236635
- type: euclidean_spearman
value: 80.52608029482144
- type: manhattan_pearson
value: 79.87881836256757
- type: manhattan_spearman
value: 79.21409642635105
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (nl-en)
config: nl-en
split: test
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
metrics:
- type: cos_sim_pearson
value: 80.08711291606406
- type: cos_sim_spearman
value: 80.50747550174945
- type: euclidean_pearson
value: 80.19128295947303
- type: euclidean_spearman
value: 79.80068556328985
- type: manhattan_pearson
value: 79.2805531467
- type: manhattan_spearman
value: 78.67459586691882
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (en)
config: en
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 63.749476793187654
- type: cos_sim_spearman
value: 62.87618960301087
- type: euclidean_pearson
value: 62.00259194547161
- type: euclidean_spearman
value: 60.14134804263504
- type: manhattan_pearson
value: 61.85663435862556
- type: manhattan_spearman
value: 60.49194043559385
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (de)
config: de
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 30.728588031668387
- type: cos_sim_spearman
value: 35.72910641917946
- type: euclidean_pearson
value: 27.727483814940634
- type: euclidean_spearman
value: 36.697908777201874
- type: manhattan_pearson
value: 26.887457740598375
- type: manhattan_spearman
value: 35.65193589164902
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (es)
config: es
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 58.515732517017895
- type: cos_sim_spearman
value: 59.34352724163223
- type: euclidean_pearson
value: 59.37822334487575
- type: euclidean_spearman
value: 59.952966536792296
- type: manhattan_pearson
value: 59.34905346132589
- type: manhattan_spearman
value: 59.58363163864109
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (pl)
config: pl
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 26.73251862968695
- type: cos_sim_spearman
value: 34.57702083368428
- type: euclidean_pearson
value: 11.555722679629111
- type: euclidean_spearman
value: 33.83302978677857
- type: manhattan_pearson
value: 11.30958607896797
- type: manhattan_spearman
value: 33.45113736058396
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (tr)
config: tr
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 50.59069907623683
- type: cos_sim_spearman
value: 54.07437321160808
- type: euclidean_pearson
value: 55.31327716542195
- type: euclidean_spearman
value: 55.862881519289
- type: manhattan_pearson
value: 55.76874086920313
- type: manhattan_spearman
value: 56.389207939925434
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (ar)
config: ar
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 43.19525519197726
- type: cos_sim_spearman
value: 49.04013064287781
- type: euclidean_pearson
value: 41.51101650799975
- type: euclidean_spearman
value: 45.69491981920255
- type: manhattan_pearson
value: 41.798306097489686
- type: manhattan_spearman
value: 45.88969916327865
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (ru)
config: ru
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 46.72887212606245
- type: cos_sim_spearman
value: 52.40251410115027
- type: euclidean_pearson
value: 42.61087105318375
- type: euclidean_spearman
value: 49.31647979068464
- type: manhattan_pearson
value: 41.971488569524226
- type: manhattan_spearman
value: 48.603948080104416
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (zh)
config: zh
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 50.282899703556204
- type: cos_sim_spearman
value: 54.31518993723914
- type: euclidean_pearson
value: 46.92686134587321
- type: euclidean_spearman
value: 50.4258942374202
- type: manhattan_pearson
value: 47.119373335384516
- type: manhattan_spearman
value: 50.290545214030644
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (fr)
config: fr
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 76.6695578258507
- type: cos_sim_spearman
value: 76.41254265129491
- type: euclidean_pearson
value: 68.10573760855496
- type: euclidean_spearman
value: 71.53756176277794
- type: manhattan_pearson
value: 67.71247571269289
- type: manhattan_spearman
value: 71.52537846395397
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (de-en)
config: de-en
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 52.39033873441029
- type: cos_sim_spearman
value: 47.50888019756861
- type: euclidean_pearson
value: 54.09329593694967
- type: euclidean_spearman
value: 46.745911343795036
- type: manhattan_pearson
value: 55.071517962875795
- type: manhattan_spearman
value: 47.82505012490346
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (es-en)
config: es-en
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 66.22856680218524
- type: cos_sim_spearman
value: 68.9583551854743
- type: euclidean_pearson
value: 69.45990476537347
- type: euclidean_spearman
value: 69.51326488176926
- type: manhattan_pearson
value: 69.2654378415376
- type: manhattan_spearman
value: 69.25549968332008
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (it)
config: it
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 63.86370050619784
- type: cos_sim_spearman
value: 65.10152541505573
- type: euclidean_pearson
value: 61.23738658178195
- type: euclidean_spearman
value: 62.77231926242124
- type: manhattan_pearson
value: 61.20141239111747
- type: manhattan_spearman
value: 62.58683030963466
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (pl-en)
config: pl-en
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 70.24310112698741
- type: cos_sim_spearman
value: 71.32608389737901
- type: euclidean_pearson
value: 69.53167907457565
- type: euclidean_spearman
value: 69.24756304760876
- type: manhattan_pearson
value: 69.4432001214127
- type: manhattan_spearman
value: 69.92998467998946
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (zh-en)
config: zh-en
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 62.96457033320131
- type: cos_sim_spearman
value: 61.750627475845285
- type: euclidean_pearson
value: 59.58377101704754
- type: euclidean_spearman
value: 55.91175172327044
- type: manhattan_pearson
value: 59.64672089274813
- type: manhattan_spearman
value: 55.93114256617111
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (es-it)
config: es-it
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 60.54093974085284
- type: cos_sim_spearman
value: 63.277246213501634
- type: euclidean_pearson
value: 59.21790717375445
- type: euclidean_spearman
value: 60.77632900198518
- type: manhattan_pearson
value: 59.572573245502824
- type: manhattan_spearman
value: 60.86391917522135
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (de-fr)
config: de-fr
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 56.2735220514599
- type: cos_sim_spearman
value: 60.76242915296164
- type: euclidean_pearson
value: 54.73358313453174
- type: euclidean_spearman
value: 59.01153256838316
- type: manhattan_pearson
value: 53.30971466711619
- type: manhattan_spearman
value: 57.427602926148516
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (de-pl)
config: de-pl
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 33.210422466959244
- type: cos_sim_spearman
value: 36.09068930156353
- type: euclidean_pearson
value: 36.72425141682268
- type: euclidean_spearman
value: 33.3808081935963
- type: manhattan_pearson
value: 35.47249118003641
- type: manhattan_spearman
value: 31.964279432613434
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (fr-pl)
config: fr-pl
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 62.721710627517034
- type: cos_sim_spearman
value: 61.97797868009122
- type: euclidean_pearson
value: 63.59898515445168
- type: euclidean_spearman
value: 84.51542547285167
- type: manhattan_pearson
value: 62.15380605376377
- type: manhattan_spearman
value: 73.24670207647144
- task:
type: STS
dataset:
type: mteb/stsbenchmark-sts
name: MTEB STSBenchmark
config: default
split: test
revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831
metrics:
- type: cos_sim_pearson
value: 81.6839488629375
- type: cos_sim_spearman
value: 80.75478754676419
- type: euclidean_pearson
value: 80.67588249670365
- type: euclidean_spearman
value: 80.2296669116562
- type: manhattan_pearson
value: 79.79275882752755
- type: manhattan_spearman
value: 79.41562131296504
- task:
type: Reranking
dataset:
type: mteb/scidocs-reranking
name: MTEB SciDocsRR
config: default
split: test
revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab
metrics:
- type: map
value: 69.21586199162861
- type: mrr
value: 88.86282290694054
- task:
type: PairClassification
dataset:
type: mteb/sprintduplicatequestions-pairclassification
name: MTEB SprintDuplicateQuestions
config: default
split: test
revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46
metrics:
- type: cos_sim_accuracy
value: 99.62079207920792
- type: cos_sim_ap
value: 87.14976457350163
- type: cos_sim_f1
value: 81.07317073170732
- type: cos_sim_precision
value: 79.14285714285715
- type: cos_sim_recall
value: 83.1
- type: dot_accuracy
value: 99.57722772277228
- type: dot_ap
value: 84.07833605976549
- type: dot_f1
value: 77.88461538461539
- type: dot_precision
value: 75
- type: dot_recall
value: 81
- type: euclidean_accuracy
value: 99.61287128712871
- type: euclidean_ap
value: 86.94165408325189
- type: euclidean_f1
value: 80.33596837944663
- type: euclidean_precision
value: 79.39453125
- type: euclidean_recall
value: 81.3
- type: manhattan_accuracy
value: 99.64653465346535
- type: manhattan_ap
value: 88.43495903247096
- type: manhattan_f1
value: 81.7193675889328
- type: manhattan_precision
value: 80.76171875
- type: manhattan_recall
value: 82.69999999999999
- type: max_accuracy
value: 99.64653465346535
- type: max_ap
value: 88.43495903247096
- type: max_f1
value: 81.7193675889328
- task:
type: Reranking
dataset:
type: mteb/stackoverflowdupquestions-reranking
name: MTEB StackOverflowDupQuestions
config: default
split: test
revision: e185fbe320c72810689fc5848eb6114e1ef5ec69
metrics:
- type: map
value: 41.92031499253617
- type: mrr
value: 42.11711389101095
- task:
type: Classification
dataset:
type: mteb/toxic_conversations_50k
name: MTEB ToxicConversationsClassification
config: default
split: test
revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c
metrics:
- type: accuracy
value: 69.0936
- type: ap
value: 13.464419132094955
- type: f1
value: 53.17756829624628
- task:
type: Classification
dataset:
type: mteb/tweet_sentiment_extraction
name: MTEB TweetSentimentExtractionClassification
config: default
split: test
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
metrics:
- type: accuracy
value: 59.968873797396704
- type: f1
value: 60.23697658216021
- task:
type: PairClassification
dataset:
type: mteb/twittersemeval2015-pairclassification
name: MTEB TwitterSemEval2015
config: default
split: test
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
metrics:
- type: cos_sim_accuracy
value: 82.7978780473267
- type: cos_sim_ap
value: 61.669291081213906
- type: cos_sim_f1
value: 57.68693665100927
- type: cos_sim_precision
value: 55.59089796917054
- type: cos_sim_recall
value: 59.94722955145119
- type: dot_accuracy
value: 81.68921738093819
- type: dot_ap
value: 57.39705387908134
- type: dot_f1
value: 54.72479298587434
- type: dot_precision
value: 50.814111261872455
- type: dot_recall
value: 59.287598944591025
- type: euclidean_accuracy
value: 82.85152291828098
- type: euclidean_ap
value: 62.456817170822255
- type: euclidean_f1
value: 58.32305795314425
- type: euclidean_precision
value: 54.745370370370374
- type: euclidean_recall
value: 62.401055408970976
- type: manhattan_accuracy
value: 82.76807534124099
- type: manhattan_ap
value: 61.85267667234618
- type: manhattan_f1
value: 57.62629336579428
- type: manhattan_precision
value: 53.49152542372882
- type: manhattan_recall
value: 62.45382585751978
- type: max_accuracy
value: 82.85152291828098
- type: max_ap
value: 62.456817170822255
- type: max_f1
value: 58.32305795314425
- task:
type: PairClassification
dataset:
type: mteb/twitterurlcorpus-pairclassification
name: MTEB TwitterURLCorpus
config: default
split: test
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
metrics:
- type: cos_sim_accuracy
value: 88.03896456708192
- type: cos_sim_ap
value: 84.0249558879327
- type: cos_sim_f1
value: 76.26290458870642
- type: cos_sim_precision
value: 72.93233082706767
- type: cos_sim_recall
value: 79.91222667077302
- type: dot_accuracy
value: 87.87402491558971
- type: dot_ap
value: 83.20076543059169
- type: dot_f1
value: 76.02826329490517
- type: dot_precision
value: 73.52898863472882
- type: dot_recall
value: 78.70341854019095
- type: euclidean_accuracy
value: 87.96328637404433
- type: euclidean_ap
value: 83.78378095020464
- type: euclidean_f1
value: 75.94917787742901
- type: euclidean_precision
value: 73.78739471391229
- type: euclidean_recall
value: 78.24145364952264
- type: manhattan_accuracy
value: 87.99239337136648
- type: manhattan_ap
value: 83.72045889779073
- type: manhattan_f1
value: 75.93527315914488
- type: manhattan_precision
value: 73.30180567497851
- type: manhattan_recall
value: 78.76501385894672
- type: max_accuracy
value: 88.03896456708192
- type: max_ap
value: 84.0249558879327
- type: max_f1
value: 76.26290458870642
sentence-transformers/distiluse-base-multilingual-cased-v2
This is a sentence-transformers model: It maps sentences & paragraphs to a 512 dimensional dense vector space and can be used for tasks like clustering or semantic search.
Usage (Sentence-Transformers)
Using this model becomes easy when you have sentence-transformers installed:
pip install -U sentence-transformers
Then you can use the model like this:
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer('sentence-transformers/distiluse-base-multilingual-cased-v2')
embeddings = model.encode(sentences)
print(embeddings)
Evaluation Results
For an automated evaluation of this model, see the Sentence Embeddings Benchmark: https://seb.sbert.net
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: DistilBertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
(2): Dense({'in_features': 768, 'out_features': 512, 'bias': True, 'activation_function': 'torch.nn.modules.activation.Tanh'})
)
Citing & Authors
This model was trained by sentence-transformers.
If you find this model helpful, feel free to cite our publication Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks:
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "http://arxiv.org/abs/1908.10084",
}