binwang commited on
Commit
049dcfc
1 Parent(s): d99b604
Files changed (4) hide show
  1. DESCRIPTION.md +1 -0
  2. README.md +5 -8
  3. app.py +1790 -0
  4. requirements.txt +4 -0
DESCRIPTION.md ADDED
@@ -0,0 +1 @@
 
 
1
+ SeaEval Leaderboard.
README.md CHANGED
@@ -1,13 +1,10 @@
1
  ---
2
- title: SeaEval Leaderboard
3
- emoji: 📈
4
- colorFrom: pink
5
- colorTo: yellow
6
  sdk: gradio
7
- sdk_version: 4.11.0
8
  app_file: app.py
9
  pinned: false
10
- license: cc-by-nc-4.0
11
  ---
12
-
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
1
  ---
2
+ title: SeaEval Leaderboard
3
+ emoji: 🥇
4
+ colorFrom: blue
5
+ colorTo: indigo
6
  sdk: gradio
7
+ sdk_version: 4.0.2
8
  app_file: app.py
9
  pinned: false
 
10
  ---
 
 
app.py ADDED
@@ -0,0 +1,1790 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from functools import partial
2
+ import json
3
+
4
+ from datasets import load_dataset
5
+ import gradio as gr
6
+ from huggingface_hub import get_hf_file_metadata, HfApi, hf_hub_download, hf_hub_url
7
+ from huggingface_hub.repocard import metadata_load
8
+ import pandas as pd
9
+
10
+ TASKS = [
11
+ "BitextMining",
12
+ "Classification",
13
+ "Clustering",
14
+ "PairClassification",
15
+ "Reranking",
16
+ "Retrieval",
17
+ "STS",
18
+ "Summarization",
19
+ ]
20
+
21
+ TASK_LIST_BITEXT_MINING = ['BUCC (de-en)', 'BUCC (fr-en)', 'BUCC (ru-en)', 'BUCC (zh-en)', 'Tatoeba (afr-eng)', 'Tatoeba (amh-eng)', 'Tatoeba (ang-eng)', 'Tatoeba (ara-eng)', 'Tatoeba (arq-eng)', 'Tatoeba (arz-eng)', 'Tatoeba (ast-eng)', 'Tatoeba (awa-eng)', 'Tatoeba (aze-eng)', 'Tatoeba (bel-eng)', 'Tatoeba (ben-eng)', 'Tatoeba (ber-eng)', 'Tatoeba (bos-eng)', 'Tatoeba (bre-eng)', 'Tatoeba (bul-eng)', 'Tatoeba (cat-eng)', 'Tatoeba (cbk-eng)', 'Tatoeba (ceb-eng)', 'Tatoeba (ces-eng)', 'Tatoeba (cha-eng)', 'Tatoeba (cmn-eng)', 'Tatoeba (cor-eng)', 'Tatoeba (csb-eng)', 'Tatoeba (cym-eng)', 'Tatoeba (dan-eng)', 'Tatoeba (deu-eng)', 'Tatoeba (dsb-eng)', 'Tatoeba (dtp-eng)', 'Tatoeba (ell-eng)', 'Tatoeba (epo-eng)', 'Tatoeba (est-eng)', 'Tatoeba (eus-eng)', 'Tatoeba (fao-eng)', 'Tatoeba (fin-eng)', 'Tatoeba (fra-eng)', 'Tatoeba (fry-eng)', 'Tatoeba (gla-eng)', 'Tatoeba (gle-eng)', 'Tatoeba (glg-eng)', 'Tatoeba (gsw-eng)', 'Tatoeba (heb-eng)', 'Tatoeba (hin-eng)', 'Tatoeba (hrv-eng)', 'Tatoeba (hsb-eng)', 'Tatoeba (hun-eng)', 'Tatoeba (hye-eng)', 'Tatoeba (ido-eng)', 'Tatoeba (ile-eng)', 'Tatoeba (ina-eng)', 'Tatoeba (ind-eng)', 'Tatoeba (isl-eng)', 'Tatoeba (ita-eng)', 'Tatoeba (jav-eng)', 'Tatoeba (jpn-eng)', 'Tatoeba (kab-eng)', 'Tatoeba (kat-eng)', 'Tatoeba (kaz-eng)', 'Tatoeba (khm-eng)', 'Tatoeba (kor-eng)', 'Tatoeba (kur-eng)', 'Tatoeba (kzj-eng)', 'Tatoeba (lat-eng)', 'Tatoeba (lfn-eng)', 'Tatoeba (lit-eng)', 'Tatoeba (lvs-eng)', 'Tatoeba (mal-eng)', 'Tatoeba (mar-eng)', 'Tatoeba (max-eng)', 'Tatoeba (mhr-eng)', 'Tatoeba (mkd-eng)', 'Tatoeba (mon-eng)', 'Tatoeba (nds-eng)', 'Tatoeba (nld-eng)', 'Tatoeba (nno-eng)', 'Tatoeba (nob-eng)', 'Tatoeba (nov-eng)', 'Tatoeba (oci-eng)', 'Tatoeba (orv-eng)', 'Tatoeba (pam-eng)', 'Tatoeba (pes-eng)', 'Tatoeba (pms-eng)', 'Tatoeba (pol-eng)', 'Tatoeba (por-eng)', 'Tatoeba (ron-eng)', 'Tatoeba (rus-eng)', 'Tatoeba (slk-eng)', 'Tatoeba (slv-eng)', 'Tatoeba (spa-eng)', 'Tatoeba (sqi-eng)', 'Tatoeba (srp-eng)', 'Tatoeba (swe-eng)', 'Tatoeba (swg-eng)', 'Tatoeba (swh-eng)', 'Tatoeba (tam-eng)', 'Tatoeba (tat-eng)', 'Tatoeba (tel-eng)', 'Tatoeba (tgl-eng)', 'Tatoeba (tha-eng)', 'Tatoeba (tuk-eng)', 'Tatoeba (tur-eng)', 'Tatoeba (tzl-eng)', 'Tatoeba (uig-eng)', 'Tatoeba (ukr-eng)', 'Tatoeba (urd-eng)', 'Tatoeba (uzb-eng)', 'Tatoeba (vie-eng)', 'Tatoeba (war-eng)', 'Tatoeba (wuu-eng)', 'Tatoeba (xho-eng)', 'Tatoeba (yid-eng)', 'Tatoeba (yue-eng)', 'Tatoeba (zsm-eng)']
22
+ TASK_LIST_BITEXT_MINING_OTHER = ["BornholmBitextMining"]
23
+
24
+ TASK_LIST_CLASSIFICATION = [
25
+ "AmazonCounterfactualClassification (en)",
26
+ "AmazonPolarityClassification",
27
+ "AmazonReviewsClassification (en)",
28
+ "Banking77Classification",
29
+ "EmotionClassification",
30
+ "ImdbClassification",
31
+ "MassiveIntentClassification (en)",
32
+ "MassiveScenarioClassification (en)",
33
+ "MTOPDomainClassification (en)",
34
+ "MTOPIntentClassification (en)",
35
+ "ToxicConversationsClassification",
36
+ "TweetSentimentExtractionClassification",
37
+ ]
38
+
39
+ TASK_LIST_CLASSIFICATION_NORM = [x.replace(" (en)", "") for x in TASK_LIST_CLASSIFICATION]
40
+
41
+ TASK_LIST_CLASSIFICATION_DA = [
42
+ "AngryTweetsClassification",
43
+ "DanishPoliticalCommentsClassification",
44
+ "DKHateClassification",
45
+ "LccSentimentClassification",
46
+ "MassiveIntentClassification (da)",
47
+ "MassiveScenarioClassification (da)",
48
+ "NordicLangClassification",
49
+ "ScalaDaClassification",
50
+ ]
51
+
52
+ TASK_LIST_CLASSIFICATION_NB = [
53
+ "NoRecClassification",
54
+ "NordicLangClassification",
55
+ "NorwegianParliament",
56
+ "MassiveIntentClassification (nb)",
57
+ "MassiveScenarioClassification (nb)",
58
+ "ScalaNbClassification",
59
+ ]
60
+
61
+ TASK_LIST_CLASSIFICATION_PL = [
62
+ "AllegroReviews",
63
+ "CBD",
64
+ "MassiveIntentClassification (pl)",
65
+ "MassiveScenarioClassification (pl)",
66
+ "PAC",
67
+ "PolEmo2.0-IN",
68
+ "PolEmo2.0-OUT",
69
+ ]
70
+
71
+ TASK_LIST_CLASSIFICATION_SV = [
72
+ "DalajClassification",
73
+ "MassiveIntentClassification (sv)",
74
+ "MassiveScenarioClassification (sv)",
75
+ "NordicLangClassification",
76
+ "ScalaSvClassification",
77
+ "SweRecClassification",
78
+ ]
79
+
80
+ TASK_LIST_CLASSIFICATION_ZH = [
81
+ "AmazonReviewsClassification (zh)",
82
+ "IFlyTek",
83
+ "JDReview",
84
+ "MassiveIntentClassification (zh-CN)",
85
+ "MassiveScenarioClassification (zh-CN)",
86
+ "MultilingualSentiment",
87
+ "OnlineShopping",
88
+ "TNews",
89
+ "Waimai",
90
+ ]
91
+
92
+ TASK_LIST_CLASSIFICATION_OTHER = ['AmazonCounterfactualClassification (de)', 'AmazonCounterfactualClassification (ja)', 'AmazonReviewsClassification (de)', 'AmazonReviewsClassification (es)', 'AmazonReviewsClassification (fr)', 'AmazonReviewsClassification (ja)', 'AmazonReviewsClassification (zh)', 'MTOPDomainClassification (de)', 'MTOPDomainClassification (es)', 'MTOPDomainClassification (fr)', 'MTOPDomainClassification (hi)', 'MTOPDomainClassification (th)', 'MTOPIntentClassification (de)', 'MTOPIntentClassification (es)', 'MTOPIntentClassification (fr)', 'MTOPIntentClassification (hi)', 'MTOPIntentClassification (th)', 'MassiveIntentClassification (af)', 'MassiveIntentClassification (am)', 'MassiveIntentClassification (ar)', 'MassiveIntentClassification (az)', 'MassiveIntentClassification (bn)', 'MassiveIntentClassification (cy)', 'MassiveIntentClassification (de)', 'MassiveIntentClassification (el)', 'MassiveIntentClassification (es)', 'MassiveIntentClassification (fa)', 'MassiveIntentClassification (fi)', 'MassiveIntentClassification (fr)', 'MassiveIntentClassification (he)', 'MassiveIntentClassification (hi)', 'MassiveIntentClassification (hu)', 'MassiveIntentClassification (hy)', 'MassiveIntentClassification (id)', 'MassiveIntentClassification (is)', 'MassiveIntentClassification (it)', 'MassiveIntentClassification (ja)', 'MassiveIntentClassification (jv)', 'MassiveIntentClassification (ka)', 'MassiveIntentClassification (km)', 'MassiveIntentClassification (kn)', 'MassiveIntentClassification (ko)', 'MassiveIntentClassification (lv)', 'MassiveIntentClassification (ml)', 'MassiveIntentClassification (mn)', 'MassiveIntentClassification (ms)', 'MassiveIntentClassification (my)', 'MassiveIntentClassification (nl)', 'MassiveIntentClassification (pt)', 'MassiveIntentClassification (ro)', 'MassiveIntentClassification (ru)', 'MassiveIntentClassification (sl)', 'MassiveIntentClassification (sq)', 'MassiveIntentClassification (sw)', 'MassiveIntentClassification (ta)', 'MassiveIntentClassification (te)', 'MassiveIntentClassification (th)', 'MassiveIntentClassification (tl)', 'MassiveIntentClassification (tr)', 'MassiveIntentClassification (ur)', 'MassiveIntentClassification (vi)', 'MassiveIntentClassification (zh-TW)', 'MassiveScenarioClassification (af)', 'MassiveScenarioClassification (am)', 'MassiveScenarioClassification (ar)', 'MassiveScenarioClassification (az)', 'MassiveScenarioClassification (bn)', 'MassiveScenarioClassification (cy)', 'MassiveScenarioClassification (de)', 'MassiveScenarioClassification (el)', 'MassiveScenarioClassification (es)', 'MassiveScenarioClassification (fa)', 'MassiveScenarioClassification (fi)', 'MassiveScenarioClassification (fr)', 'MassiveScenarioClassification (he)', 'MassiveScenarioClassification (hi)', 'MassiveScenarioClassification (hu)', 'MassiveScenarioClassification (hy)', 'MassiveScenarioClassification (id)', 'MassiveScenarioClassification (is)', 'MassiveScenarioClassification (it)', 'MassiveScenarioClassification (ja)', 'MassiveScenarioClassification (jv)', 'MassiveScenarioClassification (ka)', 'MassiveScenarioClassification (km)', 'MassiveScenarioClassification (kn)', 'MassiveScenarioClassification (ko)', 'MassiveScenarioClassification (lv)', 'MassiveScenarioClassification (ml)', 'MassiveScenarioClassification (mn)', 'MassiveScenarioClassification (ms)', 'MassiveScenarioClassification (my)', 'MassiveScenarioClassification (nl)', 'MassiveScenarioClassification (pt)', 'MassiveScenarioClassification (ro)', 'MassiveScenarioClassification (ru)', 'MassiveScenarioClassification (sl)', 'MassiveScenarioClassification (sq)', 'MassiveScenarioClassification (sw)', 'MassiveScenarioClassification (ta)', 'MassiveScenarioClassification (te)', 'MassiveScenarioClassification (th)', 'MassiveScenarioClassification (tl)', 'MassiveScenarioClassification (tr)', 'MassiveScenarioClassification (ur)', 'MassiveScenarioClassification (vi)', 'MassiveScenarioClassification (zh-TW)']
93
+
94
+ TASK_LIST_CLUSTERING = [
95
+ "ArxivClusteringP2P",
96
+ "ArxivClusteringS2S",
97
+ "BiorxivClusteringP2P",
98
+ "BiorxivClusteringS2S",
99
+ "MedrxivClusteringP2P",
100
+ "MedrxivClusteringS2S",
101
+ "RedditClustering",
102
+ "RedditClusteringP2P",
103
+ "StackExchangeClustering",
104
+ "StackExchangeClusteringP2P",
105
+ "TwentyNewsgroupsClustering",
106
+ ]
107
+
108
+
109
+ TASK_LIST_CLUSTERING_DE = [
110
+ "BlurbsClusteringP2P",
111
+ "BlurbsClusteringS2S",
112
+ "TenKGnadClusteringP2P",
113
+ "TenKGnadClusteringS2S",
114
+ ]
115
+
116
+ TASK_LIST_CLUSTERING_PL = [
117
+ "8TagsClustering",
118
+ ]
119
+
120
+ TASK_LIST_CLUSTERING_ZH = [
121
+ "CLSClusteringP2P",
122
+ "CLSClusteringS2S",
123
+ "ThuNewsClusteringP2P",
124
+ "ThuNewsClusteringS2S",
125
+ ]
126
+
127
+ TASK_LIST_PAIR_CLASSIFICATION = [
128
+ "SprintDuplicateQuestions",
129
+ "TwitterSemEval2015",
130
+ "TwitterURLCorpus",
131
+ ]
132
+
133
+ TASK_LIST_PAIR_CLASSIFICATION_PL = [
134
+ "CDSC-E",
135
+ "PPC",
136
+ "PSC",
137
+ "SICK-E-PL",
138
+ ]
139
+
140
+ TASK_LIST_PAIR_CLASSIFICATION_ZH = [
141
+ "Cmnli",
142
+ "Ocnli",
143
+ ]
144
+
145
+ TASK_LIST_RERANKING = [
146
+ "AskUbuntuDupQuestions",
147
+ "MindSmallReranking",
148
+ "SciDocsRR",
149
+ "StackOverflowDupQuestions",
150
+ ]
151
+
152
+ TASK_LIST_RERANKING_ZH = [
153
+ "CMedQAv1",
154
+ "CMedQAv2",
155
+ "MMarcoReranking",
156
+ "T2Reranking",
157
+ ]
158
+
159
+ TASK_LIST_RETRIEVAL = [
160
+ "ArguAna",
161
+ "ClimateFEVER",
162
+ "CQADupstackRetrieval",
163
+ "DBPedia",
164
+ "FEVER",
165
+ "FiQA2018",
166
+ "HotpotQA",
167
+ "MSMARCO",
168
+ "NFCorpus",
169
+ "NQ",
170
+ "QuoraRetrieval",
171
+ "SCIDOCS",
172
+ "SciFact",
173
+ "Touche2020",
174
+ "TRECCOVID",
175
+ ]
176
+
177
+ TASK_LIST_RETRIEVAL_PL = [
178
+ "ArguAna-PL",
179
+ "DBPedia-PL",
180
+ "FiQA-PL",
181
+ "HotpotQA-PL",
182
+ "MSMARCO-PL",
183
+ "NFCorpus-PL",
184
+ "NQ-PL",
185
+ "Quora-PL",
186
+ "SCIDOCS-PL",
187
+ "SciFact-PL",
188
+ "TRECCOVID-PL",
189
+ ]
190
+
191
+ TASK_LIST_RETRIEVAL_ZH = [
192
+ "CmedqaRetrieval",
193
+ "CovidRetrieval",
194
+ "DuRetrieval",
195
+ "EcomRetrieval",
196
+ "MedicalRetrieval",
197
+ "MMarcoRetrieval",
198
+ "T2Retrieval",
199
+ "VideoRetrieval",
200
+ ]
201
+
202
+ TASK_LIST_RETRIEVAL_NORM = TASK_LIST_RETRIEVAL + [
203
+ "CQADupstackAndroidRetrieval",
204
+ "CQADupstackEnglishRetrieval",
205
+ "CQADupstackGamingRetrieval",
206
+ "CQADupstackGisRetrieval",
207
+ "CQADupstackMathematicaRetrieval",
208
+ "CQADupstackPhysicsRetrieval",
209
+ "CQADupstackProgrammersRetrieval",
210
+ "CQADupstackStatsRetrieval",
211
+ "CQADupstackTexRetrieval",
212
+ "CQADupstackUnixRetrieval",
213
+ "CQADupstackWebmastersRetrieval",
214
+ "CQADupstackWordpressRetrieval"
215
+ ]
216
+
217
+ TASK_LIST_STS = [
218
+ "BIOSSES",
219
+ "SICK-R",
220
+ "STS12",
221
+ "STS13",
222
+ "STS14",
223
+ "STS15",
224
+ "STS16",
225
+ "STS17 (en-en)",
226
+ "STS22 (en)",
227
+ "STSBenchmark",
228
+ ]
229
+
230
+ TASK_LIST_STS_PL = [
231
+ "CDSC-R",
232
+ "SICK-R-PL",
233
+ "STS22 (pl)",
234
+ ]
235
+
236
+ TASK_LIST_STS_ZH = [
237
+ "AFQMC",
238
+ "ATEC",
239
+ "BQ",
240
+ "LCQMC",
241
+ "PAWSX",
242
+ "QBQTC",
243
+ "STS22 (zh)",
244
+ "STSB",
245
+ ]
246
+
247
+ TASK_LIST_STS_OTHER = ["STS17 (ar-ar)", "STS17 (en-ar)", "STS17 (en-de)", "STS17 (en-tr)", "STS17 (es-en)", "STS17 (es-es)", "STS17 (fr-en)", "STS17 (it-en)", "STS17 (ko-ko)", "STS17 (nl-en)", "STS22 (ar)", "STS22 (de)", "STS22 (de-en)", "STS22 (de-fr)", "STS22 (de-pl)", "STS22 (es)", "STS22 (es-en)", "STS22 (es-it)", "STS22 (fr)", "STS22 (fr-pl)", "STS22 (it)", "STS22 (pl)", "STS22 (pl-en)", "STS22 (ru)", "STS22 (tr)", "STS22 (zh-en)", "STSBenchmark",]
248
+ TASK_LIST_STS_NORM = [x.replace(" (en)", "").replace(" (en-en)", "") for x in TASK_LIST_STS]
249
+
250
+ TASK_LIST_SUMMARIZATION = ["SummEval",]
251
+
252
+ TASK_LIST_EN = TASK_LIST_CLASSIFICATION + TASK_LIST_CLUSTERING + TASK_LIST_PAIR_CLASSIFICATION + TASK_LIST_RERANKING + TASK_LIST_RETRIEVAL + TASK_LIST_STS + TASK_LIST_SUMMARIZATION
253
+ TASK_LIST_PL = TASK_LIST_CLASSIFICATION_PL + TASK_LIST_CLUSTERING_PL + TASK_LIST_PAIR_CLASSIFICATION_PL + TASK_LIST_RETRIEVAL_PL + TASK_LIST_STS_PL
254
+ TASK_LIST_ZH = TASK_LIST_CLASSIFICATION_ZH + TASK_LIST_CLUSTERING_ZH + TASK_LIST_PAIR_CLASSIFICATION_ZH + TASK_LIST_RERANKING_ZH + TASK_LIST_RETRIEVAL_ZH + TASK_LIST_STS_ZH
255
+
256
+ TASK_TO_METRIC = {
257
+ "BitextMining": "f1",
258
+ "Clustering": "v_measure",
259
+ "Classification": "accuracy",
260
+ "PairClassification": "cos_sim_ap",
261
+ "Reranking": "map",
262
+ "Retrieval": "ndcg_at_10",
263
+ "STS": "cos_sim_spearman",
264
+ "Summarization": "cos_sim_spearman",
265
+ }
266
+
267
+ def make_clickable_model(model_name, link=None):
268
+ if link is None:
269
+ link = "https://huggingface.co/" + model_name
270
+ # Remove user from model name
271
+ return (
272
+ f'<a target="_blank" style="text-decoration: underline" href="{link}">{model_name.split("/")[-1]}</a>'
273
+ )
274
+
275
+ # Models without metadata, thus we cannot fetch their results naturally
276
+ EXTERNAL_MODELS = [
277
+ "all-MiniLM-L12-v2",
278
+ "all-MiniLM-L6-v2",
279
+ "all-mpnet-base-v2",
280
+ "allenai-specter",
281
+ "Baichuan-text-embedding",
282
+ "bert-base-swedish-cased",
283
+ "bert-base-uncased",
284
+ "bge-base-zh-v1.5",
285
+ "bge-large-zh-v1.5",
286
+ "bge-large-zh-noinstruct",
287
+ "bge-small-zh-v1.5",
288
+ "contriever-base-msmarco",
289
+ "cross-en-de-roberta-sentence-transformer",
290
+ "dfm-encoder-large-v1",
291
+ "dfm-sentence-encoder-large-1",
292
+ "distiluse-base-multilingual-cased-v2",
293
+ "DanskBERT",
294
+ "e5-base",
295
+ "e5-large",
296
+ "e5-small",
297
+ "electra-small-nordic",
298
+ "electra-small-swedish-cased-discriminator",
299
+ "gbert-base",
300
+ "gbert-large",
301
+ "gelectra-base",
302
+ "gelectra-large",
303
+ "gottbert-base",
304
+ "glove.6B.300d",
305
+ "gtr-t5-base",
306
+ "gtr-t5-large",
307
+ "gtr-t5-xl",
308
+ "gtr-t5-xxl",
309
+ "herbert-base-retrieval-v2",
310
+ "komninos",
311
+ "luotuo-bert-medium",
312
+ "LASER2",
313
+ "LaBSE",
314
+ "m3e-base",
315
+ "m3e-large",
316
+ "msmarco-bert-co-condensor",
317
+ "multilingual-e5-base",
318
+ "multilingual-e5-large",
319
+ "multilingual-e5-small",
320
+ "nb-bert-base",
321
+ "nb-bert-large",
322
+ "norbert3-base",
323
+ "norbert3-large",
324
+ "paraphrase-multilingual-MiniLM-L12-v2",
325
+ "paraphrase-multilingual-mpnet-base-v2",
326
+ "sentence-bert-swedish-cased",
327
+ "sentence-t5-base",
328
+ "sentence-t5-large",
329
+ "sentence-t5-xl",
330
+ "sentence-t5-xxl",
331
+ "sup-simcse-bert-base-uncased",
332
+ "st-polish-paraphrase-from-distilroberta",
333
+ "st-polish-paraphrase-from-mpnet",
334
+ "text2vec-base-chinese",
335
+ "text2vec-large-chinese",
336
+ "text-embedding-ada-002",
337
+ "text-similarity-ada-001",
338
+ "text-similarity-babbage-001",
339
+ "text-similarity-curie-001",
340
+ "text-similarity-davinci-001",
341
+ "text-search-ada-doc-001",
342
+ "text-search-ada-001",
343
+ "text-search-babbage-001",
344
+ "text-search-curie-001",
345
+ "text-search-davinci-001",
346
+ "titan-embed-text-v1",
347
+ "unsup-simcse-bert-base-uncased",
348
+ "use-cmlm-multilingual",
349
+ "voyage-lite-01-instruct",
350
+ "xlm-roberta-base",
351
+ "xlm-roberta-large",
352
+ ]
353
+
354
+ EXTERNAL_MODEL_TO_LINK = {
355
+ "allenai-specter": "https://huggingface.co/sentence-transformers/allenai-specter",
356
+ "allenai-specter": "https://huggingface.co/sentence-transformers/allenai-specter",
357
+ "all-MiniLM-L12-v2": "https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2",
358
+ "all-MiniLM-L6-v2": "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2",
359
+ "all-mpnet-base-v2": "https://huggingface.co/sentence-transformers/all-mpnet-base-v2",
360
+ "Baichuan-text-embedding": "https://platform.baichuan-ai.com/docs/text-Embedding",
361
+ "bert-base-swedish-cased": "https://huggingface.co/KB/bert-base-swedish-cased",
362
+ "bert-base-uncased": "https://huggingface.co/bert-base-uncased",
363
+ "bge-base-zh-v1.5": "https://huggingface.co/BAAI/bge-base-zh-v1.5",
364
+ "bge-large-zh-v1.5": "https://huggingface.co/BAAI/bge-large-zh-v1.5",
365
+ "bge-large-zh-noinstruct": "https://huggingface.co/BAAI/bge-large-zh-noinstruct",
366
+ "bge-small-zh-v1.5": "https://huggingface.co/BAAI/bge-small-zh-v1.5",
367
+ "contriever-base-msmarco": "https://huggingface.co/nthakur/contriever-base-msmarco",
368
+ "cross-en-de-roberta-sentence-transformer": "https://huggingface.co/T-Systems-onsite/cross-en-de-roberta-sentence-transformer",
369
+ "DanskBERT": "https://huggingface.co/vesteinn/DanskBERT",
370
+ "distiluse-base-multilingual-cased-v2": "https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v2",
371
+ "dfm-encoder-large-v1": "https://huggingface.co/chcaa/dfm-encoder-large-v1",
372
+ "dfm-sentence-encoder-large-1": "https://huggingface.co/chcaa/dfm-encoder-large-v1",
373
+ "e5-base": "https://huggingface.co/intfloat/e5-base",
374
+ "e5-large": "https://huggingface.co/intfloat/e5-large",
375
+ "e5-small": "https://huggingface.co/intfloat/e5-small",
376
+ "electra-small-nordic": "https://huggingface.co/jonfd/electra-small-nordic",
377
+ "electra-small-swedish-cased-discriminator": "https://huggingface.co/KBLab/electra-small-swedish-cased-discriminator",
378
+ "gbert-base": "https://huggingface.co/deepset/gbert-base",
379
+ "gbert-large": "https://huggingface.co/deepset/gbert-large",
380
+ "gelectra-base": "https://huggingface.co/deepset/gelectra-base",
381
+ "gelectra-large": "https://huggingface.co/deepset/gelectra-large",
382
+ "glove.6B.300d": "https://huggingface.co/sentence-transformers/average_word_embeddings_glove.6B.300d",
383
+ "gottbert-base": "https://huggingface.co/uklfr/gottbert-base",
384
+ "gtr-t5-base": "https://huggingface.co/sentence-transformers/gtr-t5-base",
385
+ "gtr-t5-large": "https://huggingface.co/sentence-transformers/gtr-t5-large",
386
+ "gtr-t5-xl": "https://huggingface.co/sentence-transformers/gtr-t5-xl",
387
+ "gtr-t5-xxl": "https://huggingface.co/sentence-transformers/gtr-t5-xxl",
388
+ "herbert-base-retrieval-v2": "https://huggingface.co/ipipan/herbert-base-retrieval-v2",
389
+ "komninos": "https://huggingface.co/sentence-transformers/average_word_embeddings_komninos",
390
+ "luotuo-bert-medium": "https://huggingface.co/silk-road/luotuo-bert-medium",
391
+ "LASER2": "https://github.com/facebookresearch/LASER",
392
+ "LaBSE": "https://huggingface.co/sentence-transformers/LaBSE",
393
+ "m3e-base": "https://huggingface.co/moka-ai/m3e-base",
394
+ "m3e-large": "https://huggingface.co/moka-ai/m3e-large",
395
+ "msmarco-bert-co-condensor": "https://huggingface.co/sentence-transformers/msmarco-bert-co-condensor",
396
+ "multilingual-e5-base": "https://huggingface.co/intfloat/multilingual-e5-base",
397
+ "multilingual-e5-large": "https://huggingface.co/intfloat/multilingual-e5-large",
398
+ "multilingual-e5-small": "https://huggingface.co/intfloat/multilingual-e5-small",
399
+ "nb-bert-base": "https://huggingface.co/NbAiLab/nb-bert-base",
400
+ "nb-bert-large": "https://huggingface.co/NbAiLab/nb-bert-large",
401
+ "norbert3-base": "https://huggingface.co/ltg/norbert3-base",
402
+ "norbert3-large": "https://huggingface.co/ltg/norbert3-large",
403
+ "paraphrase-multilingual-mpnet-base-v2": "https://huggingface.co/sentence-transformers/paraphrase-multilingual-mpnet-base-v2",
404
+ "paraphrase-multilingual-MiniLM-L12-v2": "https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
405
+ "sentence-bert-swedish-cased": "https://huggingface.co/KBLab/sentence-bert-swedish-cased",
406
+ "sentence-t5-base": "https://huggingface.co/sentence-transformers/sentence-t5-base",
407
+ "sentence-t5-large": "https://huggingface.co/sentence-transformers/sentence-t5-large",
408
+ "sentence-t5-xl": "https://huggingface.co/sentence-transformers/sentence-t5-xl",
409
+ "sentence-t5-xxl": "https://huggingface.co/sentence-transformers/sentence-t5-xxl",
410
+ "sup-simcse-bert-base-uncased": "https://huggingface.co/princeton-nlp/sup-simcse-bert-base-uncased",
411
+ "st-polish-paraphrase-from-distilroberta": "https://huggingface.co/sdadas/st-polish-paraphrase-from-distilroberta",
412
+ "st-polish-paraphrase-from-mpnet": "https://huggingface.co/sdadas/st-polish-paraphrase-from-mpnet",
413
+ "text2vec-base-chinese": "https://huggingface.co/shibing624/text2vec-base-chinese",
414
+ "text2vec-large-chinese": "https://huggingface.co/GanymedeNil/text2vec-large-chinese",
415
+ "text-embedding-ada-002": "https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models",
416
+ "text-similarity-ada-001": "https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models",
417
+ "text-similarity-babbage-001": "https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models",
418
+ "text-similarity-curie-001": "https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models",
419
+ "text-similarity-davinci-001": "https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models",
420
+ "text-search-ada-doc-001": "https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models",
421
+ "text-search-ada-query-001": "https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models",
422
+ "text-search-ada-001": "https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models",
423
+ "text-search-curie-001": "https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models",
424
+ "text-search-babbage-001": "https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models",
425
+ "text-search-davinci-001": "https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models",
426
+ "titan-embed-text-v1": "https://docs.aws.amazon.com/bedrock/latest/userguide/embeddings.html",
427
+ "unsup-simcse-bert-base-uncased": "https://huggingface.co/princeton-nlp/unsup-simcse-bert-base-uncased",
428
+ "use-cmlm-multilingual": "https://huggingface.co/sentence-transformers/use-cmlm-multilingual",
429
+ "voyage-lite-01-instruct": "https://docs.voyageai.com/embeddings/",
430
+ "xlm-roberta-base": "https://huggingface.co/xlm-roberta-base",
431
+ "xlm-roberta-large": "https://huggingface.co/xlm-roberta-large",
432
+ }
433
+
434
+ EXTERNAL_MODEL_TO_DIM = {
435
+ "all-MiniLM-L12-v2": 384,
436
+ "all-MiniLM-L6-v2": 384,
437
+ "all-mpnet-base-v2": 768,
438
+ "allenai-specter": 768,
439
+ "Baichuan-text-embedding": 1024,
440
+ "bert-base-swedish-cased": 768,
441
+ "bert-base-uncased": 768,
442
+ "bge-base-zh-v1.5": 768,
443
+ "bge-large-zh-v1.5": 1024,
444
+ "bge-large-zh-noinstruct": 1024,
445
+ "bge-small-zh-v1.5": 512,
446
+ "contriever-base-msmarco": 768,
447
+ "cross-en-de-roberta-sentence-transformer": 768,
448
+ "DanskBERT": 768,
449
+ "distiluse-base-multilingual-cased-v2": 512,
450
+ "dfm-encoder-large-v1": 1024,
451
+ "dfm-sentence-encoder-large-1": 1024,
452
+ "e5-base": 768,
453
+ "e5-small": 384,
454
+ "e5-large": 1024,
455
+ "electra-small-nordic": 256,
456
+ "electra-small-swedish-cased-discriminator": 256,
457
+ "luotuo-bert-medium": 768,
458
+ "LASER2": 1024,
459
+ "LaBSE": 768,
460
+ "gbert-base": 768,
461
+ "gbert-large": 1024,
462
+ "gelectra-base": 768,
463
+ "gelectra-large": 1024,
464
+ "glove.6B.300d": 300,
465
+ "gottbert-base": 768,
466
+ "gtr-t5-base": 768,
467
+ "gtr-t5-large": 768,
468
+ "gtr-t5-xl": 768,
469
+ "gtr-t5-xxl": 768,
470
+ "herbert-base-retrieval-v2": 768,
471
+ "komninos": 300,
472
+ "m3e-base": 768,
473
+ "m3e-large": 768,
474
+ "msmarco-bert-co-condensor": 768,
475
+ "multilingual-e5-base": 768,
476
+ "multilingual-e5-small": 384,
477
+ "multilingual-e5-large": 1024,
478
+ "nb-bert-base": 768,
479
+ "nb-bert-large": 1024,
480
+ "norbert3-base": 768,
481
+ "norbert3-large": 1024,
482
+ "paraphrase-multilingual-MiniLM-L12-v2": 384,
483
+ "paraphrase-multilingual-mpnet-base-v2": 768,
484
+ "sentence-bert-swedish-cased": 768,
485
+ "sentence-t5-base": 768,
486
+ "sentence-t5-large": 768,
487
+ "sentence-t5-xl": 768,
488
+ "sentence-t5-xxl": 768,
489
+ "sup-simcse-bert-base-uncased": 768,
490
+ "st-polish-paraphrase-from-distilroberta": 768,
491
+ "st-polish-paraphrase-from-mpnet": 768,
492
+ "text2vec-base-chinese": 768,
493
+ "text2vec-large-chinese": 1024,
494
+ "text-embedding-ada-002": 1536,
495
+ "text-similarity-ada-001": 1024,
496
+ "text-similarity-babbage-001": 2048,
497
+ "text-similarity-curie-001": 4096,
498
+ "text-similarity-davinci-001": 12288,
499
+ "text-search-ada-doc-001": 1024,
500
+ "text-search-ada-query-001": 1024,
501
+ "text-search-ada-001": 1024,
502
+ "text-search-babbage-001": 2048,
503
+ "text-search-curie-001": 4096,
504
+ "text-search-davinci-001": 12288,
505
+ "titan-embed-text-v1": 1536,
506
+ "unsup-simcse-bert-base-uncased": 768,
507
+ "use-cmlm-multilingual": 768,
508
+ "voyage-lite-01-instruct": 1024,
509
+ "xlm-roberta-base": 768,
510
+ "xlm-roberta-large": 1024,
511
+ }
512
+
513
+ EXTERNAL_MODEL_TO_SEQLEN = {
514
+ "all-MiniLM-L12-v2": 512,
515
+ "all-MiniLM-L6-v2": 512,
516
+ "all-mpnet-base-v2": 514,
517
+ "allenai-specter": 512,
518
+ "Baichuan-text-embedding": 512,
519
+ "bert-base-swedish-cased": 512,
520
+ "bert-base-uncased": 512,
521
+ "bge-base-zh-v1.5": 512,
522
+ "bge-large-zh-v1.5": 512,
523
+ "bge-large-zh-noinstruct": 512,
524
+ "bge-small-zh-v1.5": 512,
525
+ "contriever-base-msmarco": 512,
526
+ "cross-en-de-roberta-sentence-transformer": 514,
527
+ "DanskBERT": 514,
528
+ "dfm-encoder-large-v1": 512,
529
+ "dfm-sentence-encoder-large-1": 512,
530
+ "distiluse-base-multilingual-cased-v2": 512,
531
+ "e5-base": 512,
532
+ "e5-large": 512,
533
+ "e5-small": 512,
534
+ "electra-small-nordic": 512,
535
+ "electra-small-swedish-cased-discriminator": 512,
536
+ "gbert-base": 512,
537
+ "gbert-large": 512,
538
+ "gelectra-base": 512,
539
+ "gelectra-large": 512,
540
+ "gottbert-base": 512,
541
+ "glove.6B.300d": "N/A",
542
+ "gtr-t5-base": 512,
543
+ "gtr-t5-large": 512,
544
+ "gtr-t5-xl": 512,
545
+ "gtr-t5-xxl": 512,
546
+ "herbert-base-retrieval-v2": 514,
547
+ "komninos": "N/A",
548
+ "luotuo-bert-medium": 512,
549
+ "LASER2": "N/A",
550
+ "LaBSE": 512,
551
+ "m3e-base": 512,
552
+ "m3e-large": 512,
553
+ "msmarco-bert-co-condensor": 512,
554
+ "multilingual-e5-base": 514,
555
+ "multilingual-e5-large": 514,
556
+ "multilingual-e5-small": 512,
557
+ "nb-bert-base": 512,
558
+ "nb-bert-large": 512,
559
+ "norbert3-base": 512,
560
+ "norbert3-large": 512,
561
+ "paraphrase-multilingual-MiniLM-L12-v2": 512,
562
+ "paraphrase-multilingual-mpnet-base-v2": 514,
563
+ "sentence-bert-swedish-cased": 512,
564
+ "sentence-t5-base": 512,
565
+ "sentence-t5-large": 512,
566
+ "sentence-t5-xl": 512,
567
+ "sentence-t5-xxl": 512,
568
+ "sup-simcse-bert-base-uncased": 512,
569
+ "st-polish-paraphrase-from-distilroberta": 514,
570
+ "st-polish-paraphrase-from-mpnet": 514,
571
+ "text2vec-base-chinese": 512,
572
+ "text2vec-large-chinese": 512,
573
+ "text-embedding-ada-002": 8191,
574
+ "text-similarity-ada-001": 2046,
575
+ "text-similarity-babbage-001": 2046,
576
+ "text-similarity-curie-001": 2046,
577
+ "text-similarity-davinci-001": 2046,
578
+ "text-search-ada-doc-001": 2046,
579
+ "text-search-ada-query-001": 2046,
580
+ "text-search-ada-001": 2046,
581
+ "text-search-babbage-001": 2046,
582
+ "text-search-curie-001": 2046,
583
+ "text-search-davinci-001": 2046,
584
+ "titan-embed-text-v1": 8000,
585
+ "use-cmlm-multilingual": 512,
586
+ "unsup-simcse-bert-base-uncased": 512,
587
+ "voyage-lite-01-instruct": 4096,
588
+ "xlm-roberta-base": 514,
589
+ "xlm-roberta-large": 514,
590
+ }
591
+
592
+ EXTERNAL_MODEL_TO_SIZE = {
593
+ "allenai-specter": 0.44,
594
+ "all-MiniLM-L12-v2": 0.13,
595
+ "all-MiniLM-L6-v2": 0.09,
596
+ "all-mpnet-base-v2": 0.44,
597
+ "bert-base-uncased": 0.44,
598
+ "bert-base-swedish-cased": 0.50,
599
+ "bge-base-zh-v1.5": 0.41,
600
+ "bge-large-zh-v1.5": 1.30,
601
+ "bge-large-zh-noinstruct": 1.30,
602
+ "bge-small-zh-v1.5": 0.10,
603
+ "cross-en-de-roberta-sentence-transformer": 1.11,
604
+ "contriever-base-msmarco": 0.44,
605
+ "DanskBERT": 0.50,
606
+ "distiluse-base-multilingual-cased-v2": 0.54,
607
+ "dfm-encoder-large-v1": 1.42,
608
+ "dfm-sentence-encoder-large-1": 1.63,
609
+ "e5-base": 0.44,
610
+ "e5-small": 0.13,
611
+ "e5-large": 1.34,
612
+ "electra-small-nordic": 0.09,
613
+ "electra-small-swedish-cased-discriminator": 0.06,
614
+ "gbert-base": 0.44,
615
+ "gbert-large": 1.35,
616
+ "gelectra-base": 0.44,
617
+ "gelectra-large": 1.34,
618
+ "glove.6B.300d": 0.48,
619
+ "gottbert-base": 0.51,
620
+ "gtr-t5-base": 0.22,
621
+ "gtr-t5-large": 0.67,
622
+ "gtr-t5-xl": 2.48,
623
+ "gtr-t5-xxl": 9.73,
624
+ "herbert-base-retrieval-v2": 0.50,
625
+ "komninos": 0.27,
626
+ "luotuo-bert-medium": 1.31,
627
+ "LASER2": 0.17,
628
+ "LaBSE": 1.88,
629
+ "m3e-base": 0.41,
630
+ "m3e-large": 0.41,
631
+ "msmarco-bert-co-condensor": 0.44,
632
+ "multilingual-e5-base": 1.11,
633
+ "multilingual-e5-small": 0.47,
634
+ "multilingual-e5-large": 2.24,
635
+ "nb-bert-base": 0.71,
636
+ "nb-bert-large": 1.42,
637
+ "norbert3-base": 0.52,
638
+ "norbert3-large": 1.47,
639
+ "paraphrase-multilingual-mpnet-base-v2": 1.11,
640
+ "paraphrase-multilingual-MiniLM-L12-v2": 0.47,
641
+ "sentence-bert-swedish-cased": 0.50,
642
+ "sentence-t5-base": 0.22,
643
+ "sentence-t5-large": 0.67,
644
+ "sentence-t5-xl": 2.48,
645
+ "sentence-t5-xxl": 9.73,
646
+ "sup-simcse-bert-base-uncased": 0.44,
647
+ "st-polish-paraphrase-from-distilroberta": 0.50,
648
+ "st-polish-paraphrase-from-mpnet": 0.50,
649
+ "text2vec-base-chinese": 0.41,
650
+ "text2vec-large-chinese": 1.30,
651
+ "unsup-simcse-bert-base-uncased": 0.44,
652
+ "use-cmlm-multilingual": 1.89,
653
+ "xlm-roberta-base": 1.12,
654
+ "xlm-roberta-large": 2.24,
655
+ }
656
+
657
+ MODELS_TO_SKIP = {
658
+ "baseplate/instructor-large-1", # Duplicate
659
+ "radames/e5-large", # Duplicate
660
+ "gentlebowl/instructor-large-safetensors", # Duplicate
661
+ "Consensus/instructor-base", # Duplicate
662
+ "GovCompete/instructor-xl", # Duplicate
663
+ "GovCompete/e5-large-v2", # Duplicate
664
+ "t12e/instructor-base", # Duplicate
665
+ "michaelfeil/ct2fast-e5-large-v2",
666
+ "michaelfeil/ct2fast-e5-large",
667
+ "michaelfeil/ct2fast-e5-small-v2",
668
+ "newsrx/instructor-xl-newsrx",
669
+ "newsrx/instructor-large-newsrx",
670
+ "fresha/e5-large-v2-endpoint",
671
+ "ggrn/e5-small-v2",
672
+ "michaelfeil/ct2fast-e5-small",
673
+ "jncraton/e5-small-v2-ct2-int8",
674
+ "anttip/ct2fast-e5-small-v2-hfie",
675
+ "newsrx/instructor-large",
676
+ "newsrx/instructor-xl",
677
+ "dmlls/all-mpnet-base-v2",
678
+ "cgldo/semanticClone",
679
+ "Malmuk1/e5-large-v2_Sharded",
680
+ "jncraton/gte-small-ct2-int8",
681
+ "Einas/einas_ashkar",
682
+ "gruber/e5-small-v2-ggml",
683
+ "jncraton/bge-small-en-ct2-int8",
684
+ "vectoriseai/bge-small-en",
685
+ "recipe/embeddings",
686
+ "dhairya0907/thenlper-get-large",
687
+ "Narsil/bge-base-en",
688
+ "kozistr/fused-large-en",
689
+ "sionic-ai/sionic-ai-v2", # Wait for https://huggingface.co/sionic-ai/sionic-ai-v2/discussions/1
690
+ "sionic-ai/sionic-ai-v1", # Wait for https://huggingface.co/sionic-ai/sionic-ai-v2/discussions/1
691
+ "BAAI/bge-large-en", # Deprecated in favor of v1.5
692
+ "BAAI/bge-base-en", # Deprecated in favor of v1.5
693
+ "BAAI/bge-small-en", # Deprecated in favor of v1.5
694
+ "d0rj/e5-large-en-ru",
695
+ "d0rj/e5-base-en-ru",
696
+ "d0rj/e5-small-en-ru",
697
+ "aident-ai/bge-base-en-onnx",
698
+ "barisaydin/bge-base-en",
699
+ "barisaydin/gte-large",
700
+ "barisaydin/gte-base",
701
+ "barisaydin/gte-small",
702
+ "barisaydin/bge-small-en",
703
+ "odunola/e5-base-v2",
704
+ "goldenrooster/multilingual-e5-large",
705
+ "davidpeer/gte-small",
706
+ "barisaydin/bge-large-en",
707
+ "jamesgpt1/english-large-v1",
708
+ "vectoriseai/bge-large-en-v1.5",
709
+ "vectoriseai/bge-base-en-v1.5",
710
+ "vectoriseai/instructor-large",
711
+ "vectoriseai/instructor-base",
712
+ "vectoriseai/gte-large",
713
+ "vectoriseai/gte-base",
714
+ "vectoriseai/e5-large-v2",
715
+ "vectoriseai/bge-small-en-v1.5",
716
+ "vectoriseai/e5-base-v2",
717
+ "vectoriseai/e5-large",
718
+ "vectoriseai/multilingual-e5-large",
719
+ "vectoriseai/gte-small",
720
+ "vectoriseai/ember-v1",
721
+ "vectoriseai/e5-base",
722
+ "vectoriseai/e5-small-v2",
723
+ "michaelfeil/ct2fast-bge-large-en-v1.5",
724
+ "michaelfeil/ct2fast-bge-large-en-v1.5",
725
+ "michaelfeil/ct2fast-bge-base-en-v1.5",
726
+ "michaelfeil/ct2fast-gte-large",
727
+ "michaelfeil/ct2fast-gte-base",
728
+ "michaelfeil/ct2fast-bge-small-en-v1.5",
729
+ "rizki/bgr-tf",
730
+ "ef-zulla/e5-multi-sml-torch",
731
+ "cherubhao/yogamodel",
732
+ "morgendigital/multilingual-e5-large-quantized",
733
+ "jncraton/gte-tiny-ct2-int8",
734
+ "Research2NLP/electrical_stella",
735
+ "Intel/bge-base-en-v1.5-sts-int8-static",
736
+ "Intel/bge-base-en-v1.5-sts-int8-dynamic",
737
+ "Intel/bge-base-en-v1.5-sst2",
738
+ "Intel/bge-base-en-v1.5-sst2-int8-static",
739
+ "Intel/bge-base-en-v1.5-sst2-int8-dynamic",
740
+ "Intel/bge-small-en-v1.5-sst2",
741
+ "Intel/bge-small-en-v1.5-sst2-int8-dynamic",
742
+ "Intel/bge-small-en-v1.5-sst2-int8-static",
743
+ "binqiangliu/EmbeddingModlebgelargeENv1.5",
744
+ "DecisionOptimizationSystem/DeepFeatEmbeddingLargeContext",
745
+ "woody72/multilingual-e5-base",
746
+ "Severian/embed",
747
+ "Frazic/udever-bloom-3b-sentence",
748
+ "jamesgpt1/zzz",
749
+ }
750
+
751
+ EXTERNAL_MODEL_RESULTS = {model: {k: {v: []} for k, v in TASK_TO_METRIC.items()} for model in EXTERNAL_MODELS}
752
+
753
+ def add_lang(examples):
754
+ if not(examples["eval_language"]):
755
+ examples["mteb_dataset_name_with_lang"] = examples["mteb_dataset_name"]
756
+ else:
757
+ examples["mteb_dataset_name_with_lang"] = examples["mteb_dataset_name"] + f' ({examples["eval_language"]})'
758
+ return examples
759
+
760
+ def add_task(examples):
761
+ # Could be added to the dataset loading script instead
762
+ if examples["mteb_dataset_name"] in TASK_LIST_CLASSIFICATION_NORM + TASK_LIST_CLASSIFICATION_DA + TASK_LIST_CLASSIFICATION_NB + TASK_LIST_CLASSIFICATION_PL + TASK_LIST_CLASSIFICATION_SV + TASK_LIST_CLASSIFICATION_ZH:
763
+ examples["mteb_task"] = "Classification"
764
+ elif examples["mteb_dataset_name"] in TASK_LIST_CLUSTERING + TASK_LIST_CLUSTERING_DE + TASK_LIST_CLUSTERING_PL + TASK_LIST_CLUSTERING_ZH:
765
+ examples["mteb_task"] = "Clustering"
766
+ elif examples["mteb_dataset_name"] in TASK_LIST_PAIR_CLASSIFICATION + TASK_LIST_PAIR_CLASSIFICATION_PL + TASK_LIST_PAIR_CLASSIFICATION_ZH:
767
+ examples["mteb_task"] = "PairClassification"
768
+ elif examples["mteb_dataset_name"] in TASK_LIST_RERANKING + TASK_LIST_RERANKING_ZH:
769
+ examples["mteb_task"] = "Reranking"
770
+ elif examples["mteb_dataset_name"] in TASK_LIST_RETRIEVAL_NORM + TASK_LIST_RETRIEVAL_PL + TASK_LIST_RETRIEVAL_ZH:
771
+ examples["mteb_task"] = "Retrieval"
772
+ elif examples["mteb_dataset_name"] in TASK_LIST_STS_NORM + TASK_LIST_STS_PL + TASK_LIST_STS_ZH:
773
+ examples["mteb_task"] = "STS"
774
+ elif examples["mteb_dataset_name"] in TASK_LIST_SUMMARIZATION:
775
+ examples["mteb_task"] = "Summarization"
776
+ elif examples["mteb_dataset_name"] in [x.split(" ")[0] for x in TASK_LIST_BITEXT_MINING + TASK_LIST_BITEXT_MINING_OTHER]:
777
+ examples["mteb_task"] = "BitextMining"
778
+ else:
779
+ print("WARNING: Task not found for dataset", examples["mteb_dataset_name"])
780
+ examples["mteb_task"] = "Unknown"
781
+ return examples
782
+
783
+ for model in EXTERNAL_MODELS:
784
+ ds = load_dataset("mteb/results", model)
785
+ # For local debugging:
786
+ #, download_mode='force_redownload', verification_mode="no_checks")
787
+ ds = ds.map(add_lang)
788
+ ds = ds.map(add_task)
789
+ base_dict = {"Model": make_clickable_model(model, link=EXTERNAL_MODEL_TO_LINK.get(model, "https://huggingface.co/spaces/mteb/leaderboard"))}
790
+ # For now only one metric per task - Could add more metrics lateron
791
+ for task, metric in TASK_TO_METRIC.items():
792
+ ds_dict = ds.filter(lambda x: (x["mteb_task"] == task) and (x["metric"] == metric))["test"].to_dict()
793
+ ds_dict = {k: round(v, 2) for k, v in zip(ds_dict["mteb_dataset_name_with_lang"], ds_dict["score"])}
794
+ EXTERNAL_MODEL_RESULTS[model][task][metric].append({**base_dict, **ds_dict})
795
+
796
+ def get_dim_seq_size(model):
797
+ filenames = [sib.rfilename for sib in model.siblings]
798
+ dim, seq, size = "", "", ""
799
+ if "1_Pooling/config.json" in filenames:
800
+ st_config_path = hf_hub_download(model.modelId, filename="1_Pooling/config.json")
801
+ dim = json.load(open(st_config_path)).get("word_embedding_dimension", "")
802
+ elif "2_Pooling/config.json" in filenames:
803
+ st_config_path = hf_hub_download(model.modelId, filename="2_Pooling/config.json")
804
+ dim = json.load(open(st_config_path)).get("word_embedding_dimension", "")
805
+ if "config.json" in filenames:
806
+ config_path = hf_hub_download(model.modelId, filename="config.json")
807
+ config = json.load(open(config_path))
808
+ if not dim:
809
+ dim = config.get("hidden_dim", config.get("hidden_size", config.get("d_model", "")))
810
+ seq = config.get("n_positions", config.get("max_position_embeddings", config.get("n_ctx", config.get("seq_length", ""))))
811
+ # Get model file size without downloading
812
+ if "pytorch_model.bin" in filenames:
813
+ url = hf_hub_url(model.modelId, filename="pytorch_model.bin")
814
+ meta = get_hf_file_metadata(url)
815
+ size = round(meta.size / 1e9, 2)
816
+ elif "pytorch_model.bin.index.json" in filenames:
817
+ index_path = hf_hub_download(model.modelId, filename="pytorch_model.bin.index.json")
818
+ """
819
+ {
820
+ "metadata": {
821
+ "total_size": 28272820224
822
+ },....
823
+ """
824
+ size = json.load(open(index_path))
825
+ if ("metadata" in size) and ("total_size" in size["metadata"]):
826
+ size = round(size["metadata"]["total_size"] / 1e9, 2)
827
+ elif "model.safetensors" in filenames:
828
+ url = hf_hub_url(model.modelId, filename="model.safetensors")
829
+ meta = get_hf_file_metadata(url)
830
+ size = round(meta.size / 1e9, 2)
831
+ return dim, seq, size
832
+
833
+ def make_datasets_clickable(df):
834
+ """Does not work"""
835
+ if "BornholmBitextMining" in df.columns:
836
+ link = "https://huggingface.co/datasets/strombergnlp/bornholmsk_parallel"
837
+ df = df.rename(
838
+ columns={f'BornholmBitextMining': '<a target="_blank" style="text-decoration: underline" href="{link}">BornholmBitextMining</a>',})
839
+ return df
840
+
841
+ def add_rank(df):
842
+ cols_to_rank = [col for col in df.columns if col not in ["Model", "Model Size (GB)", "Embedding Dimensions", "Sequence Length"]]
843
+ if len(cols_to_rank) == 1:
844
+ df.sort_values(cols_to_rank[0], ascending=False, inplace=True)
845
+ else:
846
+ df.insert(1, "Average", df[cols_to_rank].mean(axis=1, skipna=False))
847
+ df.sort_values("Average", ascending=False, inplace=True)
848
+ df.insert(0, "Rank", list(range(1, len(df) + 1)))
849
+ df = df.round(2)
850
+ # Fill NaN after averaging
851
+ df.fillna("", inplace=True)
852
+ return df
853
+
854
+ def get_mteb_data(tasks=["Clustering"], langs=[], datasets=[], fillna=True, add_emb_dim=False, task_to_metric=TASK_TO_METRIC, rank=True):
855
+ api = HfApi()
856
+ models = api.list_models(filter="mteb")
857
+ # Initialize list to models that we cannot fetch metadata from
858
+ df_list = []
859
+ for model in EXTERNAL_MODEL_RESULTS:
860
+ results_list = [res for task in tasks for res in EXTERNAL_MODEL_RESULTS[model][task][task_to_metric[task]]]
861
+ if len(datasets) > 0:
862
+ res = {k: v for d in results_list for k, v in d.items() if (k == "Model") or any([x in k for x in datasets])}
863
+ elif langs:
864
+ # Would be cleaner to rely on an extra language column instead
865
+ langs_format = [f"({lang})" for lang in langs]
866
+ res = {k: v for d in results_list for k, v in d.items() if any([k.split(" ")[-1] in (k, x) for x in langs_format])}
867
+ else:
868
+ res = {k: v for d in results_list for k, v in d.items()}
869
+ # Model & at least one result
870
+ if len(res) > 1:
871
+ if add_emb_dim:
872
+ res["Model Size (GB)"] = EXTERNAL_MODEL_TO_SIZE.get(model, "")
873
+ res["Embedding Dimensions"] = EXTERNAL_MODEL_TO_DIM.get(model, "")
874
+ res["Sequence Length"] = EXTERNAL_MODEL_TO_SEQLEN.get(model, "")
875
+ df_list.append(res)
876
+
877
+ for model in models:
878
+ if model.modelId in MODELS_TO_SKIP: continue
879
+ print("MODEL", model)
880
+ readme_path = hf_hub_download(model.modelId, filename="README.md")
881
+ meta = metadata_load(readme_path)
882
+ # meta['model-index'][0]["results"] is list of elements like:
883
+ # {
884
+ # "task": {"type": "Classification"},
885
+ # "dataset": {
886
+ # "type": "mteb/amazon_massive_intent",
887
+ # "name": "MTEB MassiveIntentClassification (nb)",
888
+ # "config": "nb",
889
+ # "split": "test",
890
+ # },
891
+ # "metrics": [
892
+ # {"type": "accuracy", "value": 39.81506388702084},
893
+ # {"type": "f1", "value": 38.809586587791664},
894
+ # ],
895
+ # },
896
+ # Use "get" instead of dict indexing to skip incompat metadata instead of erroring out
897
+ if len(datasets) > 0:
898
+ task_results = [sub_res for sub_res in meta["model-index"][0]["results"] if (sub_res.get("task", {}).get("type", "") in tasks) and any([x in sub_res.get("dataset", {}).get("name", "") for x in datasets])]
899
+ elif langs:
900
+ task_results = [sub_res for sub_res in meta["model-index"][0]["results"] if (sub_res.get("task", {}).get("type", "") in tasks) and (sub_res.get("dataset", {}).get("config", "default") in ("default", *langs))]
901
+ else:
902
+ task_results = [sub_res for sub_res in meta["model-index"][0]["results"] if (sub_res.get("task", {}).get("type", "") in tasks)]
903
+ out = [{res["dataset"]["name"].replace("MTEB ", ""): [round(score["value"], 2) for score in res["metrics"] if score["type"] == task_to_metric.get(res["task"]["type"])][0]} for res in task_results]
904
+ out = {k: v for d in out for k, v in d.items()}
905
+ out["Model"] = make_clickable_model(model.modelId)
906
+ # Model & at least one result
907
+ if len(out) > 1:
908
+ if add_emb_dim:
909
+ out["Embedding Dimensions"], out["Sequence Length"], out["Model Size (GB)"] = get_dim_seq_size(model)
910
+ df_list.append(out)
911
+ df = pd.DataFrame(df_list)
912
+ # If there are any models that are the same, merge them
913
+ # E.g. if out["Model"] has the same value in two places, merge & take whichever one is not NaN else just take the first one
914
+ df = df.groupby("Model", as_index=False).first()
915
+ # Put 'Model' column first
916
+ cols = sorted(list(df.columns))
917
+ cols.insert(0, cols.pop(cols.index("Model")))
918
+ df = df[cols]
919
+ if rank:
920
+ df = add_rank(df)
921
+ if fillna:
922
+ df.fillna("", inplace=True)
923
+ return df
924
+
925
+ def get_mteb_average():
926
+ global DATA_OVERALL, DATA_CLASSIFICATION_EN, DATA_CLUSTERING, DATA_PAIR_CLASSIFICATION, DATA_RERANKING, DATA_RETRIEVAL, DATA_STS_EN, DATA_SUMMARIZATION
927
+
928
+ DATA_OVERALL = get_mteb_data(
929
+ tasks=[
930
+ "Classification",
931
+ "Clustering",
932
+ "PairClassification",
933
+ "Reranking",
934
+ "Retrieval",
935
+ "STS",
936
+ "Summarization",
937
+ ],
938
+ datasets=TASK_LIST_CLASSIFICATION + TASK_LIST_CLUSTERING + TASK_LIST_PAIR_CLASSIFICATION + TASK_LIST_RERANKING + TASK_LIST_RETRIEVAL + TASK_LIST_STS + TASK_LIST_SUMMARIZATION,
939
+ fillna=False,
940
+ add_emb_dim=True,
941
+ rank=False,
942
+ )
943
+ # Debugging:
944
+ # DATA_OVERALL.to_csv("overall.csv")
945
+
946
+ DATA_OVERALL.insert(1, f"Average ({len(TASK_LIST_EN)} datasets)", DATA_OVERALL[TASK_LIST_EN].mean(axis=1, skipna=False))
947
+ DATA_OVERALL.insert(2, f"Classification Average ({len(TASK_LIST_CLASSIFICATION)} datasets)", DATA_OVERALL[TASK_LIST_CLASSIFICATION].mean(axis=1, skipna=False))
948
+ DATA_OVERALL.insert(3, f"Clustering Average ({len(TASK_LIST_CLUSTERING)} datasets)", DATA_OVERALL[TASK_LIST_CLUSTERING].mean(axis=1, skipna=False))
949
+ DATA_OVERALL.insert(4, f"Pair Classification Average ({len(TASK_LIST_PAIR_CLASSIFICATION)} datasets)", DATA_OVERALL[TASK_LIST_PAIR_CLASSIFICATION].mean(axis=1, skipna=False))
950
+ DATA_OVERALL.insert(5, f"Reranking Average ({len(TASK_LIST_RERANKING)} datasets)", DATA_OVERALL[TASK_LIST_RERANKING].mean(axis=1, skipna=False))
951
+ DATA_OVERALL.insert(6, f"Retrieval Average ({len(TASK_LIST_RETRIEVAL)} datasets)", DATA_OVERALL[TASK_LIST_RETRIEVAL].mean(axis=1, skipna=False))
952
+ DATA_OVERALL.insert(7, f"STS Average ({len(TASK_LIST_STS)} datasets)", DATA_OVERALL[TASK_LIST_STS].mean(axis=1, skipna=False))
953
+ DATA_OVERALL.insert(8, f"Summarization Average ({len(TASK_LIST_SUMMARIZATION)} dataset)", DATA_OVERALL[TASK_LIST_SUMMARIZATION].mean(axis=1, skipna=False))
954
+ DATA_OVERALL.sort_values(f"Average ({len(TASK_LIST_EN)} datasets)", ascending=False, inplace=True)
955
+ # Start ranking from 1
956
+ DATA_OVERALL.insert(0, "Rank", list(range(1, len(DATA_OVERALL) + 1)))
957
+
958
+ DATA_OVERALL = DATA_OVERALL.round(2)
959
+
960
+ DATA_CLASSIFICATION_EN = add_rank(DATA_OVERALL[["Model"] + TASK_LIST_CLASSIFICATION])
961
+ # Only keep rows with at least one score in addition to the "Model" & rank column
962
+ DATA_CLASSIFICATION_EN = DATA_CLASSIFICATION_EN[DATA_CLASSIFICATION_EN.iloc[:, 2:].ne("").any(axis=1)]
963
+
964
+ DATA_CLUSTERING = add_rank(DATA_OVERALL[["Model"] + TASK_LIST_CLUSTERING])
965
+ DATA_CLUSTERING = DATA_CLUSTERING[DATA_CLUSTERING.iloc[:, 2:].ne("").any(axis=1)]
966
+
967
+ DATA_PAIR_CLASSIFICATION = add_rank(DATA_OVERALL[["Model"] + TASK_LIST_PAIR_CLASSIFICATION])
968
+ DATA_PAIR_CLASSIFICATION = DATA_PAIR_CLASSIFICATION[DATA_PAIR_CLASSIFICATION.iloc[:, 2:].ne("").any(axis=1)]
969
+
970
+ DATA_RERANKING = add_rank(DATA_OVERALL[["Model"] + TASK_LIST_RERANKING])
971
+ DATA_RERANKING = DATA_RERANKING[DATA_RERANKING.iloc[:, 2:].ne("").any(axis=1)]
972
+
973
+ DATA_RETRIEVAL = add_rank(DATA_OVERALL[["Model"] + TASK_LIST_RETRIEVAL])
974
+ DATA_RETRIEVAL = DATA_RETRIEVAL[DATA_RETRIEVAL.iloc[:, 2:].ne("").any(axis=1)]
975
+
976
+ DATA_STS_EN = add_rank(DATA_OVERALL[["Model"] + TASK_LIST_STS])
977
+ DATA_STS_EN = DATA_STS_EN[DATA_STS_EN.iloc[:, 2:].ne("").any(axis=1)]
978
+
979
+ DATA_SUMMARIZATION = add_rank(DATA_OVERALL[["Model"] + TASK_LIST_SUMMARIZATION])
980
+ DATA_SUMMARIZATION = DATA_SUMMARIZATION[DATA_SUMMARIZATION.iloc[:, 1:].ne("").any(axis=1)]
981
+
982
+ # Fill NaN after averaging
983
+ DATA_OVERALL.fillna("", inplace=True)
984
+
985
+ DATA_OVERALL = DATA_OVERALL[["Rank", "Model", "Model Size (GB)", "Embedding Dimensions", "Sequence Length", f"Average ({len(TASK_LIST_EN)} datasets)", f"Classification Average ({len(TASK_LIST_CLASSIFICATION)} datasets)", f"Clustering Average ({len(TASK_LIST_CLUSTERING)} datasets)", f"Pair Classification Average ({len(TASK_LIST_PAIR_CLASSIFICATION)} datasets)", f"Reranking Average ({len(TASK_LIST_RERANKING)} datasets)", f"Retrieval Average ({len(TASK_LIST_RETRIEVAL)} datasets)", f"STS Average ({len(TASK_LIST_STS)} datasets)", f"Summarization Average ({len(TASK_LIST_SUMMARIZATION)} dataset)"]]
986
+ DATA_OVERALL = DATA_OVERALL[DATA_OVERALL.iloc[:, 5:].ne("").any(axis=1)]
987
+
988
+ return DATA_OVERALL
989
+
990
+ def get_mteb_average_zh():
991
+ global DATA_OVERALL_ZH, DATA_CLASSIFICATION_ZH, DATA_CLUSTERING_ZH, DATA_PAIR_CLASSIFICATION_ZH, DATA_RERANKING_ZH, DATA_RETRIEVAL_ZH, DATA_STS_ZH
992
+ DATA_OVERALL_ZH = get_mteb_data(
993
+ tasks=[
994
+ "Classification",
995
+ "Clustering",
996
+ "PairClassification",
997
+ "Reranking",
998
+ "Retrieval",
999
+ "STS",
1000
+ ],
1001
+ datasets=TASK_LIST_CLASSIFICATION_ZH + TASK_LIST_CLUSTERING_ZH + TASK_LIST_PAIR_CLASSIFICATION_ZH + TASK_LIST_RERANKING_ZH + TASK_LIST_RETRIEVAL_ZH + TASK_LIST_STS_ZH,
1002
+ fillna=False,
1003
+ add_emb_dim=True,
1004
+ rank=False,
1005
+ )
1006
+ # Debugging:
1007
+ # DATA_OVERALL_ZH.to_csv("overall.csv")
1008
+
1009
+ DATA_OVERALL_ZH.insert(1, f"Average ({len(TASK_LIST_ZH)} datasets)", DATA_OVERALL_ZH[TASK_LIST_ZH].mean(axis=1, skipna=False))
1010
+ DATA_OVERALL_ZH.insert(2, f"Classification Average ({len(TASK_LIST_CLASSIFICATION_ZH)} datasets)", DATA_OVERALL_ZH[TASK_LIST_CLASSIFICATION_ZH].mean(axis=1, skipna=False))
1011
+ DATA_OVERALL_ZH.insert(3, f"Clustering Average ({len(TASK_LIST_CLUSTERING_ZH)} datasets)", DATA_OVERALL_ZH[TASK_LIST_CLUSTERING_ZH].mean(axis=1, skipna=False))
1012
+ DATA_OVERALL_ZH.insert(4, f"Pair Classification Average ({len(TASK_LIST_PAIR_CLASSIFICATION_ZH)} datasets)", DATA_OVERALL_ZH[TASK_LIST_PAIR_CLASSIFICATION_ZH].mean(axis=1, skipna=False))
1013
+ DATA_OVERALL_ZH.insert(5, f"Reranking Average ({len(TASK_LIST_RERANKING_ZH)} datasets)", DATA_OVERALL_ZH[TASK_LIST_RERANKING_ZH].mean(axis=1, skipna=False))
1014
+ DATA_OVERALL_ZH.insert(6, f"Retrieval Average ({len(TASK_LIST_RETRIEVAL_ZH)} datasets)", DATA_OVERALL_ZH[TASK_LIST_RETRIEVAL_ZH].mean(axis=1, skipna=False))
1015
+ DATA_OVERALL_ZH.insert(7, f"STS Average ({len(TASK_LIST_STS_ZH)} datasets)", DATA_OVERALL_ZH[TASK_LIST_STS_ZH].mean(axis=1, skipna=False))
1016
+ DATA_OVERALL_ZH.sort_values(f"Average ({len(TASK_LIST_ZH)} datasets)", ascending=False, inplace=True)
1017
+ # Start ranking from 1
1018
+ DATA_OVERALL_ZH.insert(0, "Rank", list(range(1, len(DATA_OVERALL_ZH) + 1)))
1019
+
1020
+ DATA_OVERALL_ZH = DATA_OVERALL_ZH.round(2)
1021
+
1022
+ DATA_CLASSIFICATION_ZH = add_rank(DATA_OVERALL_ZH[["Model"] + TASK_LIST_CLASSIFICATION_ZH])
1023
+ # Only keep rows with at least one score in addition to the "Model" & rank column
1024
+ DATA_CLASSIFICATION_ZH = DATA_CLASSIFICATION_ZH[DATA_CLASSIFICATION_ZH.iloc[:, 2:].ne("").any(axis=1)]
1025
+
1026
+ DATA_CLUSTERING_ZH = add_rank(DATA_OVERALL_ZH[["Model"] + TASK_LIST_CLUSTERING_ZH])
1027
+ DATA_CLUSTERING_ZH = DATA_CLUSTERING_ZH[DATA_CLUSTERING_ZH.iloc[:, 2:].ne("").any(axis=1)]
1028
+
1029
+ DATA_PAIR_CLASSIFICATION_ZH = add_rank(DATA_OVERALL_ZH[["Model"] + TASK_LIST_PAIR_CLASSIFICATION_ZH])
1030
+ DATA_PAIR_CLASSIFICATION_ZH = DATA_PAIR_CLASSIFICATION_ZH[DATA_PAIR_CLASSIFICATION_ZH.iloc[:, 2:].ne("").any(axis=1)]
1031
+
1032
+ DATA_RERANKING_ZH = add_rank(DATA_OVERALL_ZH[["Model"] + TASK_LIST_RERANKING_ZH])
1033
+ DATA_RERANKING_ZH = DATA_RERANKING_ZH[DATA_RERANKING_ZH.iloc[:, 2:].ne("").any(axis=1)]
1034
+
1035
+ DATA_RETRIEVAL_ZH = add_rank(DATA_OVERALL_ZH[["Model"] + TASK_LIST_RETRIEVAL_ZH])
1036
+ DATA_RETRIEVAL_ZH = DATA_RETRIEVAL_ZH[DATA_RETRIEVAL_ZH.iloc[:, 2:].ne("").any(axis=1)]
1037
+
1038
+ DATA_STS_ZH = add_rank(DATA_OVERALL_ZH[["Model"] + TASK_LIST_STS_ZH])
1039
+ DATA_STS_ZH = DATA_STS_ZH[DATA_STS_ZH.iloc[:, 2:].ne("").any(axis=1)]
1040
+
1041
+ # Fill NaN after averaging
1042
+ DATA_OVERALL_ZH.fillna("", inplace=True)
1043
+
1044
+ DATA_OVERALL_ZH = DATA_OVERALL_ZH[["Rank", "Model", "Model Size (GB)", "Embedding Dimensions", "Sequence Length", f"Average ({len(TASK_LIST_ZH)} datasets)", f"Classification Average ({len(TASK_LIST_CLASSIFICATION_ZH)} datasets)", f"Clustering Average ({len(TASK_LIST_CLUSTERING_ZH)} datasets)", f"Pair Classification Average ({len(TASK_LIST_PAIR_CLASSIFICATION_ZH)} datasets)", f"Reranking Average ({len(TASK_LIST_RERANKING_ZH)} datasets)", f"Retrieval Average ({len(TASK_LIST_RETRIEVAL_ZH)} datasets)", f"STS Average ({len(TASK_LIST_STS_ZH)} datasets)"]]
1045
+ DATA_OVERALL_ZH = DATA_OVERALL_ZH[DATA_OVERALL_ZH.iloc[:, 5:].ne("").any(axis=1)]
1046
+
1047
+ return DATA_OVERALL_ZH
1048
+
1049
+ def get_mteb_average_pl():
1050
+ global DATA_OVERALL_PL, DATA_CLASSIFICATION_PL, DATA_CLUSTERING_PL, DATA_PAIR_CLASSIFICATION_PL, DATA_RETRIEVAL_PL, DATA_STS_PL
1051
+ DATA_OVERALL_PL = get_mteb_data(
1052
+ tasks=[
1053
+ "Classification",
1054
+ "Clustering",
1055
+ "PairClassification",
1056
+ "Retrieval",
1057
+ "STS",
1058
+ ],
1059
+ datasets=TASK_LIST_CLASSIFICATION_PL + TASK_LIST_CLUSTERING_PL + TASK_LIST_PAIR_CLASSIFICATION_PL + TASK_LIST_RETRIEVAL_PL + TASK_LIST_STS_PL,
1060
+ fillna=False,
1061
+ add_emb_dim=True,
1062
+ rank=False,
1063
+ )
1064
+ # Debugging:
1065
+ # DATA_OVERALL_PL.to_csv("overall.csv")
1066
+
1067
+ DATA_OVERALL_PL.insert(1, f"Average ({len(TASK_LIST_PL)} datasets)", DATA_OVERALL_PL[TASK_LIST_PL].mean(axis=1, skipna=False))
1068
+ DATA_OVERALL_PL.insert(2, f"Classification Average ({len(TASK_LIST_CLASSIFICATION_PL)} datasets)", DATA_OVERALL_PL[TASK_LIST_CLASSIFICATION_PL].mean(axis=1, skipna=False))
1069
+ DATA_OVERALL_PL.insert(3, f"Clustering Average ({len(TASK_LIST_CLUSTERING_PL)} datasets)", DATA_OVERALL_PL[TASK_LIST_CLUSTERING_PL].mean(axis=1, skipna=False))
1070
+ DATA_OVERALL_PL.insert(4, f"Pair Classification Average ({len(TASK_LIST_PAIR_CLASSIFICATION_PL)} datasets)", DATA_OVERALL_PL[TASK_LIST_PAIR_CLASSIFICATION_PL].mean(axis=1, skipna=False))
1071
+ DATA_OVERALL_PL.insert(5, f"Retrieval Average ({len(TASK_LIST_RETRIEVAL_PL)} datasets)", DATA_OVERALL_PL[TASK_LIST_RETRIEVAL_PL].mean(axis=1, skipna=False))
1072
+ DATA_OVERALL_PL.insert(6, f"STS Average ({len(TASK_LIST_STS_PL)} datasets)", DATA_OVERALL_PL[TASK_LIST_STS_PL].mean(axis=1, skipna=False))
1073
+ DATA_OVERALL_PL.sort_values(f"Average ({len(TASK_LIST_PL)} datasets)", ascending=False, inplace=True)
1074
+ # Start ranking from 1
1075
+ DATA_OVERALL_PL.insert(0, "Rank", list(range(1, len(DATA_OVERALL_PL) + 1)))
1076
+
1077
+ DATA_OVERALL_PL = DATA_OVERALL_PL.round(2)
1078
+
1079
+ DATA_CLASSIFICATION_PL = add_rank(DATA_OVERALL_PL[["Model"] + TASK_LIST_CLASSIFICATION_PL])
1080
+ # Only keep rows with at least one score in addition to the "Model" & rank column
1081
+ DATA_CLASSIFICATION_PL = DATA_CLASSIFICATION_PL[DATA_CLASSIFICATION_PL.iloc[:, 2:].ne("").any(axis=1)]
1082
+
1083
+ DATA_CLUSTERING_PL = add_rank(DATA_OVERALL_PL[["Model"] + TASK_LIST_CLUSTERING_PL])
1084
+ DATA_CLUSTERING_PL = DATA_CLUSTERING_PL[DATA_CLUSTERING_PL.iloc[:, 2:].ne("").any(axis=1)]
1085
+
1086
+ DATA_PAIR_CLASSIFICATION_PL = add_rank(DATA_OVERALL_PL[["Model"] + TASK_LIST_PAIR_CLASSIFICATION_PL])
1087
+ DATA_PAIR_CLASSIFICATION_PL = DATA_PAIR_CLASSIFICATION_PL[DATA_PAIR_CLASSIFICATION_PL.iloc[:, 2:].ne("").any(axis=1)]
1088
+
1089
+ DATA_RETRIEVAL_PL = add_rank(DATA_OVERALL_PL[["Model"] + TASK_LIST_RETRIEVAL_PL])
1090
+ DATA_RETRIEVAL_PL = DATA_RETRIEVAL_PL[DATA_RETRIEVAL_PL.iloc[:, 2:].ne("").any(axis=1)]
1091
+
1092
+ DATA_STS_PL = add_rank(DATA_OVERALL_PL[["Model"] + TASK_LIST_STS_PL])
1093
+ DATA_STS_PL = DATA_STS_PL[DATA_STS_PL.iloc[:, 2:].ne("").any(axis=1)]
1094
+
1095
+ # Fill NaN after averaging
1096
+ DATA_OVERALL_PL.fillna("", inplace=True)
1097
+
1098
+ DATA_OVERALL_PL = DATA_OVERALL_PL[["Rank", "Model", "Model Size (GB)", "Embedding Dimensions", "Sequence Length", f"Average ({len(TASK_LIST_PL)} datasets)", f"Classification Average ({len(TASK_LIST_CLASSIFICATION_PL)} datasets)", f"Clustering Average ({len(TASK_LIST_CLUSTERING_PL)} datasets)", f"Pair Classification Average ({len(TASK_LIST_PAIR_CLASSIFICATION_PL)} datasets)", f"Retrieval Average ({len(TASK_LIST_RETRIEVAL_PL)} datasets)", f"STS Average ({len(TASK_LIST_STS_PL)} datasets)"]]
1099
+ DATA_OVERALL_PL = DATA_OVERALL_PL[DATA_OVERALL_PL.iloc[:, 5:].ne("").any(axis=1)]
1100
+
1101
+ return DATA_OVERALL_PL
1102
+
1103
+ get_mteb_average()
1104
+
1105
+
1106
+ get_mteb_average_pl()
1107
+ get_mteb_average_zh()
1108
+ DATA_BITEXT_MINING = get_mteb_data(["BitextMining"], [], TASK_LIST_BITEXT_MINING)
1109
+ DATA_BITEXT_MINING_OTHER = get_mteb_data(["BitextMining"], [], TASK_LIST_BITEXT_MINING_OTHER)
1110
+ DATA_CLASSIFICATION_DA = get_mteb_data(["Classification"], [], TASK_LIST_CLASSIFICATION_DA)
1111
+ DATA_CLASSIFICATION_NB = get_mteb_data(["Classification"], [], TASK_LIST_CLASSIFICATION_NB)
1112
+ DATA_CLASSIFICATION_SV = get_mteb_data(["Classification"], [], TASK_LIST_CLASSIFICATION_SV)
1113
+ DATA_CLASSIFICATION_OTHER = get_mteb_data(["Classification"], [], TASK_LIST_CLASSIFICATION_OTHER)
1114
+ DATA_CLUSTERING_DE = get_mteb_data(["Clustering"], [], TASK_LIST_CLUSTERING_DE)
1115
+ DATA_STS_OTHER = get_mteb_data(["STS"], [], TASK_LIST_STS_OTHER)
1116
+
1117
+ # Exact, add all non-nan integer values for every dataset
1118
+ NUM_SCORES = 0
1119
+ DATASETS = []
1120
+ MODELS = []
1121
+ # LANGUAGES = []
1122
+ for d in [
1123
+ DATA_BITEXT_MINING,
1124
+ DATA_BITEXT_MINING_OTHER,
1125
+ DATA_CLASSIFICATION_EN,
1126
+ DATA_CLASSIFICATION_DA,
1127
+ DATA_CLASSIFICATION_NB,
1128
+ DATA_CLASSIFICATION_PL,
1129
+ DATA_CLASSIFICATION_SV,
1130
+ DATA_CLASSIFICATION_ZH,
1131
+ DATA_CLASSIFICATION_OTHER,
1132
+ DATA_CLUSTERING,
1133
+ DATA_CLUSTERING_DE,
1134
+ DATA_CLUSTERING_PL,
1135
+ DATA_CLUSTERING_ZH,
1136
+ DATA_PAIR_CLASSIFICATION,
1137
+ DATA_PAIR_CLASSIFICATION_PL,
1138
+ DATA_PAIR_CLASSIFICATION_ZH,
1139
+ DATA_RERANKING,
1140
+ DATA_RERANKING_ZH,
1141
+ DATA_RETRIEVAL,
1142
+ DATA_RETRIEVAL_PL,
1143
+ DATA_RETRIEVAL_ZH,
1144
+ DATA_STS_EN,
1145
+ DATA_STS_PL,
1146
+ DATA_STS_ZH,
1147
+ DATA_STS_OTHER,
1148
+ DATA_SUMMARIZATION,
1149
+ ]:
1150
+ # NUM_SCORES += d.iloc[:, 1:].apply(lambda x: sum([1 for y in x if isinstance(y, float) and not np.isnan(y)]), axis=1).sum()
1151
+ cols_to_ignore = 3 if "Average" in d.columns else 2
1152
+ # Count number of scores including only non-nan floats & excluding the rank column
1153
+ NUM_SCORES += d.iloc[:, cols_to_ignore:].notna().sum().sum()
1154
+ # Exclude rank & model name column (first two); Do not count different language versions as different datasets
1155
+ DATASETS += [i.split(" ")[0] for i in d.columns[cols_to_ignore:]]
1156
+ # LANGUAGES += [i.split(" ")[-1] for i in d.columns[cols_to_ignore:]]
1157
+ MODELS += d["Model"].tolist()
1158
+
1159
+ NUM_DATASETS = len(set(DATASETS))
1160
+ # NUM_LANGUAGES = len(set(LANGUAGES))
1161
+ NUM_MODELS = len(set(MODELS))
1162
+
1163
+
1164
+
1165
+
1166
+
1167
+ block = gr.Blocks()
1168
+ with block:
1169
+ gr.Markdown(f"""
1170
+ SeaEval Leaderboard. To submit, refer to the <a href="https://seaeval.github.io/" target="_blank" style="text-decoration: underline">SeaEval Website</a> Refer to the [SeaEval paper](https://arxiv.org/abs/2309.04766) for details on metrics, tasks and models.
1171
+
1172
+ - **Total Datasets**: 31
1173
+ - **Total Languages**: 8
1174
+ - **Total Models**: 5
1175
+ """)
1176
+ with gr.Tabs():
1177
+
1178
+
1179
+ # dataset 1: cross-mmlu
1180
+ with gr.TabItem("Cross-MMLU"):
1181
+ with gr.Row():
1182
+ gr.Markdown("""
1183
+ **Overall Cross-MMLU Leaderboard** 🔮
1184
+
1185
+ - **Metric:** Cross-Lingual Consistency, Accuracy, AC3
1186
+ - **Languages:** English, Chinese, Malay, Indonesian, Spanish, Vietnamese, Filipino
1187
+ """)
1188
+
1189
+ with gr.TabItem("Zero-Shot"):
1190
+
1191
+
1192
+ with gr.TabItem("Overall"):
1193
+
1194
+ with gr.Row():
1195
+ data_bitext_mining = gr.components.Dataframe(
1196
+ DATA_BITEXT_MINING,
1197
+ datatype=["number", "markdown"] + ["number"] * len(DATA_BITEXT_MINING.columns),
1198
+ type="pandas",
1199
+ )
1200
+ with gr.Row():
1201
+ data_run_bitext_mining = gr.Button("Refresh")
1202
+ data_run_bitext_mining.click(
1203
+ partial(get_mteb_data, tasks=["BitextMining"], datasets=TASK_LIST_BITEXT_MINING),
1204
+ outputs=data_bitext_mining,
1205
+ )
1206
+
1207
+ with gr.TabItem("Detailed Consistency"):
1208
+
1209
+ with gr.Row():
1210
+ data_bitext_mining = gr.components.Dataframe(
1211
+ DATA_BITEXT_MINING,
1212
+ datatype=["number", "markdown"] + ["number"] * len(DATA_BITEXT_MINING.columns),
1213
+ type="pandas",
1214
+ )
1215
+ with gr.Row():
1216
+ data_run_bitext_mining = gr.Button("Refresh")
1217
+ data_run_bitext_mining.click(
1218
+ partial(get_mteb_data, tasks=["BitextMining"], datasets=TASK_LIST_BITEXT_MINING),
1219
+ outputs=data_bitext_mining,
1220
+ )
1221
+
1222
+ with gr.TabItem("Five-Shot"):
1223
+
1224
+ with gr.TabItem("Overall"):
1225
+
1226
+ with gr.Row():
1227
+ data_bitext_mining = gr.components.Dataframe(
1228
+ DATA_BITEXT_MINING,
1229
+ datatype=["number", "markdown"] + ["number"] * len(DATA_BITEXT_MINING.columns),
1230
+ type="pandas",
1231
+ )
1232
+ with gr.Row():
1233
+ data_run_bitext_mining = gr.Button("Refresh")
1234
+ data_run_bitext_mining.click(
1235
+ partial(get_mteb_data, tasks=["BitextMining"], datasets=TASK_LIST_BITEXT_MINING),
1236
+ outputs=data_bitext_mining,
1237
+ )
1238
+
1239
+ with gr.TabItem("Detailed Consistency"):
1240
+
1241
+ with gr.Row():
1242
+ data_bitext_mining = gr.components.Dataframe(
1243
+ DATA_BITEXT_MINING,
1244
+ datatype=["number", "markdown"] + ["number"] * len(DATA_BITEXT_MINING.columns),
1245
+ type="pandas",
1246
+ )
1247
+ with gr.Row():
1248
+ data_run_bitext_mining = gr.Button("Refresh")
1249
+ data_run_bitext_mining.click(
1250
+ partial(get_mteb_data, tasks=["BitextMining"], datasets=TASK_LIST_BITEXT_MINING),
1251
+ outputs=data_bitext_mining,
1252
+ )
1253
+
1254
+
1255
+
1256
+ with gr.TabItem("Classification"):
1257
+ with gr.TabItem("English"):
1258
+ with gr.Row():
1259
+ gr.Markdown("""
1260
+ **Classification English Leaderboard** ❤️
1261
+
1262
+ - **Metric:** [Accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy)
1263
+ - **Languages:** English
1264
+ """)
1265
+ with gr.Row():
1266
+ data_classification_en = gr.components.Dataframe(
1267
+ DATA_CLASSIFICATION_EN,
1268
+ datatype=["number", "markdown"] + ["number"] * len(DATA_CLASSIFICATION_EN.columns),
1269
+ type="pandas",
1270
+ )
1271
+ with gr.Row():
1272
+ data_run_classification_en = gr.Button("Refresh")
1273
+ data_run_classification_en.click(
1274
+ partial(get_mteb_data, tasks=["Classification"], langs=["en"]),
1275
+ outputs=data_classification_en,
1276
+ )
1277
+ with gr.TabItem("Chinese"):
1278
+ with gr.Row():
1279
+ gr.Markdown("""
1280
+ **Classification Chinese Leaderboard** 🧡🇨🇳
1281
+
1282
+ - **Metric:** [Accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy)
1283
+ - **Languages:** Chinese
1284
+ - **Credits:** [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding)
1285
+ """)
1286
+ with gr.Row():
1287
+ data_classification_zh = gr.components.Dataframe(
1288
+ DATA_CLASSIFICATION_ZH,
1289
+ datatype=["number", "markdown"] + ["number"] * len(DATA_CLASSIFICATION_ZH.columns),
1290
+ type="pandas",
1291
+ )
1292
+ with gr.Row():
1293
+ data_run_classification_zh = gr.Button("Refresh")
1294
+ data_run_classification_zh.click(
1295
+ partial(get_mteb_data, tasks=["Classification"], datasets=TASK_LIST_CLASSIFICATION_ZH),
1296
+ outputs=data_classification_zh,
1297
+ )
1298
+ with gr.TabItem("Danish"):
1299
+ with gr.Row():
1300
+ gr.Markdown("""
1301
+ **Classification Danish Leaderboard** 🤍🇩🇰
1302
+
1303
+ - **Metric:** [Accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy)
1304
+ - **Languages:** Danish
1305
+ - **Credits:** [Kenneth Enevoldsen](https://github.com/KennethEnevoldsen), [scandinavian-embedding-benchmark](https://kennethenevoldsen.github.io/scandinavian-embedding-benchmark/)
1306
+ """)
1307
+ with gr.Row():
1308
+ data_classification_da = gr.components.Dataframe(
1309
+ DATA_CLASSIFICATION_DA,
1310
+ datatype=["number", "markdown"] + ["number"] * len(DATA_CLASSIFICATION_DA.columns),
1311
+ type="pandas",
1312
+ )
1313
+ with gr.Row():
1314
+ data_run_classification_da = gr.Button("Refresh")
1315
+ data_run_classification_da.click(
1316
+ partial(get_mteb_data, tasks=["Classification"], datasets=TASK_LIST_CLASSIFICATION_DA),
1317
+ outputs=data_run_classification_da,
1318
+ )
1319
+ with gr.TabItem("Norwegian"):
1320
+ with gr.Row():
1321
+ gr.Markdown("""
1322
+ **Classification Norwegian Leaderboard** 💙🇳🇴
1323
+
1324
+ - **Metric:** [Accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy)
1325
+ - **Languages:** Norwegian Bokmål
1326
+ - **Credits:** [Kenneth Enevoldsen](https://github.com/KennethEnevoldsen), [scandinavian-embedding-benchmark](https://kennethenevoldsen.github.io/scandinavian-embedding-benchmark/)
1327
+ """)
1328
+ with gr.Row():
1329
+ data_classification_nb = gr.components.Dataframe(
1330
+ DATA_CLASSIFICATION_NB,
1331
+ datatype=["number", "markdown"] + ["number"] * len(DATA_CLASSIFICATION_NB.columns),
1332
+ type="pandas",
1333
+ )
1334
+ with gr.Row():
1335
+ data_run_classification_nb = gr.Button("Refresh")
1336
+ data_run_classification_nb.click(
1337
+ partial(get_mteb_data, tasks=["Classification"], datasets=TASK_LIST_CLASSIFICATION_NB),
1338
+ outputs=data_classification_nb,
1339
+ )
1340
+ with gr.TabItem("Polish"):
1341
+ with gr.Row():
1342
+ gr.Markdown("""
1343
+ **Classification Polish Leaderboard** 🤍🇵🇱
1344
+
1345
+ - **Metric:** [Accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy)
1346
+ - **Languages:** Polish
1347
+ - **Credits:** [Rafał Poświata](https://github.com/rafalposwiata)
1348
+ """)
1349
+ with gr.Row():
1350
+ data_classification_pl = gr.components.Dataframe(
1351
+ DATA_CLASSIFICATION_PL,
1352
+ datatype=["number", "markdown"] + ["number"] * len(DATA_CLASSIFICATION_PL.columns),
1353
+ type="pandas",
1354
+ )
1355
+ with gr.Row():
1356
+ data_run_classification_pl = gr.Button("Refresh")
1357
+ data_run_classification_pl.click(
1358
+ partial(get_mteb_data, tasks=["Classification"], datasets=TASK_LIST_CLASSIFICATION_PL),
1359
+ outputs=data_classification_pl,
1360
+ )
1361
+ with gr.TabItem("Swedish"):
1362
+ with gr.Row():
1363
+ gr.Markdown("""
1364
+ **Classification Swedish Leaderboard** 💛🇸🇪
1365
+
1366
+ - **Metric:** [Accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy)
1367
+ - **Languages:** Swedish
1368
+ - **Credits:** [Kenneth Enevoldsen](https://github.com/KennethEnevoldsen), [scandinavian-embedding-benchmark](https://kennethenevoldsen.github.io/scandinavian-embedding-benchmark/)
1369
+ """)
1370
+ with gr.Row():
1371
+ data_classification_sv = gr.components.Dataframe(
1372
+ DATA_CLASSIFICATION_SV,
1373
+ datatype=["number", "markdown"] + ["number"] * len(DATA_CLASSIFICATION_SV.columns),
1374
+ type="pandas",
1375
+ )
1376
+ with gr.Row():
1377
+ data_run_classification_sv = gr.Button("Refresh")
1378
+ data_run_classification_sv.click(
1379
+ partial(get_mteb_data, tasks=["Classification"], datasets=TASK_LIST_CLASSIFICATION_SV),
1380
+ outputs=data_classification_sv,
1381
+ )
1382
+ with gr.TabItem("Other"):
1383
+ with gr.Row():
1384
+ gr.Markdown("""
1385
+ **Classification Other Languages Leaderboard** 💜💚💙
1386
+
1387
+ - **Metric:** [Accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy)
1388
+ - **Languages:** 47 (Only languages not included in the other tabs)
1389
+ """)
1390
+ with gr.Row():
1391
+ data_classification = gr.components.Dataframe(
1392
+ DATA_CLASSIFICATION_OTHER,
1393
+ datatype=["number", "markdown"] + ["number"] * len(DATA_CLASSIFICATION_OTHER) * 10,
1394
+ type="pandas",
1395
+ )
1396
+ with gr.Row():
1397
+ data_run_classification = gr.Button("Refresh")
1398
+ data_run_classification.click(
1399
+ partial(get_mteb_data, tasks=["Classification"], datasets=TASK_LIST_CLASSIFICATION_OTHER),
1400
+ outputs=data_classification,
1401
+ )
1402
+ with gr.TabItem("Clustering"):
1403
+ with gr.TabItem("English"):
1404
+ with gr.Row():
1405
+ gr.Markdown("""
1406
+ **Clustering Leaderboard** ✨
1407
+
1408
+ - **Metric:** Validity Measure (v_measure)
1409
+ - **Languages:** English
1410
+ """)
1411
+ with gr.Row():
1412
+ data_clustering = gr.components.Dataframe(
1413
+ DATA_CLUSTERING,
1414
+ datatype=["number", "markdown"] + ["number"] * len(DATA_CLUSTERING.columns),
1415
+ type="pandas",
1416
+ )
1417
+ with gr.Row():
1418
+ data_run_clustering_en = gr.Button("Refresh")
1419
+ data_run_clustering_en.click(
1420
+ partial(get_mteb_data, tasks=["Clustering"], datasets=TASK_LIST_CLUSTERING),
1421
+ outputs=data_clustering,
1422
+ )
1423
+ with gr.TabItem("Chinese"):
1424
+ with gr.Row():
1425
+ gr.Markdown("""
1426
+ **Clustering Chinese Leaderboard** ✨🇨🇳
1427
+
1428
+ - **Metric:** Validity Measure (v_measure)
1429
+ - **Languages:** Chinese
1430
+ - **Credits:** [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding)
1431
+ """)
1432
+ with gr.Row():
1433
+ data_clustering_zh = gr.components.Dataframe(
1434
+ DATA_CLUSTERING_ZH,
1435
+ datatype=["number", "markdown"] + ["number"] * len(DATA_CLUSTERING_ZH.columns),
1436
+ type="pandas",
1437
+ )
1438
+ with gr.Row():
1439
+ data_run_clustering_zh = gr.Button("Refresh")
1440
+ data_run_clustering_zh.click(
1441
+ partial(get_mteb_data, tasks=["Clustering"], datasets=TASK_LIST_CLUSTERING_ZH),
1442
+ outputs=data_clustering_zh,
1443
+ )
1444
+ with gr.TabItem("German"):
1445
+ with gr.Row():
1446
+ gr.Markdown("""
1447
+ **Clustering German Leaderboard** ✨🇩🇪
1448
+
1449
+ - **Metric:** Validity Measure (v_measure)
1450
+ - **Languages:** German
1451
+ - **Credits:** [Silvan](https://github.com/slvnwhrl)
1452
+ """)
1453
+ with gr.Row():
1454
+ data_clustering_de = gr.components.Dataframe(
1455
+ DATA_CLUSTERING_DE,
1456
+ datatype=["number", "markdown"] + ["number"] * len(DATA_CLUSTERING_DE.columns) * 2,
1457
+ type="pandas",
1458
+ )
1459
+ with gr.Row():
1460
+ data_run_clustering_de = gr.Button("Refresh")
1461
+ data_run_clustering_de.click(
1462
+ partial(get_mteb_data, tasks=["Clustering"], datasets=TASK_LIST_CLUSTERING_DE),
1463
+ outputs=data_clustering_de,
1464
+ )
1465
+ with gr.TabItem("Polish"):
1466
+ with gr.Row():
1467
+ gr.Markdown("""
1468
+ **Clustering Polish Leaderboard** ✨🇵🇱
1469
+
1470
+ - **Metric:** Validity Measure (v_measure)
1471
+ - **Languages:** Polish
1472
+ - **Credits:** [Rafał Poświata](https://github.com/rafalposwiata)
1473
+ """)
1474
+ with gr.Row():
1475
+ data_clustering_pl = gr.components.Dataframe(
1476
+ DATA_CLUSTERING_PL,
1477
+ datatype=["number", "markdown"] + ["number"] * len(DATA_CLUSTERING_PL.columns) * 2,
1478
+ type="pandas",
1479
+ )
1480
+ with gr.Row():
1481
+ data_run_clustering_pl = gr.Button("Refresh")
1482
+ data_run_clustering_pl.click(
1483
+ partial(get_mteb_data, tasks=["Clustering"], datasets=TASK_LIST_CLUSTERING_PL),
1484
+ outputs=data_clustering_pl,
1485
+ )
1486
+ with gr.TabItem("Pair Classification"):
1487
+ with gr.TabItem("English"):
1488
+ with gr.Row():
1489
+ gr.Markdown("""
1490
+ **Pair Classification English Leaderboard** 🎭
1491
+
1492
+ - **Metric:** Average Precision based on Cosine Similarities (cos_sim_ap)
1493
+ - **Languages:** English
1494
+ """)
1495
+ with gr.Row():
1496
+ data_pair_classification = gr.components.Dataframe(
1497
+ DATA_PAIR_CLASSIFICATION,
1498
+ datatype=["number", "markdown"] + ["number"] * len(DATA_PAIR_CLASSIFICATION.columns),
1499
+ type="pandas",
1500
+ )
1501
+ with gr.Row():
1502
+ data_run_pair_classification = gr.Button("Refresh")
1503
+ data_run_pair_classification.click(
1504
+ partial(get_mteb_data, tasks=["PairClassification"], datasets=TASK_LIST_PAIR_CLASSIFICATION),
1505
+ outputs=data_pair_classification,
1506
+ )
1507
+ with gr.TabItem("Chinese"):
1508
+ with gr.Row():
1509
+ gr.Markdown("""
1510
+ **Pair Classification Chinese Leaderboard** 🎭🇨🇳
1511
+
1512
+ - **Metric:** Average Precision based on Cosine Similarities (cos_sim_ap)
1513
+ - **Languages:** Chinese
1514
+ - **Credits:** [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding)
1515
+ """)
1516
+ with gr.Row():
1517
+ data_pair_classification_zh = gr.components.Dataframe(
1518
+ DATA_PAIR_CLASSIFICATION_ZH,
1519
+ datatype=["number", "markdown"] + ["number"] * len(DATA_PAIR_CLASSIFICATION_ZH.columns),
1520
+ type="pandas",
1521
+ )
1522
+ with gr.Row():
1523
+ data_run_pair_classification_zh = gr.Button("Refresh")
1524
+ data_run_pair_classification_zh.click(
1525
+ partial(get_mteb_data, tasks=["PairClassification"], datasets=TASK_LIST_PAIR_CLASSIFICATION_ZH),
1526
+ outputs=data_pair_classification_zh,
1527
+ )
1528
+ with gr.TabItem("Polish"):
1529
+ with gr.Row():
1530
+ gr.Markdown("""
1531
+ **Pair Classification Polish Leaderboard** 🎭🇵🇱
1532
+
1533
+ - **Metric:** Average Precision based on Cosine Similarities (cos_sim_ap)
1534
+ - **Languages:** Polish
1535
+ - **Credits:** [Rafał Poświata](https://github.com/rafalposwiata)
1536
+ """)
1537
+ with gr.Row():
1538
+ data_pair_classification_pl = gr.components.Dataframe(
1539
+ DATA_PAIR_CLASSIFICATION_PL,
1540
+ datatype=["number", "markdown"] + ["number"] * len(DATA_PAIR_CLASSIFICATION_PL.columns),
1541
+ type="pandas",
1542
+ )
1543
+ with gr.Row():
1544
+ data_run_pair_classification_pl = gr.Button("Refresh")
1545
+ data_run_pair_classification_pl.click(
1546
+ partial(get_mteb_data, tasks=["PairClassification"], datasets=TASK_LIST_PAIR_CLASSIFICATION_PL),
1547
+ outputs=data_pair_classification_pl,
1548
+ )
1549
+ with gr.TabItem("Reranking"):
1550
+ with gr.TabItem("English"):
1551
+ with gr.Row():
1552
+ gr.Markdown("""
1553
+ **Reranking English Leaderboard** 🥈
1554
+
1555
+ - **Metric:** Mean Average Precision (MAP)
1556
+ - **Languages:** English
1557
+ """)
1558
+ with gr.Row():
1559
+ data_reranking = gr.components.Dataframe(
1560
+ DATA_RERANKING,
1561
+ datatype=["number", "markdown"] + ["number"] * len(DATA_RERANKING.columns),
1562
+ type="pandas",
1563
+ )
1564
+ with gr.Row():
1565
+ data_run_reranking = gr.Button("Refresh")
1566
+ data_run_reranking.click(
1567
+ partial(get_mteb_data, tasks=["Reranking"], datasets=TASK_LIST_RERANKING),
1568
+ outputs=data_reranking,
1569
+ )
1570
+ with gr.TabItem("Chinese"):
1571
+ with gr.Row():
1572
+ gr.Markdown("""
1573
+ **Reranking Chinese Leaderboard** 🥈🇨🇳
1574
+
1575
+ - **Metric:** Mean Average Precision (MAP)
1576
+ - **Languages:** Chinese
1577
+ - **Credits:** [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding)
1578
+ """)
1579
+ with gr.Row():
1580
+ data_reranking_zh = gr.components.Dataframe(
1581
+ DATA_RERANKING_ZH,
1582
+ datatype=["number", "markdown"] + ["number"] * len(DATA_RERANKING_ZH.columns),
1583
+ type="pandas",
1584
+ )
1585
+ with gr.Row():
1586
+ data_run_reranking_zh = gr.Button("Refresh")
1587
+ data_run_reranking_zh.click(
1588
+ partial(get_mteb_data, tasks=["Reranking"], datasets=TASK_LIST_RERANKING_ZH),
1589
+ outputs=data_reranking_zh,
1590
+ )
1591
+ with gr.TabItem("Retrieval"):
1592
+ with gr.TabItem("English"):
1593
+ with gr.Row():
1594
+ gr.Markdown("""
1595
+ **Retrieval English Leaderboard** 🔎
1596
+
1597
+ - **Metric:** Normalized Discounted Cumulative Gain @ k (ndcg_at_10)
1598
+ - **Languages:** English
1599
+ """)
1600
+ with gr.Row():
1601
+ data_retrieval = gr.components.Dataframe(
1602
+ DATA_RETRIEVAL,
1603
+ # Add support for more columns than existing as a buffer for CQADupstack & other Retrieval tasks (e.g. MSMARCOv2)
1604
+ datatype=["number", "markdown"] + ["number"] * len(DATA_RETRIEVAL.columns) * 2,
1605
+ type="pandas",
1606
+ )
1607
+ with gr.Row():
1608
+ data_run_retrieval = gr.Button("Refresh")
1609
+ data_run_retrieval.click(
1610
+ partial(get_mteb_data, tasks=["Retrieval"], datasets=TASK_LIST_RETRIEVAL),
1611
+ outputs=data_retrieval,
1612
+ )
1613
+ with gr.TabItem("Chinese"):
1614
+ with gr.Row():
1615
+ gr.Markdown("""
1616
+ **Retrieval Chinese Leaderboard** 🔎🇨🇳
1617
+
1618
+ - **Metric:** Normalized Discounted Cumulative Gain @ k (ndcg_at_10)
1619
+ - **Languages:** Chinese
1620
+ - **Credits:** [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding)
1621
+ """)
1622
+ with gr.Row():
1623
+ data_retrieval_zh = gr.components.Dataframe(
1624
+ DATA_RETRIEVAL_ZH,
1625
+ # Add support for more columns than existing as a buffer for CQADupstack & other Retrieval tasks (e.g. MSMARCOv2)
1626
+ datatype=["number", "markdown"] + ["number"] * len(DATA_RETRIEVAL_ZH.columns) * 2,
1627
+ type="pandas",
1628
+ )
1629
+ with gr.Row():
1630
+ data_run_retrieval_zh = gr.Button("Refresh")
1631
+ data_run_retrieval_zh.click(
1632
+ partial(get_mteb_data, tasks=["Retrieval"], datasets=TASK_LIST_RETRIEVAL_ZH),
1633
+ outputs=data_retrieval_zh,
1634
+ )
1635
+ with gr.TabItem("Polish"):
1636
+ with gr.Row():
1637
+ gr.Markdown("""
1638
+ **Retrieval Polish Leaderboard** 🔎🇵🇱
1639
+
1640
+ - **Metric:** Normalized Discounted Cumulative Gain @ k (ndcg_at_10)
1641
+ - **Languages:** Polish
1642
+ - **Credits:** [Konrad Wojtasik](https://github.com/kwojtasi) & [BEIR-PL](https://arxiv.org/abs/2305.19840)
1643
+ """)
1644
+ with gr.Row():
1645
+ data_retrieval_pl = gr.components.Dataframe(
1646
+ DATA_RETRIEVAL_PL,
1647
+ # Add support for more columns than existing as a buffer for CQADupstack & other Retrieval tasks (e.g. MSMARCOv2)
1648
+ datatype=["number", "markdown"] + ["number"] * len(DATA_RETRIEVAL_PL.columns) * 2,
1649
+ type="pandas",
1650
+ )
1651
+ with gr.Row():
1652
+ data_run_retrieval_pl = gr.Button("Refresh")
1653
+ data_run_retrieval_pl.click(
1654
+ partial(get_mteb_data, tasks=["Retrieval"], datasets=TASK_LIST_RETRIEVAL_PL),
1655
+ outputs=data_retrieval_pl,
1656
+ )
1657
+ with gr.TabItem("STS"):
1658
+ with gr.TabItem("English"):
1659
+ with gr.Row():
1660
+ gr.Markdown("""
1661
+ **STS English Leaderboard** 🤖
1662
+
1663
+ - **Metric:** Spearman correlation based on cosine similarity
1664
+ - **Languages:** English
1665
+ """)
1666
+ with gr.Row():
1667
+ data_sts_en = gr.components.Dataframe(
1668
+ DATA_STS_EN,
1669
+ datatype=["number", "markdown"] + ["number"] * len(DATA_STS_EN.columns),
1670
+ type="pandas",
1671
+ )
1672
+ with gr.Row():
1673
+ data_run_sts_en = gr.Button("Refresh")
1674
+ data_run_sts_en.click(
1675
+ partial(get_mteb_data, tasks=["STS"], datasets=TASK_LIST_STS),
1676
+ outputs=data_sts_en,
1677
+ )
1678
+ with gr.TabItem("Chinese"):
1679
+ with gr.Row():
1680
+ gr.Markdown("""
1681
+ **STS Chinese Leaderboard** 🤖🇨🇳
1682
+
1683
+ - **Metric:** Spearman correlation based on cosine similarity
1684
+ - **Languages:** Chinese
1685
+ - **Credits:** [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding)
1686
+ """)
1687
+ with gr.Row():
1688
+ data_sts_zh = gr.components.Dataframe(
1689
+ DATA_STS_ZH,
1690
+ datatype=["number", "markdown"] + ["number"] * len(DATA_STS_ZH.columns),
1691
+ type="pandas",
1692
+ )
1693
+ with gr.Row():
1694
+ data_run_sts_zh = gr.Button("Refresh")
1695
+ data_run_sts_zh.click(
1696
+ partial(get_mteb_data, tasks=["STS"], datasets=TASK_LIST_STS_ZH),
1697
+ outputs=data_sts_zh,
1698
+ )
1699
+ with gr.TabItem("Polish"):
1700
+ with gr.Row():
1701
+ gr.Markdown("""
1702
+ **STS Polish Leaderboard** 🤖🇵🇱
1703
+
1704
+ - **Metric:** Spearman correlation based on cosine similarity
1705
+ - **Languages:** Polish
1706
+ - **Credits:** [Rafał Poświata](https://github.com/rafalposwiata)
1707
+ """)
1708
+ with gr.Row():
1709
+ data_sts_pl = gr.components.Dataframe(
1710
+ DATA_STS_PL,
1711
+ datatype=["number", "markdown"] + ["number"] * len(DATA_STS_PL.columns),
1712
+ type="pandas",
1713
+ )
1714
+ with gr.Row():
1715
+ data_run_sts_pl = gr.Button("Refresh")
1716
+ data_run_sts_pl.click(
1717
+ partial(get_mteb_data, tasks=["STS"], datasets=TASK_LIST_STS_PL),
1718
+ outputs=data_sts_pl,
1719
+ )
1720
+ with gr.TabItem("Other"):
1721
+ with gr.Row():
1722
+ gr.Markdown("""
1723
+ **STS Other Leaderboard** 👽
1724
+
1725
+ - **Metric:** Spearman correlation based on cosine similarity
1726
+ - **Languages:** Arabic, Chinese, Dutch, English, French, German, Italian, Korean, Polish, Russian, Spanish (Only language combos not included in the other tabs)
1727
+ """)
1728
+ with gr.Row():
1729
+ data_sts_other = gr.components.Dataframe(
1730
+ DATA_STS_OTHER,
1731
+ datatype=["number", "markdown"] + ["number"] * len(DATA_STS_OTHER.columns) * 2,
1732
+ type="pandas",
1733
+ )
1734
+ with gr.Row():
1735
+ data_run_sts_other = gr.Button("Refresh")
1736
+ data_run_sts_other.click(
1737
+ partial(get_mteb_data, tasks=["STS"], datasets=TASK_LIST_STS_OTHER),
1738
+ outputs=data_sts_other,
1739
+ )
1740
+ with gr.TabItem("Summarization"):
1741
+ with gr.Row():
1742
+ gr.Markdown("""
1743
+ **Summarization Leaderboard** 📜
1744
+
1745
+ - **Metric:** Spearman correlation based on cosine similarity
1746
+ - **Languages:** English
1747
+ """)
1748
+ with gr.Row():
1749
+ data_summarization = gr.components.Dataframe(
1750
+ DATA_SUMMARIZATION,
1751
+ datatype=["number", "markdown"] + ["number"] * 2,
1752
+ type="pandas",
1753
+ )
1754
+ with gr.Row():
1755
+ data_run = gr.Button("Refresh")
1756
+ data_run.click(
1757
+ partial(get_mteb_data, tasks=["Summarization"]),
1758
+ outputs=data_summarization,
1759
+ )
1760
+ gr.Markdown(r"""
1761
+
1762
+ If this work is useful to you, please consider citing:
1763
+
1764
+ ```bibtex
1765
+ @article{SeaEval2023,
1766
+ title={SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning},
1767
+ author={Wang, Bin and Liu, Zhengyuan and Huang, Xin and Jiao, Fangkai and Ding, Yang and Aw, Ai Ti and Chen, Nancy F.},
1768
+ journal={arXiv preprint arXiv:2309.04766},
1769
+ year={2023}
1770
+ }
1771
+ ```
1772
+ """)
1773
+ # Running the functions on page load in addition to when the button is clicked
1774
+ # This is optional - If deactivated the data loaded at "Build time" is shown like for Overall tab
1775
+ """
1776
+ block.load(get_mteb_data, inputs=[task_bitext_mining], outputs=data_bitext_mining)
1777
+ """
1778
+
1779
+ block.queue(max_size=10)
1780
+ block.launch(server_name="0.0.0.0", share=True)
1781
+
1782
+
1783
+ # Possible changes:
1784
+ # Could add graphs / other visual content
1785
+ # Could add verification marks
1786
+
1787
+ # Sources:
1788
+ # https://huggingface.co/spaces/gradio/leaderboard
1789
+ # https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard
1790
+ # https://getemoji.com/
requirements.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ gradio
2
+ datasets
3
+ pandas
4
+ huggingface_hub