|
--- |
|
pipeline_tag: sentence-similarity |
|
tags: |
|
- finetuner |
|
- sentence-transformers |
|
- feature-extraction |
|
- sentence-similarity |
|
- mteb |
|
datasets: |
|
- jinaai/negation-dataset |
|
language: en |
|
license: apache-2.0 |
|
model-index: |
|
- name: jina-embedding-b-en-v1 |
|
results: |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (en) |
|
config: en |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 66.73134328358208 |
|
- type: ap |
|
value: 28.30575908745204 |
|
- type: f1 |
|
value: 60.02420130946191 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_polarity |
|
name: MTEB AmazonPolarityClassification |
|
config: default |
|
split: test |
|
revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
|
metrics: |
|
- type: accuracy |
|
value: 67.6068 |
|
- type: ap |
|
value: 63.5899352938589 |
|
- type: f1 |
|
value: 65.64285334357656 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (en) |
|
config: en |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 31.178 |
|
- type: f1 |
|
value: 29.68460843733487 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana |
|
name: MTEB ArguAna |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 24.964 |
|
- type: map_at_10 |
|
value: 40.217999999999996 |
|
- type: map_at_100 |
|
value: 41.263 |
|
- type: map_at_1000 |
|
value: 41.277 |
|
- type: map_at_3 |
|
value: 35.183 |
|
- type: map_at_5 |
|
value: 38.045 |
|
- type: mrr_at_1 |
|
value: 25.107000000000003 |
|
- type: mrr_at_10 |
|
value: 40.272999999999996 |
|
- type: mrr_at_100 |
|
value: 41.318 |
|
- type: mrr_at_1000 |
|
value: 41.333 |
|
- type: mrr_at_3 |
|
value: 35.242000000000004 |
|
- type: mrr_at_5 |
|
value: 38.101 |
|
- type: ndcg_at_1 |
|
value: 24.964 |
|
- type: ndcg_at_10 |
|
value: 49.006 |
|
- type: ndcg_at_100 |
|
value: 53.446000000000005 |
|
- type: ndcg_at_1000 |
|
value: 53.813 |
|
- type: ndcg_at_3 |
|
value: 38.598 |
|
- type: ndcg_at_5 |
|
value: 43.74 |
|
- type: precision_at_1 |
|
value: 24.964 |
|
- type: precision_at_10 |
|
value: 7.724 |
|
- type: precision_at_100 |
|
value: 0.966 |
|
- type: precision_at_1000 |
|
value: 0.099 |
|
- type: precision_at_3 |
|
value: 16.169 |
|
- type: precision_at_5 |
|
value: 12.191 |
|
- type: recall_at_1 |
|
value: 24.964 |
|
- type: recall_at_10 |
|
value: 77.24 |
|
- type: recall_at_100 |
|
value: 96.586 |
|
- type: recall_at_1000 |
|
value: 99.431 |
|
- type: recall_at_3 |
|
value: 48.506 |
|
- type: recall_at_5 |
|
value: 60.953 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-p2p |
|
name: MTEB ArxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
|
metrics: |
|
- type: v_measure |
|
value: 39.25203906042786 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-s2s |
|
name: MTEB ArxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 |
|
metrics: |
|
- type: v_measure |
|
value: 29.07648348376354 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/askubuntudupquestions-reranking |
|
name: MTEB AskUbuntuDupQuestions |
|
config: default |
|
split: test |
|
revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
|
metrics: |
|
- type: map |
|
value: 62.4029266143623 |
|
- type: mrr |
|
value: 75.45750340764191 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/biosses-sts |
|
name: MTEB BIOSSES |
|
config: default |
|
split: test |
|
revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 85.92280995704714 |
|
- type: cos_sim_spearman |
|
value: 83.58082010833608 |
|
- type: euclidean_pearson |
|
value: 48.64744162695948 |
|
- type: euclidean_spearman |
|
value: 48.817377397301556 |
|
- type: manhattan_pearson |
|
value: 48.87684776623195 |
|
- type: manhattan_spearman |
|
value: 48.94268145725884 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/banking77 |
|
name: MTEB Banking77Classification |
|
config: default |
|
split: test |
|
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
|
metrics: |
|
- type: accuracy |
|
value: 84.05519480519482 |
|
- type: f1 |
|
value: 83.94978356890618 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-p2p |
|
name: MTEB BiorxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 |
|
metrics: |
|
- type: v_measure |
|
value: 32.2033276486685 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-s2s |
|
name: MTEB BiorxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 |
|
metrics: |
|
- type: v_measure |
|
value: 26.631954164406014 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackAndroidRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 29.625 |
|
- type: map_at_10 |
|
value: 40.037 |
|
- type: map_at_100 |
|
value: 41.52 |
|
- type: map_at_1000 |
|
value: 41.654 |
|
- type: map_at_3 |
|
value: 36.818 |
|
- type: map_at_5 |
|
value: 38.426 |
|
- type: mrr_at_1 |
|
value: 35.336 |
|
- type: mrr_at_10 |
|
value: 45.395 |
|
- type: mrr_at_100 |
|
value: 46.221000000000004 |
|
- type: mrr_at_1000 |
|
value: 46.264 |
|
- type: mrr_at_3 |
|
value: 42.823 |
|
- type: mrr_at_5 |
|
value: 44.204 |
|
- type: ndcg_at_1 |
|
value: 35.336 |
|
- type: ndcg_at_10 |
|
value: 46.326 |
|
- type: ndcg_at_100 |
|
value: 51.795 |
|
- type: ndcg_at_1000 |
|
value: 53.834 |
|
- type: ndcg_at_3 |
|
value: 41.299 |
|
- type: ndcg_at_5 |
|
value: 43.247 |
|
- type: precision_at_1 |
|
value: 35.336 |
|
- type: precision_at_10 |
|
value: 8.627 |
|
- type: precision_at_100 |
|
value: 1.428 |
|
- type: precision_at_1000 |
|
value: 0.197 |
|
- type: precision_at_3 |
|
value: 19.647000000000002 |
|
- type: precision_at_5 |
|
value: 13.733999999999998 |
|
- type: recall_at_1 |
|
value: 29.625 |
|
- type: recall_at_10 |
|
value: 59.165 |
|
- type: recall_at_100 |
|
value: 81.675 |
|
- type: recall_at_1000 |
|
value: 94.17 |
|
- type: recall_at_3 |
|
value: 44.485 |
|
- type: recall_at_5 |
|
value: 50.198 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackEnglishRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 26.687 |
|
- type: map_at_10 |
|
value: 36.062 |
|
- type: map_at_100 |
|
value: 37.263000000000005 |
|
- type: map_at_1000 |
|
value: 37.397999999999996 |
|
- type: map_at_3 |
|
value: 32.967 |
|
- type: map_at_5 |
|
value: 34.75 |
|
- type: mrr_at_1 |
|
value: 33.885 |
|
- type: mrr_at_10 |
|
value: 42.632999999999996 |
|
- type: mrr_at_100 |
|
value: 43.305 |
|
- type: mrr_at_1000 |
|
value: 43.354 |
|
- type: mrr_at_3 |
|
value: 39.958 |
|
- type: mrr_at_5 |
|
value: 41.63 |
|
- type: ndcg_at_1 |
|
value: 33.885 |
|
- type: ndcg_at_10 |
|
value: 42.001 |
|
- type: ndcg_at_100 |
|
value: 46.436 |
|
- type: ndcg_at_1000 |
|
value: 48.774 |
|
- type: ndcg_at_3 |
|
value: 37.183 |
|
- type: ndcg_at_5 |
|
value: 39.605000000000004 |
|
- type: precision_at_1 |
|
value: 33.885 |
|
- type: precision_at_10 |
|
value: 7.962 |
|
- type: precision_at_100 |
|
value: 1.283 |
|
- type: precision_at_1000 |
|
value: 0.18 |
|
- type: precision_at_3 |
|
value: 17.855999999999998 |
|
- type: precision_at_5 |
|
value: 13.083 |
|
- type: recall_at_1 |
|
value: 26.687 |
|
- type: recall_at_10 |
|
value: 52.75 |
|
- type: recall_at_100 |
|
value: 71.324 |
|
- type: recall_at_1000 |
|
value: 86.356 |
|
- type: recall_at_3 |
|
value: 38.83 |
|
- type: recall_at_5 |
|
value: 45.23 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGamingRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 34.02 |
|
- type: map_at_10 |
|
value: 45.751999999999995 |
|
- type: map_at_100 |
|
value: 46.867 |
|
- type: map_at_1000 |
|
value: 46.93 |
|
- type: map_at_3 |
|
value: 42.409 |
|
- type: map_at_5 |
|
value: 44.464999999999996 |
|
- type: mrr_at_1 |
|
value: 38.307 |
|
- type: mrr_at_10 |
|
value: 48.718 |
|
- type: mrr_at_100 |
|
value: 49.509 |
|
- type: mrr_at_1000 |
|
value: 49.542 |
|
- type: mrr_at_3 |
|
value: 46.007999999999996 |
|
- type: mrr_at_5 |
|
value: 47.766999999999996 |
|
- type: ndcg_at_1 |
|
value: 38.307 |
|
- type: ndcg_at_10 |
|
value: 51.666999999999994 |
|
- type: ndcg_at_100 |
|
value: 56.242000000000004 |
|
- type: ndcg_at_1000 |
|
value: 57.477999999999994 |
|
- type: ndcg_at_3 |
|
value: 45.912 |
|
- type: ndcg_at_5 |
|
value: 49.106 |
|
- type: precision_at_1 |
|
value: 38.307 |
|
- type: precision_at_10 |
|
value: 8.476 |
|
- type: precision_at_100 |
|
value: 1.176 |
|
- type: precision_at_1000 |
|
value: 0.133 |
|
- type: precision_at_3 |
|
value: 20.522000000000002 |
|
- type: precision_at_5 |
|
value: 14.557999999999998 |
|
- type: recall_at_1 |
|
value: 34.02 |
|
- type: recall_at_10 |
|
value: 66.046 |
|
- type: recall_at_100 |
|
value: 85.817 |
|
- type: recall_at_1000 |
|
value: 94.453 |
|
- type: recall_at_3 |
|
value: 51.059 |
|
- type: recall_at_5 |
|
value: 58.667 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGisRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.939 |
|
- type: map_at_10 |
|
value: 32.627 |
|
- type: map_at_100 |
|
value: 33.617999999999995 |
|
- type: map_at_1000 |
|
value: 33.701 |
|
- type: map_at_3 |
|
value: 30.11 |
|
- type: map_at_5 |
|
value: 31.380000000000003 |
|
- type: mrr_at_1 |
|
value: 25.989 |
|
- type: mrr_at_10 |
|
value: 34.655 |
|
- type: mrr_at_100 |
|
value: 35.502 |
|
- type: mrr_at_1000 |
|
value: 35.563 |
|
- type: mrr_at_3 |
|
value: 32.109 |
|
- type: mrr_at_5 |
|
value: 33.426 |
|
- type: ndcg_at_1 |
|
value: 25.989 |
|
- type: ndcg_at_10 |
|
value: 37.657000000000004 |
|
- type: ndcg_at_100 |
|
value: 42.467 |
|
- type: ndcg_at_1000 |
|
value: 44.677 |
|
- type: ndcg_at_3 |
|
value: 32.543 |
|
- type: ndcg_at_5 |
|
value: 34.74 |
|
- type: precision_at_1 |
|
value: 25.989 |
|
- type: precision_at_10 |
|
value: 5.876 |
|
- type: precision_at_100 |
|
value: 0.8710000000000001 |
|
- type: precision_at_1000 |
|
value: 0.11 |
|
- type: precision_at_3 |
|
value: 13.861 |
|
- type: precision_at_5 |
|
value: 9.626999999999999 |
|
- type: recall_at_1 |
|
value: 23.939 |
|
- type: recall_at_10 |
|
value: 51.28 |
|
- type: recall_at_100 |
|
value: 73.428 |
|
- type: recall_at_1000 |
|
value: 90.309 |
|
- type: recall_at_3 |
|
value: 37.245 |
|
- type: recall_at_5 |
|
value: 42.541000000000004 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackMathematicaRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 15.082 |
|
- type: map_at_10 |
|
value: 22.486 |
|
- type: map_at_100 |
|
value: 23.687 |
|
- type: map_at_1000 |
|
value: 23.807000000000002 |
|
- type: map_at_3 |
|
value: 20.076 |
|
- type: map_at_5 |
|
value: 21.362000000000002 |
|
- type: mrr_at_1 |
|
value: 18.532 |
|
- type: mrr_at_10 |
|
value: 26.605 |
|
- type: mrr_at_100 |
|
value: 27.628999999999998 |
|
- type: mrr_at_1000 |
|
value: 27.698 |
|
- type: mrr_at_3 |
|
value: 23.964 |
|
- type: mrr_at_5 |
|
value: 25.319000000000003 |
|
- type: ndcg_at_1 |
|
value: 18.532 |
|
- type: ndcg_at_10 |
|
value: 27.474999999999998 |
|
- type: ndcg_at_100 |
|
value: 33.357 |
|
- type: ndcg_at_1000 |
|
value: 36.361 |
|
- type: ndcg_at_3 |
|
value: 22.851 |
|
- type: ndcg_at_5 |
|
value: 24.87 |
|
- type: precision_at_1 |
|
value: 18.532 |
|
- type: precision_at_10 |
|
value: 5.210999999999999 |
|
- type: precision_at_100 |
|
value: 0.9329999999999999 |
|
- type: precision_at_1000 |
|
value: 0.134 |
|
- type: precision_at_3 |
|
value: 11.235000000000001 |
|
- type: precision_at_5 |
|
value: 8.134 |
|
- type: recall_at_1 |
|
value: 15.082 |
|
- type: recall_at_10 |
|
value: 38.759 |
|
- type: recall_at_100 |
|
value: 64.621 |
|
- type: recall_at_1000 |
|
value: 86.162 |
|
- type: recall_at_3 |
|
value: 26.055 |
|
- type: recall_at_5 |
|
value: 31.208999999999996 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackPhysicsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 24.759999999999998 |
|
- type: map_at_10 |
|
value: 33.706 |
|
- type: map_at_100 |
|
value: 35.0 |
|
- type: map_at_1000 |
|
value: 35.134 |
|
- type: map_at_3 |
|
value: 30.789 |
|
- type: map_at_5 |
|
value: 32.427 |
|
- type: mrr_at_1 |
|
value: 29.548000000000002 |
|
- type: mrr_at_10 |
|
value: 38.521 |
|
- type: mrr_at_100 |
|
value: 39.432 |
|
- type: mrr_at_1000 |
|
value: 39.494 |
|
- type: mrr_at_3 |
|
value: 35.691 |
|
- type: mrr_at_5 |
|
value: 37.424 |
|
- type: ndcg_at_1 |
|
value: 29.548000000000002 |
|
- type: ndcg_at_10 |
|
value: 39.301 |
|
- type: ndcg_at_100 |
|
value: 44.907000000000004 |
|
- type: ndcg_at_1000 |
|
value: 47.494 |
|
- type: ndcg_at_3 |
|
value: 34.08 |
|
- type: ndcg_at_5 |
|
value: 36.649 |
|
- type: precision_at_1 |
|
value: 29.548000000000002 |
|
- type: precision_at_10 |
|
value: 7.084 |
|
- type: precision_at_100 |
|
value: 1.169 |
|
- type: precision_at_1000 |
|
value: 0.158 |
|
- type: precision_at_3 |
|
value: 15.881 |
|
- type: precision_at_5 |
|
value: 11.53 |
|
- type: recall_at_1 |
|
value: 24.759999999999998 |
|
- type: recall_at_10 |
|
value: 51.202000000000005 |
|
- type: recall_at_100 |
|
value: 74.542 |
|
- type: recall_at_1000 |
|
value: 91.669 |
|
- type: recall_at_3 |
|
value: 36.892 |
|
- type: recall_at_5 |
|
value: 43.333 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackProgrammersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.247999999999998 |
|
- type: map_at_10 |
|
value: 31.878 |
|
- type: map_at_100 |
|
value: 33.135 |
|
- type: map_at_1000 |
|
value: 33.263999999999996 |
|
- type: map_at_3 |
|
value: 29.406 |
|
- type: map_at_5 |
|
value: 30.602 |
|
- type: mrr_at_1 |
|
value: 28.767 |
|
- type: mrr_at_10 |
|
value: 36.929 |
|
- type: mrr_at_100 |
|
value: 37.844 |
|
- type: mrr_at_1000 |
|
value: 37.913000000000004 |
|
- type: mrr_at_3 |
|
value: 34.589 |
|
- type: mrr_at_5 |
|
value: 35.908 |
|
- type: ndcg_at_1 |
|
value: 28.767 |
|
- type: ndcg_at_10 |
|
value: 37.172 |
|
- type: ndcg_at_100 |
|
value: 42.842 |
|
- type: ndcg_at_1000 |
|
value: 45.534 |
|
- type: ndcg_at_3 |
|
value: 32.981 |
|
- type: ndcg_at_5 |
|
value: 34.628 |
|
- type: precision_at_1 |
|
value: 28.767 |
|
- type: precision_at_10 |
|
value: 6.678000000000001 |
|
- type: precision_at_100 |
|
value: 1.1199999999999999 |
|
- type: precision_at_1000 |
|
value: 0.155 |
|
- type: precision_at_3 |
|
value: 15.715000000000002 |
|
- type: precision_at_5 |
|
value: 10.913 |
|
- type: recall_at_1 |
|
value: 23.247999999999998 |
|
- type: recall_at_10 |
|
value: 48.16 |
|
- type: recall_at_100 |
|
value: 72.753 |
|
- type: recall_at_1000 |
|
value: 90.8 |
|
- type: recall_at_3 |
|
value: 35.961999999999996 |
|
- type: recall_at_5 |
|
value: 40.504 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.825583333333334 |
|
- type: map_at_10 |
|
value: 32.2845 |
|
- type: map_at_100 |
|
value: 33.48566666666667 |
|
- type: map_at_1000 |
|
value: 33.60833333333333 |
|
- type: map_at_3 |
|
value: 29.604916666666664 |
|
- type: map_at_5 |
|
value: 31.015333333333334 |
|
- type: mrr_at_1 |
|
value: 27.850916666666663 |
|
- type: mrr_at_10 |
|
value: 36.122416666666666 |
|
- type: mrr_at_100 |
|
value: 37.01275 |
|
- type: mrr_at_1000 |
|
value: 37.07566666666667 |
|
- type: mrr_at_3 |
|
value: 33.665749999999996 |
|
- type: mrr_at_5 |
|
value: 35.00916666666667 |
|
- type: ndcg_at_1 |
|
value: 27.850916666666663 |
|
- type: ndcg_at_10 |
|
value: 37.47625 |
|
- type: ndcg_at_100 |
|
value: 42.74433333333334 |
|
- type: ndcg_at_1000 |
|
value: 45.21991666666667 |
|
- type: ndcg_at_3 |
|
value: 32.70916666666667 |
|
- type: ndcg_at_5 |
|
value: 34.80658333333333 |
|
- type: precision_at_1 |
|
value: 27.850916666666663 |
|
- type: precision_at_10 |
|
value: 6.5761666666666665 |
|
- type: precision_at_100 |
|
value: 1.0879999999999999 |
|
- type: precision_at_1000 |
|
value: 0.15058333333333332 |
|
- type: precision_at_3 |
|
value: 14.933833333333336 |
|
- type: precision_at_5 |
|
value: 10.607249999999999 |
|
- type: recall_at_1 |
|
value: 23.825583333333334 |
|
- type: recall_at_10 |
|
value: 49.100500000000004 |
|
- type: recall_at_100 |
|
value: 72.21133333333334 |
|
- type: recall_at_1000 |
|
value: 89.34791666666666 |
|
- type: recall_at_3 |
|
value: 35.90525 |
|
- type: recall_at_5 |
|
value: 41.24583333333334 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackStatsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 21.343 |
|
- type: map_at_10 |
|
value: 27.313 |
|
- type: map_at_100 |
|
value: 28.316999999999997 |
|
- type: map_at_1000 |
|
value: 28.406 |
|
- type: map_at_3 |
|
value: 25.06 |
|
- type: map_at_5 |
|
value: 26.409 |
|
- type: mrr_at_1 |
|
value: 23.313 |
|
- type: mrr_at_10 |
|
value: 29.467 |
|
- type: mrr_at_100 |
|
value: 30.348999999999997 |
|
- type: mrr_at_1000 |
|
value: 30.42 |
|
- type: mrr_at_3 |
|
value: 27.173000000000002 |
|
- type: mrr_at_5 |
|
value: 28.461 |
|
- type: ndcg_at_1 |
|
value: 23.313 |
|
- type: ndcg_at_10 |
|
value: 31.183 |
|
- type: ndcg_at_100 |
|
value: 36.252 |
|
- type: ndcg_at_1000 |
|
value: 38.582 |
|
- type: ndcg_at_3 |
|
value: 26.838 |
|
- type: ndcg_at_5 |
|
value: 29.042 |
|
- type: precision_at_1 |
|
value: 23.313 |
|
- type: precision_at_10 |
|
value: 4.9079999999999995 |
|
- type: precision_at_100 |
|
value: 0.808 |
|
- type: precision_at_1000 |
|
value: 0.109 |
|
- type: precision_at_3 |
|
value: 11.299 |
|
- type: precision_at_5 |
|
value: 8.097999999999999 |
|
- type: recall_at_1 |
|
value: 21.343 |
|
- type: recall_at_10 |
|
value: 41.047 |
|
- type: recall_at_100 |
|
value: 64.372 |
|
- type: recall_at_1000 |
|
value: 81.499 |
|
- type: recall_at_3 |
|
value: 29.337000000000003 |
|
- type: recall_at_5 |
|
value: 34.756 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackTexRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 16.595 |
|
- type: map_at_10 |
|
value: 23.433 |
|
- type: map_at_100 |
|
value: 24.578 |
|
- type: map_at_1000 |
|
value: 24.709999999999997 |
|
- type: map_at_3 |
|
value: 21.268 |
|
- type: map_at_5 |
|
value: 22.393 |
|
- type: mrr_at_1 |
|
value: 20.131 |
|
- type: mrr_at_10 |
|
value: 27.026 |
|
- type: mrr_at_100 |
|
value: 28.003 |
|
- type: mrr_at_1000 |
|
value: 28.083999999999996 |
|
- type: mrr_at_3 |
|
value: 24.966 |
|
- type: mrr_at_5 |
|
value: 26.064999999999998 |
|
- type: ndcg_at_1 |
|
value: 20.131 |
|
- type: ndcg_at_10 |
|
value: 27.846 |
|
- type: ndcg_at_100 |
|
value: 33.318999999999996 |
|
- type: ndcg_at_1000 |
|
value: 36.403 |
|
- type: ndcg_at_3 |
|
value: 23.883 |
|
- type: ndcg_at_5 |
|
value: 25.595000000000002 |
|
- type: precision_at_1 |
|
value: 20.131 |
|
- type: precision_at_10 |
|
value: 5.034000000000001 |
|
- type: precision_at_100 |
|
value: 0.9079999999999999 |
|
- type: precision_at_1000 |
|
value: 0.13699999999999998 |
|
- type: precision_at_3 |
|
value: 11.23 |
|
- type: precision_at_5 |
|
value: 8.032 |
|
- type: recall_at_1 |
|
value: 16.595 |
|
- type: recall_at_10 |
|
value: 37.576 |
|
- type: recall_at_100 |
|
value: 62.044 |
|
- type: recall_at_1000 |
|
value: 83.97 |
|
- type: recall_at_3 |
|
value: 26.631 |
|
- type: recall_at_5 |
|
value: 31.002000000000002 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackUnixRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 24.85 |
|
- type: map_at_10 |
|
value: 32.762 |
|
- type: map_at_100 |
|
value: 33.896 |
|
- type: map_at_1000 |
|
value: 34.006 |
|
- type: map_at_3 |
|
value: 29.965000000000003 |
|
- type: map_at_5 |
|
value: 31.485999999999997 |
|
- type: mrr_at_1 |
|
value: 28.731 |
|
- type: mrr_at_10 |
|
value: 36.504999999999995 |
|
- type: mrr_at_100 |
|
value: 37.364999999999995 |
|
- type: mrr_at_1000 |
|
value: 37.431 |
|
- type: mrr_at_3 |
|
value: 34.033 |
|
- type: mrr_at_5 |
|
value: 35.4 |
|
- type: ndcg_at_1 |
|
value: 28.731 |
|
- type: ndcg_at_10 |
|
value: 37.788 |
|
- type: ndcg_at_100 |
|
value: 43.1 |
|
- type: ndcg_at_1000 |
|
value: 45.623999999999995 |
|
- type: ndcg_at_3 |
|
value: 32.717 |
|
- type: ndcg_at_5 |
|
value: 35.024 |
|
- type: precision_at_1 |
|
value: 28.731 |
|
- type: precision_at_10 |
|
value: 6.371 |
|
- type: precision_at_100 |
|
value: 1.02 |
|
- type: precision_at_1000 |
|
value: 0.135 |
|
- type: precision_at_3 |
|
value: 14.521 |
|
- type: precision_at_5 |
|
value: 10.41 |
|
- type: recall_at_1 |
|
value: 24.85 |
|
- type: recall_at_10 |
|
value: 49.335 |
|
- type: recall_at_100 |
|
value: 72.792 |
|
- type: recall_at_1000 |
|
value: 90.525 |
|
- type: recall_at_3 |
|
value: 35.698 |
|
- type: recall_at_5 |
|
value: 41.385 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWebmastersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.016000000000002 |
|
- type: map_at_10 |
|
value: 32.126 |
|
- type: map_at_100 |
|
value: 33.786 |
|
- type: map_at_1000 |
|
value: 34.012 |
|
- type: map_at_3 |
|
value: 29.256 |
|
- type: map_at_5 |
|
value: 30.552 |
|
- type: mrr_at_1 |
|
value: 27.272999999999996 |
|
- type: mrr_at_10 |
|
value: 35.967 |
|
- type: mrr_at_100 |
|
value: 37.082 |
|
- type: mrr_at_1000 |
|
value: 37.146 |
|
- type: mrr_at_3 |
|
value: 33.531 |
|
- type: mrr_at_5 |
|
value: 34.697 |
|
- type: ndcg_at_1 |
|
value: 27.272999999999996 |
|
- type: ndcg_at_10 |
|
value: 37.945 |
|
- type: ndcg_at_100 |
|
value: 43.928 |
|
- type: ndcg_at_1000 |
|
value: 46.772999999999996 |
|
- type: ndcg_at_3 |
|
value: 33.111000000000004 |
|
- type: ndcg_at_5 |
|
value: 34.794000000000004 |
|
- type: precision_at_1 |
|
value: 27.272999999999996 |
|
- type: precision_at_10 |
|
value: 7.53 |
|
- type: precision_at_100 |
|
value: 1.512 |
|
- type: precision_at_1000 |
|
value: 0.241 |
|
- type: precision_at_3 |
|
value: 15.547 |
|
- type: precision_at_5 |
|
value: 11.146 |
|
- type: recall_at_1 |
|
value: 23.016000000000002 |
|
- type: recall_at_10 |
|
value: 49.576 |
|
- type: recall_at_100 |
|
value: 75.74600000000001 |
|
- type: recall_at_1000 |
|
value: 94.069 |
|
- type: recall_at_3 |
|
value: 35.964 |
|
- type: recall_at_5 |
|
value: 40.455999999999996 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWordpressRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 22.742 |
|
- type: map_at_10 |
|
value: 29.232000000000003 |
|
- type: map_at_100 |
|
value: 30.160999999999998 |
|
- type: map_at_1000 |
|
value: 30.278 |
|
- type: map_at_3 |
|
value: 27.134999999999998 |
|
- type: map_at_5 |
|
value: 27.932000000000002 |
|
- type: mrr_at_1 |
|
value: 24.399 |
|
- type: mrr_at_10 |
|
value: 31.048 |
|
- type: mrr_at_100 |
|
value: 31.912000000000003 |
|
- type: mrr_at_1000 |
|
value: 31.999 |
|
- type: mrr_at_3 |
|
value: 29.144 |
|
- type: mrr_at_5 |
|
value: 29.809 |
|
- type: ndcg_at_1 |
|
value: 24.399 |
|
- type: ndcg_at_10 |
|
value: 33.354 |
|
- type: ndcg_at_100 |
|
value: 38.287 |
|
- type: ndcg_at_1000 |
|
value: 41.105000000000004 |
|
- type: ndcg_at_3 |
|
value: 29.112 |
|
- type: ndcg_at_5 |
|
value: 30.379 |
|
- type: precision_at_1 |
|
value: 24.399 |
|
- type: precision_at_10 |
|
value: 5.157 |
|
- type: precision_at_100 |
|
value: 0.828 |
|
- type: precision_at_1000 |
|
value: 0.11800000000000001 |
|
- type: precision_at_3 |
|
value: 11.892 |
|
- type: precision_at_5 |
|
value: 8.022 |
|
- type: recall_at_1 |
|
value: 22.742 |
|
- type: recall_at_10 |
|
value: 44.31 |
|
- type: recall_at_100 |
|
value: 67.422 |
|
- type: recall_at_1000 |
|
value: 88.193 |
|
- type: recall_at_3 |
|
value: 32.705 |
|
- type: recall_at_5 |
|
value: 35.669000000000004 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: climate-fever |
|
name: MTEB ClimateFEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 9.067 |
|
- type: map_at_10 |
|
value: 14.821000000000002 |
|
- type: map_at_100 |
|
value: 16.195 |
|
- type: map_at_1000 |
|
value: 16.359 |
|
- type: map_at_3 |
|
value: 12.666 |
|
- type: map_at_5 |
|
value: 13.675999999999998 |
|
- type: mrr_at_1 |
|
value: 20.326 |
|
- type: mrr_at_10 |
|
value: 29.798000000000002 |
|
- type: mrr_at_100 |
|
value: 30.875000000000004 |
|
- type: mrr_at_1000 |
|
value: 30.928 |
|
- type: mrr_at_3 |
|
value: 26.678 |
|
- type: mrr_at_5 |
|
value: 28.433000000000003 |
|
- type: ndcg_at_1 |
|
value: 20.326 |
|
- type: ndcg_at_10 |
|
value: 21.477 |
|
- type: ndcg_at_100 |
|
value: 27.637 |
|
- type: ndcg_at_1000 |
|
value: 30.953000000000003 |
|
- type: ndcg_at_3 |
|
value: 17.456 |
|
- type: ndcg_at_5 |
|
value: 18.789 |
|
- type: precision_at_1 |
|
value: 20.326 |
|
- type: precision_at_10 |
|
value: 6.482 |
|
- type: precision_at_100 |
|
value: 1.302 |
|
- type: precision_at_1000 |
|
value: 0.191 |
|
- type: precision_at_3 |
|
value: 12.53 |
|
- type: precision_at_5 |
|
value: 9.603 |
|
- type: recall_at_1 |
|
value: 9.067 |
|
- type: recall_at_10 |
|
value: 26.246000000000002 |
|
- type: recall_at_100 |
|
value: 47.837 |
|
- type: recall_at_1000 |
|
value: 66.637 |
|
- type: recall_at_3 |
|
value: 16.468 |
|
- type: recall_at_5 |
|
value: 20.088 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: dbpedia-entity |
|
name: MTEB DBPedia |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 7.563000000000001 |
|
- type: map_at_10 |
|
value: 15.22 |
|
- type: map_at_100 |
|
value: 20.048 |
|
- type: map_at_1000 |
|
value: 21.17 |
|
- type: map_at_3 |
|
value: 11.627 |
|
- type: map_at_5 |
|
value: 13.239 |
|
- type: mrr_at_1 |
|
value: 56.25 |
|
- type: mrr_at_10 |
|
value: 64.846 |
|
- type: mrr_at_100 |
|
value: 65.405 |
|
- type: mrr_at_1000 |
|
value: 65.41799999999999 |
|
- type: mrr_at_3 |
|
value: 63.125 |
|
- type: mrr_at_5 |
|
value: 64.1 |
|
- type: ndcg_at_1 |
|
value: 45.0 |
|
- type: ndcg_at_10 |
|
value: 32.437 |
|
- type: ndcg_at_100 |
|
value: 35.483 |
|
- type: ndcg_at_1000 |
|
value: 42.186 |
|
- type: ndcg_at_3 |
|
value: 37.297000000000004 |
|
- type: ndcg_at_5 |
|
value: 34.697 |
|
- type: precision_at_1 |
|
value: 56.25 |
|
- type: precision_at_10 |
|
value: 25.15 |
|
- type: precision_at_100 |
|
value: 7.539999999999999 |
|
- type: precision_at_1000 |
|
value: 1.678 |
|
- type: precision_at_3 |
|
value: 40.666999999999994 |
|
- type: precision_at_5 |
|
value: 33.45 |
|
- type: recall_at_1 |
|
value: 7.563000000000001 |
|
- type: recall_at_10 |
|
value: 19.969 |
|
- type: recall_at_100 |
|
value: 40.113 |
|
- type: recall_at_1000 |
|
value: 61.72299999999999 |
|
- type: recall_at_3 |
|
value: 12.950999999999999 |
|
- type: recall_at_5 |
|
value: 15.690999999999999 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/emotion |
|
name: MTEB EmotionClassification |
|
config: default |
|
split: test |
|
revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
|
metrics: |
|
- type: accuracy |
|
value: 44.675000000000004 |
|
- type: f1 |
|
value: 40.779372586075105 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fever |
|
name: MTEB FEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 57.406 |
|
- type: map_at_10 |
|
value: 67.69500000000001 |
|
- type: map_at_100 |
|
value: 68.08 |
|
- type: map_at_1000 |
|
value: 68.095 |
|
- type: map_at_3 |
|
value: 65.688 |
|
- type: map_at_5 |
|
value: 66.93 |
|
- type: mrr_at_1 |
|
value: 61.941 |
|
- type: mrr_at_10 |
|
value: 72.513 |
|
- type: mrr_at_100 |
|
value: 72.83699999999999 |
|
- type: mrr_at_1000 |
|
value: 72.844 |
|
- type: mrr_at_3 |
|
value: 70.60499999999999 |
|
- type: mrr_at_5 |
|
value: 71.807 |
|
- type: ndcg_at_1 |
|
value: 61.941 |
|
- type: ndcg_at_10 |
|
value: 73.29 |
|
- type: ndcg_at_100 |
|
value: 74.96300000000001 |
|
- type: ndcg_at_1000 |
|
value: 75.28200000000001 |
|
- type: ndcg_at_3 |
|
value: 69.491 |
|
- type: ndcg_at_5 |
|
value: 71.573 |
|
- type: precision_at_1 |
|
value: 61.941 |
|
- type: precision_at_10 |
|
value: 9.388 |
|
- type: precision_at_100 |
|
value: 1.0290000000000001 |
|
- type: precision_at_1000 |
|
value: 0.107 |
|
- type: precision_at_3 |
|
value: 27.423 |
|
- type: precision_at_5 |
|
value: 17.627000000000002 |
|
- type: recall_at_1 |
|
value: 57.406 |
|
- type: recall_at_10 |
|
value: 85.975 |
|
- type: recall_at_100 |
|
value: 93.29899999999999 |
|
- type: recall_at_1000 |
|
value: 95.531 |
|
- type: recall_at_3 |
|
value: 75.624 |
|
- type: recall_at_5 |
|
value: 80.78999999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fiqa |
|
name: MTEB FiQA2018 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 16.314999999999998 |
|
- type: map_at_10 |
|
value: 26.678 |
|
- type: map_at_100 |
|
value: 28.322000000000003 |
|
- type: map_at_1000 |
|
value: 28.519 |
|
- type: map_at_3 |
|
value: 23.105 |
|
- type: map_at_5 |
|
value: 24.808 |
|
- type: mrr_at_1 |
|
value: 33.333 |
|
- type: mrr_at_10 |
|
value: 41.453 |
|
- type: mrr_at_100 |
|
value: 42.339 |
|
- type: mrr_at_1000 |
|
value: 42.39 |
|
- type: mrr_at_3 |
|
value: 38.863 |
|
- type: mrr_at_5 |
|
value: 40.159 |
|
- type: ndcg_at_1 |
|
value: 33.333 |
|
- type: ndcg_at_10 |
|
value: 34.062 |
|
- type: ndcg_at_100 |
|
value: 40.595 |
|
- type: ndcg_at_1000 |
|
value: 44.124 |
|
- type: ndcg_at_3 |
|
value: 30.689 |
|
- type: ndcg_at_5 |
|
value: 31.255 |
|
- type: precision_at_1 |
|
value: 33.333 |
|
- type: precision_at_10 |
|
value: 9.722 |
|
- type: precision_at_100 |
|
value: 1.6480000000000001 |
|
- type: precision_at_1000 |
|
value: 0.22699999999999998 |
|
- type: precision_at_3 |
|
value: 20.936 |
|
- type: precision_at_5 |
|
value: 15.154 |
|
- type: recall_at_1 |
|
value: 16.314999999999998 |
|
- type: recall_at_10 |
|
value: 41.221000000000004 |
|
- type: recall_at_100 |
|
value: 65.857 |
|
- type: recall_at_1000 |
|
value: 87.327 |
|
- type: recall_at_3 |
|
value: 27.435 |
|
- type: recall_at_5 |
|
value: 32.242 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: hotpotqa |
|
name: MTEB HotpotQA |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 31.978 |
|
- type: map_at_10 |
|
value: 43.784 |
|
- type: map_at_100 |
|
value: 44.547 |
|
- type: map_at_1000 |
|
value: 44.614 |
|
- type: map_at_3 |
|
value: 41.317 |
|
- type: map_at_5 |
|
value: 42.812 |
|
- type: mrr_at_1 |
|
value: 63.956999999999994 |
|
- type: mrr_at_10 |
|
value: 70.502 |
|
- type: mrr_at_100 |
|
value: 70.845 |
|
- type: mrr_at_1000 |
|
value: 70.865 |
|
- type: mrr_at_3 |
|
value: 69.192 |
|
- type: mrr_at_5 |
|
value: 69.994 |
|
- type: ndcg_at_1 |
|
value: 63.956999999999994 |
|
- type: ndcg_at_10 |
|
value: 52.782 |
|
- type: ndcg_at_100 |
|
value: 55.78999999999999 |
|
- type: ndcg_at_1000 |
|
value: 57.289 |
|
- type: ndcg_at_3 |
|
value: 48.864000000000004 |
|
- type: ndcg_at_5 |
|
value: 50.964 |
|
- type: precision_at_1 |
|
value: 63.956999999999994 |
|
- type: precision_at_10 |
|
value: 10.809000000000001 |
|
- type: precision_at_100 |
|
value: 1.319 |
|
- type: precision_at_1000 |
|
value: 0.152 |
|
- type: precision_at_3 |
|
value: 30.2 |
|
- type: precision_at_5 |
|
value: 19.787 |
|
- type: recall_at_1 |
|
value: 31.978 |
|
- type: recall_at_10 |
|
value: 54.045 |
|
- type: recall_at_100 |
|
value: 65.928 |
|
- type: recall_at_1000 |
|
value: 75.976 |
|
- type: recall_at_3 |
|
value: 45.300000000000004 |
|
- type: recall_at_5 |
|
value: 49.467 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/imdb |
|
name: MTEB ImdbClassification |
|
config: default |
|
split: test |
|
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
|
metrics: |
|
- type: accuracy |
|
value: 63.8708 |
|
- type: ap |
|
value: 59.02002684158838 |
|
- type: f1 |
|
value: 63.650055896985315 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: msmarco |
|
name: MTEB MSMARCO |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 19.834 |
|
- type: map_at_10 |
|
value: 31.317 |
|
- type: map_at_100 |
|
value: 32.576 |
|
- type: map_at_1000 |
|
value: 32.631 |
|
- type: map_at_3 |
|
value: 27.728 |
|
- type: map_at_5 |
|
value: 29.720000000000002 |
|
- type: mrr_at_1 |
|
value: 20.43 |
|
- type: mrr_at_10 |
|
value: 31.868999999999996 |
|
- type: mrr_at_100 |
|
value: 33.074999999999996 |
|
- type: mrr_at_1000 |
|
value: 33.123999999999995 |
|
- type: mrr_at_3 |
|
value: 28.333000000000002 |
|
- type: mrr_at_5 |
|
value: 30.305 |
|
- type: ndcg_at_1 |
|
value: 20.43 |
|
- type: ndcg_at_10 |
|
value: 37.769000000000005 |
|
- type: ndcg_at_100 |
|
value: 43.924 |
|
- type: ndcg_at_1000 |
|
value: 45.323 |
|
- type: ndcg_at_3 |
|
value: 30.422 |
|
- type: ndcg_at_5 |
|
value: 33.98 |
|
- type: precision_at_1 |
|
value: 20.43 |
|
- type: precision_at_10 |
|
value: 6.027 |
|
- type: precision_at_100 |
|
value: 0.9119999999999999 |
|
- type: precision_at_1000 |
|
value: 0.10300000000000001 |
|
- type: precision_at_3 |
|
value: 12.985 |
|
- type: precision_at_5 |
|
value: 9.593 |
|
- type: recall_at_1 |
|
value: 19.834 |
|
- type: recall_at_10 |
|
value: 57.647000000000006 |
|
- type: recall_at_100 |
|
value: 86.276 |
|
- type: recall_at_1000 |
|
value: 97.065 |
|
- type: recall_at_3 |
|
value: 37.616 |
|
- type: recall_at_5 |
|
value: 46.171 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (en) |
|
config: en |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 91.52530779753762 |
|
- type: f1 |
|
value: 91.4004687820246 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 72.82717738258093 |
|
- type: f1 |
|
value: 56.791387113030346 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 71.09280430396772 |
|
- type: f1 |
|
value: 68.92843467363518 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (en) |
|
config: en |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 76.2542030934768 |
|
- type: f1 |
|
value: 76.22211319699834 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-p2p |
|
name: MTEB MedrxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
|
metrics: |
|
- type: v_measure |
|
value: 29.604407852989457 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-s2s |
|
name: MTEB MedrxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
|
metrics: |
|
- type: v_measure |
|
value: 25.011863718751183 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/mind_small |
|
name: MTEB MindSmallReranking |
|
config: default |
|
split: test |
|
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
|
metrics: |
|
- type: map |
|
value: 31.55552172383111 |
|
- type: mrr |
|
value: 32.65475731770242 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nfcorpus |
|
name: MTEB NFCorpus |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 4.968 |
|
- type: map_at_10 |
|
value: 10.703999999999999 |
|
- type: map_at_100 |
|
value: 13.316 |
|
- type: map_at_1000 |
|
value: 14.674000000000001 |
|
- type: map_at_3 |
|
value: 7.809000000000001 |
|
- type: map_at_5 |
|
value: 9.268 |
|
- type: mrr_at_1 |
|
value: 41.796 |
|
- type: mrr_at_10 |
|
value: 50.558 |
|
- type: mrr_at_100 |
|
value: 51.125 |
|
- type: mrr_at_1000 |
|
value: 51.184 |
|
- type: mrr_at_3 |
|
value: 48.349 |
|
- type: mrr_at_5 |
|
value: 49.572 |
|
- type: ndcg_at_1 |
|
value: 39.783 |
|
- type: ndcg_at_10 |
|
value: 30.375999999999998 |
|
- type: ndcg_at_100 |
|
value: 27.648 |
|
- type: ndcg_at_1000 |
|
value: 36.711 |
|
- type: ndcg_at_3 |
|
value: 35.053 |
|
- type: ndcg_at_5 |
|
value: 33.278999999999996 |
|
- type: precision_at_1 |
|
value: 41.796 |
|
- type: precision_at_10 |
|
value: 22.663 |
|
- type: precision_at_100 |
|
value: 7.210999999999999 |
|
- type: precision_at_1000 |
|
value: 1.984 |
|
- type: precision_at_3 |
|
value: 33.127 |
|
- type: precision_at_5 |
|
value: 29.102 |
|
- type: recall_at_1 |
|
value: 4.968 |
|
- type: recall_at_10 |
|
value: 14.469999999999999 |
|
- type: recall_at_100 |
|
value: 28.188000000000002 |
|
- type: recall_at_1000 |
|
value: 60.769 |
|
- type: recall_at_3 |
|
value: 8.737 |
|
- type: recall_at_5 |
|
value: 11.539000000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nq |
|
name: MTEB NQ |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 26.958 |
|
- type: map_at_10 |
|
value: 40.6 |
|
- type: map_at_100 |
|
value: 41.754000000000005 |
|
- type: map_at_1000 |
|
value: 41.792 |
|
- type: map_at_3 |
|
value: 36.521 |
|
- type: map_at_5 |
|
value: 38.866 |
|
- type: mrr_at_1 |
|
value: 30.330000000000002 |
|
- type: mrr_at_10 |
|
value: 43.013 |
|
- type: mrr_at_100 |
|
value: 43.89 |
|
- type: mrr_at_1000 |
|
value: 43.917 |
|
- type: mrr_at_3 |
|
value: 39.489000000000004 |
|
- type: mrr_at_5 |
|
value: 41.504999999999995 |
|
- type: ndcg_at_1 |
|
value: 30.330000000000002 |
|
- type: ndcg_at_10 |
|
value: 47.878 |
|
- type: ndcg_at_100 |
|
value: 52.761 |
|
- type: ndcg_at_1000 |
|
value: 53.69500000000001 |
|
- type: ndcg_at_3 |
|
value: 40.061 |
|
- type: ndcg_at_5 |
|
value: 43.980000000000004 |
|
- type: precision_at_1 |
|
value: 30.330000000000002 |
|
- type: precision_at_10 |
|
value: 8.048 |
|
- type: precision_at_100 |
|
value: 1.076 |
|
- type: precision_at_1000 |
|
value: 0.117 |
|
- type: precision_at_3 |
|
value: 18.299000000000003 |
|
- type: precision_at_5 |
|
value: 13.25 |
|
- type: recall_at_1 |
|
value: 26.958 |
|
- type: recall_at_10 |
|
value: 67.72399999999999 |
|
- type: recall_at_100 |
|
value: 89.02600000000001 |
|
- type: recall_at_1000 |
|
value: 96.029 |
|
- type: recall_at_3 |
|
value: 47.332 |
|
- type: recall_at_5 |
|
value: 56.36600000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: quora |
|
name: MTEB QuoraRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 69.926 |
|
- type: map_at_10 |
|
value: 83.797 |
|
- type: map_at_100 |
|
value: 84.42699999999999 |
|
- type: map_at_1000 |
|
value: 84.446 |
|
- type: map_at_3 |
|
value: 80.78 |
|
- type: map_at_5 |
|
value: 82.669 |
|
- type: mrr_at_1 |
|
value: 80.44 |
|
- type: mrr_at_10 |
|
value: 86.79 |
|
- type: mrr_at_100 |
|
value: 86.90299999999999 |
|
- type: mrr_at_1000 |
|
value: 86.904 |
|
- type: mrr_at_3 |
|
value: 85.753 |
|
- type: mrr_at_5 |
|
value: 86.478 |
|
- type: ndcg_at_1 |
|
value: 80.44 |
|
- type: ndcg_at_10 |
|
value: 87.634 |
|
- type: ndcg_at_100 |
|
value: 88.9 |
|
- type: ndcg_at_1000 |
|
value: 89.03 |
|
- type: ndcg_at_3 |
|
value: 84.622 |
|
- type: ndcg_at_5 |
|
value: 86.29 |
|
- type: precision_at_1 |
|
value: 80.44 |
|
- type: precision_at_10 |
|
value: 13.305 |
|
- type: precision_at_100 |
|
value: 1.524 |
|
- type: precision_at_1000 |
|
value: 0.157 |
|
- type: precision_at_3 |
|
value: 36.957 |
|
- type: precision_at_5 |
|
value: 24.328 |
|
- type: recall_at_1 |
|
value: 69.926 |
|
- type: recall_at_10 |
|
value: 94.99300000000001 |
|
- type: recall_at_100 |
|
value: 99.345 |
|
- type: recall_at_1000 |
|
value: 99.97 |
|
- type: recall_at_3 |
|
value: 86.465 |
|
- type: recall_at_5 |
|
value: 91.121 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering |
|
name: MTEB RedditClustering |
|
config: default |
|
split: test |
|
revision: 24640382cdbf8abc73003fb0fa6d111a705499eb |
|
metrics: |
|
- type: v_measure |
|
value: 42.850644235471144 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering-p2p |
|
name: MTEB RedditClusteringP2P |
|
config: default |
|
split: test |
|
revision: 282350215ef01743dc01b456c7f5241fa8937f16 |
|
metrics: |
|
- type: v_measure |
|
value: 52.547875398320734 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scidocs |
|
name: MTEB SCIDOCS |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 4.328 |
|
- type: map_at_10 |
|
value: 10.479 |
|
- type: map_at_100 |
|
value: 12.25 |
|
- type: map_at_1000 |
|
value: 12.522 |
|
- type: map_at_3 |
|
value: 7.548000000000001 |
|
- type: map_at_5 |
|
value: 9.039 |
|
- type: mrr_at_1 |
|
value: 21.3 |
|
- type: mrr_at_10 |
|
value: 30.678 |
|
- type: mrr_at_100 |
|
value: 31.77 |
|
- type: mrr_at_1000 |
|
value: 31.831 |
|
- type: mrr_at_3 |
|
value: 27.500000000000004 |
|
- type: mrr_at_5 |
|
value: 29.375 |
|
- type: ndcg_at_1 |
|
value: 21.3 |
|
- type: ndcg_at_10 |
|
value: 17.626 |
|
- type: ndcg_at_100 |
|
value: 25.03 |
|
- type: ndcg_at_1000 |
|
value: 30.055 |
|
- type: ndcg_at_3 |
|
value: 16.744999999999997 |
|
- type: ndcg_at_5 |
|
value: 14.729999999999999 |
|
- type: precision_at_1 |
|
value: 21.3 |
|
- type: precision_at_10 |
|
value: 9.09 |
|
- type: precision_at_100 |
|
value: 1.989 |
|
- type: precision_at_1000 |
|
value: 0.32 |
|
- type: precision_at_3 |
|
value: 15.467 |
|
- type: precision_at_5 |
|
value: 12.879999999999999 |
|
- type: recall_at_1 |
|
value: 4.328 |
|
- type: recall_at_10 |
|
value: 18.412 |
|
- type: recall_at_100 |
|
value: 40.363 |
|
- type: recall_at_1000 |
|
value: 64.997 |
|
- type: recall_at_3 |
|
value: 9.408 |
|
- type: recall_at_5 |
|
value: 13.048000000000002 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sickr-sts |
|
name: MTEB SICK-R |
|
config: default |
|
split: test |
|
revision: a6ea5a8cab320b040a23452cc28066d9beae2cee |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 84.1338589503896 |
|
- type: cos_sim_spearman |
|
value: 79.1378154534123 |
|
- type: euclidean_pearson |
|
value: 73.17857462509251 |
|
- type: euclidean_spearman |
|
value: 70.79268955610539 |
|
- type: manhattan_pearson |
|
value: 72.8280251705823 |
|
- type: manhattan_spearman |
|
value: 70.60323787229834 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts12-sts |
|
name: MTEB STS12 |
|
config: default |
|
split: test |
|
revision: a0d554a64d88156834ff5ae9920b964011b16384 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 84.21604641858598 |
|
- type: cos_sim_spearman |
|
value: 75.06080146054282 |
|
- type: euclidean_pearson |
|
value: 69.44429285856924 |
|
- type: euclidean_spearman |
|
value: 58.240130690046456 |
|
- type: manhattan_pearson |
|
value: 69.07597314234852 |
|
- type: manhattan_spearman |
|
value: 58.08224335836159 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts13-sts |
|
name: MTEB STS13 |
|
config: default |
|
split: test |
|
revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 80.2252849321165 |
|
- type: cos_sim_spearman |
|
value: 80.85907200101076 |
|
- type: euclidean_pearson |
|
value: 70.85619832878055 |
|
- type: euclidean_spearman |
|
value: 71.59417341887324 |
|
- type: manhattan_pearson |
|
value: 70.55842192345895 |
|
- type: manhattan_spearman |
|
value: 71.30332994715893 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts14-sts |
|
name: MTEB STS14 |
|
config: default |
|
split: test |
|
revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 80.50469360654135 |
|
- type: cos_sim_spearman |
|
value: 76.12917164308409 |
|
- type: euclidean_pearson |
|
value: 70.4070213910491 |
|
- type: euclidean_spearman |
|
value: 66.97320451942113 |
|
- type: manhattan_pearson |
|
value: 70.24834290119863 |
|
- type: manhattan_spearman |
|
value: 66.9047074173091 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts15-sts |
|
name: MTEB STS15 |
|
config: default |
|
split: test |
|
revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 84.70140350059746 |
|
- type: cos_sim_spearman |
|
value: 85.55427877110485 |
|
- type: euclidean_pearson |
|
value: 63.4780453371435 |
|
- type: euclidean_spearman |
|
value: 64.65485395077273 |
|
- type: manhattan_pearson |
|
value: 63.64869846572011 |
|
- type: manhattan_spearman |
|
value: 64.87219311596813 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts16-sts |
|
name: MTEB STS16 |
|
config: default |
|
split: test |
|
revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 79.4416477676503 |
|
- type: cos_sim_spearman |
|
value: 81.2094925260351 |
|
- type: euclidean_pearson |
|
value: 68.372257553367 |
|
- type: euclidean_spearman |
|
value: 69.47792807911692 |
|
- type: manhattan_pearson |
|
value: 68.17773583183664 |
|
- type: manhattan_spearman |
|
value: 69.31505452732998 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-en) |
|
config: en-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 88.94688403351994 |
|
- type: cos_sim_spearman |
|
value: 88.97626967707933 |
|
- type: euclidean_pearson |
|
value: 74.09942728422159 |
|
- type: euclidean_spearman |
|
value: 72.91022362666948 |
|
- type: manhattan_pearson |
|
value: 74.11262432880199 |
|
- type: manhattan_spearman |
|
value: 72.82115894578564 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (en) |
|
config: en |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 67.42605802805606 |
|
- type: cos_sim_spearman |
|
value: 66.22330559222408 |
|
- type: euclidean_pearson |
|
value: 50.15272876367891 |
|
- type: euclidean_spearman |
|
value: 60.695400782452715 |
|
- type: manhattan_pearson |
|
value: 50.17076569264417 |
|
- type: manhattan_spearman |
|
value: 60.3761281869747 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/stsbenchmark-sts |
|
name: MTEB STSBenchmark |
|
config: default |
|
split: test |
|
revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 82.85939227596093 |
|
- type: cos_sim_spearman |
|
value: 82.57071649593358 |
|
- type: euclidean_pearson |
|
value: 72.18291316100125 |
|
- type: euclidean_spearman |
|
value: 70.70702024402348 |
|
- type: manhattan_pearson |
|
value: 72.36789718833687 |
|
- type: manhattan_spearman |
|
value: 70.92789721402387 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/scidocs-reranking |
|
name: MTEB SciDocsRR |
|
config: default |
|
split: test |
|
revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
|
metrics: |
|
- type: map |
|
value: 79.31107201598611 |
|
- type: mrr |
|
value: 93.66321314850727 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scifact |
|
name: MTEB SciFact |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 45.428000000000004 |
|
- type: map_at_10 |
|
value: 54.730000000000004 |
|
- type: map_at_100 |
|
value: 55.421 |
|
- type: map_at_1000 |
|
value: 55.47299999999999 |
|
- type: map_at_3 |
|
value: 52.333 |
|
- type: map_at_5 |
|
value: 53.72 |
|
- type: mrr_at_1 |
|
value: 48.333 |
|
- type: mrr_at_10 |
|
value: 56.601 |
|
- type: mrr_at_100 |
|
value: 57.106 |
|
- type: mrr_at_1000 |
|
value: 57.154 |
|
- type: mrr_at_3 |
|
value: 54.611 |
|
- type: mrr_at_5 |
|
value: 55.87800000000001 |
|
- type: ndcg_at_1 |
|
value: 48.333 |
|
- type: ndcg_at_10 |
|
value: 59.394999999999996 |
|
- type: ndcg_at_100 |
|
value: 62.549 |
|
- type: ndcg_at_1000 |
|
value: 63.941 |
|
- type: ndcg_at_3 |
|
value: 55.096000000000004 |
|
- type: ndcg_at_5 |
|
value: 57.325 |
|
- type: precision_at_1 |
|
value: 48.333 |
|
- type: precision_at_10 |
|
value: 8.1 |
|
- type: precision_at_100 |
|
value: 0.983 |
|
- type: precision_at_1000 |
|
value: 0.11 |
|
- type: precision_at_3 |
|
value: 21.889 |
|
- type: precision_at_5 |
|
value: 14.533 |
|
- type: recall_at_1 |
|
value: 45.428000000000004 |
|
- type: recall_at_10 |
|
value: 71.806 |
|
- type: recall_at_100 |
|
value: 86.533 |
|
- type: recall_at_1000 |
|
value: 97.5 |
|
- type: recall_at_3 |
|
value: 60.228 |
|
- type: recall_at_5 |
|
value: 65.90599999999999 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/sprintduplicatequestions-pairclassification |
|
name: MTEB SprintDuplicateQuestions |
|
config: default |
|
split: test |
|
revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 99.8029702970297 |
|
- type: cos_sim_ap |
|
value: 95.48085242816634 |
|
- type: cos_sim_f1 |
|
value: 89.86653484923382 |
|
- type: cos_sim_precision |
|
value: 88.85630498533725 |
|
- type: cos_sim_recall |
|
value: 90.9 |
|
- type: dot_accuracy |
|
value: 99.21881188118812 |
|
- type: dot_ap |
|
value: 55.14126603018576 |
|
- type: dot_f1 |
|
value: 55.22458628841608 |
|
- type: dot_precision |
|
value: 52.37668161434977 |
|
- type: dot_recall |
|
value: 58.4 |
|
- type: euclidean_accuracy |
|
value: 99.64356435643565 |
|
- type: euclidean_ap |
|
value: 84.52487064474103 |
|
- type: euclidean_f1 |
|
value: 80.53908355795149 |
|
- type: euclidean_precision |
|
value: 87.36842105263159 |
|
- type: euclidean_recall |
|
value: 74.7 |
|
- type: manhattan_accuracy |
|
value: 99.63861386138613 |
|
- type: manhattan_ap |
|
value: 84.1994288662172 |
|
- type: manhattan_f1 |
|
value: 80.38482095136291 |
|
- type: manhattan_precision |
|
value: 86.33754305396096 |
|
- type: manhattan_recall |
|
value: 75.2 |
|
- type: max_accuracy |
|
value: 99.8029702970297 |
|
- type: max_ap |
|
value: 95.48085242816634 |
|
- type: max_f1 |
|
value: 89.86653484923382 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering |
|
name: MTEB StackExchangeClustering |
|
config: default |
|
split: test |
|
revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
|
metrics: |
|
- type: v_measure |
|
value: 48.06508273111389 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering-p2p |
|
name: MTEB StackExchangeClusteringP2P |
|
config: default |
|
split: test |
|
revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
|
metrics: |
|
- type: v_measure |
|
value: 31.36169910951664 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/stackoverflowdupquestions-reranking |
|
name: MTEB StackOverflowDupQuestions |
|
config: default |
|
split: test |
|
revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
|
metrics: |
|
- type: map |
|
value: 50.110601218420356 |
|
- type: mrr |
|
value: 50.90277777777777 |
|
- task: |
|
type: Summarization |
|
dataset: |
|
type: mteb/summeval |
|
name: MTEB SummEval |
|
config: default |
|
split: test |
|
revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 29.63669555287747 |
|
- type: cos_sim_spearman |
|
value: 30.708042454053853 |
|
- type: dot_pearson |
|
value: 20.309025749838924 |
|
- type: dot_spearman |
|
value: 21.511758746817165 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: trec-covid |
|
name: MTEB TRECCOVID |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.201 |
|
- type: map_at_10 |
|
value: 1.405 |
|
- type: map_at_100 |
|
value: 7.359999999999999 |
|
- type: map_at_1000 |
|
value: 17.858 |
|
- type: map_at_3 |
|
value: 0.494 |
|
- type: map_at_5 |
|
value: 0.757 |
|
- type: mrr_at_1 |
|
value: 74.0 |
|
- type: mrr_at_10 |
|
value: 84.89999999999999 |
|
- type: mrr_at_100 |
|
value: 84.89999999999999 |
|
- type: mrr_at_1000 |
|
value: 84.89999999999999 |
|
- type: mrr_at_3 |
|
value: 84.0 |
|
- type: mrr_at_5 |
|
value: 84.89999999999999 |
|
- type: ndcg_at_1 |
|
value: 68.0 |
|
- type: ndcg_at_10 |
|
value: 60.571 |
|
- type: ndcg_at_100 |
|
value: 46.016 |
|
- type: ndcg_at_1000 |
|
value: 41.277 |
|
- type: ndcg_at_3 |
|
value: 63.989 |
|
- type: ndcg_at_5 |
|
value: 61.41 |
|
- type: precision_at_1 |
|
value: 74.0 |
|
- type: precision_at_10 |
|
value: 65.2 |
|
- type: precision_at_100 |
|
value: 47.04 |
|
- type: precision_at_1000 |
|
value: 18.416 |
|
- type: precision_at_3 |
|
value: 68.0 |
|
- type: precision_at_5 |
|
value: 66.4 |
|
- type: recall_at_1 |
|
value: 0.201 |
|
- type: recall_at_10 |
|
value: 1.763 |
|
- type: recall_at_100 |
|
value: 11.008999999999999 |
|
- type: recall_at_1000 |
|
value: 38.509 |
|
- type: recall_at_3 |
|
value: 0.551 |
|
- type: recall_at_5 |
|
value: 0.881 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: webis-touche2020 |
|
name: MTEB Touche2020 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 1.4040000000000001 |
|
- type: map_at_10 |
|
value: 7.847999999999999 |
|
- type: map_at_100 |
|
value: 12.908 |
|
- type: map_at_1000 |
|
value: 14.37 |
|
- type: map_at_3 |
|
value: 3.6450000000000005 |
|
- type: map_at_5 |
|
value: 4.93 |
|
- type: mrr_at_1 |
|
value: 18.367 |
|
- type: mrr_at_10 |
|
value: 32.576 |
|
- type: mrr_at_100 |
|
value: 34.163 |
|
- type: mrr_at_1000 |
|
value: 34.18 |
|
- type: mrr_at_3 |
|
value: 28.571 |
|
- type: mrr_at_5 |
|
value: 30.918 |
|
- type: ndcg_at_1 |
|
value: 15.306000000000001 |
|
- type: ndcg_at_10 |
|
value: 18.59 |
|
- type: ndcg_at_100 |
|
value: 30.394 |
|
- type: ndcg_at_1000 |
|
value: 42.198 |
|
- type: ndcg_at_3 |
|
value: 18.099 |
|
- type: ndcg_at_5 |
|
value: 16.955000000000002 |
|
- type: precision_at_1 |
|
value: 16.326999999999998 |
|
- type: precision_at_10 |
|
value: 17.959 |
|
- type: precision_at_100 |
|
value: 6.755 |
|
- type: precision_at_1000 |
|
value: 1.4529999999999998 |
|
- type: precision_at_3 |
|
value: 20.408 |
|
- type: precision_at_5 |
|
value: 18.367 |
|
- type: recall_at_1 |
|
value: 1.4040000000000001 |
|
- type: recall_at_10 |
|
value: 14.048 |
|
- type: recall_at_100 |
|
value: 42.150999999999996 |
|
- type: recall_at_1000 |
|
value: 77.85600000000001 |
|
- type: recall_at_3 |
|
value: 4.819 |
|
- type: recall_at_5 |
|
value: 7.13 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/toxic_conversations_50k |
|
name: MTEB ToxicConversationsClassification |
|
config: default |
|
split: test |
|
revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
|
metrics: |
|
- type: accuracy |
|
value: 66.1456 |
|
- type: ap |
|
value: 11.631023858569064 |
|
- type: f1 |
|
value: 50.128196455722254 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/tweet_sentiment_extraction |
|
name: MTEB TweetSentimentExtractionClassification |
|
config: default |
|
split: test |
|
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
|
metrics: |
|
- type: accuracy |
|
value: 56.850594227504246 |
|
- type: f1 |
|
value: 56.82313689360827 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/twentynewsgroups-clustering |
|
name: MTEB TwentyNewsgroupsClustering |
|
config: default |
|
split: test |
|
revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
|
metrics: |
|
- type: v_measure |
|
value: 38.060423744064764 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twittersemeval2015-pairclassification |
|
name: MTEB TwitterSemEval2015 |
|
config: default |
|
split: test |
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 84.43702688204088 |
|
- type: cos_sim_ap |
|
value: 68.30176948820142 |
|
- type: cos_sim_f1 |
|
value: 64.25430330443524 |
|
- type: cos_sim_precision |
|
value: 61.33365315423362 |
|
- type: cos_sim_recall |
|
value: 67.46701846965699 |
|
- type: dot_accuracy |
|
value: 77.76718126005842 |
|
- type: dot_ap |
|
value: 37.510516716176305 |
|
- type: dot_f1 |
|
value: 43.53859496964441 |
|
- type: dot_precision |
|
value: 32.428940568475454 |
|
- type: dot_recall |
|
value: 66.2269129287599 |
|
- type: euclidean_accuracy |
|
value: 82.10049472492102 |
|
- type: euclidean_ap |
|
value: 61.64354520687271 |
|
- type: euclidean_f1 |
|
value: 59.804144841721694 |
|
- type: euclidean_precision |
|
value: 52.604166666666664 |
|
- type: euclidean_recall |
|
value: 69.28759894459104 |
|
- type: manhattan_accuracy |
|
value: 82.22566609048101 |
|
- type: manhattan_ap |
|
value: 61.753431124879974 |
|
- type: manhattan_f1 |
|
value: 59.77735297424941 |
|
- type: manhattan_precision |
|
value: 52.0870076425632 |
|
- type: manhattan_recall |
|
value: 70.13192612137203 |
|
- type: max_accuracy |
|
value: 84.43702688204088 |
|
- type: max_ap |
|
value: 68.30176948820142 |
|
- type: max_f1 |
|
value: 64.25430330443524 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twitterurlcorpus-pairclassification |
|
name: MTEB TwitterURLCorpus |
|
config: default |
|
split: test |
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 88.81515116233942 |
|
- type: cos_sim_ap |
|
value: 85.33305785100573 |
|
- type: cos_sim_f1 |
|
value: 78.11202938475667 |
|
- type: cos_sim_precision |
|
value: 74.68567816253424 |
|
- type: cos_sim_recall |
|
value: 81.86787804126887 |
|
- type: dot_accuracy |
|
value: 82.50475414289595 |
|
- type: dot_ap |
|
value: 69.87015340174045 |
|
- type: dot_f1 |
|
value: 65.94174480373633 |
|
- type: dot_precision |
|
value: 61.40362525728703 |
|
- type: dot_recall |
|
value: 71.20418848167539 |
|
- type: euclidean_accuracy |
|
value: 83.05778709201692 |
|
- type: euclidean_ap |
|
value: 70.54206653977498 |
|
- type: euclidean_f1 |
|
value: 62.98969847356943 |
|
- type: euclidean_precision |
|
value: 61.55033063923585 |
|
- type: euclidean_recall |
|
value: 64.49799815214044 |
|
- type: manhattan_accuracy |
|
value: 83.0034540303489 |
|
- type: manhattan_ap |
|
value: 70.53997987198404 |
|
- type: manhattan_f1 |
|
value: 62.95875898600075 |
|
- type: manhattan_precision |
|
value: 61.89555125725339 |
|
- type: manhattan_recall |
|
value: 64.05913150600554 |
|
- type: max_accuracy |
|
value: 88.81515116233942 |
|
- type: max_ap |
|
value: 85.33305785100573 |
|
- type: max_f1 |
|
value: 78.11202938475667 |
|
--- |
|
--- |
|
|
|
<br><br> |
|
|
|
<p align="center"> |
|
<img src="https://github.com/jina-ai/finetuner/blob/main/docs/_static/finetuner-logo-ani.svg?raw=true" alt="Finetuner logo: Finetuner helps you to create experiments in order to improve embeddings on search tasks. It accompanies you to deliver the last mile of performance-tuning for neural search applications." width="150px"> |
|
</p> |
|
|
|
|
|
<p align="center"> |
|
<b>The text embedding set trained by <a href="https://jina.ai/"><b>Jina AI</b></a>, <a href="https://github.com/jina-ai/finetuner"><b>Finetuner</b></a> team.</b> |
|
</p> |
|
|
|
|
|
## Intented Usage & Model Info |
|
|
|
`jina-embedding-b-en-v1` is a language model that has been trained using Jina AI's Linnaeus-Clean dataset. |
|
This dataset consists of 380 million pairs of sentences, which include both query-document pairs. |
|
These pairs were obtained from various domains and were carefully selected through a thorough cleaning process. |
|
The Linnaeus-Full dataset, from which the Linnaeus-Clean dataset is derived, originally contained 1.6 billion sentence pairs. |
|
|
|
The model has a range of use cases, including information retrieval, semantic textual similarity, text reranking, and more. |
|
|
|
With a standard size of 110 million parameters, |
|
the model enables fast inference while delivering better performance than our small model. |
|
It is recommended to use a single GPU for inference. |
|
Additionally, we provide the following options: |
|
|
|
- [`jina-embedding-t-en-v1`](https://huggingface.co/jinaai/jina-embedding-t-en-v1): 14 million parameters. |
|
- [`jina-embedding-s-en-v1`](https://huggingface.co/jinaai/jina-embedding-s-en-v1): 35 million parameters |
|
- [`jina-embedding-b-en-v1`](https://huggingface.co/jinaai/jina-embedding-b-en-v1): 110 million parameters **(you are here)**. |
|
- [`jina-embedding-l-en-v1`](https://huggingface.co/jinaai/jina-embedding-l-en-v1): 330 million parameters. |
|
- `jina-embedding-1b-en-v1`: 1.2 billion parameters, 10 times bert-base (soon). |
|
- `jina-embedding-6b-en-v1`: 6 billion parameters, 30 times bert-base (soon). |
|
|
|
## Data & Parameters |
|
|
|
Please checkout our [technical blog](https://arxiv.org/abs/2307.11224). |
|
|
|
## Metrics |
|
|
|
We compared the model against `all-minilm-l6-v2`/`all-mpnet-base-v2` from sbert and `text-embeddings-ada-002` from OpenAI: |
|
|
|
|Name|param |dimension| |
|
|------------------------------|-----|------| |
|
|all-minilm-l6-v2|23m |384| |
|
|all-mpnet-base-v2 |110m |768| |
|
|ada-embedding-002|Unknown/OpenAI API |1536| |
|
|jina-embedding-t-en-v1|14m |312| |
|
|jina-embedding-s-en-v1|35m |512| |
|
|jina-embedding-b-en-v1|110m |768| |
|
|jina-embedding-l-en-v1|330m |1024| |
|
|
|
|
|
|Name|STS12|STS13|STS14|STS15|STS16|STS17|TRECOVID|Quora|SciFact| |
|
|------------------------------|-----|-----|-----|-----|-----|-----|--------|-----|-----| |
|
|all-minilm-l6-v2|0.724|0.806|0.756|0.854|0.79 |0.876|0.473 |0.876|0.645 | |
|
|all-mpnet-base-v2|0.726|**0.835**|0.78 |0.857|0.8 |**0.906**|0.513 |0.875|0.656 | |
|
|ada-embedding-002|0.698|0.833|0.761|0.861|**0.86** |0.903|**0.685** |0.876|**0.726** | |
|
|jina-embedding-t-en-v1|0.717|0.773|0.731|0.829|0.777|0.860|0.482 |0.840|0.522 | |
|
|jina-embedding-s-en-v1|0.743|0.786|0.738|0.837|0.80|0.875|0.523 |0.857|0.524 | |
|
|jina-embedding-b-en-v1|**0.751**|0.809|0.761|0.856|0.812|0.890|0.606 |0.876|0.594 | |
|
|jina-embedding-l-en-v1|0.745|0.832|**0.781**|**0.869**|0.837|0.902|0.573 |**0.881**|0.598 | |
|
|
|
## Usage |
|
|
|
Usage with Jina AI Finetuner: |
|
|
|
```python |
|
!pip install finetuner |
|
import finetuner |
|
|
|
model = finetuner.build_model('jinaai/jina-embedding-b-en-v1') |
|
embeddings = finetuner.encode( |
|
model=model, |
|
data=['how is the weather today', 'What is the current weather like today?'] |
|
) |
|
print(finetuner.cos_sim(embeddings[0], embeddings[1])) |
|
``` |
|
|
|
Use with sentence-transformers: |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
from sentence_transformers.util import cos_sim |
|
|
|
sentences = ['how is the weather today', 'What is the current weather like today?'] |
|
|
|
model = SentenceTransformer('jinaai/jina-embedding-b-en-v1') |
|
embeddings = model.encode(sentences) |
|
print(cos_sim(embeddings[0], embeddings[1])) |
|
``` |
|
|
|
|
|
## Fine-tuning |
|
|
|
Please consider [Finetuner](https://github.com/jina-ai/finetuner). |
|
|
|
## Plans |
|
|
|
1. The development of `jina-embedding-s-en-v2` is currently underway with two main objectives: improving performance and increasing the maximum sequence length. |
|
2. We are currently working on a bilingual embedding model that combines English and X language. The upcoming model will be called `jina-embedding-s/b/l-de-v1`. |
|
|
|
## Contact |
|
|
|
Join our [Discord community](https://discord.jina.ai) and chat with other community members about ideas. |
|
|
|
## Citation |
|
|
|
If you find Jina Embeddings useful in your research, please cite the following paper: |
|
|
|
``` latex |
|
@misc{günther2023jina, |
|
title={Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Models}, |
|
author={Michael Günther and Louis Milliken and Jonathan Geuter and Georgios Mastrapas and Bo Wang and Han Xiao}, |
|
year={2023}, |
|
eprint={2307.11224}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
|
