|
--- |
|
pipeline_tag: sentence-similarity |
|
tags: |
|
- sentence-transformers |
|
- feature-extraction |
|
- sentence-similarity |
|
- transformers |
|
- mteb |
|
model-index: |
|
- name: mmlw-roberta-large |
|
results: |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: PL-MTEB/8tags-clustering |
|
name: MTEB 8TagsClustering |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 31.16472823814849 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/allegro-reviews |
|
name: MTEB AllegroReviews |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 47.48508946322067 |
|
- type: f1 |
|
value: 42.33327527584009 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana-pl |
|
name: MTEB ArguAna-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 38.834 |
|
- type: map_at_10 |
|
value: 55.22899999999999 |
|
- type: map_at_100 |
|
value: 55.791999999999994 |
|
- type: map_at_1000 |
|
value: 55.794 |
|
- type: map_at_3 |
|
value: 51.233 |
|
- type: map_at_5 |
|
value: 53.772 |
|
- type: mrr_at_1 |
|
value: 39.687 |
|
- type: mrr_at_10 |
|
value: 55.596000000000004 |
|
- type: mrr_at_100 |
|
value: 56.157000000000004 |
|
- type: mrr_at_1000 |
|
value: 56.157999999999994 |
|
- type: mrr_at_3 |
|
value: 51.66 |
|
- type: mrr_at_5 |
|
value: 54.135 |
|
- type: ndcg_at_1 |
|
value: 38.834 |
|
- type: ndcg_at_10 |
|
value: 63.402 |
|
- type: ndcg_at_100 |
|
value: 65.78 |
|
- type: ndcg_at_1000 |
|
value: 65.816 |
|
- type: ndcg_at_3 |
|
value: 55.349000000000004 |
|
- type: ndcg_at_5 |
|
value: 59.892 |
|
- type: precision_at_1 |
|
value: 38.834 |
|
- type: precision_at_10 |
|
value: 8.905000000000001 |
|
- type: precision_at_100 |
|
value: 0.9939999999999999 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 22.428 |
|
- type: precision_at_5 |
|
value: 15.647 |
|
- type: recall_at_1 |
|
value: 38.834 |
|
- type: recall_at_10 |
|
value: 89.047 |
|
- type: recall_at_100 |
|
value: 99.36 |
|
- type: recall_at_1000 |
|
value: 99.644 |
|
- type: recall_at_3 |
|
value: 67.283 |
|
- type: recall_at_5 |
|
value: 78.236 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/cbd |
|
name: MTEB CBD |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 69.33 |
|
- type: ap |
|
value: 22.972409521444508 |
|
- type: f1 |
|
value: 58.91072163784952 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/cdsce-pairclassification |
|
name: MTEB CDSC-E |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 89.8 |
|
- type: cos_sim_ap |
|
value: 79.87039801032493 |
|
- type: cos_sim_f1 |
|
value: 68.53932584269663 |
|
- type: cos_sim_precision |
|
value: 73.49397590361446 |
|
- type: cos_sim_recall |
|
value: 64.21052631578948 |
|
- type: dot_accuracy |
|
value: 86.1 |
|
- type: dot_ap |
|
value: 63.684975861694035 |
|
- type: dot_f1 |
|
value: 63.61746361746362 |
|
- type: dot_precision |
|
value: 52.57731958762887 |
|
- type: dot_recall |
|
value: 80.52631578947368 |
|
- type: euclidean_accuracy |
|
value: 89.8 |
|
- type: euclidean_ap |
|
value: 79.7527126811392 |
|
- type: euclidean_f1 |
|
value: 68.46361185983827 |
|
- type: euclidean_precision |
|
value: 70.1657458563536 |
|
- type: euclidean_recall |
|
value: 66.84210526315789 |
|
- type: manhattan_accuracy |
|
value: 89.7 |
|
- type: manhattan_ap |
|
value: 79.64632771093657 |
|
- type: manhattan_f1 |
|
value: 68.4931506849315 |
|
- type: manhattan_precision |
|
value: 71.42857142857143 |
|
- type: manhattan_recall |
|
value: 65.78947368421053 |
|
- type: max_accuracy |
|
value: 89.8 |
|
- type: max_ap |
|
value: 79.87039801032493 |
|
- type: max_f1 |
|
value: 68.53932584269663 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: PL-MTEB/cdscr-sts |
|
name: MTEB CDSC-R |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 92.1088892402831 |
|
- type: cos_sim_spearman |
|
value: 92.54126377343101 |
|
- type: euclidean_pearson |
|
value: 91.99022371986013 |
|
- type: euclidean_spearman |
|
value: 92.55235973775511 |
|
- type: manhattan_pearson |
|
value: 91.92170171331357 |
|
- type: manhattan_spearman |
|
value: 92.47797623672449 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: dbpedia-pl |
|
name: MTEB DBPedia-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 8.683 |
|
- type: map_at_10 |
|
value: 18.9 |
|
- type: map_at_100 |
|
value: 26.933 |
|
- type: map_at_1000 |
|
value: 28.558 |
|
- type: map_at_3 |
|
value: 13.638 |
|
- type: map_at_5 |
|
value: 15.9 |
|
- type: mrr_at_1 |
|
value: 63.74999999999999 |
|
- type: mrr_at_10 |
|
value: 73.566 |
|
- type: mrr_at_100 |
|
value: 73.817 |
|
- type: mrr_at_1000 |
|
value: 73.824 |
|
- type: mrr_at_3 |
|
value: 71.875 |
|
- type: mrr_at_5 |
|
value: 73.2 |
|
- type: ndcg_at_1 |
|
value: 53.125 |
|
- type: ndcg_at_10 |
|
value: 40.271 |
|
- type: ndcg_at_100 |
|
value: 45.51 |
|
- type: ndcg_at_1000 |
|
value: 52.968 |
|
- type: ndcg_at_3 |
|
value: 45.122 |
|
- type: ndcg_at_5 |
|
value: 42.306 |
|
- type: precision_at_1 |
|
value: 63.74999999999999 |
|
- type: precision_at_10 |
|
value: 31.55 |
|
- type: precision_at_100 |
|
value: 10.440000000000001 |
|
- type: precision_at_1000 |
|
value: 2.01 |
|
- type: precision_at_3 |
|
value: 48.333 |
|
- type: precision_at_5 |
|
value: 40.5 |
|
- type: recall_at_1 |
|
value: 8.683 |
|
- type: recall_at_10 |
|
value: 24.63 |
|
- type: recall_at_100 |
|
value: 51.762 |
|
- type: recall_at_1000 |
|
value: 75.64999999999999 |
|
- type: recall_at_3 |
|
value: 15.136 |
|
- type: recall_at_5 |
|
value: 18.678 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fiqa-pl |
|
name: MTEB FiQA-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 19.872999999999998 |
|
- type: map_at_10 |
|
value: 32.923 |
|
- type: map_at_100 |
|
value: 34.819 |
|
- type: map_at_1000 |
|
value: 34.99 |
|
- type: map_at_3 |
|
value: 28.500999999999998 |
|
- type: map_at_5 |
|
value: 31.087999999999997 |
|
- type: mrr_at_1 |
|
value: 40.432 |
|
- type: mrr_at_10 |
|
value: 49.242999999999995 |
|
- type: mrr_at_100 |
|
value: 50.014 |
|
- type: mrr_at_1000 |
|
value: 50.05500000000001 |
|
- type: mrr_at_3 |
|
value: 47.144999999999996 |
|
- type: mrr_at_5 |
|
value: 48.171 |
|
- type: ndcg_at_1 |
|
value: 40.586 |
|
- type: ndcg_at_10 |
|
value: 40.887 |
|
- type: ndcg_at_100 |
|
value: 47.701 |
|
- type: ndcg_at_1000 |
|
value: 50.624 |
|
- type: ndcg_at_3 |
|
value: 37.143 |
|
- type: ndcg_at_5 |
|
value: 38.329 |
|
- type: precision_at_1 |
|
value: 40.586 |
|
- type: precision_at_10 |
|
value: 11.497 |
|
- type: precision_at_100 |
|
value: 1.838 |
|
- type: precision_at_1000 |
|
value: 0.23700000000000002 |
|
- type: precision_at_3 |
|
value: 25.0 |
|
- type: precision_at_5 |
|
value: 18.549 |
|
- type: recall_at_1 |
|
value: 19.872999999999998 |
|
- type: recall_at_10 |
|
value: 48.073 |
|
- type: recall_at_100 |
|
value: 73.473 |
|
- type: recall_at_1000 |
|
value: 90.94 |
|
- type: recall_at_3 |
|
value: 33.645 |
|
- type: recall_at_5 |
|
value: 39.711 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: hotpotqa-pl |
|
name: MTEB HotpotQA-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 39.399 |
|
- type: map_at_10 |
|
value: 62.604000000000006 |
|
- type: map_at_100 |
|
value: 63.475 |
|
- type: map_at_1000 |
|
value: 63.534 |
|
- type: map_at_3 |
|
value: 58.870999999999995 |
|
- type: map_at_5 |
|
value: 61.217 |
|
- type: mrr_at_1 |
|
value: 78.758 |
|
- type: mrr_at_10 |
|
value: 84.584 |
|
- type: mrr_at_100 |
|
value: 84.753 |
|
- type: mrr_at_1000 |
|
value: 84.759 |
|
- type: mrr_at_3 |
|
value: 83.65700000000001 |
|
- type: mrr_at_5 |
|
value: 84.283 |
|
- type: ndcg_at_1 |
|
value: 78.798 |
|
- type: ndcg_at_10 |
|
value: 71.04 |
|
- type: ndcg_at_100 |
|
value: 74.048 |
|
- type: ndcg_at_1000 |
|
value: 75.163 |
|
- type: ndcg_at_3 |
|
value: 65.862 |
|
- type: ndcg_at_5 |
|
value: 68.77600000000001 |
|
- type: precision_at_1 |
|
value: 78.798 |
|
- type: precision_at_10 |
|
value: 14.949000000000002 |
|
- type: precision_at_100 |
|
value: 1.7309999999999999 |
|
- type: precision_at_1000 |
|
value: 0.188 |
|
- type: precision_at_3 |
|
value: 42.237 |
|
- type: precision_at_5 |
|
value: 27.634999999999998 |
|
- type: recall_at_1 |
|
value: 39.399 |
|
- type: recall_at_10 |
|
value: 74.747 |
|
- type: recall_at_100 |
|
value: 86.529 |
|
- type: recall_at_1000 |
|
value: 93.849 |
|
- type: recall_at_3 |
|
value: 63.356 |
|
- type: recall_at_5 |
|
value: 69.08800000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: msmarco-pl |
|
name: MTEB MSMARCO-PL |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 19.598 |
|
- type: map_at_10 |
|
value: 30.453999999999997 |
|
- type: map_at_100 |
|
value: 31.601000000000003 |
|
- type: map_at_1000 |
|
value: 31.66 |
|
- type: map_at_3 |
|
value: 27.118 |
|
- type: map_at_5 |
|
value: 28.943 |
|
- type: mrr_at_1 |
|
value: 20.1 |
|
- type: mrr_at_10 |
|
value: 30.978 |
|
- type: mrr_at_100 |
|
value: 32.057 |
|
- type: mrr_at_1000 |
|
value: 32.112 |
|
- type: mrr_at_3 |
|
value: 27.679 |
|
- type: mrr_at_5 |
|
value: 29.493000000000002 |
|
- type: ndcg_at_1 |
|
value: 20.158 |
|
- type: ndcg_at_10 |
|
value: 36.63 |
|
- type: ndcg_at_100 |
|
value: 42.291000000000004 |
|
- type: ndcg_at_1000 |
|
value: 43.828 |
|
- type: ndcg_at_3 |
|
value: 29.744999999999997 |
|
- type: ndcg_at_5 |
|
value: 33.024 |
|
- type: precision_at_1 |
|
value: 20.158 |
|
- type: precision_at_10 |
|
value: 5.811999999999999 |
|
- type: precision_at_100 |
|
value: 0.868 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 12.689 |
|
- type: precision_at_5 |
|
value: 9.295 |
|
- type: recall_at_1 |
|
value: 19.598 |
|
- type: recall_at_10 |
|
value: 55.596999999999994 |
|
- type: recall_at_100 |
|
value: 82.143 |
|
- type: recall_at_1000 |
|
value: 94.015 |
|
- type: recall_at_3 |
|
value: 36.720000000000006 |
|
- type: recall_at_5 |
|
value: 44.606 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 74.8117014122394 |
|
- type: f1 |
|
value: 72.0259730121889 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 77.84465366509752 |
|
- type: f1 |
|
value: 77.73439218970051 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nfcorpus-pl |
|
name: MTEB NFCorpus-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 5.604 |
|
- type: map_at_10 |
|
value: 12.684000000000001 |
|
- type: map_at_100 |
|
value: 16.274 |
|
- type: map_at_1000 |
|
value: 17.669 |
|
- type: map_at_3 |
|
value: 9.347 |
|
- type: map_at_5 |
|
value: 10.752 |
|
- type: mrr_at_1 |
|
value: 43.963 |
|
- type: mrr_at_10 |
|
value: 52.94 |
|
- type: mrr_at_100 |
|
value: 53.571000000000005 |
|
- type: mrr_at_1000 |
|
value: 53.613 |
|
- type: mrr_at_3 |
|
value: 51.032 |
|
- type: mrr_at_5 |
|
value: 52.193 |
|
- type: ndcg_at_1 |
|
value: 41.486000000000004 |
|
- type: ndcg_at_10 |
|
value: 33.937 |
|
- type: ndcg_at_100 |
|
value: 31.726 |
|
- type: ndcg_at_1000 |
|
value: 40.331 |
|
- type: ndcg_at_3 |
|
value: 39.217 |
|
- type: ndcg_at_5 |
|
value: 36.521 |
|
- type: precision_at_1 |
|
value: 43.034 |
|
- type: precision_at_10 |
|
value: 25.324999999999996 |
|
- type: precision_at_100 |
|
value: 8.022 |
|
- type: precision_at_1000 |
|
value: 2.0629999999999997 |
|
- type: precision_at_3 |
|
value: 36.945 |
|
- type: precision_at_5 |
|
value: 31.517 |
|
- type: recall_at_1 |
|
value: 5.604 |
|
- type: recall_at_10 |
|
value: 16.554 |
|
- type: recall_at_100 |
|
value: 33.113 |
|
- type: recall_at_1000 |
|
value: 62.832 |
|
- type: recall_at_3 |
|
value: 10.397 |
|
- type: recall_at_5 |
|
value: 12.629999999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nq-pl |
|
name: MTEB NQ-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 26.642 |
|
- type: map_at_10 |
|
value: 40.367999999999995 |
|
- type: map_at_100 |
|
value: 41.487 |
|
- type: map_at_1000 |
|
value: 41.528 |
|
- type: map_at_3 |
|
value: 36.292 |
|
- type: map_at_5 |
|
value: 38.548 |
|
- type: mrr_at_1 |
|
value: 30.156 |
|
- type: mrr_at_10 |
|
value: 42.853 |
|
- type: mrr_at_100 |
|
value: 43.742 |
|
- type: mrr_at_1000 |
|
value: 43.772 |
|
- type: mrr_at_3 |
|
value: 39.47 |
|
- type: mrr_at_5 |
|
value: 41.366 |
|
- type: ndcg_at_1 |
|
value: 30.214000000000002 |
|
- type: ndcg_at_10 |
|
value: 47.620000000000005 |
|
- type: ndcg_at_100 |
|
value: 52.486 |
|
- type: ndcg_at_1000 |
|
value: 53.482 |
|
- type: ndcg_at_3 |
|
value: 39.864 |
|
- type: ndcg_at_5 |
|
value: 43.645 |
|
- type: precision_at_1 |
|
value: 30.214000000000002 |
|
- type: precision_at_10 |
|
value: 8.03 |
|
- type: precision_at_100 |
|
value: 1.0739999999999998 |
|
- type: precision_at_1000 |
|
value: 0.117 |
|
- type: precision_at_3 |
|
value: 18.183 |
|
- type: precision_at_5 |
|
value: 13.105 |
|
- type: recall_at_1 |
|
value: 26.642 |
|
- type: recall_at_10 |
|
value: 67.282 |
|
- type: recall_at_100 |
|
value: 88.632 |
|
- type: recall_at_1000 |
|
value: 96.109 |
|
- type: recall_at_3 |
|
value: 47.048 |
|
- type: recall_at_5 |
|
value: 55.791000000000004 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: laugustyniak/abusive-clauses-pl |
|
name: MTEB PAC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 64.69446857804807 |
|
- type: ap |
|
value: 75.58028779280512 |
|
- type: f1 |
|
value: 62.3610392963539 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/ppc-pairclassification |
|
name: MTEB PPC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 88.4 |
|
- type: cos_sim_ap |
|
value: 93.56462741831817 |
|
- type: cos_sim_f1 |
|
value: 90.73634204275535 |
|
- type: cos_sim_precision |
|
value: 86.94992412746586 |
|
- type: cos_sim_recall |
|
value: 94.86754966887418 |
|
- type: dot_accuracy |
|
value: 75.3 |
|
- type: dot_ap |
|
value: 83.06945936688015 |
|
- type: dot_f1 |
|
value: 81.50887573964496 |
|
- type: dot_precision |
|
value: 73.66310160427807 |
|
- type: dot_recall |
|
value: 91.22516556291392 |
|
- type: euclidean_accuracy |
|
value: 88.8 |
|
- type: euclidean_ap |
|
value: 93.53974198044985 |
|
- type: euclidean_f1 |
|
value: 90.87947882736157 |
|
- type: euclidean_precision |
|
value: 89.42307692307693 |
|
- type: euclidean_recall |
|
value: 92.3841059602649 |
|
- type: manhattan_accuracy |
|
value: 88.8 |
|
- type: manhattan_ap |
|
value: 93.54209967780366 |
|
- type: manhattan_f1 |
|
value: 90.85072231139645 |
|
- type: manhattan_precision |
|
value: 88.1619937694704 |
|
- type: manhattan_recall |
|
value: 93.70860927152319 |
|
- type: max_accuracy |
|
value: 88.8 |
|
- type: max_ap |
|
value: 93.56462741831817 |
|
- type: max_f1 |
|
value: 90.87947882736157 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/psc-pairclassification |
|
name: MTEB PSC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 97.03153988868274 |
|
- type: cos_sim_ap |
|
value: 98.63208302459417 |
|
- type: cos_sim_f1 |
|
value: 95.06172839506173 |
|
- type: cos_sim_precision |
|
value: 96.25 |
|
- type: cos_sim_recall |
|
value: 93.90243902439023 |
|
- type: dot_accuracy |
|
value: 86.82745825602969 |
|
- type: dot_ap |
|
value: 83.77450133931302 |
|
- type: dot_f1 |
|
value: 79.3053545586107 |
|
- type: dot_precision |
|
value: 75.48209366391184 |
|
- type: dot_recall |
|
value: 83.53658536585365 |
|
- type: euclidean_accuracy |
|
value: 97.03153988868274 |
|
- type: euclidean_ap |
|
value: 98.80678168225653 |
|
- type: euclidean_f1 |
|
value: 95.20958083832335 |
|
- type: euclidean_precision |
|
value: 93.52941176470588 |
|
- type: euclidean_recall |
|
value: 96.95121951219512 |
|
- type: manhattan_accuracy |
|
value: 97.21706864564007 |
|
- type: manhattan_ap |
|
value: 98.82279484224186 |
|
- type: manhattan_f1 |
|
value: 95.44072948328268 |
|
- type: manhattan_precision |
|
value: 95.15151515151516 |
|
- type: manhattan_recall |
|
value: 95.73170731707317 |
|
- type: max_accuracy |
|
value: 97.21706864564007 |
|
- type: max_ap |
|
value: 98.82279484224186 |
|
- type: max_f1 |
|
value: 95.44072948328268 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/polemo2_in |
|
name: MTEB PolEmo2.0-IN |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 76.84210526315789 |
|
- type: f1 |
|
value: 75.49713789106988 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/polemo2_out |
|
name: MTEB PolEmo2.0-OUT |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 53.7246963562753 |
|
- type: f1 |
|
value: 43.060592194322986 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: quora-pl |
|
name: MTEB Quora-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 67.021 |
|
- type: map_at_10 |
|
value: 81.362 |
|
- type: map_at_100 |
|
value: 82.06700000000001 |
|
- type: map_at_1000 |
|
value: 82.084 |
|
- type: map_at_3 |
|
value: 78.223 |
|
- type: map_at_5 |
|
value: 80.219 |
|
- type: mrr_at_1 |
|
value: 77.17 |
|
- type: mrr_at_10 |
|
value: 84.222 |
|
- type: mrr_at_100 |
|
value: 84.37599999999999 |
|
- type: mrr_at_1000 |
|
value: 84.379 |
|
- type: mrr_at_3 |
|
value: 83.003 |
|
- type: mrr_at_5 |
|
value: 83.834 |
|
- type: ndcg_at_1 |
|
value: 77.29 |
|
- type: ndcg_at_10 |
|
value: 85.506 |
|
- type: ndcg_at_100 |
|
value: 87.0 |
|
- type: ndcg_at_1000 |
|
value: 87.143 |
|
- type: ndcg_at_3 |
|
value: 82.17 |
|
- type: ndcg_at_5 |
|
value: 84.057 |
|
- type: precision_at_1 |
|
value: 77.29 |
|
- type: precision_at_10 |
|
value: 13.15 |
|
- type: precision_at_100 |
|
value: 1.522 |
|
- type: precision_at_1000 |
|
value: 0.156 |
|
- type: precision_at_3 |
|
value: 36.173 |
|
- type: precision_at_5 |
|
value: 23.988 |
|
- type: recall_at_1 |
|
value: 67.021 |
|
- type: recall_at_10 |
|
value: 93.943 |
|
- type: recall_at_100 |
|
value: 99.167 |
|
- type: recall_at_1000 |
|
value: 99.929 |
|
- type: recall_at_3 |
|
value: 84.55799999999999 |
|
- type: recall_at_5 |
|
value: 89.697 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scidocs-pl |
|
name: MTEB SCIDOCS-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 4.523 |
|
- type: map_at_10 |
|
value: 11.584 |
|
- type: map_at_100 |
|
value: 13.705 |
|
- type: map_at_1000 |
|
value: 14.038999999999998 |
|
- type: map_at_3 |
|
value: 8.187999999999999 |
|
- type: map_at_5 |
|
value: 9.922 |
|
- type: mrr_at_1 |
|
value: 22.1 |
|
- type: mrr_at_10 |
|
value: 32.946999999999996 |
|
- type: mrr_at_100 |
|
value: 34.11 |
|
- type: mrr_at_1000 |
|
value: 34.163 |
|
- type: mrr_at_3 |
|
value: 29.633 |
|
- type: mrr_at_5 |
|
value: 31.657999999999998 |
|
- type: ndcg_at_1 |
|
value: 22.2 |
|
- type: ndcg_at_10 |
|
value: 19.466 |
|
- type: ndcg_at_100 |
|
value: 27.725 |
|
- type: ndcg_at_1000 |
|
value: 33.539 |
|
- type: ndcg_at_3 |
|
value: 18.26 |
|
- type: ndcg_at_5 |
|
value: 16.265 |
|
- type: precision_at_1 |
|
value: 22.2 |
|
- type: precision_at_10 |
|
value: 10.11 |
|
- type: precision_at_100 |
|
value: 2.204 |
|
- type: precision_at_1000 |
|
value: 0.36 |
|
- type: precision_at_3 |
|
value: 17.1 |
|
- type: precision_at_5 |
|
value: 14.44 |
|
- type: recall_at_1 |
|
value: 4.523 |
|
- type: recall_at_10 |
|
value: 20.497 |
|
- type: recall_at_100 |
|
value: 44.757000000000005 |
|
- type: recall_at_1000 |
|
value: 73.14699999999999 |
|
- type: recall_at_3 |
|
value: 10.413 |
|
- type: recall_at_5 |
|
value: 14.638000000000002 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/sicke-pl-pairclassification |
|
name: MTEB SICK-E-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 87.4235629841011 |
|
- type: cos_sim_ap |
|
value: 84.46531935663157 |
|
- type: cos_sim_f1 |
|
value: 77.18910963944077 |
|
- type: cos_sim_precision |
|
value: 79.83257229832572 |
|
- type: cos_sim_recall |
|
value: 74.71509971509973 |
|
- type: dot_accuracy |
|
value: 81.10476966979209 |
|
- type: dot_ap |
|
value: 71.12231750543143 |
|
- type: dot_f1 |
|
value: 68.13455657492355 |
|
- type: dot_precision |
|
value: 59.69989281886387 |
|
- type: dot_recall |
|
value: 79.34472934472934 |
|
- type: euclidean_accuracy |
|
value: 87.21973094170403 |
|
- type: euclidean_ap |
|
value: 84.33077991405355 |
|
- type: euclidean_f1 |
|
value: 76.81931132410365 |
|
- type: euclidean_precision |
|
value: 76.57466383581033 |
|
- type: euclidean_recall |
|
value: 77.06552706552706 |
|
- type: manhattan_accuracy |
|
value: 87.21973094170403 |
|
- type: manhattan_ap |
|
value: 84.35651252115137 |
|
- type: manhattan_f1 |
|
value: 76.87004481213376 |
|
- type: manhattan_precision |
|
value: 74.48229792919172 |
|
- type: manhattan_recall |
|
value: 79.41595441595442 |
|
- type: max_accuracy |
|
value: 87.4235629841011 |
|
- type: max_ap |
|
value: 84.46531935663157 |
|
- type: max_f1 |
|
value: 77.18910963944077 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: PL-MTEB/sickr-pl-sts |
|
name: MTEB SICK-R-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 83.05629619004273 |
|
- type: cos_sim_spearman |
|
value: 79.90632583043678 |
|
- type: euclidean_pearson |
|
value: 81.56426663515931 |
|
- type: euclidean_spearman |
|
value: 80.05439220131294 |
|
- type: manhattan_pearson |
|
value: 81.52958181013108 |
|
- type: manhattan_spearman |
|
value: 80.0387467163383 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (pl) |
|
config: pl |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 35.93847200513348 |
|
- type: cos_sim_spearman |
|
value: 39.31543525546526 |
|
- type: euclidean_pearson |
|
value: 30.19743936591465 |
|
- type: euclidean_spearman |
|
value: 39.966612599252095 |
|
- type: manhattan_pearson |
|
value: 30.195614462473387 |
|
- type: manhattan_spearman |
|
value: 39.822552043685754 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scifact-pl |
|
name: MTEB SciFact-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 56.05 |
|
- type: map_at_10 |
|
value: 65.93299999999999 |
|
- type: map_at_100 |
|
value: 66.571 |
|
- type: map_at_1000 |
|
value: 66.60000000000001 |
|
- type: map_at_3 |
|
value: 63.489 |
|
- type: map_at_5 |
|
value: 64.91799999999999 |
|
- type: mrr_at_1 |
|
value: 59.0 |
|
- type: mrr_at_10 |
|
value: 67.026 |
|
- type: mrr_at_100 |
|
value: 67.559 |
|
- type: mrr_at_1000 |
|
value: 67.586 |
|
- type: mrr_at_3 |
|
value: 65.444 |
|
- type: mrr_at_5 |
|
value: 66.278 |
|
- type: ndcg_at_1 |
|
value: 59.0 |
|
- type: ndcg_at_10 |
|
value: 70.233 |
|
- type: ndcg_at_100 |
|
value: 72.789 |
|
- type: ndcg_at_1000 |
|
value: 73.637 |
|
- type: ndcg_at_3 |
|
value: 66.40700000000001 |
|
- type: ndcg_at_5 |
|
value: 68.206 |
|
- type: precision_at_1 |
|
value: 59.0 |
|
- type: precision_at_10 |
|
value: 9.367 |
|
- type: precision_at_100 |
|
value: 1.06 |
|
- type: precision_at_1000 |
|
value: 0.11299999999999999 |
|
- type: precision_at_3 |
|
value: 26.222 |
|
- type: precision_at_5 |
|
value: 17.067 |
|
- type: recall_at_1 |
|
value: 56.05 |
|
- type: recall_at_10 |
|
value: 82.089 |
|
- type: recall_at_100 |
|
value: 93.167 |
|
- type: recall_at_1000 |
|
value: 100.0 |
|
- type: recall_at_3 |
|
value: 71.822 |
|
- type: recall_at_5 |
|
value: 76.483 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: trec-covid-pl |
|
name: MTEB TRECCOVID-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.21 |
|
- type: map_at_10 |
|
value: 1.7680000000000002 |
|
- type: map_at_100 |
|
value: 9.447999999999999 |
|
- type: map_at_1000 |
|
value: 21.728 |
|
- type: map_at_3 |
|
value: 0.603 |
|
- type: map_at_5 |
|
value: 0.9610000000000001 |
|
- type: mrr_at_1 |
|
value: 80.0 |
|
- type: mrr_at_10 |
|
value: 88.667 |
|
- type: mrr_at_100 |
|
value: 88.667 |
|
- type: mrr_at_1000 |
|
value: 88.667 |
|
- type: mrr_at_3 |
|
value: 87.667 |
|
- type: mrr_at_5 |
|
value: 88.667 |
|
- type: ndcg_at_1 |
|
value: 77.0 |
|
- type: ndcg_at_10 |
|
value: 70.814 |
|
- type: ndcg_at_100 |
|
value: 52.532000000000004 |
|
- type: ndcg_at_1000 |
|
value: 45.635999999999996 |
|
- type: ndcg_at_3 |
|
value: 76.542 |
|
- type: ndcg_at_5 |
|
value: 73.24000000000001 |
|
- type: precision_at_1 |
|
value: 80.0 |
|
- type: precision_at_10 |
|
value: 75.0 |
|
- type: precision_at_100 |
|
value: 53.879999999999995 |
|
- type: precision_at_1000 |
|
value: 20.002 |
|
- type: precision_at_3 |
|
value: 80.0 |
|
- type: precision_at_5 |
|
value: 76.4 |
|
- type: recall_at_1 |
|
value: 0.21 |
|
- type: recall_at_10 |
|
value: 2.012 |
|
- type: recall_at_100 |
|
value: 12.781999999999998 |
|
- type: recall_at_1000 |
|
value: 42.05 |
|
- type: recall_at_3 |
|
value: 0.644 |
|
- type: recall_at_5 |
|
value: 1.04 |
|
language: pl |
|
license: apache-2.0 |
|
widget: |
|
- source_sentence: "zapytanie: Jak dożyć 100 lat?" |
|
sentences: |
|
- "Trzeba zdrowo się odżywiać i uprawiać sport." |
|
- "Trzeba pić alkohol, imprezować i jeździć szybkimi autami." |
|
- "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu." |
|
|
|
--- |
|
|
|
<h1 align="center">MMLW-roberta-large</h1> |
|
|
|
MMLW (muszę mieć lepszą wiadomość) are neural text encoders for Polish. |
|
This is a distilled model that can be used to generate embeddings applicable to many tasks such as semantic similarity, clustering, information retrieval. The model can also serve as a base for further fine-tuning. |
|
It transforms texts to 1024 dimensional vectors. |
|
The model was initialized with Polish RoBERTa checkpoint, and then trained with [multilingual knowledge distillation method](https://aclanthology.org/2020.emnlp-main.365/) on a diverse corpus of 60 million Polish-English text pairs. We utilised [English FlagEmbeddings (BGE)](https://huggingface.co/BAAI/bge-base-en) as teacher models for distillation. |
|
|
|
## Usage (Sentence-Transformers) |
|
|
|
⚠️ Our embedding models require the use of specific prefixes and suffixes when encoding texts. For this model, each query should be preceded by the prefix **"zapytanie: "** ⚠️ |
|
|
|
You can use the model like this with [sentence-transformers](https://www.SBERT.net): |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
from sentence_transformers.util import cos_sim |
|
|
|
query_prefix = "zapytanie: " |
|
answer_prefix = "" |
|
queries = [query_prefix + "Jak dożyć 100 lat?"] |
|
answers = [ |
|
answer_prefix + "Trzeba zdrowo się odżywiać i uprawiać sport.", |
|
answer_prefix + "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.", |
|
answer_prefix + "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu." |
|
] |
|
model = SentenceTransformer("sdadas/mmlw-roberta-large") |
|
queries_emb = model.encode(queries, convert_to_tensor=True, show_progress_bar=False) |
|
answers_emb = model.encode(answers, convert_to_tensor=True, show_progress_bar=False) |
|
|
|
best_answer = cos_sim(queries_emb, answers_emb).argmax().item() |
|
print(answers[best_answer]) |
|
# Trzeba zdrowo się odżywiać i uprawiać sport. |
|
``` |
|
|
|
## Evaluation Results |
|
|
|
- The model achieves an **Average Score** of **63.23** on the Polish Massive Text Embedding Benchmark (MTEB). See [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) for detailed results. |
|
- The model achieves **NDCG@10** of **55.95** on the Polish Information Retrieval Benchmark. See [PIRB Leaderboard](https://huggingface.co/spaces/sdadas/pirb) for detailed results. |
|
|
|
## Acknowledgements |
|
This model was trained with the A100 GPU cluster support delivered by the Gdansk University of Technology within the TASK center initiative. |
|
|
|
## Citation |
|
|
|
```bibtex |
|
@article{dadas2024pirb, |
|
title={{PIRB}: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods}, |
|
author={Sławomir Dadas and Michał Perełkiewicz and Rafał Poświata}, |
|
year={2024}, |
|
eprint={2402.13350}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
