MMLW-e5-small
MMLW (muszę mieć lepszą wiadomość) are neural text encoders for Polish. This is a distilled model that can be used to generate embeddings applicable to many tasks such as semantic similarity, clustering, information retrieval. The model can also serve as a base for further fine-tuning. It transforms texts to 384 dimensional vectors. The model was initialized with multilingual E5 checkpoint, and then trained with multilingual knowledge distillation method on a diverse corpus of 60 million Polish-English text pairs. We utilised English FlagEmbeddings (BGE) as teacher models for distillation.
Usage (Sentence-Transformers)
⚠️ Our embedding models require the use of specific prefixes and suffixes when encoding texts. For this model, queries should be prefixed with "query: " and passages with "passage: " ⚠️
You can use the model like this with sentence-transformers:
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
query_prefix = "query: "
answer_prefix = "passage: "
queries = [query_prefix + "Jak dożyć 100 lat?"]
answers = [
answer_prefix + "Trzeba zdrowo się odżywiać i uprawiać sport.",
answer_prefix + "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.",
answer_prefix + "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu."
]
model = SentenceTransformer("sdadas/mmlw-e5-small")
queries_emb = model.encode(queries, convert_to_tensor=True, show_progress_bar=False)
answers_emb = model.encode(answers, convert_to_tensor=True, show_progress_bar=False)
best_answer = cos_sim(queries_emb, answers_emb).argmax().item()
print(answers[best_answer])
# Trzeba zdrowo się odżywiać i uprawiać sport.
Evaluation Results
- The model achieves an Average Score of 55.84 on the Polish Massive Text Embedding Benchmark (MTEB). See MTEB Leaderboard for detailed results.
- The model achieves NDCG@10 of 47.64 on the Polish Information Retrieval Benchmark. See PIRB Leaderboard for detailed results.
Acknowledgements
This model was trained with the A100 GPU cluster support delivered by the Gdansk University of Technology within the TASK center initiative.
Citation
@article{dadas2024pirb,
title={{PIRB}: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods},
author={Sławomir Dadas and Michał Perełkiewicz and Rafał Poświata},
year={2024},
eprint={2402.13350},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
- Downloads last month
- 4,034
Evaluation results
- v_measure on MTEB 8TagsClusteringtest set self-reported31.772
- accuracy on MTEB AllegroReviewstest set self-reported33.032
- f1 on MTEB AllegroReviewstest set self-reported29.800
- map_at_1 on MTEB ArguAna-PLtest set self-reported28.805
- map_at_10 on MTEB ArguAna-PLtest set self-reported45.327
- map_at_100 on MTEB ArguAna-PLtest set self-reported46.170
- map_at_1000 on MTEB ArguAna-PLtest set self-reported46.177
- map_at_3 on MTEB ArguAna-PLtest set self-reported40.529
- map_at_5 on MTEB ArguAna-PLtest set self-reported43.335
- mrr_at_1 on MTEB ArguAna-PLtest set self-reported30.299
- mrr_at_10 on MTEB ArguAna-PLtest set self-reported45.763
- mrr_at_100 on MTEB ArguAna-PLtest set self-reported46.641
- mrr_at_1000 on MTEB ArguAna-PLtest set self-reported46.648
- mrr_at_3 on MTEB ArguAna-PLtest set self-reported41.074
- mrr_at_5 on MTEB ArguAna-PLtest set self-reported43.837
- ndcg_at_1 on MTEB ArguAna-PLtest set self-reported28.805
- ndcg_at_10 on MTEB ArguAna-PLtest set self-reported54.308
- ndcg_at_100 on MTEB ArguAna-PLtest set self-reported57.879
- ndcg_at_1000 on MTEB ArguAna-PLtest set self-reported58.048
- ndcg_at_3 on MTEB ArguAna-PLtest set self-reported44.502
- ndcg_at_5 on MTEB ArguAna-PLtest set self-reported49.519
- precision_at_1 on MTEB ArguAna-PLtest set self-reported28.805
- precision_at_10 on MTEB ArguAna-PLtest set self-reported8.286
- precision_at_100 on MTEB ArguAna-PLtest set self-reported0.984
- precision_at_1000 on MTEB ArguAna-PLtest set self-reported0.100
- precision_at_3 on MTEB ArguAna-PLtest set self-reported18.682
- precision_at_5 on MTEB ArguAna-PLtest set self-reported13.627
- recall_at_1 on MTEB ArguAna-PLtest set self-reported28.805
- recall_at_10 on MTEB ArguAna-PLtest set self-reported82.859
- recall_at_100 on MTEB ArguAna-PLtest set self-reported98.364
- recall_at_1000 on MTEB ArguAna-PLtest set self-reported99.644
- recall_at_3 on MTEB ArguAna-PLtest set self-reported56.046
- recall_at_5 on MTEB ArguAna-PLtest set self-reported68.137
- accuracy on MTEB CBDtest set self-reported64.240
- ap on MTEB CBDtest set self-reported17.967
- f1 on MTEB CBDtest set self-reported52.974
- cos_sim_accuracy on MTEB CDSC-Etest set self-reported88.800
- cos_sim_ap on MTEB CDSC-Etest set self-reported76.680
- cos_sim_f1 on MTEB CDSC-Etest set self-reported66.824
- cos_sim_precision on MTEB CDSC-Etest set self-reported60.426
- cos_sim_recall on MTEB CDSC-Etest set self-reported74.737
- dot_accuracy on MTEB CDSC-Etest set self-reported88.100
- dot_ap on MTEB CDSC-Etest set self-reported72.049
- dot_f1 on MTEB CDSC-Etest set self-reported66.667
- dot_precision on MTEB CDSC-Etest set self-reported69.318
- dot_recall on MTEB CDSC-Etest set self-reported64.211
- euclidean_accuracy on MTEB CDSC-Etest set self-reported88.800
- euclidean_ap on MTEB CDSC-Etest set self-reported76.636
- euclidean_f1 on MTEB CDSC-Etest set self-reported67.133
- euclidean_precision on MTEB CDSC-Etest set self-reported60.251
- euclidean_recall on MTEB CDSC-Etest set self-reported75.789
- manhattan_accuracy on MTEB CDSC-Etest set self-reported88.900
- manhattan_ap on MTEB CDSC-Etest set self-reported76.546
- manhattan_f1 on MTEB CDSC-Etest set self-reported66.667
- manhattan_precision on MTEB CDSC-Etest set self-reported60.515
- manhattan_recall on MTEB CDSC-Etest set self-reported74.211
- max_accuracy on MTEB CDSC-Etest set self-reported88.900
- max_ap on MTEB CDSC-Etest set self-reported76.680
- max_f1 on MTEB CDSC-Etest set self-reported67.133
- cos_sim_pearson on MTEB CDSC-Rtest set self-reported91.642
- cos_sim_spearman on MTEB CDSC-Rtest set self-reported91.976
- euclidean_pearson on MTEB CDSC-Rtest set self-reported90.875
- euclidean_spearman on MTEB CDSC-Rtest set self-reported91.924
- manhattan_pearson on MTEB CDSC-Rtest set self-reported90.842
- manhattan_spearman on MTEB CDSC-Rtest set self-reported91.859
- map_at_1 on MTEB DBPedia-PLtest set self-reported6.148
- map_at_10 on MTEB DBPedia-PLtest set self-reported12.871
- map_at_100 on MTEB DBPedia-PLtest set self-reported18.040
- map_at_1000 on MTEB DBPedia-PLtest set self-reported19.286
- map_at_3 on MTEB DBPedia-PLtest set self-reported9.156
- map_at_5 on MTEB DBPedia-PLtest set self-reported10.858
- mrr_at_1 on MTEB DBPedia-PLtest set self-reported53.250
- mrr_at_10 on MTEB DBPedia-PLtest set self-reported61.017
- mrr_at_100 on MTEB DBPedia-PLtest set self-reported61.484
- mrr_at_1000 on MTEB DBPedia-PLtest set self-reported61.508
- mrr_at_3 on MTEB DBPedia-PLtest set self-reported58.750
- mrr_at_5 on MTEB DBPedia-PLtest set self-reported60.375
- ndcg_at_1 on MTEB DBPedia-PLtest set self-reported41.000
- ndcg_at_10 on MTEB DBPedia-PLtest set self-reported30.281
- ndcg_at_100 on MTEB DBPedia-PLtest set self-reported33.956
- ndcg_at_1000 on MTEB DBPedia-PLtest set self-reported40.770
- ndcg_at_3 on MTEB DBPedia-PLtest set self-reported34.127
- ndcg_at_5 on MTEB DBPedia-PLtest set self-reported32.274
- precision_at_1 on MTEB DBPedia-PLtest set self-reported52.500
- precision_at_10 on MTEB DBPedia-PLtest set self-reported24.525
- precision_at_100 on MTEB DBPedia-PLtest set self-reported8.125
- precision_at_1000 on MTEB DBPedia-PLtest set self-reported1.728
- precision_at_3 on MTEB DBPedia-PLtest set self-reported37.083
- precision_at_5 on MTEB DBPedia-PLtest set self-reported32.150
- recall_at_1 on MTEB DBPedia-PLtest set self-reported6.148
- recall_at_10 on MTEB DBPedia-PLtest set self-reported17.866
- recall_at_100 on MTEB DBPedia-PLtest set self-reported39.213
- recall_at_1000 on MTEB DBPedia-PLtest set self-reported61.604
- recall_at_3 on MTEB DBPedia-PLtest set self-reported10.084
- recall_at_5 on MTEB DBPedia-PLtest set self-reported13.334
- map_at_1 on MTEB FiQA-PLtest set self-reported14.643
- map_at_10 on MTEB FiQA-PLtest set self-reported23.166
- map_at_100 on MTEB FiQA-PLtest set self-reported24.725
- map_at_1000 on MTEB FiQA-PLtest set self-reported24.920
- map_at_3 on MTEB FiQA-PLtest set self-reported20.166
- map_at_5 on MTEB FiQA-PLtest set self-reported22.003
- mrr_at_1 on MTEB FiQA-PLtest set self-reported29.630
- mrr_at_10 on MTEB FiQA-PLtest set self-reported37.632
- mrr_at_100 on MTEB FiQA-PLtest set self-reported38.512
- mrr_at_1000 on MTEB FiQA-PLtest set self-reported38.578
- mrr_at_3 on MTEB FiQA-PLtest set self-reported35.391
- mrr_at_5 on MTEB FiQA-PLtest set self-reported36.857
- ndcg_at_1 on MTEB FiQA-PLtest set self-reported29.167
- ndcg_at_10 on MTEB FiQA-PLtest set self-reported29.749
- ndcg_at_100 on MTEB FiQA-PLtest set self-reported35.983
- ndcg_at_1000 on MTEB FiQA-PLtest set self-reported39.817
- ndcg_at_3 on MTEB FiQA-PLtest set self-reported26.739
- ndcg_at_5 on MTEB FiQA-PLtest set self-reported27.993
- precision_at_1 on MTEB FiQA-PLtest set self-reported29.167
- precision_at_10 on MTEB FiQA-PLtest set self-reported8.333
- precision_at_100 on MTEB FiQA-PLtest set self-reported1.448
- precision_at_1000 on MTEB FiQA-PLtest set self-reported0.213
- precision_at_3 on MTEB FiQA-PLtest set self-reported17.747
- precision_at_5 on MTEB FiQA-PLtest set self-reported13.580
- recall_at_1 on MTEB FiQA-PLtest set self-reported14.643
- recall_at_10 on MTEB FiQA-PLtest set self-reported35.247
- recall_at_100 on MTEB FiQA-PLtest set self-reported59.151
- recall_at_1000 on MTEB FiQA-PLtest set self-reported82.565
- recall_at_3 on MTEB FiQA-PLtest set self-reported24.006
- recall_at_5 on MTEB FiQA-PLtest set self-reported29.383
- map_at_1 on MTEB HotpotQA-PLtest set self-reported32.627
- map_at_10 on MTEB HotpotQA-PLtest set self-reported48.041
- map_at_100 on MTEB HotpotQA-PLtest set self-reported49.008
- map_at_1000 on MTEB HotpotQA-PLtest set self-reported49.093
- map_at_3 on MTEB HotpotQA-PLtest set self-reported44.774
- map_at_5 on MTEB HotpotQA-PLtest set self-reported46.791
- mrr_at_1 on MTEB HotpotQA-PLtest set self-reported65.280
- mrr_at_10 on MTEB HotpotQA-PLtest set self-reported72.535
- mrr_at_100 on MTEB HotpotQA-PLtest set self-reported72.892
- mrr_at_1000 on MTEB HotpotQA-PLtest set self-reported72.909
- mrr_at_3 on MTEB HotpotQA-PLtest set self-reported71.083
- mrr_at_5 on MTEB HotpotQA-PLtest set self-reported71.985
- ndcg_at_1 on MTEB HotpotQA-PLtest set self-reported65.253
- ndcg_at_10 on MTEB HotpotQA-PLtest set self-reported57.137
- ndcg_at_100 on MTEB HotpotQA-PLtest set self-reported60.783
- ndcg_at_1000 on MTEB HotpotQA-PLtest set self-reported62.507
- ndcg_at_3 on MTEB HotpotQA-PLtest set self-reported52.170
- ndcg_at_5 on MTEB HotpotQA-PLtest set self-reported54.896
- precision_at_1 on MTEB HotpotQA-PLtest set self-reported65.253
- precision_at_10 on MTEB HotpotQA-PLtest set self-reported12.088
- precision_at_100 on MTEB HotpotQA-PLtest set self-reported1.496
- precision_at_1000 on MTEB HotpotQA-PLtest set self-reported0.172
- precision_at_3 on MTEB HotpotQA-PLtest set self-reported32.960
- precision_at_5 on MTEB HotpotQA-PLtest set self-reported21.931
- recall_at_1 on MTEB HotpotQA-PLtest set self-reported32.627
- recall_at_10 on MTEB HotpotQA-PLtest set self-reported60.439
- recall_at_100 on MTEB HotpotQA-PLtest set self-reported74.808
- recall_at_1000 on MTEB HotpotQA-PLtest set self-reported86.219
- recall_at_3 on MTEB HotpotQA-PLtest set self-reported49.440
- recall_at_5 on MTEB HotpotQA-PLtest set self-reported54.828
- map_at_1 on MTEB MSMARCO-PLvalidation set self-reported13.151
- map_at_10 on MTEB MSMARCO-PLvalidation set self-reported21.179
- map_at_100 on MTEB MSMARCO-PLvalidation set self-reported22.227
- map_at_1000 on MTEB MSMARCO-PLvalidation set self-reported22.308
- map_at_3 on MTEB MSMARCO-PLvalidation set self-reported18.473
- map_at_5 on MTEB MSMARCO-PLvalidation set self-reported19.943
- mrr_at_1 on MTEB MSMARCO-PLvalidation set self-reported13.467
- mrr_at_10 on MTEB MSMARCO-PLvalidation set self-reported21.471
- mrr_at_100 on MTEB MSMARCO-PLvalidation set self-reported22.509
- mrr_at_1000 on MTEB MSMARCO-PLvalidation set self-reported22.585
- mrr_at_3 on MTEB MSMARCO-PLvalidation set self-reported18.789
- mrr_at_5 on MTEB MSMARCO-PLvalidation set self-reported20.262
- ndcg_at_1 on MTEB MSMARCO-PLvalidation set self-reported13.539
- ndcg_at_10 on MTEB MSMARCO-PLvalidation set self-reported25.943
- ndcg_at_100 on MTEB MSMARCO-PLvalidation set self-reported31.387
- ndcg_at_1000 on MTEB MSMARCO-PLvalidation set self-reported33.641
- ndcg_at_3 on MTEB MSMARCO-PLvalidation set self-reported20.368
- ndcg_at_5 on MTEB MSMARCO-PLvalidation set self-reported23.004
- precision_at_1 on MTEB MSMARCO-PLvalidation set self-reported13.539
- precision_at_10 on MTEB MSMARCO-PLvalidation set self-reported4.249
- precision_at_100 on MTEB MSMARCO-PLvalidation set self-reported0.704
- precision_at_1000 on MTEB MSMARCO-PLvalidation set self-reported0.090
- precision_at_3 on MTEB MSMARCO-PLvalidation set self-reported8.782
- precision_at_5 on MTEB MSMARCO-PLvalidation set self-reported6.605
- recall_at_1 on MTEB MSMARCO-PLvalidation set self-reported13.151
- recall_at_10 on MTEB MSMARCO-PLvalidation set self-reported40.698
- recall_at_100 on MTEB MSMARCO-PLvalidation set self-reported66.710
- recall_at_1000 on MTEB MSMARCO-PLvalidation set self-reported84.491
- recall_at_3 on MTEB MSMARCO-PLvalidation set self-reported25.452
- recall_at_5 on MTEB MSMARCO-PLvalidation set self-reported31.791