Add metrics on Polish datasets

README.md

@@ -7,9 +7,9 @@ language:
   - it
   - ja
   - nl
-  - pl
   - pt
   - zh
+  - pl
 ---
 
 # Model Card for `passage-ranker.pistachio`
@@ -22,27 +22,28 @@ Model name: `passage-ranker.pistachio`
 
 The model was trained and tested in the following languages:
 
-- Chinese (simplified)
-- Dutch
 - English
 - French
 - German
+- Spanish
 - Italian
+- Dutch
 - Japanese
-- Polish
 - Portuguese
-- Spanish
+- Chinese (simplified)
+- Polish
 
 Besides the aforementioned languages, basic support can be expected for additional 93 languages that were used during the pretraining of the base model (see
 [list of languages](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages)).
 
 ## Scores
 
-| Metric              | Value |
-|:--------------------|------:|
-| Relevance (NDCG@10) | 0.480 |
+| Metric                      | Value |
+|:----------------------------|------:|
+| English Relevance (NDCG@10) | 0.474 |
+| Polish Relevance (NDCG@10)  | 0.380 |
 
-Note that the relevance score is computed as an average over 14 retrieval datasets (see
+Note that the relevance score is computed as an average over several retrieval datasets (see
 [details below](#evaluation-metrics)).
 
 ## Inference Times
@@ -93,6 +94,8 @@ can be around 0.5 to 1 GiB depending on the used GPU.
 
 ### Evaluation Metrics
 
+##### English
+
 To determine the relevance score, we averaged the results that we obtained when evaluating on the datasets of the
 [BEIR benchmark](https://github.com/beir-cellar/beir). Note that all these datasets are in English.
 
@@ -115,12 +118,38 @@ To determine the relevance score, we averaged the results that we obtained when
 | TREC-COVID        |   0.651 |
 | Webis-Touche-2020 |   0.312 |
 
-We evaluated the model on the datasets of the [MIRACL benchmark](https://github.com/project-miracl/miracl) to test its multilingual capacities. Note that not all training languages are part of the benchmark, so we only report the metrics for the existing languages.
+#### Polish
+
+This model has polish capacities, that are being evaluated over a subset of 
+the [PIRBenchmark](https://github.com/sdadas/pirb) with BM25 as the first stage retrieval.
+
+
+| Dataset       | NDCG@10 |
+|:--------------|--------:|
+| Average       |   0.380 |
+|               |         |
+| arguana-pl    |   0.285 |
+| dbpedia-pl    |   0.283 |
+| fiqa-pl       |   0.223 |
+| hotpotqa-pl   |   0.603 |
+| msmarco-pl    |   0.259 |
+| nfcorpus-pl   |   0.293 |
+| nq-pl         |   0.355 |
+| quora-pl      |   0.613 |
+| scidocs-pl    |   0.128 |
+| scifact-pl    |   0.581 |
+| trec-covid-pl |   0.560 |
+
+#### Other languages
+
+We evaluated the model on the datasets of the [MIRACL benchmark](https://github.com/project-miracl/miracl) to test its
+multilingual capacities. Note that not all training languages are part of the benchmark, so we only report the metrics
+for the existing languages.
 
 | Language              | NDCG@10 |
 |:----------------------|--------:|
-| Chinese (simplified)  |   0.454 |
 | French                |   0.439 |
 | German                |   0.418 |
+| Spanish               |   0.487 |
 | Japanese              |   0.517 |
-| Spanish               |   0.487 |
+| Chinese (simplified)  |   0.454 |