piotr-rybak
commited on
Commit
•
10982b1
1
Parent(s):
9c0ae12
add evaluation
Browse files
README.md
CHANGED
@@ -26,14 +26,51 @@ Silver Retriever model encodes the Polish sentences or paragraphs into a 768-dim
|
|
26 |
|
27 |
It was initialized from the [HerBERT-base](https://huggingface.co/allegro/herbert-base-cased) model and fine-tuned on the [PolQA](https://huggingface.co/ipipan/polqa) and [MAUPQA](https://huggingface.co/ipipan/maupqa) datasets for 15,000 steps with a batch size of 1,024.
|
28 |
|
29 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
30 |
|
31 |
The model was trained on question-passage pairs and works best when the input is the same format as that used during training:
|
32 |
-
- We added the phrase `Pytanie
|
33 |
- The training passages consisted of `title` and `text` concatenated with the special token `</s>`. Even if your passages don't have a `title`, it is still beneficial to prefix a passage with the `</s>` token.
|
34 |
- Although we used the dot product during training, the model usually works better with the cosine distance.
|
35 |
|
36 |
-
|
37 |
|
38 |
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
|
39 |
|
@@ -55,7 +92,7 @@ embeddings = model.encode(sentences)
|
|
55 |
print(embeddings)
|
56 |
```
|
57 |
|
58 |
-
|
59 |
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
|
60 |
|
61 |
```python
|
|
|
26 |
|
27 |
It was initialized from the [HerBERT-base](https://huggingface.co/allegro/herbert-base-cased) model and fine-tuned on the [PolQA](https://huggingface.co/ipipan/polqa) and [MAUPQA](https://huggingface.co/ipipan/maupqa) datasets for 15,000 steps with a batch size of 1,024.
|
28 |
|
29 |
+
## Evaluation
|
30 |
+
|
31 |
+
|
32 |
+
### Accuracy@10
|
33 |
+
|
34 |
+
| **Model** | [**PolQA**](https://huggingface.co/datasets/ipipan/polqa) | [**Allegro FAQ**](https://huggingface.co/datasets/piotr-rybak/allegro-faq) | [**Legal Questions**](https://huggingface.co/datasets/piotr-rybak/legal-questions) | **Average** |
|
35 |
+
|:-----------------------|------------:|------------:|------------:|------------:|
|
36 |
+
| BM25 | 61.35 | 66.89 | **96.38** | 74.87 |
|
37 |
+
| BM25 (lemma) | 71.49 | 75.33 | 94.57 | 80.46 |
|
38 |
+
| MiniLM-L12-v2 | 37.24 | 71.67 | 78.97 | 62.62 |
|
39 |
+
| LaBSE | 46.23 | 67.11 | 81.34 | 64.89 |
|
40 |
+
| mContriever-Base | 78.66 | 84.44 | 95.82 | 86.31 |
|
41 |
+
| E5-Base | 86.61 | 91.89 | 96.24 | 91.58 |
|
42 |
+
| ST-DistilRoBERTa | 48.43 | 84.89 | 88.02 | 73.78 |
|
43 |
+
| ST-MPNet | 56.80 | 86.00 | 87.19 | 76.66 |
|
44 |
+
| HerBERT-QA | 75.84 | 85.78 | 91.09 | 84.23 |
|
45 |
+
| **SilverRetriever** | **87.24** | **94.56** | 95.54 | **92.45** |
|
46 |
+
|
47 |
+
### NDCG@10
|
48 |
+
|
49 |
+
| **Model** | **PolQA** | **Allegro FAQ** | **Legal Questions** | **Average** |
|
50 |
+
|:-----------------------|-------------:|-------------:|-------------:|-------------:|
|
51 |
+
| BM25 | 24.51 | 48.71 | **82.21** | 51.81 |
|
52 |
+
| BM25 (lemma) | 31.97 | 55.70 | 78.65 | 55.44 |
|
53 |
+
| MiniLM-L12-v2 | 11.93 | 51.25 | 54.44 | 39.21 |
|
54 |
+
| LaBSE | 15.53 | 46.71 | 56.16 | 39.47 |
|
55 |
+
| mContriever-Base | 36.30 | 67.38 | 77.42 | 60.37 |
|
56 |
+
| E5-Base | **46.08** | 75.90 | 77.69 | 66.56 |
|
57 |
+
| ST-DistilRoBERTa | 16.73 | 64.39 | 63.76 | 48.29 |
|
58 |
+
| ST-MPNet | 21.55 | 65.44 | 62.99 | 49.99 |
|
59 |
+
| HerBERT-QA | 32.52 | 63.58 | 66.99 | 54.36 |
|
60 |
+
| **SilverRetriever** | 43.40 | **79.66** | 77.10 | **66.72** |
|
61 |
+
|
62 |
+
|
63 |
+
|
64 |
+
## Usage
|
65 |
+
|
66 |
+
### Preparing inputs
|
67 |
|
68 |
The model was trained on question-passage pairs and works best when the input is the same format as that used during training:
|
69 |
+
- We added the phrase `Pytanie:` to the beginning of the question.
|
70 |
- The training passages consisted of `title` and `text` concatenated with the special token `</s>`. Even if your passages don't have a `title`, it is still beneficial to prefix a passage with the `</s>` token.
|
71 |
- Although we used the dot product during training, the model usually works better with the cosine distance.
|
72 |
|
73 |
+
### Inference with Sentence-Transformers
|
74 |
|
75 |
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
|
76 |
|
|
|
92 |
print(embeddings)
|
93 |
```
|
94 |
|
95 |
+
### Inference with HuggingFace Transformers
|
96 |
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
|
97 |
|
98 |
```python
|