Update README.md
Browse files
README.md
CHANGED
@@ -13,11 +13,15 @@ Sentence Transformer for Assurance & Risk Question-Answering (STAR-QA) is a fine
|
|
13 |
|
14 |
## Evaluation Results
|
15 |
|
16 |
-
The model was evaluated on a held-out sample from the STAR-QA dataset (see below) using `sentence-transformers.InformationRetrievalEvaluator`. Reported metrics include P/R @ 3 candidates, as well as MRR @ 10, MAP @ 10 and NDCG @ 100. This fine-tuned model was also benchmarked against its base model using the same methodology.
|
17 |
-
|
18 |
-
|
|
19 |
-
|
20 |
-
|
|
|
|
|
|
|
|
|
|
21 |
|
22 |
## Training Data
|
23 |
|
|
|
13 |
|
14 |
## Evaluation Results
|
15 |
|
16 |
+
The model was evaluated on a held-out sample from the STAR-QA dataset (see below) using `sentence-transformers.InformationRetrievalEvaluator`. Reported metrics include cosine similarity of retrieved documents w/r/t ground truth P/R @ 3 candidates, as well as MRR @ 10, MAP @ 10 and NDCG @ 100. This fine-tuned model was also benchmarked against its base model using the same methodology.
|
17 |
+
|
18 |
+
| Metric | STAR-QA Score | ALL-MPNET-BASE-V2 Score |
|
19 |
+
|--------------|---------------|-------------------------|
|
20 |
+
|Precision @ 3 | 0.315| 0.215|
|
21 |
+
|Recall @ 3 | 0.324| 0.223|
|
22 |
+
|MRR @ 10 | 0.887| 0.578|
|
23 |
+
|NDCG @ 10 | 0.44| 0.303|
|
24 |
+
|MAP @ 100 | 0.316| 0.209|
|
25 |
|
26 |
## Training Data
|
27 |
|