dptrsa commited on
Commit
35fb03b
·
verified ·
1 Parent(s): de23f7c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -5
README.md CHANGED
@@ -13,11 +13,15 @@ Sentence Transformer for Assurance & Risk Question-Answering (STAR-QA) is a fine
13
 
14
  ## Evaluation Results
15
 
16
- The model was evaluated on a held-out sample from the STAR-QA dataset (see below) using `sentence-transformers.InformationRetrievalEvaluator`. Reported metrics include P/R @ 3 candidates, as well as MRR @ 10, MAP @ 10 and NDCG @ 100. This fine-tuned model was also benchmarked against its base model using the same methodology.
17
-
18
- | Model | Metric | Score |
19
- |-------|--------|-------|
20
- |test |test | 0.0|
 
 
 
 
21
 
22
  ## Training Data
23
 
 
13
 
14
  ## Evaluation Results
15
 
16
+ The model was evaluated on a held-out sample from the STAR-QA dataset (see below) using `sentence-transformers.InformationRetrievalEvaluator`. Reported metrics include cosine similarity of retrieved documents w/r/t ground truth P/R @ 3 candidates, as well as MRR @ 10, MAP @ 10 and NDCG @ 100. This fine-tuned model was also benchmarked against its base model using the same methodology.
17
+
18
+ | Metric | STAR-QA Score | ALL-MPNET-BASE-V2 Score |
19
+ |--------------|---------------|-------------------------|
20
+ |Precision @ 3 | 0.315| 0.215|
21
+ |Recall @ 3 | 0.324| 0.223|
22
+ |MRR @ 10 | 0.887| 0.578|
23
+ |NDCG @ 10 | 0.44| 0.303|
24
+ |MAP @ 100 | 0.316| 0.209|
25
 
26
  ## Training Data
27