julianrisch commited on
Commit
04c190f
1 Parent(s): 3bda274

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +11 -15
README.md CHANGED
@@ -26,29 +26,25 @@ lr_schedule = LinearWarmup
26
  embeds_dropout_prob = 0.1
27
  ```
28
  ## Performance
29
- We use the GermanDPR test dataset as ground truth labels and run two experiments to compare how a BM25 retriever performs with or without reranking with our model. The first experiment runs retrieval on the full German Wikipedia (>2million passages) and second experiment runs retrieval on the GermanDPR dataset only (<5000 passages). Both experiments use 1025 queries. Note that the second experiment is evaluating on a much simpler task because of the smaller dataset size, which explains strong BM25 retrieval performance.
30
 
31
- Full German Wikipedia:
32
  BM25 Retriever without Reranking
33
- -----------------
34
- recall@3: 0.4088 (419 / 1025)
35
- mean_reciprocal_rank@3: 0.3322
36
 
37
  BM25 Retriever with Reranking Top 10 Documents
38
- -----------------
39
- recall@3: 0.5200 (533 / 1025)
40
- mean_reciprocal_rank@3: 0.4800
41
 
42
- Germandpr only:
43
  BM25 Retriever without Reranking
44
- -----------------
45
- recall@3: 0.9102 (933 / 1025)
46
- mean_reciprocal_rank@3: 0.8528
47
 
48
  BM25 Retriever with Reranking Top 10 Documents
49
- -----------------
50
- recall@3: 0.9298 (953 / 1025)
51
- mean_reciprocal_rank@3: 0.8813
52
 
53
 
54
 
 
26
  embeds_dropout_prob = 0.1
27
  ```
28
  ## Performance
29
+ We use the GermanDPR test dataset as ground truth labels and run two experiments to compare how a BM25 retriever performs with or without reranking with our model. The first experiment runs retrieval on the full German Wikipedia (more than 2 million passages) and second experiment runs retrieval on the GermanDPR dataset only (not more than 5000 passages). Both experiments use 1025 queries. Note that the second experiment is evaluating on a much simpler task because of the smaller dataset size, which explains strong BM25 retrieval performance.
30
 
31
+ ### Full German Wikipedia (more than 2 million passages):
32
  BM25 Retriever without Reranking
33
+ - recall@3: 0.4088 (419 / 1025)
34
+ - mean_reciprocal_rank@3: 0.3322
 
35
 
36
  BM25 Retriever with Reranking Top 10 Documents
37
+ - recall@3: 0.5200 (533 / 1025)
38
+ - mean_reciprocal_rank@3: 0.4800
 
39
 
40
+ ### GermanDPR Dataset only (not more than 5000 passages):
41
  BM25 Retriever without Reranking
42
+ - recall@3: 0.9102 (933 / 1025)
43
+ - mean_reciprocal_rank@3: 0.8528
 
44
 
45
  BM25 Retriever with Reranking Top 10 Documents
46
+ - recall@3: 0.9298 (953 / 1025)
47
+ - mean_reciprocal_rank@3: 0.8813
 
48
 
49
 
50