sebastian-hofstaetter commited on
Commit
a03882c
1 Parent(s): 5323213

add model & infos

Browse files
README.md ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: "en"
3
+ tags:
4
+ - dpr
5
+ - dense-passage-retrieval
6
+ - knowledge-distillation
7
+ datasets:
8
+ - ms_marco
9
+ ---
10
+
11
+ # DistilBert for Dense Passage Retrieval trained with Balanced Topic Aware Sampling (TAS-B)
12
+
13
+ We provide a retrieval trained DistilBert-based model (we call the *dual-encoder then dot-product scoring* architecture BERT_Dot) trained with Balanced Topic Aware Sampling on MSMARCO-Passage.
14
+
15
+ This instance was trained with a batch size of 256 and can be used to **re-rank a candidate set** or **directly for a vector index based dense retrieval**. The architecture is a 6-layer DistilBERT, without architecture additions or modifications (we only change the weights during training) - to receive a query/passage representation we pool the CLS vector. We use the same BERT layers for both query and passage encoding (yields better results, and lowers memory requirements).
16
+
17
+ If you want to know more about our efficient (can be done on a single consumer GPU in 48 hours) batch composition procedure and dual supervision for dense retrieval training, check out our paper: https://arxiv.org/abs/2104.06967 🎉
18
+
19
+ For more information and a minimal usage example please visit: https://github.com/sebastian-hofstaetter/tas-balanced-dense-retrieval
20
+
21
+ ## Effectiveness on MSMARCO Passage & TREC-DL'19
22
+
23
+ We trained our model on the MSMARCO standard ("small"-400K query) training triples re-sampled with our TAS-B method. As teacher models we used the BERT_CAT pairwise scores as well as the ColBERT model for in-batch-negative signals published here: https://github.com/sebastian-hofstaetter/neural-ranking-kd
24
+
25
+ ### MSMARCO-DEV (7K)
26
+
27
+ | | MRR@10 | NDCG@10 | Recall@1K |
28
+ |----------------------------------|--------|---------|-----------------------------|
29
+ | BM25 | .194 | .241 | .857 |
30
+ | **TAS-B BERT_Dot** (Retrieval) | .347 | .410 | .978 |
31
+
32
+ ### TREC-DL'19
33
+
34
+ For MRR and Recall we use the recommended binarization point of the graded relevance of 2. This might skew the results when compared to other binarization point numbers.
35
+
36
+ | | MRR@10 | NDCG@10 | Recall@1K |
37
+ |----------------------------------|--------|---------|-----------------------------|
38
+ | BM25 | .689 | .501 | .739 |
39
+ | **TAS-B BERT_Dot** (Retrieval) | .883 | .717 | .843 |
40
+
41
+ ### TREC-DL'20
42
+
43
+ For MRR and Recall we use the recommended binarization point of the graded relevance of 2. This might skew the results when compared to other binarization point numbers.
44
+
45
+ | | MRR@10 | NDCG@10 | Recall@1K |
46
+ |----------------------------------|--------|---------|-----------------------------|
47
+ | BM25 | .649 | .475 | .806 |
48
+ | **TAS-B BERT_Dot** (Retrieval) | .843 | .686 | .875 |
49
+
50
+
51
+ For more baselines, info and analysis, please see the paper: https://arxiv.org/abs/2104.06967
52
+
53
+ ## Limitations & Bias
54
+
55
+ - The model inherits social biases from both DistilBERT and MSMARCO.
56
+
57
+ - The model is only trained on relatively short passages of MSMARCO (avg. 60 words length), so it might struggle with longer text.
58
+
59
+
60
+ ## Citation
61
+
62
+ If you use our model checkpoint please cite our work as:
63
+
64
+ ```
65
+ @inproceedings{Hofstaetter2021_tasb_dense_retrieval,
66
+ author = {Sebastian Hofst{\"a}tter and Sheng-Chieh Lin and Jheng-Hong Yang and Jimmy Lin and Allan Hanbury},
67
+ title = {{Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling}},
68
+ booktitle = {Proc. of SIGIR},
69
+ year = {2021},
70
+ }
71
+ ```
config.json ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "distilbert-base-uncased",
3
+ "activation": "gelu",
4
+ "architectures": [
5
+ "DistilBertModel"
6
+ ],
7
+ "attention_dropout": 0.1,
8
+ "dim": 768,
9
+ "dropout": 0.1,
10
+ "hidden_dim": 3072,
11
+ "initializer_range": 0.02,
12
+ "max_position_embeddings": 512,
13
+ "model_type": "distilbert",
14
+ "n_heads": 12,
15
+ "n_layers": 6,
16
+ "pad_token_id": 0,
17
+ "qa_dropout": 0.1,
18
+ "seq_classif_dropout": 0.2,
19
+ "sinusoidal_pos_embds": false,
20
+ "tie_weights_": true,
21
+ "vocab_size": 30522
22
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6dc56a56d0f65ec2d057be4507775c47a79713f0b25f6c832a00eb6f59004a49
3
+ size 265472230
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
1
+ {"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
1
+ {"do_lower_case": true, "do_basic_tokenize": true, "never_split": null, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "name_or_path": "distilbert-base-uncased"}
vocab.txt ADDED
The diff for this file is too large to render. See raw diff