File size: 1,722 Bytes
68b08cf
 
096a82e
68b08cf
 
 
096a82e
68b08cf
c2a3101
68b08cf
 
 
 
4ab835d
68b08cf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c2a3101
68b08cf
 
 
 
 
 
 
 
 
 
096a82e
68b08cf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
### Putting it all together

You can use Doc2Query or Doc2Query-- in an indexing pipeline to build an index of the expanded documents:

<div class="pipeline">
  <div class="df" title="Document Frame">D</div>
  <div class="transformer attn" title="Doc2Query or Doc2Query&minus;&minus; Transformer">Doc2Query[&minus;&minus;]</div>
  <div class="df" title="Document Frame">D</div>
  <div class="transformer" title="Indexer">Indexer</div>
  <div class="artefact" title="Doc2Query Index">IDX</div>
</div>

```python
import pyterrier as pt
pt.init()
import pyterrier_doc2query
doc2query = pyterrier_doc2query.Doc2Query(append=True)

dataset = pt.get_dataset('irds:msmarco-passage')

indexer = pt.IterDictIndexer('./msmarco_psg')

indxer_pipe = doc2query >> indexer
indxer_pipe.index(dataset.get_corpus_iter())
```

Once you built an index, you can retrieve from it using any retrieval function (often BM25):

<div class="pipeline">
  <div class="df" title="Query Frame">Q</div>
  <div class="transformer" title="BM25 Transformer">BM25 Retriever <div class="artefact" title="Doc2Query Index">IDX</div></div>
  <div class="df" title="Result Frame">R</div>
</div>

```python
bm25 = pt.BatchRetrieve('./msmarco_psg', wmodel="BM25")
```

### References & Credits

 - Rodrigo Nogueira and Jimmy Lin. [From doc2query to docTTTTTquery](https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery-v2.pdf).
 - Mitko Gospodinov, Sean MacAvaney, and Craig Macdonald. Doc2Query--: When Less is More. ECIR 2023.
 - Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, Iadh Ounis. [PyTerrier: Declarative Experimentation in Python from BM25 to Dense Retrieval](https://dl.acm.org/doi/abs/10.1145/3459637.3482013). CIKM 2021.