wrapup.md

Putting it all together

You can use Doc2Query or Doc2Query-- in an indexing pipeline to build an index of the expanded documents:

D

Doc2Query[−−]

D

Indexer

IDX

import pyterrer as pt
pt.init()
import pyterrier_doc2query
doc2query = pyterrier_doc2query.Doc2Query(append=True)

dataset = pt.get_dataset('irds:msmarco-passage')

indexer = pt.IterDictIndexer('./msmarco_psg')

indxer_pipe = doc2query >> indexer
indxer_pipe.index(dataset.get_corpus_iter())

Once you built an index, you can retrieve from it using any retrieval function (often BM25):

Q

BM25 Retriever

IDX

R

bm25 = pt.BatchRetrieve('./msmarco_psg', wmodel="BM25")

References & Credits

• Rodrigo Nogueira and Jimmy Lin. From doc2query to docTTTTTquery.
• Mitko Gospodinov, Sean MacAvaney, and Craig Macdonald. Doc2Query--: When Less is More. ECIR 2023.
• Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, Iadh Ounis. PyTerrier: Declarative Experimentation in Python from BM25 to Dense Retrieval. CIKM 2021.