  Recently introduced pre-trained contextualized autoregressive models like
BERT have shown improvements in document retrieval tasks. One of the major
limitations of the current approaches can be attributed to the manner they deal
with variable-size document lengths using a fixed input BERT model. Common
approaches either truncate or split longer documents into small
sentences/passages and subsequently label them - using the original document
label or from another externally trained model. In this paper, we conduct a
detailed study of the design decisions about splitting and label transfer on
retrieval effectiveness and efficiency. We find that direct transfer of
relevance labels from documents to passages introduces label noise that
strongly affects retrieval effectiveness for large training datasets. We also
find that query processing times are adversely affected by fine-grained
splitting schemes. As a remedy, we propose a careful passage level labelling
scheme using weak supervision that delivers improved performance (3-14% in
terms of nDCG score) over most of the recently proposed models for ad-hoc
retrieval while maintaining manageable computational complexity on four diverse
document retrieval datasets.
