Longtriever: a Pre-trained Long Text Encoder for Dense Document Retrieval (2023)

Abstract: Pre-trained language models (PLMs) have achieved the preeminent position in dense retrieval due to their powerful capacity in modeling intrinsic semantics. However, most existing PLM-based retrieval models encounter substantial computational costs and are infeasible for processing long documents. In this paper, a novel retrieval model Longtriever is proposed to embrace three core challenges of long document retrieval: substantial computational cost, incomprehensive document understanding, and scarce annotations. Longtriever splits long documents into short blocks and then efficiently models the local semantics within a block and the global context semantics across blocks in a tightly-coupled manner. A pre-training phase is further proposed to empower Longtriever to achieve a better understanding of underlying semantic correlations. Experimental results on two popular benchmark datasets demonstrate the superiority of our proposal.

## introduction document retrieval aims at retrieving the most relevant documents from a vast corpus in response to an input query @xcite , facilitating a myriad of applications such as web search @xcite and question answering @xcite . recent works @xcite generally first embed queries and documents into low-dimensional dense vectors, and then subsequently calculate their relevance based on these vectors, dubbed as dense retrieval. due to their powerful capacity in modeling the intrinsic semantics, pre-trained language models (e.g., bert @xcite , roberta @xcite and deberta @xcite ) have attained a preeminent position in dense retrieval. due to the high computational complexity @xcite , most existing plm-based retrieval models @xcite are designed for passage retrieval (passages are short texts generally no longer than 100 words @xcite ). nevertheless, long documents are ubiquitous in real life. for example, the average number of words and tokens in the documents of the ms marco dataset @xcite are 1,165.46 and 1,631.30 respectively, significantly larger than the maximum input length (e.g. 512) of passage retrieval. in this paper, we aim to investigate the crucial task of long document retrieval, which is challenging due to the following three reasons. in this paper, we propose a novel dense retrieval model for long documents, dubbed as longtriever. longtriever follows the hierarchical paradigm, in which the long document is split into multiple short blocks to ensure the model's efficiency. each layer of longtriever consists of two core modules: the intra-block encoder to convey messages among tokens belonging to the same block, and the inter-block encoder to pass information across different blocks. after iteratively stacking multiple longtriever layers, the tokens in a block are capable of attending to tokens in other blocks, resulting in a tightly-coupled paradigm. longtriever models the local semantics within a single block and the global context correlations across multiple blocks simultaneously under a desirable time complexity. different from previous fine-tuning based methods @xcite @xcite , here we further design a pre-training phase to gain a better understanding of inherent semantics within the long documents. a novel pre-training task, local masked autoencoder (lmae), is proposed to mine the intrinsic semantics by reconstructing the raw input of each block based on the global document representations, the block representations, and the original context tokens. the pre-training phase empowers longtriever with the capability of capturing the unsupervised semantic correlations and contributes to alleviating the reliance on the annotations. longtriever is extensively evaluated over two popular benchmark datasets, and the experimental results demonstrate the superiority of our proposal. ## conclusion we propose longtriever, a novel dense retrieval model for long documents. longtriever effectively incorporates both local and global semantic modeling, while maintaining a desirable time complexity. besides, longtriever is pre-trained with a novel task lmae to gain a better understanding of inherent semantics within the long documents. experimental results demonstrate that longtriever consistently outperforms existing retrieval methods on various document retrieval datasets.