S im LM : Pre-training with Representation Bottleneck for Dense Passage Retrieval (2023)

Abstract: In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA (Clark et al., 2020), to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning. SimLM only requires access to an unlabeled corpus and is more broadly applicable when there are no labeled data or queries. We conduct experiments on several large-scale passage retrieval datasets and show substantial improvements over strong baselines under various settings. Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 (Santhanam et al., 2021) which incurs significantly more storage cost. Our code and model checkpoints are available at https://github.com/microsoft/unilm/tree/master/simlm .

## introduction passage retrieval is an important component in applications like ad-hoc information retrieval, opendomain question answering @xcite , retrieval-augmented generation @xcite and fact verification @xcite . sparse retrieval methods such as bm25 were the dominant approach for several decades, and still play a vital role nowadays. with the emergence of large-scale pre-trained language models (plm) @xcite , increasing attention is being paid to neural dense retrieval methods @xcite . dense retrieval methods map both queries and passages into a low-dimensional vector space, where the relevance between the queries and passages are measured by the dot product or cosine similarity between their respective vectors. ## related work dense retrieval the field of information retrieval (ir) @xcite aims to find the relevant information given an ad-hoc query and has played a key role in the success of modern search engines. in recent years, ir has witnessed a paradigm shift from traditional bm25-based inverted index retrieval to neural dense retrieval @xcite . bm25-based retrieval, though efficient and interpretable, suffers from the issue of lexical mismatch between the query and passages. methods like document expansion @xcite or query expansion @xcite are proposed to help mitigate this issue. in contrast, neural dense retrievers first map the query and passages to a low-dimensional vector space, and then perform semantic matching. popular methods include dssm @xcite , c-dssm @xcite , and dpr @xcite etc. 1 also called dual-encoder / two-tower encoder. inference can be done efficiently with approximate nearest neighbor (ann) search algorithms such as hnsw @xcite . some recent works @xcite @xcite show that neural dense retrievers may fail to capture some exact lexical match information. to mitigate this issue, @xcite proposes to use bm25 as a complementary teacher model, colbert @xcite instead replaces simple dot-product matching with a more complex token-level maxsim interaction, while coil @xcite incorporates lexical match information into the scoring component of neural retrievers. our proposed pre-training method aims to adapt the underlying text encoders for retrieval tasks, and can be easily integrated with existing approaches. pre-training for dense retrieval with the development of large-scale language model pre-training @xcite , transformerbased models such as bert @xcite have become the de facto backbone architecture for learning text representations. however, most pre-training tasks are designed without any prior knowledge of downstream applications. @xcite presents three heuristically constructed pre-training tasks tailored for text retrieval: inverse cloze task (ict), body first selection (bfs), and wiki link prediction (wlp). these tasks exploit the document structure of wikipedia pages to automatically generate contrastive pairs. other related pretraining tasks include representative words prediction @xcite , contrastive span prediction @xcite , contrastive learning with independent cropping @xcite , domainmatched pre-training @xcite or neighboring text pairs @xcite etc. another line of research builds upon the intuition that the [cls] vector should encode all the important information in the given text for robust matching, which is also one major motivation for this paper. such methods include condenser @xcite , cocondenser @xcite , seed @xcite , diffcse @xcite , and retromae (liu and shao, 2022) etc. compared with condenser and cocondenser, our pre-training architecture does not have skip connections between the encoder and decoder, and therefore forces the [cls] vector to encode as encoder shallow decoder you never [mask] what you're going to [mask] in life . you never know what you're going to get in life . random mask bottleneck l enc : replaced language model loss generator [cls] you never tell what you're going to learn in life . you never expect what you're going to be in life . much information as possible. retromae @xcite ) is a concurrent work at the time of writing that combines a bottleneck architecture and the masked auto-encoding objective. ## conclusion this paper proposes a novel pre-training method simlm for dense passage retrieval. it follows an encoder-decoder architecture with a representation bottleneck in between. the encoder learns to compress all the semantic information into a dense vector and passes it to the decoder to perform well on the replaced language modeling task. when used as initialization in a dense retriever training pipeline, our model achieves competitive results on several large-scale passage retrieval datasets. for future work, we would like to increase the model size and the corpus size to examine the scaling effects. it is also interesting to explore other pre-training mechanisms to support unsupervised dense retrieval and multilingual retrieval.