Investigating Multi-layer Representations for Dense Passage Retrieval (2025)

Abstract: Dense retrieval models usually adopt vectors from the last hidden layer of the document encoder to represent a document, which is in contrast to the fact that representations in different layers of a pre-trained language model usually contain different kinds of linguistic knowledge, and behave differently during fine-tuning. Therefore, we propose to investigate utilizing representations from multiple encoder layers to make up the representation of a document, which we denote Multi-layer Representations (MLR). We first investigate how representations in different layers affect MLR’s performance under the multi-vector retrieval setting, and then propose to leverage pooling strategies to reduce multi-vector models to single-vector ones to improve retrieval efficiency. Experiments demonstrate the effectiveness of MLR over dual encoder, ME-BERT and ColBERT in the single-vector retrieval setting, as well as demonstrate that it works well with other advanced training techniques such as retrieval-oriented pre-training and hard negative mining.

## introduction dense passage retrieval is adopted in open-domain question answering @xcite to retrieve relevant passages from a large corpus for the reader model to extract answers. the dense retrieval technique encodes queries and documents into dense embeddings, and has wide applications in other knowledgeintensive tasks @xcite as well as retrieval-augmented generation @xcite . dense retrieval enjoys many advantages over sparse retrieval methods @xcite , such as alleviation of the term mismatch problem @xcite , and improved retrieval performance through supervised learning @xcite . dense retrieval models typically adopt a dualencoder architecture @xcite . figure 1 (a) ), where the query and document are encoded by two encoders usually fine-tuned from a pre-trained language model. however, current dense retrieval architectures @xcite @xcite represent documents with vectors foot_0 taken only from the encoder's last hidden layer (figure 1 (b) ). this is in contrast to studies which have shown that, for a pre-trained language model like bert @xcite , representations in different layers contain different kinds of linguistic knowledge, and behave differently during fine-tuning @xcite . for example, syntactic information resides mainly in the middle layers of bert, while the semantics spreads across all the layers @xcite @xcite , and the final layers are most task-specific after fine-tuning @xcite @xcite . given this observation, we propose to investigate utilizing representations from multiple encoder layers, instead of those only from the last layer, to make up the representation of a document, which we denote multi-layer representations (mlr, figure 1 (c) ). a straightforward way to utilize representations from multiple encoder layers to represent a document is to retrieve in a multi-vector setting, which is illustrated in figure 1 (c ). unlike the vanilla dual-encoders where each document is represented by a single vector, multi-vector models @xcite @xcite represent each document with multiple vectors, and thus enjoy more representational capacity. we propose to investigate how representations in different layers affect this representational capacity, and how this compares to previous "last-layer" multi-vector models like me-bert @xcite . on the other hand, multi-vector models' improved retrieval performance over single-vector models usually comes with the cost of a decreased retrieval efficiency, because the number of document embeddings to be searched is proportional to the average number of vectors used to represent each document. for example, for an 8-vector me-bert @xcite , it takes 481gb disk space to store the document embeddings for ∼21m documents, as well as 3.0321 seconds per query to build and retrieve the entire index. in contrast, for a single-vector dual-encoder, it only takes 60gb and 0.2056 seconds on the same machine. given this, we further propose to pool the multi-vector representation of a document into a single one so that multi-vector models can be reduced to singlevector models during inference time (figure 1 (d) ). thus, we can enjoy an improved retrieval performance over a single-vector dual-encoder with exactly the same retrieval efficiency. our contributions are as follows: single-vector retrieval models. dpr @xcite adopts a dual-encoder architecture and a contrastive loss to learn dense representations for queries and documents. improved training techniques are further developed to learn better single-vector representations, and can be roughly divided into three categories: (i) hard negative mining @xcite , (ii) retrieval-oriented pre-training @xcite @xcite , and (iii) knowledge distillation from cross encoders @xcite @xcite . these techniques are in general orthogonal to our proposed method, and we empirically investigate mlr's compatibility with retrieval-oriented pre-training and hard negative mining in § 4.3.2. ## multi-layer representations given a text query q, we aim to find the relevant documents from a large document collection @xmath0, d 2 , . . . , d n }, where n can range from millions to billions. we adopt one query encoder e q (•) and one document encoder e d (•) to represent the corresponding text sequences as realvalued dense vectors.. for the query/document encoders, we will experiment with two pre-trained language models, bert @xcite and t5 @xcite . for convenience, we will describe our method using bert, and highlight the adaptations for t5 in § 3.3. for a query q, the [cls] representation in the last layer is adopted as the query embedding e q (q) . = h @xmath1. for a document d with t tokens, assume the output of the document encoder with l transformer layers is a set of hidden states {h ## conclusions in this study, we proposed to leverage representations from different encoder layers to represent a document in text retrieval. for multi-vector retrieval, we investigated how representations in different layers affect mlr's performance. for singlevector retrieval, we found that mlr can outperform single-vector dual encoder by a large margin, and that pooling representations from different layers is more effective than pooling from representations only in the last layer. we also showed that advanced training techniques such as retrieval-oriented pretraining and hard negative mining can further boost mlr's performance.